Two benchmarks built from primary experimental records. ChemBench-Lit asks an agent to determine a compound's structure from its full published characterization. HRMS-LLM asks an agent to infer elemental composition and audit high-resolution mass spectra for injected errors. Neither rewards recall: both require reading the evidence and committing to a single, checkable answer.
Frontier models can recite reactions by the thousand and still stop in front of a real spectrum. We build tasks that expose this gap: primary data, objective answers, difficulty calibrated against models, not against human intuition.
Every item is taken verbatim from a paper's experimental section or an original mass-spectrometry record, with DOI, page and quoted source retained. Numbers are never altered or generated.
Gold answers come from the source value, a recomputable derivation, or a candidate that is uniquely supported by all data. Judgments of "most plausible" are excluded.
A task is only kept if it survives a sequence of independent failures. Tasks that strong models solve are retired rather than weakened, so the held-out set keeps its edge.
Both define success strictly: a strong model failing once is not evidence. It has to fail reliably.
An agent receives the full characterization of one compound (¹H, ¹³C and ¹⁹F NMR, IR, HRMS) with the compound name removed, and must return the complete structure as SMILES. Regioisomer traps, peri interactions and long-range electronic effects are part of the task.
(a) Given a high-resolution MS1 spectrum and isotope pattern, infer the elemental composition. (e) Given a spectrum, decide whether the data are self-consistent and locate the controlled anomaly. Two task types, the two most common judgments in a real mass-spec workflow.
Three examples from the live set. Data and gold answers have been checked against the original publication or record.
¹H NMR (500 MHz, CDCl₃)
δ 7.79 (s, 1H), 7.56 (d, J = 8.8 Hz, 1H),
6.99 - 6.96 (m, 1H), 3.87 (s, 3H), ...
¹³C{¹H} NMR (126 MHz, CDCl₃)
δ 168.2, 158.4, 157.2, 136.4, 132.2, 126.3 (q, 278.3 Hz),
120.1, 114.3, 99.7, 56.0, 42.1 (q, 27.6 Hz), ...
¹⁹F NMR δ -59.22 (t, 10.9 Hz)
HRMS [M+H]⁺ calcd for C₁₅H₁₆F₃N₂O₂ 313.1158
Same skeleton, same molecular formula, one variable: whether the methoxy group sits at the 7- or the 8-position. The 7-isomer gives an isolated singlet at 7.79; the 8-isomer gives an ortho-coupled doublet of doublets at 8.11. An agent that pattern-matches spectra without comparing the peri-nitrogen environments will answer the wrong isomer.
Formula C₈H₈O₃ · HRMS 153.0546 [M+H]⁺ ¹³C equivalent nuclei: 8 HMBC pair: (6.81, 155.0) Candidates: S1 2-methoxy-3-hydroxybenzaldehyde S2 2-hydroxy-3-methoxybenzaldehyde S3 3-methoxy-4-hydroxybenzaldehyde S4 3-hydroxy-4-methoxybenzaldehyde S5 2-hydroxy-4-methoxybenzaldehyde S6 2-methoxy-4-hydroxybenzaldehyde
Only two pieces of evidence are given for six isomeric candidates. Eliminating five of them requires chaining the ¹³C count, the HMBC assignment and substituent effects together. Similarity to a familiar spectrum is not enough.
ESI- HRMS, m/z: [C34H28Cl3FeP2Pd]- theoretical 828.9452 measured isotope pattern (M, M+2, M+4, ...) peak-by-peak Task: infer elemental composition, report the candidate space and exclusions
Metal complex, multiple chlorine isotopes, Fe and Pd in the same ion. The agent must combine exact-mass matching, isotope-envelope comparison and element-space narrowing. Skipping any step produces a plausible but wrong composition.
No prompt engineering, no difficulty reduction. ChemBench-Lit runs in a sandboxed agent harness with RDKit and standard chemistry tooling; HRMS-LLM uses the model panel directly with staged prompts. Time and cost limits are part of the protocol and recorded in every report.
Extract data from a paper or original record, pin the unique answer, and store DOI, page and verbatim quote.
Automated checks for prompt leakage, gold self-consistency, formula/characterization agreement and candidate uniqueness.
Two strong models solve independently. If either succeeds, the task is retired before any further evaluation.
A stronger gate model runs twice, on separate sessions. Two failures are required before the task enters the held-out set, followed by independent blind review.
On the NMR structure task, both models got the full skeleton right and the methoxy position wrong. Their written reasoning scored in the high 80s. The structure score stayed at zero.
Both models generated the correct structure as a candidate and checked it with RDKit. Both then submitted the neighboring isomer. In both write-ups, the aromatic assignment leans on the acylated nitrogen and carbonyl side of the ring; the imine-type C=N environment is not given comparable weight. This is a stable, repeatable reasoning shortcut, not a random miss.
Original test reports are written in Chinese; this page summarizes them in English. Full reports can be provided with an English translation on request.
| Model | Attempts | Solved | Rate | Verdict |
|---|---|---|---|---|
| claude-opus-5-5 | 11 | 9 | pass | |
| gemini-4-argon | 11 | 8 | below threshold | |
| gemini-3.8-flash | 9 | 6 | below threshold | |
| qwen3.8-max | 6 | 4 | below threshold | |
| MiniMax-M2.5 | 9 | 4 | below threshold |
Pass threshold 80% across 11 formal tasks; all models run under the same protocol. Every other model in the panel stayed below 50%. In an 8-model quick screen (control plus two anomaly tasks plus one held-out task), no model passed all three.
A low conversion rate is the design, not a defect. A task must pass two solver models, two gate runs and a blind review before it is held out. Rejected candidates are kept with their full screening record.
These are failure-mode distributions, not leaderboard positions. They decide what the next batch of tasks targets.
Imine-type and lactam-type nitrogens deshield an adjacent aromatic proton by different amounts. Agents that treat them as equivalent systematically flip mirror-pair regioisomers.
In HRMS anomaly checks, converting peak width into resolution was the only dimension that reliably separated stronger models. Most models skip the R proportional to 1/√m conversion and miss the anomaly.
In a recent large rerun, a large share of failures were budget exhaustion or timeout with no submitted answer. Strong models often give up rather than guess.
On clean control spectra, multiple models still raised alerts. Any "find the anomaly" score needs a false-positive penalty, or strategic guessing can farm points.
Every claim here is checkable against a report. The current evidence base is deliberately small and labeled as such.
ChemBench-Lit currently holds 11 hard tasks; the two NMR model tests are single runs. Results are descriptive case evidence, not a statistically powered model ranking.
Only a small number of sample tasks and full reports are shared, to limit training-data contamination. The held-out set and its screening traces are available to reviewers on request.
Model outcomes depend on the routed provider, reasoning effort and run date recorded in each report. Panel or gateway changes trigger re-runs, and older conclusions are versioned rather than silently kept.
This is independent work. It is not affiliated with or endorsed by OpenAI, Anthropic or any model provider. Model names are those exposed by the evaluation gateway at test time. Sample tasks, protocols and full reports are available for review on request.