Syntelligence · Independent benchmark · Public preview v0.1 · November 2026

Can AI agents reason from real chemistry data?

Two benchmarks built from primary experimental records. ChemBench-Lit asks an agent to determine a compound's structure from its full published characterization. HRMS-LLM asks an agent to infer elemental composition and audit high-resolution mass spectra for injected errors. Neither rewards recall: both require reading the evidence and committing to a single, checkable answer.

1,936,648
records in the literature index
2 benchmarks
structure elucidation / mass-spec inference
11 hard tasks
held out from ChemBench-Lit
46 models
on the HRMS-LLM panel
Overview

The gap is not memorization. It is evidence handling.

Frontier models can recite reactions by the thousand and still stop in front of a real spectrum. We build tasks that expose this gap: primary data, objective answers, difficulty calibrated against models, not against human intuition.

Data from real records

Every item is taken verbatim from a paper's experimental section or an original mass-spectrometry record, with DOI, page and quoted source retained. Numbers are never altered or generated.

Answers that can be checked

Gold answers come from the source value, a recomputable derivation, or a candidate that is uniquely supported by all data. Judgments of "most plausible" are excluded.

Difficulty measured on models

A task is only kept if it survives a sequence of independent failures. Tasks that strong models solve are retired rather than weakened, so the held-out set keeps its edge.

Two benchmarks

One measures structure. One measures composition.

Both define success strictly: a strong model failing once is not evidence. It has to fail reliably.

ChemBench-Lit

Structure elucidation from the literature

An agent receives the full characterization of one compound (¹H, ¹³C and ¹⁹F NMR, IR, HRMS) with the compound name removed, and must return the complete structure as SMILES. Regioisomer traps, peri interactions and long-range electronic effects are part of the task.

100
target tasks
11
currently held out
7
task directions
A1 spectroscopy assignment 7 A2 structure elucidation 3 B quantitative 1 C mechanism D stereochemistry E figures / multimodal F multi-step reasoning
HRMS-LLM

Composition and data integrity from mass spectra

(a) Given a high-resolution MS1 spectrum and isotope pattern, infer the elemental composition. (e) Given a spectrum, decide whether the data are self-consistent and locate the controlled anomaly. Two task types, the two most common judgments in a real mass-spec workflow.

58
question pool
12
formal task set
80%
pass threshold
(a) composition inference (e) consistency audit 46-model panel staged testing 7 anomaly types
Task examples

What the tasks look like.

Three examples from the live set. Data and gold answers have been checked against the original publication or record.

ChemBench-Lit / A2 · Mirror isomers: one skeleton, two methoxy positionsLIT-12-i
Reference structure versus the structure submitted by both models
Left: reference structure, methoxy at the 7-position. Right: the structure returned by both Astra and Fable, methoxy at the 8-position. The only difference is one bond to the aromatic ring.
Task data (excerpt)
¹H NMR (500 MHz, CDCl₃)
δ 7.79 (s, 1H), 7.56 (d, J = 8.8 Hz, 1H),
  6.99 - 6.96 (m, 1H), 3.87 (s, 3H), ...
¹³C{¹H} NMR (126 MHz, CDCl₃)
δ 168.2, 158.4, 157.2, 136.4, 132.2, 126.3 (q, 278.3 Hz),
  120.1, 114.3, 99.7, 56.0, 42.1 (q, 27.6 Hz), ...
¹⁹F NMR δ -59.22 (t, 10.9 Hz)
HRMS [M+H]⁺ calcd for C₁₅H₁₆F₃N₂O₂ 313.1158
The prompt gives characterization data and molecular formula only. No compound name, no DOI.
What makes it hard

Same skeleton, same molecular formula, one variable: whether the methoxy group sits at the 7- or the 8-position. The 7-isomer gives an isolated singlet at 7.79; the 8-isomer gives an ortho-coupled doublet of doublets at 8.11. An agent that pattern-matches spectra without comparing the peri-nitrogen environments will answer the wrong isomer.

Gold
COc1ccc2c(c1)nc1n2C(=O)CCC1(C)CC(F)(F)F
Panel result
k3.1-pro wrong · qwen4-max wrong · gpt-6-astra 2/2 wrong
The models do not fail in different ways. They swap the two mirror tasks for each other.
ChemBench-Lit / A1 · Sparse evidence: choose 1 of 6 isomersGQ9-001
Task data (excerpt)
Formula C₈H₈O₃ · HRMS 153.0546 [M+H]⁺
¹³C equivalent nuclei: 8
HMBC pair: (6.81, 155.0)

Candidates:
S1  2-methoxy-3-hydroxybenzaldehyde
S2  2-hydroxy-3-methoxybenzaldehyde
S3  3-methoxy-4-hydroxybenzaldehyde
S4  3-hydroxy-4-methoxybenzaldehyde
S5  2-hydroxy-4-methoxybenzaldehyde
S6  2-methoxy-4-hydroxybenzaldehyde
What makes it hard

Only two pieces of evidence are given for six isomeric candidates. Eliminating five of them requires chaining the ¹³C count, the HMBC assignment and substituent effects together. Similarity to a familiar spectrum is not enough.

Gold
S1
Panel result
k3.1-pro wrong · qwen4-max wrong · gpt-6-astra wrong
HRMS-LLM / type (a) · Infer composition from exact mass and isotope patternqxt4-0002
Task data (excerpt)
ESI- HRMS, m/z:

  [C34H28Cl3FeP2Pd]-   theoretical 828.9452
  measured isotope pattern
  (M, M+2, M+4, ...) peak-by-peak

Task: infer elemental composition,
report the candidate space and exclusions
What makes it hard

Metal complex, multiple chlorine isotopes, Fe and Pd in the same ion. The agent must combine exact-mass matching, isotope-envelope comparison and element-space narrowing. Skipping any step produces a plausible but wrong composition.

Gold
[C₃₄H₂₈Cl₃FeP₂Pd]⁻
Result
1 model out of 46 reached rank 1
Protocol

First prove the task is hard. Then talk about scores.

No prompt engineering, no difficulty reduction. ChemBench-Lit runs in a sandboxed agent harness with RDKit and standard chemistry tooling; HRMS-LLM uses the model panel directly with staged prompts. Time and cost limits are part of the protocol and recorded in every report.

01

Build

Extract data from a paper or original record, pin the unique answer, and store DOI, page and verbatim quote.

02

Mechanical checks

Automated checks for prompt leakage, gold self-consistency, formula/characterization agreement and candidate uniqueness.

03

Solver screening

Two strong models solve independently. If either succeeds, the task is retired before any further evaluation.

04

Gate replication

A stronger gate model runs twice, on separate sessions. Two failures are required before the task enters the held-out set, followed by independent blind review.

ChemBench-Lit scoring

  • Structures compared as normalized SMILES; stereochemistry falls back to an InChIKey layer
  • Numeric answers must fall within tolerance; no partial credit
  • Multiple choice must select the single correct option
  • Blind review answers one question: capability failure or task defect?

HRMS-LLM scoring

  • Type (e): conclusion 30 / anomaly recall 40 / precision 15 / evidence 10 / confidence 5
  • Type (a): candidate hit 50 / element space 20 / evidence chain 20 / format 10
  • False alarms on clean controls are penalized, so "always alert" does not score
  • The whole panel runs at its highest reasoning effort, no downgrade retries
Results

Two strong agents miss the same bond.

On the NMR structure task, both models got the full skeleton right and the methoxy position wrong. Their written reasoning scored in the high 80s. The structure score stayed at zero.

Astra high test report
NMR structure task 01 · v0.2.0
Astra high · single run
0 / 1
structure
89.0
analysis
446 s
runtime
$1.39
cost
Open full report →
Fable 5.2 high test report
NMR structure task 01 · v0.2.0
Fable 5.2 high · single run
0 / 1
structure
90.5
analysis
467 s
runtime
$2.14
cost
Open full report →

Two reports, one failure mode

Both models generated the correct structure as a candidate and checked it with RDKit. Both then submitted the neighboring isomer. In both write-ups, the aromatic assignment leans on the acylated nitrogen and carbonyl side of the ring; the imine-type C=N environment is not given comparable weight. This is a stable, repeatable reasoning shortcut, not a random miss.

"Aromatic position selection relies mainly on qualitative deshielding arguments from the acylated N / carbonyl side, without establishing the relative influence of the imine C=N end." — Astra report

Original test reports are written in Chinese; this page summarizes them in English. Full reports can be provided with an English translation on request.

HRMS-LLM formal set
ModelAttemptsSolvedRateVerdict
claude-opus-5-5119
82%
pass
gemini-4-argon118
73%
below threshold
gemini-3.8-flash96
67%
below threshold
qwen3.8-max64
67%
below threshold
MiniMax-M2.594
44%
below threshold

Pass threshold 80% across 11 formal tasks; all models run under the same protocol. Every other model in the panel stayed below 50%. In an 8-model quick screen (control plus two anomaly tasks plus one held-out task), no model passed all three.

ChemBench-Lit funnel
Literature index
1.94M
Papers used
21
Candidates built
169
Held out
11
Target
100

A low conversion rate is the design, not a defect. A task must pass two solver models, two gate runs and a blind review before it is held out. Rejected candidates are kept with their full screening record.

Failure analysis

Where agents stop.

These are failure-mode distributions, not leaderboard positions. They decide what the next batch of tasks targets.

0.23 - 0.91 ppm

Peri electronic effects, treated as one thing

Imine-type and lactam-type nitrogens deshield an adjacent aromatic proton by different amounts. Agents that treat them as equivalent systematically flip mirror-pair regioisomers.

15 - 20%

Peak width and resolution conversion

In HRMS anomaly checks, converting peak width into resolution was the only dimension that reliably separated stronger models. Most models skip the R proportional to 1/√m conversion and miss the anomaly.

43 / 85

No answer, not a wrong answer

In a recent large rerun, a large share of failures were budget exhaustion or timeout with no submitted answer. Strong models often give up rather than guess.

4+ models

False alarms on clean controls

On clean control spectra, multiple models still raised alerts. Any "find the anomaly" score needs a false-positive penalty, or strategic guessing can farm points.

Limitations & availability

What this does and does not show.

Every claim here is checkable against a report. The current evidence base is deliberately small and labeled as such.

Pilot scale, not a ranking

ChemBench-Lit currently holds 11 hard tasks; the two NMR model tests are single runs. Results are descriptive case evidence, not a statistically powered model ranking.

Full set is withheld

Only a small number of sample tasks and full reports are shared, to limit training-data contamination. The held-out set and its screening traces are available to reviewers on request.

Fixed panel, dated results

Model outcomes depend on the routed provider, reasoning effort and run date recorded in each report. Panel or gateway changes trigger re-runs, and older conclusions are versioned rather than silently kept.

This is independent work. It is not affiliated with or endorsed by OpenAI, Anthropic or any model provider. Model names are those exposed by the evaluation gateway at test time. Sample tasks, protocols and full reports are available for review on request.