Syntelligence · Chemistry benchmarks · November 2026

Four chemistry benchmark cases, built from real data.

A small public set of evaluation artifacts: three case studies of frontier models on primary chemistry records, and one task package ready for evaluation. Each case is described together with its task definition, output format, scoring rules and verification summary.

4
benchmark artifacts
2 models
single runs on the NMR task
3 trials
independent runs in the MS case
0/1
structure score for both NMR runs
Cases

What is in this release.

Each case is independent: its own task, evidence and verifier. Click through for the full case description.

*The 3/3 result follows a user turn that supplied the reference candidate and asked for a Gaussian re-ranking. It is not an independent solve and is reported separately in the case.

Method

Every case is checkable end to end.

The common thread across the four cases: primary data, a defined output format, independent verification, and preserved records.

Primary records, not reconstructions

The NMR case uses a paper's characterization entry verbatim. The MS case uses the original raw spectrum. The TS case keeps the exact prompt and the model's exact XYZ. Nothing is paraphrased into a synthetic example.

Outputs are machine-checkable

Structure answers are compared as molecular graphs; explanations are reviewed on a published 100-point rubric, separately from the structure metric. Platform cases are validated by their own verifiers: batch records, optimisation and frequency checks, IRC.

Full records kept for review

Prompts, raw model replies and verification records are preserved. Summaries are published here; complete records are available to reviewers on request.

Results

What the case studies found.

Four separate evaluations. Read rows independently; the sample sizes are too small to compare models.

CaseEvaluationOutcomeStatus
NMR structure task 01 Model A and Model B, one run each Both returned the same neighbouring aromatic isomer; structure 0/1. Explanations scored 89.0 and 90.5 / 100. structure wrong
MS-bench composition case Three independent trials, same prompt Initial best candidate matched the reference 0/3. After an answer-level hint, 3/3; not an independent solve. 0/3 independent
TS guess case One model call, one platform validation Guess was parseable and plausible; optimisation, frequency and IRC completed; final validation still failed. not verified
SNAr salt ranking Task package complete; no model run published yet Verifier requires a recorded batch covering all nine blind codes; literature-only rankings are rejected. pending
Availability

What can be shared, and how.

The public summaries are designed to be enough to judge the method and the result; the underlying records stay controlled.

Published here

Task definitions, output formats, scoring rules, result summaries and validation verdicts for the four cases. Model identities are labelled Model A–D in public pages and are disclosed to reviewers on request.

On request

Full task records for a specific case: prompts, raw model outputs, verification logs and the supporting spectrum or geometry files. Requests are reviewed case by case.

Not published

Task pools intended for future evaluation, reference solutions, and internal evaluation records are not part of this release.

Limitations

What these cases do not show.

The public artifacts are deliberately small. Their value is traceability, not statistical power.

No model ranking

One task, one or three runs per model. Differences between models are not established by these cases, and none of the numbers should be read as a general capability estimate.

Some settings unpinned

The TS prompt did not specify charge and multiplicity; the model assumed charge 0, singlet. Sampling parameters for the MS trials were not recorded. The case pages mark these limits explicitly.

Model identity as routed

Model identifiers refer to the evaluation endpoints used at run time. Provider-side versions or snapshots were not independently verified; public pages use anonymised labels.

This is independent work by Syntelligence. It is not affiliated with or endorsed by OpenAI, Anthropic or any model provider.