A small public set of evaluation artifacts: three case studies of frontier models on primary chemistry records, and one task package ready for evaluation. Each case is described together with its task definition, output format, scoring rules and verification summary.
Each case is independent: its own task, evidence and verifier. Click through for the full case description.
Full ¹H/¹³C/¹⁹F NMR, HRMS and IR entry from a paper, compound name withheld. Two frontier models each get one run to return a SMILES and a separately reviewed explanation.
A 10,160-point raw spectrum and a target m/z; the model must propose one best and two next candidate formulas with mass errors. Three fresh conversations under the same prompt.
An 8-atom system, a specified forming and breaking bond. The model outputs one XYZ guess; a quantum-chemistry platform runs optimisation, frequency and IRC validation.
A simulated lab platform with blind-coded salts. The agent must actually run batches and rank nine salts by measured 10-minute conversion; a literature-only ordering cannot pass the verifier.
*The 3/3 result follows a user turn that supplied the reference candidate and asked for a Gaussian re-ranking. It is not an independent solve and is reported separately in the case.
The common thread across the four cases: primary data, a defined output format, independent verification, and preserved records.
The NMR case uses a paper's characterization entry verbatim. The MS case uses the original raw spectrum. The TS case keeps the exact prompt and the model's exact XYZ. Nothing is paraphrased into a synthetic example.
Structure answers are compared as molecular graphs; explanations are reviewed on a published 100-point rubric, separately from the structure metric. Platform cases are validated by their own verifiers: batch records, optimisation and frequency checks, IRC.
Prompts, raw model replies and verification records are preserved. Summaries are published here; complete records are available to reviewers on request.
Four separate evaluations. Read rows independently; the sample sizes are too small to compare models.
| Case | Evaluation | Outcome | Status |
|---|---|---|---|
| NMR structure task 01 | Model A and Model B, one run each | Both returned the same neighbouring aromatic isomer; structure 0/1. Explanations scored 89.0 and 90.5 / 100. | structure wrong |
| MS-bench composition case | Three independent trials, same prompt | Initial best candidate matched the reference 0/3. After an answer-level hint, 3/3; not an independent solve. | 0/3 independent |
| TS guess case | One model call, one platform validation | Guess was parseable and plausible; optimisation, frequency and IRC completed; final validation still failed. | not verified |
| SNAr salt ranking | Task package complete; no model run published yet | Verifier requires a recorded batch covering all nine blind codes; literature-only rankings are rejected. | pending |
The public summaries are designed to be enough to judge the method and the result; the underlying records stay controlled.
Task definitions, output formats, scoring rules, result summaries and validation verdicts for the four cases. Model identities are labelled Model A–D in public pages and are disclosed to reviewers on request.
Full task records for a specific case: prompts, raw model outputs, verification logs and the supporting spectrum or geometry files. Requests are reviewed case by case.
Task pools intended for future evaluation, reference solutions, and internal evaluation records are not part of this release.
The public artifacts are deliberately small. Their value is traceability, not statistical power.
One task, one or three runs per model. Differences between models are not established by these cases, and none of the numbers should be read as a general capability estimate.
The TS prompt did not specify charge and multiplicity; the model assumed charge 0, singlet. Sampling parameters for the MS trials were not recorded. The case pages mark these limits explicitly.
Model identifiers refer to the evaluation endpoints used at run time. Provider-side versions or snapshots were not independently verified; public pages use anonymised labels.
This is independent work by Syntelligence. It is not affiliated with or endorsed by OpenAI, Anthropic or any model provider.