Infer a structure from a published characterization entry
The task gives one compound's complete characterization entry from a paper's experimental section, with the compound name withheld: 1H, 13C and 19F NMR, HRMS and IR. The agent must infer the complete structure and write it as SMILES, plus a separate evidence-based explanation. Two frontier models each received one run under the same protocol.
The task
- Data
- One paper's characterization entry, supplied as text and as a local data file
- Question
- Infer the neutral molecular structure and return it as SMILES
- Inputs allowed
- Local computational tools (e.g. RDKit) for analysis; no web search, databases, paper lookup or hidden solutions
- Outputs
- A JSON file with a single SMILES, and a Markdown explanation
- Scoring
- Structure: correct / incorrect by molecular-graph comparison. Explanation: 7 dimensions, 100 points, reviewed on the submitted text only
Data supplied (verbatim excerpt)
¹H NMR (500 MHz, chloroform-d) δ 7.79 (s, 1H), 7.56 (d, J = 8.8 Hz, 1H),
6.99−6.96 (m, 1H), 3.87 (s, 3H), 3.04−2.90 (m, 3H), 2.83−2.73 (m, 1H),
2.37−2.31 (m, 1H), 2.26−2.21 (m, 1H), 1.61 (s, 3H);
¹³C{¹H} NMR (126 MHz, chloroform-d) δ 168.2, 158.4, 157.2, 136.4,
132.2, 126.3 (q, 278.3 Hz), 120.1, 114.3, 99.7, 56.0, 42.1 (q, 27.6 Hz),
34.5 (q, 1.7 Hz), 31.0, 30.4, 24.9;
¹⁹F NMR δ −59.22 (t, 10.9 Hz);
HRMS (ESI) [M+H]⁺ calcd for C₁₅H₁₆F₃N₂O₂ 313.1158, found 313.1153
Neutral molecular formula: C₁₅H₁₅F₃N₂O₂. Absolute stereochemistry is not required; grading compares connectivity and bond orders after removing stereo annotations.
Results
| Run | Structure | Explanation |
|---|---|---|
| Model A | 0 / 1 | 89.0 / 100 |
| Model B | 0 / 1 | 90.5 / 100 |
One answer each, no retries, no exceptions. Model identities are available to reviewers on request.
What happened
Both runs produced a structure identical to the reference except for one aromatic substituent position. Both models generated and checked the correct structure as a candidate during the run, and then submitted the neighbouring isomer. Both explanations leaned on the acylated-nitrogen / carbonyl side of the ring for the aromatic assignment and did not give the imine-type C=N environment comparable weight.
The recurring pattern
“Aromatic position selection relies mainly on qualitative deshielding arguments from the acylated N / carbonyl side, without establishing the relative influence of the imine C=N end.”
Explanation scores were high despite the wrong structure. The rubric is designed that way: it rewards correct fragments, atom accounting and calibrated evidence, and a correct final structure gives no automatic explanation credit. Conversely, a wrong final structure does not erase valid partial reasoning.
Explanation dimensions (out of 100)
| Dimension | Model A | Model B |
|---|---|---|
| Formula and unsaturation | 10 / 10 | 10 / 10 |
| Fragments | 20 / 20 | 20 / 20 |
| Scaffold assembly | 20 / 20 | 20 / 20 |
| Aliphatic placement | 10.5 / 15 | 13.5 / 15 |
| Aromatic placement | 15 / 20 | 15 / 20 |
| Final consistency check | 8.5 / 10 | 8.5 / 10 |
| Evidence calibration | 5 / 5 | 3.5 / 5 |
Task validation
Before release, the task package was run end to end with its reference solution and a separate verifier container: the reference structure scored 1, both required output files were collected, and structure and explanation metrics are reported independently. The comparison program treats equivalent atom orderings, Kekulé forms, explicit hydrogens and stereochemical annotations consistently, and the analysis-file checks cover missing, empty, invalid and oversized submissions. The original characterization data and the reference structure are unchanged from the source record.
Availability
The full task record for this case (prompt, model outputs, verification logs, reference solution) is available to reviewers on request. Model identities are disclosed with the record.