VQM24 (Vector-QM24) is the first exhaustive quantum mechanical dataset covering all possible neutral closed-shell small molecules with up to five heavy atoms from nine p-block elements (C, N, O, F, Si, P, S, Cl, Br). It provides DFT-level properties for all 836k structures and diffusion quantum Monte Carlo (DMC) energies for a 10,793-molecule subset, constituting the largest QMC dataset in chemical space to date. ML benchmarking reveals that VQM24 is significantly more challenging than QM9 despite containing smaller molecules.

Overview

Most existing QM datasets (QM7, QM9, ANI-1x) are derived from string-based molecular lists and are restricted to a few elements (typically CHONF), introducing selection bias and limiting ML model generalizability. VQM24 addresses this by exhaustively enumerating all valid stoichiometries, Lewis-rule-consistent graphs, and stable conformers for molecules composed of 9 elements with their most common valencies:

Dataset Subsets

Including saddle points, the full dataset contains 835,947 converged structures. Extrapolation suggests ~33 million geometries at 6 heavy atoms.

Generation Pipeline

1. Stoichiometry enumeration: All combinations of up to 5 heavy atoms from the 13 element/valency types, with hydrogen counts determined by integer partitioning of total valency
2. Graph generation: Constitutional isomers enumerated using Surge for each stoichiometry
3. Geometry initialization: RDKit MMFF94 force field generates initial 3D coordinates
4. Semi-empirical optimization: GFN2-xTB geometry optimization
5. Conformer search: CREST identifies conformational isomers (~1.1M initial geometries)
6. DFT optimization: Three-pass $\omega$B97X-D3/cc-pVDZ optimization in PSI4 v1.7, all using Gaussian Tight convergence criteria with density fitting (cc-pVDZ-JKFIT auxiliary basis):
   • Pass 1: Default PSI4 settings (DIIS for SCF, RFO optimizer in redundant internal coordinates), max 100 steps
   • Pass 2: SOSCF with full Newton step, ultrafine Lebedev-Treutler grid (590 spherical, 99 radial points), max 100 steps
   • Pass 3: Full Hessian evaluation at initial geometry and every 20th step, Cartesian coordinates, max 50 steps
7. DMC calculations: For 10,793 lowest-energy conformers with up to 4 heavy atoms, using QMCPACK with PBE0/ccECP/cc-pVQZ trial wavefunctions. Slater-Jastrow trial wavefunctions with Jastrow terms for 1-body (16 params/atom type, 8 Bohr cutoff), 2-body (20 params/spin-channel, 10 Bohr cutoff), and 3-body (26 params, 5 Bohr cutoff) interactions. DMC used a timestep of 0.001 a.u., 16,000 walkers, and 1,500 blocks of 40 imaginary time steps. ccECP pseudopotentials with the determinant-localization approximation and t-moves (DLTM) handled core electrons.

The $\omega$B97X-D3 functional was chosen for its strong GMTKN55 benchmark performance and for compatibility with ANI-1, ANI-1x, OrbNet Denali, QMugs, SPICE, and MultiXC-QM9, all of which use $\omega$B97X variants with double-zeta basis sets. This enables transfer learning across datasets.

Data Files and Access

The Zenodo dataset contains separate .npz files, loadable via NumPy:

All molecules are ordered consistently across every array within a file. Properties are accessed by key:

import numpy as np
data = np.load('DFT_all.npz', allow_pickle=True)
print(data.files)  # list all available properties
freqs = data['freqs']  # vibrational frequencies

Computed Properties

DFT ($\omega$B97X-D3/cc-pVDZ) properties and their NPZ access keys:

(H) indicates thermodynamic properties computed via the harmonic approximation. Molecular orbital energies are available in the wavefunction .molden files.

DMC properties (DMC.npz) include total energy (Etot) and statistical error bar (std) for each molecule.

DMC energies (PBE0/ccECP/cc-pVQZ nodal surfaces, Slater-Jastrow trial wavefunctions) achieve average statistical uncertainty of 0.4 mHa across ~2.3 billion samples per molecule.

ML Benchmarking: Harder Than QM9

Learning curves for atomization energy prediction show that VQM24 is substantially more challenging than QM9 for all tested models:

• KRR models (CM, ACSF, LMBTR, FCHL19, cMBDF) and GNNs (SchNet, PaiNN) all show up to ~8x larger mean errors on VQM24 than QM9 at the same training set size
• None of the tested models achieve chemical accuracy (1 kcal/mol) on VQM24, even with 128k training molecules
• The atomization energy range in VQM24 (1,545 kcal/mol) is smaller than QM9 (2,427 kcal/mol), so the higher errors reflect greater chemical diversity rather than a wider property range
• For a fair comparison with QM9 (which has no conformational isomers), learning curves use only the 258k unique constitutional isomers from VQM24

Benchmark methodology: KRR models use an atomic Gaussian kernel with hyperparameters (length-scale $l$, regularizer $\lambda$) optimized via grid search and 5-fold cross-validation. Both GNNs (SchNet, PaiNN) use 128 atomic basis functions (589k total parameters), trained for 1,000 epochs with Adam (lr = $10^{-4}$). Test set size is 10,000 randomly selected molecules, with results averaged over 5 runs. Training and evaluation scripts are available in the GitHub repository.

Prediction error analysis with the best KRR model (cMBDF, trained on 200k across 4 disjoint training sets on all 784,875 equilibrium geometries) yields an overall MAE of 0.75 kcal/mol (standard deviation 1.55 kcal/mol). The largest individual error reaches 167.3 kcal/mol, and the 25 largest outliers have a mean absolute error of 85.9 kcal/mol.

Strengths & Limitations

Strengths:

• Exhaustive coverage of 1-5 heavy atom chemical space across 9 elements
• Both DFT and DMC-level data (largest QMC dataset in chemical space)
• Includes conformational isomers (average 3 per constitutional isomer)
• Extensive property set including wavefunctions and multipole moments up to hexadecapole
• More challenging ML benchmark than QM9, exposing model limitations

Limitations:

• Limited to 5 heavy atoms (very small molecules)
• 262,542 structures (~24%) failed DFT convergence, with a strong silicon bias in failures
• 51,072 structures converged to saddle points rather than minima
• DMC subset limited to 4 heavy atoms (10,793 molecules)
• Does not include metals, rare gases, or heavier halogens (I)

Reproducibility Details

Status: Highly Reproducible

The paper, dataset, and code are all publicly available.

Software stack: Surge (graph enumeration), RDKit/MMFF94 (initial geometries), GFN2-xTB (semi-empirical optimization), CREST (conformer search), PSI4 v1.7 (DFT), PySCF (trial wavefunctions), QMCPACK (DMC), QMLcode (KRR models), SchNetPack (GNN models).

Hardware requirements:

• DFT: Three-pass $\omega$B97X-D3/cc-pVDZ optimization in PSI4 (compute details not specified per-molecule for DFT)
• DMC trial wavefunctions: Argonne LCRC Improv, single node (2x AMD EPYC 7713, 64 cores, 2 GHz), ~45 seconds per molecule, ~134 node-hours total
• DMC calculations: Argonne Polaris HPC (AMD EPYC 7543P, 64 cores, 2.8 GHz), 20 nodes per molecule, ~15 minutes each, ~54,000 node-hours total

Citation

@article{khan2025quantum,
  title={Quantum mechanical dataset of 836k neutral closed-shell molecules
         with up to 5 heavy atoms from C, N, O, F, Si, P, S, Cl, Br},
  author={Khan, Danish and Benali, Anouar and Kim, Scott Y. H.
          and von Rudorff, Guido Falk and von Lilienfeld, O. Anatole},
  journal={Scientific Data},
  volume={12},
  number={1},
  pages={1551},
  year={2025},
  publisher={Nature Portfolio},
  doi={10.1038/s41597-025-05428-4}
}

VEHICLe (Virtual Exploratory Heterocyclic Library) is a complete enumeration of all possible heteroaromatic ring systems under a set of constraints designed to capture the ring types most relevant to medicinal chemistry. The library contains 24,867 ring systems (23,895 after collapsing tautomers), yet only 1,701 of these have ever appeared in published compounds across databases totaling over 10 million molecules. The authors use this complete library to predict which unsynthesized ring systems could plausibly be made and to challenge organic chemists to conquer them.

Why Heteroaromatic Rings Matter for Drug Design

Heteroaromatic rings are central to synthetic bioactive small molecules for several reasons: they bind proteins efficiently through shape and hydrophobicity, their rigidity combined with heteroatom hydrogen bonding provides target selectivity, they support parallelizable coupling reactions (Suzuki, Stille) for rapid SAR exploration, multiple substitution positions can be explored without introducing stereocenters, and unusual ring systems or substitution patterns provide patent novelty. These advantages come with tradeoffs: low aqueous solubility, restricted SAR from rigidity, tendency toward molecular bloat during optimization, and difficulty achieving patent novelty with well-explored ring systems.

VEHICLe Construction

The library is built through a simple combinatorial pipeline implemented in Pipeline Pilot (Accelrys Software Inc.) that runs in about 3 minutes on a single-core 3 GHz Intel Xeon workstation:

1. Building blocks: Six atomic units (C, N, O, S variants with appropriate bond types) serve as starting materials.
2. Chain formation: Building blocks are combined into all possible chains of length 5 and 6 using two bond-forming rules (single and double bond).
3. Ring closure: Chains are closed into five- and six-membered rings using three closure rules. Only rings satisfying Hückel’s $4n + 2$ aromaticity rule are retained.
4. Ring fusion: Monocyclic rings are fused pairwise into all possible bicyclic combinations using four fusion rules. Aromatic bicycles are retained.

The enumeration constraints are: mono- and bicyclic rings only, five- and six-membered rings only, atoms restricted to C, N, O, S, and H, all neutral, all aromatic by Hückel’s rule, and only exocyclic carbonyls allowed. Including the carbonyl building block expands the library from 2,986 to 24,867 ring systems. Within this count, 1,744 tautomeric pairs exist in 772 clusters. Building blocks are input as MDL mol files, chains formed using MDL REACCS rxn format reactions, and duplicates removed by canonical SMILES comparison.

The following table summarizes VEHICLe ring system coverage across the compound datasets used for analysis:

Synthetic Tractability Prediction

Many VEHICLe ring systems are clearly impractical (e.g., rings composed almost entirely of nitrogen). To separate plausible candidates from outlandish ones, the authors train a random forest classifier using the NovoD ArborPharm decision tree software (NovoDynamics, Inc.) within Pipeline Pilot:

• Features: ECFP_2 circular fingerprints (346 unique fragment types across VEHICLe), recording the presence or absence of each small substructure fragment per ring system
• Training labels: “Good” (769 ring systems found in compound databases totaling 3M+ molecules) vs. “bad” (24,098 remaining)
• Method: 100 trees using the Buja pure-bucket split method, optimized to minimize false negatives (GoodBias = 32, the ratio of bad to good examples). The PreserveMinority parameter was set to true, ensuring that training data selected for exclusion came exclusively from the “bad” class.
• Tree depth: 200 layers, chosen by systematic variation (50 to 250 in steps of 50) showing diminishing returns beyond this depth
• Node parameters: EnrichmentThreshold = 0.2 (if $\geq 20%$ of molecules in a node are “good”, the whole node is classified as good); minimum bucket size = 10 molecules per node ($0.04%$ of the dataset)

The classifier produces a $p(\text{good})$ score for each ring system. All 769 known ring systems scored $p(\text{good}) > 0.9$. Of the unknown ring systems, 2,185 (9%) were predicted tractable ($p(\text{good}) > 0.5$).

Validation: 36 VEHICLe rings from UCB’s corporate collection (not in the training set) were all correctly classified as good ($p(\text{good}) \geq 0.95$). Against the Beilstein database, 663 of 2,185 predicted-good unknowns had at least one substructure hit (30% minimum true positive rate), compared to only 374 of 21,913 predicted-bad unknowns (2% false negative rate), a 15-fold improvement over random. Selecting only $p(\text{good}) = 1.0$ predictions raised this ratio to 56-fold.

A final random forest incorporating Beilstein data predicted 3,288 unique unknown ring systems as tractable, with 232 having fewer than five heteroatoms and $p(\text{good}) > 0.95$. The authors manually selected 22 of these as “unconquered” challenges for synthetic chemists.

Ring System Usage Patterns

Analysis of ring system frequency across compound databases reveals striking concentration:

• Phenyl dominance: 2% of ring systems (15 types) account for 90% of occurrences, with phenyl alone at 70%.
• Heteroatom penalty: The significance of ring system usage drops sharply with increasing heteroatom count, quantified as:

$$ \text{significance}_{i,j} = \frac{\text{nobs}_{i,j} / \text{nobs}_{j}}{\text{ntot}_{i,j} / \text{ntot}_{j}} $$

where $i$ is the number of heteroatoms, $j$ is the compound set, $\text{nobs}$ is the frequency of observation, and $\text{ntot}$ is the total count in VEHICLe. Drug molecules in clinical trials show an even steeper drop-off than the broader compound set.

• Frequency distribution: Ring system frequency does not follow Zipf’s power law across the full range. Only ring systems occurring fewer than 500 times follow a power-law distribution.
• Publication rate decline: The rate of first publication of novel heteroaromatic ring systems peaked at about 41 per year in the late 1970s and declined to 5-10 per year by the early 2000s.

The concentration likely reflects the “principle of least effort,” the phylogenetic nature of drug discovery, and conservative risk management in pharma, rather than inherent unsuitability of the unused ring systems.

Reproducibility Details

The enumeration method is fully described and could be reimplemented, but the original implementation relies on proprietary software. The random forest model also uses proprietary tools but is specified in sufficient detail for reproduction with open-source alternatives.

• Software dependencies: Pipeline Pilot (Accelrys Software Inc.) for enumeration; NovoD ArborPharm (NovoDynamics, Inc.) for decision trees. Both are proprietary.
• Hardware: 3 GHz Intel Xeon workstation (enumeration completes in ~3 minutes).
• Missing components: Original Pipeline Pilot protocols and rxn files are not publicly released. ECFP_2 fingerprints used a proprietary Accelrys implementation, though open-source equivalents (RDKit Morgan fingerprints with radius 1) exist.
• Reproducibility status: Partially Reproducible. The VEHICLe library itself is publicly available, and the method is described in sufficient detail for reimplementation with modern open-source tools, but the original code and protocols are not released.

Paper Information

• Journal: Journal of Medicinal Chemistry, Vol. 52, No. 9, pp. 2952-2963
• Published: April 6, 2009

@article{pitt2009heteroaromatic,
  title={Heteroaromatic Rings of the Future},
  author={Pitt, William R. and Parry, David M. and Perry, Benjamin G. and Groom, Colin R.},
  journal={Journal of Medicinal Chemistry},
  volume={52},
  number={9},
  pages={2952--2963},
  year={2009},
  publisher={American Chemical Society},
  doi={10.1021/jm801513z}
}

QM9 provides a consistent, comprehensive set of quantum chemical properties for 133,885 small organic molecules (up to 9 heavy atoms of C, N, O, F) from the GDB-17 chemical universe. It is among the most widely used benchmark datasets in molecular machine learning, enabling systematic development and evaluation of structure-property prediction methods.

Overview

The dataset corresponds to the GDB-9 subset of the GDB-17 chemical universe: all neutral molecules with up to nine heavy atoms (C, O, N, F), not counting hydrogen. Cations, anions, and molecules containing S, Br, Cl, or I were excluded, though 1,705 zwitterions (relevant for small biomolecules like amino acids) were retained. The dataset spans 621 stoichiometries. It includes small amino acids (glycine, alanine), nucleobases (cytosine, uracil, thymine), and pharmaceutically relevant building blocks (pyruvic acid, piperazine, hydroxy urea).

Computed Properties

All properties were calculated at the B3LYP/6-31G(2df,p) level of DFT. The 15 scalar properties per molecule are:

Each molecule is stored in an extended XYZ file. The first line gives the atom count, and the second (comment) line packs all 15 scalar properties. Lines 3 through $n_a + 2$ contain element type, Cartesian coordinates (x, y, z in Angstroms), and Mulliken partial charges as a fifth column. Three trailing lines append harmonic vibrational frequencies ($3n_a - 5$ or $3n_a - 6$ modes, in cm$^{-1}$), SMILES strings (from GDB-17 and from the B3LYP-relaxed geometry), and InChI strings (from Corina and B3LYP geometries).

Dataset Subsets

Geometry Generation Pipeline

Starting from GDB-17 SMILES strings, initial 3D coordinates were generated with Corina, then relaxed at the PM7 semi-empirical level (MOPAC), followed by B3LYP/6-31G(2df,p) geometry optimization (Gaussian 09). A five-stage iterative convergence procedure handled difficult cases: default thresholds, then ultrafine grids, tighter SCF criteria, Hessian-guided optimization (calcfc), and full Hessian optimization (calcall). After all stages, 11 molecules still failed to converge to true minima (6 converged with loose thresholds, 2 near-linear molecules converged to saddle points with very low imaginary frequencies below $i10 \text{ cm}^{-1}$).

Validation

Geometry consistency: B3LYP-relaxed geometries were converted back to InChI strings and compared against the original GDB-17 InChI. 3,054 molecules failed this round-trip test, primarily due to implementation-specific artifacts in SMILES/InChI conversion rather than actual geometry problems. Coulomb-matrix distances between Corina and B3LYP geometries quantified the magnitude of geometric changes.

Energy accuracy: For 100 randomly selected molecules, B3LYP atomization enthalpies were compared against higher-level composite methods. These reference methods are themselves near experimental accuracy: G4MP2 achieves MAE 1.0 and RMSE 1.5 kcal/mol against the G3/05 test set of 454 experimental energies, while G4 achieves MAE 0.8 and RMSE 1.2 kcal/mol on the same set. G4MP2 also deviates by only 1.4 kcal/mol from the highly accurate W1w composite procedure on 261 bond dissociation enthalpies (BDE261 dataset). Against these references, B3LYP shows:

All 6,095 C7H10O2 isomers passed the geometry consistency check, and their G4MP2-level energetics provide a higher-accuracy benchmark within a fixed stoichiometry.

Strengths & Limitations

Strengths:

• Comprehensive and consistent: same level of theory across all 134k molecules
• Derived from a systematically enumerated chemical space (GDB-17), reducing selection bias
• Rich property set covering geometric, electronic, energetic, and thermodynamic quantities
• Widely adopted benchmark enabling reproducible comparisons across ML methods

Limitations:

• Restricted to very small molecules (up to 9 heavy atoms), limiting relevance to drug-sized compounds
• Only CHONF elements, excluding sulfur, halogens (Cl, Br, I), and metals
• B3LYP/6-31G(2df,p) has known systematic errors (~5 kcal/mol MAE for atomization enthalpies)
• 3,054 molecules have geometry consistency issues in SMILES/InChI round-tripping
• Single conformer per molecule (energy-minimized geometry only)

Reproducibility Details

The Figshare deposit contains four files:

• dsgdb9nsd.xyz.tar.bz2: All 133,885 GDB-1 through GDB-9 molecules with B3LYP properties
• dsC7O2H10nsd.xyz.tar.bz2: 6,095 C7H10O2 constitutional isomers with G4MP2 energetics
• validation.txt: Atomization enthalpies at B3LYP, G4MP2, G4, and CBS-QB3 for 100 random molecules
• atomref.txt: Atomic reference energies for computing atomization energies from total energies

All data is in extended XYZ plain-text format. The paper and its metadata are open access (CC BY-NC-SA 4.0 for the article, CC0 for metadata).

No source code is provided. The computational pipeline relies on commercial and semi-commercial software: Corina (3D coordinate generation), MOPAC (PM7 semi-empirical relaxation), and Gaussian 09 (B3LYP DFT calculations). Specific convergence keywords and iteration procedures are documented in the paper. Hardware requirements are not reported.

Reproducibility status: Partially Reproducible. The dataset itself is fully available, but regenerating it requires commercial licenses for Corina and Gaussian 09.

Citation

@article{ramakrishnan2014quantum,
  title={Quantum chemistry structures and properties of 134 kilo molecules},
  author={Ramakrishnan, Raghunathan and Dral, Pavlo O. and Rupp, Matthias and von Lilienfeld, O. Anatole},
  journal={Scientific Data},
  volume={1},
  number={1},
  pages={140022},
  year={2014},
  publisher={Nature Portfolio},
  doi={10.1038/sdata.2014.22}
}

GDBMedChem is a 10 million molecule subset of GDB-17 selected using medicinal chemistry criteria rather than the fragment-likeness rules used for FDB-17. The resulting database has reduced complexity and better synthetic accessibility than the full GDB-17, while retaining higher Fsp3 carbon fraction and natural product likeness compared to known drugs. Critically, 97% of its MHFP6 substructure shingles are absent from DrugBank, ChEMBL, and ZINC, making it an unprecedented source of structural diversity for drug design.

Overview

GDB-17 enumerates 166.4 billion molecules following chemical stability and synthetic feasibility rules, but does not consider medicinal chemistry criteria such as acceptable functional group types, overall structural complexity, or drug-likeness. GDBMedChem addresses this gap with a different filtering philosophy than FDB-17: instead of enforcing fragment-likeness (rotatable bond limits, small size), it applies medicinal chemistry-inspired rules that allow larger, more flexible molecules while excluding problematic functional groups and overly complex scaffolds.

Assembly Pipeline

Stage 1: Medicinal chemistry filters (166.4B to 17.8B, ~9.4x reduction)

Three categories of filters, each benchmarked against ChEMBL, DrugBank, and UNPD (natural products) to ensure low elimination of known bioactives:

These filters eliminate 86% of GDB-17 but only 36% of ChEMBL molecules and 50% of DrugBank drugs (the higher DrugBank rate is driven mainly by the heteroatom-to-carbon ratio filter removing highly polar drugs with negative clogP values).

Of the 21 filters, 16 are implemented as SMARTS queries and 5 (stereocenters, ring count, avalon density, heteroatom-to-carbon ratio, largest aromatic ring size) use other RDKit functions. Filters were applied progressively (simplest first), not in the order listed above. The benchmarking percentages for ChEMBL and DrugBank refer to ChEMBL 22 and DrugBank 5.011 molecules with HAC ≤ 17.

Stage 2: Even sampling (17.8B to 10M)

The 17,804,900,000 molecules in the filtered set are binned into 425 possible triplet combinations of HAC (1-17), heteroatoms (≤1, 2, 3, 4, ≥5), and stereocenters (0, 1, 2, 3, 4). Of these, 181 bins are unoccupied, leaving 244 bins. PySpark’s sampleBy function performs stratified sampling without replacement, using a round-robin allocation that increments each bin’s quota by one until the total reaches 10M. The resulting distribution is uniform except in low-HAC bins (HAC ≤ 10) where all available molecules are taken.

Comparison with FDB-17

GDBMedChem and FDB-17 are both 10M-molecule subsets of GDB-17 but take fundamentally different approaches:

Both databases retain GDB-17’s characteristic high Fsp3 fraction and 3D molecular shape diversity compared to predominantly planar known molecules.

Substructure Novelty

MHFP6 (MinHash fingerprint with diameter 6) shingle analysis reveals striking structural novelty:

GDBMedChem contains 17.3 million unique shingles, roughly 10x more than the 15 million-molecule ZINC database, with 97% appearing in no other database. The cumulative unique shingle count grows faster and more steadily with database size for GDBMedChem than for known molecule databases, reflecting greater internal diversity. Among the most frequent shingles, oxygen-containing saturated or singly unsaturated substructures dominate GDBMedChem, in contrast to aromatic and nitrogen heterocycles in ZINC.

Property Profiles

Compared to known drugs (DrugBank17, ChEMBL17):

• Synthetic accessibility: Slightly better than GDB-17 due to complexity filters, but still lower than known molecules
• Natural product likeness: Significantly higher than drugs, approaching natural products (UNPD17)
• Fsp3 fraction: Higher than drugs, reflecting more 3D-shaped molecules
• Compound categories: Much higher fraction of heterocyclic molecules, much lower fraction of aromatic molecules (a consequence of combinatorial enumeration favoring heteroatom-in-ring combinations)

Strengths & Limitations

Strengths:

• 97% structurally novel substructures provide unprecedented diversity for drug design
• Medicinal chemistry filters retain drug-relevant functional group patterns
• Even sampling corrects GDB-17’s combinatorial bias toward large, complex molecules
• Higher Fsp3 and natural product likeness compared to known drugs
• Available with interactive 3D visualization, MQN/MHFP6 similarity search, and download

Limitations:

• Synthetic accessibility scores remain lower than for known molecules
• Excludes Br, I, and Cl/F on heterocycles, which are common in medicinal chemistry
• Random sampling means specific molecules of interest from the 17.8B parent set may be absent
• Overlap with FDB-17 is limited (different filtering philosophies), so both databases complement rather than replace each other

Technical Notes

Molecule Preprocessing

Before filtering, each molecule undergoes: counter-ion removal, largest-fragment retention, conversion to non-chiral SMILES, valence-error checking, and protonation at pH 7.4 (using ChemAxon JChem). Duplicates are removed by canonical SMILES comparison within each database.

Reference Databases

The comparison databases used specific versions: ChEMBL 22 (1.4M compounds with HAC ≤ 50; 105,423 with HAC ≤ 17), DrugBank 5.011 (8,299 approved/experimental drugs with HAC ≤ 50; 2,284 with HAC ≤ 17), UNPD (20,302 natural products with HAC ≤ 17), and ZINC 12 (15M commercially available compounds).

MHFP6 Shingle Computation

Shingles were computed using the mhfp Python package (also on PyPI), specifically the shingling_from_smiles function from the MHFPEncoder class. Each shingle represents an extended-connectivity substructure around an atom with a diameter of up to 6 bonds, plus all ring structures, encoded as rooted SMILES strings.

Avalon Fingerprint Density

The avalon fingerprint density, used as the overall structural complexity filter (max 18), is defined as the number of on-bits in the avalon fingerprint scaled to the heavy atom count.

Reproducibility Details

Status: Partially Reproducible. The dataset itself is publicly available for download, and the paper describes the filtering and sampling pipeline in detail (RDKit 2017_09_03, PySpark 2.3.2, 98-node cluster with 252 GB RAM). The mhfp package for shingle analysis is open-source. However, no standalone filtering/sampling code is released: reproducing the pipeline from scratch requires reimplementing the 16 SMARTS filters and 5 RDKit-based filters, plus the PySpark stratified sampling procedure. The molecule preprocessing step also depends on ChemAxon JChem (commercial) for pH 7.4 protonation and MQN calculation.

The paper is published in the closed-access journal Molecular Informatics. An open-access preprint is available on ChemRxiv.

Citation

@article{awale2019medicinal,
  title={Medicinal Chemistry Aware Database GDBMedChem},
  author={Awale, Mahendra and Sirockin, Finton and Stiefl, Nikolaus and Reymond, Jean-Louis},
  journal={Molecular Informatics},
  volume={38},
  number={8-9},
  pages={e1900031},
  year={2019},
  publisher={Wiley},
  doi={10.1002/minf.201900031}
}

FDB-17 is a curated subset of 10 million fragment-like molecules extracted from the 166.4 billion molecules in GDB-17. It corrects the combinatorial bias of exhaustive enumeration (which overwhelmingly produces large, complex molecules) by evenly sampling across molecular size, polarity, and stereochemical complexity. The result is a database sized for practical virtual screening tools while retaining GDB-17’s distinctive 3D molecular shape diversity.

Overview

GDB-17 exhaustively enumerates molecules up to 17 heavy atoms, but the combinatorial explosion means the database is dominated by the largest, most functionalized, and stereochemically most complex entries. This makes it impractical for most virtual screening workflows and poorly suited for identifying simple, synthetically accessible fragments. FDB-17 addresses both problems through a two-stage reduction.

Assembly Pipeline

Stage 1: Fragment-likeness filters (166.4B to 4.6B, 36x reduction)

Criteria limiting structural and functional group complexity:

Stage 2: Even sampling (4.6B to 10M, 460x reduction)

The 4.6B fragment subset is binned into 175 cells defined by value triplets of (HAC, heteroatoms, stereocenters):

Individual bins ranged from 3,359 to 446,322,188 molecules, reflecting the extreme combinatorial skew toward large, complex structures. Bins with ≤70,000 molecules are taken entirely; larger bins are randomly sampled to approximately 60,000 molecules each. The filtering was implemented in Java using ChemAxon’s JChem libraries and executed on a 500-node cluster in 10,000 CPU hours. The resulting even distribution across molecular size, polarity, and complexity replaces the exponentially skewed distribution of the parent database.

Property Profiles vs. Commercial Fragments

FDB-17 was compared against 40,986 commercial fragments collected from 8 vendors (AnalytiCon, ChemBridge, Enamine, FRAGMENTA, BIONET, LifeChemical, Maybridge, Vitas) and filtered by Congreve’s rule of three (mass ≤300, HBA ≤3, HBD ≤3, logP ≤3, RBC ≤3, PSA ≤60). Only 31% (12,847) of these commercial fragments appeared in the 4.6B fragment subset at all, due to functional groups absent from GDB-17 (halogens, thiols, azides, thioethers). Of those, only 6.7% (2,740) appeared in FDB-17 due to the random sampling step.

Key differences:

• Size and polarity: FDB-17’s even sampling produces distributions comparable to commercial fragments, unlike the parent GDB-17 which peaks sharply at HAC = 17
• Compound categories: Half are heteroaromatic in both sets, but FDB-17’s second half is predominantly heterocyclic vs. aromatic for commercial fragments
• 3D character: FDB-17 retains GDB-17’s coverage of the full PMI (principal moments of inertia) shape triangle, with a frequency peak at center-left (PMI computed from single low-energy CORINA conformers). Commercial fragments are predominantly planar. FDB-17 has significantly higher Fsp3 values
• Ring count: Fragment subsets of GDB-17 are enriched in 2- and 3-ring molecules (a consequence of the rotatable bond limit, which constrains monocyclic molecules more than polycyclic ones)

Virtual Screening Validation

Nearest-neighbor searches were performed using two fingerprint spaces: MQN (42-dimensional molecular quantum numbers counting atoms, bonds, polarity, and topology) and Xfp (55-dimensional extended pharmacophore fingerprint capturing shape and pharmacophore features). Four fragment-like drugs were used as queries: fencamfamine, gabapentin, rimantadine, and levetiracetam. For each drug, 10,000 nearest neighbors were retrieved and scored by 3D-shape similarity using ROCS (Rapid Overlay of Chemical Structures). 3D conformers were generated with OMEGA (all possible stereoisomers, keeping the highest-scoring one). Molecules with ROCS Tanimoto Combo > 1.4 were considered virtual hits.

FDB-17 delivered comparable numbers of virtual hits to the full 4.6B fragment subset and the entire GDB-17, despite being 460x and 16,640x smaller respectively. Both close analogs (high substructure similarity, Tsfp > 0.7) and scaffold-hopping compounds (low substructure similarity but high shape similarity) were identified. Random sampling from FDB-17 and searches in the 41k commercial fragment set returned far fewer hits.

Strengths & Limitations

Strengths:

• Manageable size (10M) compatible with docking and 3D-shape virtual screening tools
• Even coverage of molecular size, polarity, and complexity avoids combinatorial bias
• High 3D shape diversity compared to predominantly flat commercial fragment libraries
• Available with interactive visualization (MQN/SMIfp-mapplet) and web-based nearest neighbor search

Limitations:

• Only the 10M FDB-17 is released, not the 4.6B fragment-filtered intermediate. Practitioners who want a different sampling strategy or the full fragment subset cannot access it
• Random sampling means specific molecules of interest from the 4.6B subset may be absent
• Excludes halogens, non-aromatic unsaturations, and several functional group classes present in commercial fragments
• Only 6.7% overlap with commercial fragments limits direct comparison
• Still derived from GDB-17’s enumeration rules, so molecules outside those rules (e.g., containing metals or larger rings) are excluded

Reproducibility Details

FDB-17 is publicly available for download from the GDB project page as a single SMILES file (62.2 MB), hosted on Zenodo. Interactive visualization via the MQN/SMIfp-mapplet and web-based nearest neighbor search tools are also accessible through the same site. The multi-fingerprint browser supports nearest-neighbor search across six fingerprints: MQN (42D), SMIfp (34D), APfp (21D), Xfp (55D), Sfp (1024-bit Daylight-type), and ECfp4 (1024-bit circular). The filtering code was written in Java using JChem libraries (ChemAxon) and executed on a 500-node cluster in 10,000 CPU hours. The filtering code itself is not publicly released. Virtual screening additionally requires OMEGA (conformer generation) and ROCS (3D-shape scoring), both commercial tools from OpenEye.

Reproducibility status: Partially Reproducible. The 10M FDB-17 is freely downloadable, but the 4.6B fragment-filtered intermediate is not released. The filtering criteria are fully documented, but the Java filtering code is not released and depends on proprietary ChemAxon libraries. Reproducing the virtual screening experiments requires commercial tools (OMEGA, ROCS from OpenEye; CORINA for PMI analysis).

Citation

@article{visini2017fragment,
  title={Fragment Database FDB-17},
  author={Visini, Ricardo and Awale, Mahendra and Reymond, Jean-Louis},
  journal={Journal of Chemical Information and Modeling},
  volume={57},
  number={4},
  pages={700--709},
  year={2017},
  publisher={American Chemical Society},
  doi={10.1021/acs.jcim.7b00020}
}

CHX8 is the first dataset to fully enumerate all closed-shell hydrocarbons with up to eight carbon atoms, deliberately including strained, anti-Bredt, and unconventional architectures that prior enumerations (e.g., GDB-13, GDB-17) excluded. Of 77,524 enumerated structures, 31,497 are stable under DFT optimization, covering 16x more C8 hydrocarbons than GDB-13. A universal relative strain energy (RSE) metric provides a quantitative synthesizability proxy for every molecule.

Motivation: Strained Scaffolds Are No Longer Inaccessible

GDB-series databases applied strict filters during enumeration, excluding highly strained polycyclic systems, cyclic allenes, anti-Bredt frameworks, and other “unconventional” motifs. Recent synthetic advances have shown that many of these structures can be accessed and exploited: 3D strained bioisosteres improve pharmacokinetic properties, cyclic allenes enable rapid construction of complex skeletons, and anti-Bredt olefins can be generated and trapped stereospecifically. CHX8 deliberately retains all of these motifs to provide a future-proofed database that remains relevant as synthetic capabilities expand.

Enumeration and Optimization

CHX8-enum (77,524 structures): All mathematically feasible hydrocarbons generated by exhaustively enumerating saturated carbon frameworks using the GENG tool from the nauty graph-isomorphism package (all 1-to-8-node connected graphs with 1-4 edges per node), then converting graphs to 3D coordinates via OpenBabel’s --Gen3D with the MMFF94 force field. Unsaturations (double bonds, triple bonds, allenes) were introduced iteratively in all valid positions by identifying C-C bonds flanked by hydrogen atoms (SMARTS: [#1]~[#6]~[#6]~[#1]), removing H atoms, and incrementing bond order. Point diastereoisomers and E/Z isomers were generated by manipulating InChI chiral layers. Duplicate detection relied on canonical InChI strings; residual duplicates account for no more than 1.5% of CHX8.

DFT optimization: All structures were geometry-optimized at the PBE0-D4/def2-TZVP level of theory. 66.5% of structures converged after a single optimization; the remainder required one or two additional passes. 59% of CHX8-enum structures underwent $\sigma$-framework rearrangements during optimization and were classified as unstable. Rearranged structures were identified by comparing input and output InChI strings. Analysis confirmed that all rearrangement products (closed-shell, zwitterionic, or carbene species) were already present in the enumeration, so no new compounds were missed.

Relative Strain Energy as a Synthesizability Proxy

A universal RSE metric, referenced to cyclohexane (zero strain), was developed and assigned to every molecule. The RSE for a molecule of interest (subscript $n$) relative to a reference structure (subscript $r$) is:

$$ \text{RSE} = E_{n} - E_{r} - (c_{n} - c_{r}),E_{\text{CH}_2} + E_{\text{unsat}} $$

where $E_{n}$ and $E_{r}$ are Gibbs energies, $c_{n}$ and $c_{r}$ are carbon counts, $E_{\text{CH}_2}$ is the average energy cost of adding an unstrained CH$_2$ unit, computed from the Gibbs energy differences between consecutive linear alkanes (ethane through octane, six increments), and $E_{\text{unsat}}$ corrects for differences in unsaturation:

$$ E_{\text{unsat}} = (r_{n} - r_{r}),E_{\text{ring}} + (d_{n} - d_{r}),E_{\text{double}} + (t_{n} - t_{r}),E_{\text{triple}} $$

$E_{\text{double}}$ and $E_{\text{triple}}$ are each derived from internal transformations between the second and third carbon of linear chains, averaged over four chain lengths (n-butane through n-octane). Initial attempts using terminal unsaturations systematically underestimated RSE for structures containing double and triple bonds. $E_{\text{ring}}$ is derived separately using the Dudev-Lim homolytic bond dissociation approach:

$$ E_{\text{ring}} = 2E_{\text{C-H}} - E_{\text{C-C}} $$

where the individual bond energies are obtained from ethane:

$$ E_{\text{C-H}} = E_{\text{ethane}} - E_{\text{ethyl radical}}, \quad E_{\text{C-C}} = E_{\text{ethane}} - 2E_{\text{methyl radical}} $$

The highest-RSE molecule with synthetic precedent (a C6 structure detected by atomic force microscopy on a metal surface) has an RSE of 201.4 kcal/mol. Using this as a threshold, over 90% of the novel structures in CHX8 should be considered synthetically accessible in principle.

Notable reference points on the RSE scale:

• Cyclopropane: 27.5 kcal/mol
• Tetrahedrane: 140.1 kcal/mol (substituted variants synthesized, unsubstituted not yet)
• Cubane: 157.4 kcal/mol (synthesized)
• Highest synthesized: 201.4 kcal/mol (C6 structure on metal surface)

Key Findings on Strained Motifs

The exhaustive enumeration enables systematic analysis of structural classes previously excluded:

1. Trans-cycloalkenes: All trans-cycloalkenes in 6-membered rings or larger should be synthetically feasible. The stability of multi-trans systems depends on the relative position of double bonds: parallel trans-double bonds in a ring can undergo thermally accessible 4$\pi$-electrocyclisation, while non-parallel arrangements may be conformationally locked and stable.
2. Cyclic alkynes and allenes: 37% of the CHX8 dataset consists of cyclic alkynes or allenes. All cyclic alkynes except cyclopropyne, and all cyclic allenes, should be synthesizable (in singlet or triplet states), with RSE values below cubane.
3. Trans-fused rings: All but [3,3]- and [3,4]-unsubstituted trans-fused rings should be accessible. The proposed lower limit for trans-ring junctions is either (i) a 3-membered ring trans-fused to a ring of five or more atoms, or (ii) a 4-membered ring trans-fused to another 4-membered ring.
4. Anti-Bredt structures: CHX8 contains seven hydrocarbon skeletons with a bridging section, yielding fourteen possible anti-Bredt (bridgehead-unsaturated) derivatives. Of these, thirteen are stable under DFT optimization, and over 200 substituted anti-Bredt structures are present in the dataset. All stable anti-Bredt structures have RSE values below cubane. Stability is classified using Fawcett’s S parameter (the number of non-bridgehead ring atoms): CHX8 finds structures with S $\geq$ 4 are stable to optimization, consistent with recent experimental work that has accessed anti-Bredt intermediates at S values as low as 4.

Comparison to Existing Databases

• vs. GDB-13: CHX8 contains 31,497 C1-C8 hydrocarbons vs. 1,966 in GDB-13 (16x more). For C8 hydrocarbons specifically, GDB-13 has more coverage than GDB-17 (1,966 vs. 1,121). All GDB-13 hydrocarbons appear in CHX8-enum, though some were unstable to DFT optimization.
• vs. VQM24: For C1-C5 hydrocarbons, VQM24 contains 123 closed-shell isomers vs. 154 in CHX8 (14-25% more). Many missing structures in VQM24 are diastereoisomers not generated by the SURGE process.
• vs. PubChem: Less than 44% of CHX8 structures appear in PubChem
• vs. Reaxys: Only 25% of CHX7 (up to 7 carbons) structures are commercially available

Reproducibility Details

The enumeration pipeline uses open-source tools: GENG from the nauty package for graph generation, RDKit for molecular manipulation and InChI canonicalization, and OpenBabel for 3D coordinate generation (MMFF94). DFT calculations used the PBE0-D4/def2-TZVP level of theory via the ORCA quantum chemistry package. The paper does not report total compute time or hardware specifications.

Missing components for full reproduction: No source code for the enumeration or unsaturation-introduction scripts is released. The RSE calculation scripts and DFT input templates are not provided. Hardware/compute requirements are not reported.

Reproducibility status: Partially Reproducible. The dataset itself is deposited, but the enumeration and analysis code is not released.

Paper Information

• Preprint: ChemRxiv, January 2, 2026

@article{harman2026complete,
  title={Complete Computational Exploration of Eight-Carbon Hydrocarbon Chemical Space},
  author={Harman, Stephen J. and Ermanis, Kristaps},
  journal={ChemRxiv},
  year={2026},
  doi={10.26434/chemrxiv-2026-qjr5r}
}

AllChem is a computer-aided molecular design system that generates and searches an unprecedentedly large space of synthetically accessible structures (on the order of $10^{20}$). Rather than enumerating molecules from mathematical graphs (as in the GDB databases), AllChem builds its chemical space from real synthetic chemistry: it recursively applies known reactions to commercial building blocks, producing synthons (structures with open valences of defined reactivity) that combinatorially assemble into complete molecules. Every structure found by a search comes paired with a proposed synthetic route.

Motivation: Costs and Benefits Together

Most computer-aided molecular design methods focus on predicting biological activity (the benefit) while leaving synthesis feasibility (the cost) to the laboratory chemist. AllChem addresses both simultaneously. Its predecessor, ChemSpace, accessed $\sim 10^{14}$ structures built from simple combinatorial libraries (chemist-proposed scaffolds plus commercial side chains), but only about 5% of structures in the medicinal chemistry literature fit that template. AllChem aims to cover roughly 50% of published structures by allowing multi-step synthon generation that produces more complex, non-trivial scaffolds.

The gensyn Synthon Generator

The core component is gensyn, a program that recursively applies a curated set of approximately 100 reactions to approximately 7,000 commercially available building blocks. Each product becomes a new building block for subsequent reaction steps, with recursion bounded primarily by a cumulative synthesis “cost” limit (roughly five AllChem-type steps per sequence). Structures bearing open valences are collected as synthons. A typical run produces around $5 \times 10^6$ synthons, which combinatorially represent $(5 \times 10^6)^3 = 10^{20}$ complete structures with an A-B-C topology.

Key design decisions in gensyn:

• Reaction curation: All reactions come from external human-readable text files, based on reactions already practiced by laboratory chemists. Scope constraints are calibrated so that at least 90% of randomly sampled reaction applications appear unchallengeable to synthetic chemists.
• Reactive intermediates: Explicitly represented. For example, amide formation requires three steps: acid chloride to electrophilic synthon, amine to nucleophilic synthon, then coupling.
• Protective groups: Addition and removal are treated as standard reactions.
• Concerted cyclizations: Represented by splitting the ring formation across two complementary synthons with specially labeled open valences.
• Bimolecular reactions: In addition to unimolecular transformations, gensyn performs reactions that combine selected synthons with other synthons, increasing overall structural diversity.
• Constraints: Maximum of one prochiral center (to avoid diastereomeric mixtures), heavy atom count limits for lead-likeness, and a cumulative cost bound on synthetic routes. Each reaction step has a default cost of $-5$, and the maximum allowed cumulative cost is $-25$ (roughly five steps per sequence).

Reaction Description Language

Reactions are described using an extension of Sybyl Line Notation (SLN), a SMILES-like notation. Each reaction description specifies the structural pattern required in the substrate, the transformation to apply, the reactivity class of resulting open valences, the relative cost, incompatible functional groups, and rules for handling multiple equivalent reactive sites. A separate reactivity table defines which valence classes can react with each other (e.g., nucleophilic with electrophilic).

Topomer Similarity Search

Searching among $10^{20}$ complete structures relies on topomer shape similarity as a branch-and-bound filter. A query structure is fragmented by breaking acyclic single bonds (individually and pairwise), each fragment is converted to a topomer (a canonical 3D shape), and the topomer is compared against all stored synthons. Topomer comparisons run at tens of thousands per second. Because the vast majority of synthons are individually shape-dissimilar enough to eliminate every complete structure containing them, the search space collapses rapidly. To be acceptable, a product must also have been formed by joining open valences with complementary reactivity.

Validation used repeated “self-searches,” in which a query structure is assembled from randomly chosen synthons and searched for in the database. On the 250,000-synthon leadhopping database, average self-search time was 7.1 minutes; complete searches of the full-scale database take several hours on standard hardware.

Applications: Lead Hopping and Scaffold Generation

Lead hopping: Finding structurally novel molecules that are shape-similar (and therefore likely biologically similar) to a query lead. Using a 250,000-synthon leadhopping database, 18 of 19 self-search queries recovered the query structure perfectly (shape difference of 0 topomer units). The remaining query also recovered itself as the closest hit.

Scaffold idea generation: Filtering the synthon collection for small ($\leq$ 14 heavy atoms), low-chirality scaffolds with at least two diversification sites (primarily through nucleophilic heteroatom reactions on activated carbon electrophiles or Suzuki-type couplings), UV chromophores, minimal freely rotatable bonds (especially between diversification sites and rings), a ring, and short synthetic paths (all branches fewer than about six AllChem steps). Over 20% of gensyn-proposed synthons pass these scaffold filters, suggesting on the order of $10^6$ accessible and structurally distinct scaffolds, compared to the few thousand scaffolds typically represented in large screening collections.

Compute and Infrastructure

Full-scale synthon database recreation takes approximately one week using two standard workstations (one Oracle database server, one compute engine). The codebase was rewritten from Java to Python for portability and performance. All data is managed through an Oracle relational database, including synthons, intermediates, and a reactions table recording every gensyn conversion.

Limitations

• Variable reactivity of open valences (e.g., weakly nucleophilic amines may not form the implied bond readily) is handled only approximately via reagent class annotations.
• Stereospecificity and most aromatic electrophilic substitution reactions are omitted.
• The system was described as under active development at the time of publication, giving the paper the character of an interim progress report.
• Drug-likeness of 3-synthon products (average MW ~800, CLOGP ~8.0) requires careful filtering of the synthon distribution toward smaller, less lipophilic components.

Reproducibility Details

AllChem was developed as proprietary software at Tripos Inc. (Tripos Discovery Research, Bude, Cornwall, UK). No source code, synthon databases, or reaction files have been publicly released. The paper functions as a description of the system’s architecture and early results rather than a reproducibility-oriented publication.

• Code: Not publicly available. The system was proprietary to Tripos Inc.
• Data: Synthon databases and reaction description files are not shared.
• Hardware: Two standard workstations (one Oracle server, one compute engine); no specialized hardware required.
• Funding: NIH/GMS SBIR grant 2 R44 GM068359-02.

Reproducibility status: Closed.

Paper Information

• Journal: Journal of Computer-Aided Molecular Design, Vol. 21, No. 6, pp. 341-350
• Published: January 25, 2007

@article{cramer2007allchem,
  title={AllChem: generating and searching 10^{20} synthetically accessible structures},
  author={Cramer, Richard D. and Soltanshahi, Farhad and Jilek, Robert J. and Campbell, Brian},
  journal={Journal of Computer-Aided Molecular Design},
  volume={21},
  number={6},
  pages={341--350},
  year={2007},
  publisher={Springer Science+Business Media},
  doi={10.1007/s10822-006-9093-8}
}

This is a discovery paper that empirically investigates how different combinations and proportions of data domains affect language model pretraining. Using the SlimPajama dataset (a globally deduplicated, 627B token refinement of RedPajama), the study trains seven 1.3B model configurations with varying domain mixtures to identify which combinations and deduplication strategies produce the best downstream performance.

Why data combination strategy matters

Multi-source pretraining datasets combine data from web crawls, code repositories, books, academic papers, and other sources. Two underexplored questions drive this work: (1) Does deduplication within each source (local) versus across all sources (global) meaningfully affect model quality? (2) When sources are thoroughly deduplicated, how does the combination and proportion of domains affect downstream performance? Most open-source LLM training datasets (RedPajama, The Pile) perform only local deduplication, leaving cross-source redundancy unaddressed.

Global deduplication and the SlimPajama dataset

SlimPajama applies global MinHashLSH deduplication (Jaccard similarity threshold 0.8, 13-gram signatures) across all seven data sources simultaneously. This reduces RedPajama’s 1.2T tokens to 627B tokens, a roughly 48% reduction. The heaviest deduplication hits CommonCrawl and GitHub, which had the most cross-source overlap.

The key processing steps:

1. Low-length document filtering: Remove documents below a minimum length threshold.
2. Global deduplication: MinHashLSH across all sources simultaneously, requiring 64 CPU cores and 1.4TB peak memory. This removes both within-source and between-source duplicates.

The resulting dataset composition:

Seven domain combination configurations

All configurations train 1.3B parameter models on 330B tokens with identical architecture and hyperparameters. The configurations systematically vary domain diversity:

• DC-1: CommonCrawl only (single source)
• DC-2: CommonCrawl + C4 (two web sources)
• DC-3: CommonCrawl + C4 with adjusted proportions
• DC-4: Wikipedia + Books + GitHub + ArXiv + StackExchange (no web crawl)
• DC-5: CommonCrawl + C4 + Wikipedia + Books (four sources, no code/academic)
• DC-6: All seven SlimPajama sources (maximum diversity)
• DC-7: RefinedWeb CommonCrawl (external single-source baseline)

The experimental design probes: incremental diversity (DC-1 to DC-2 to DC-5 to DC-6), proportion sensitivity (DC-2 vs DC-3), source importance (DC-3 vs DC-4), and specialization vs generalization (individual vs combined).

Diversity after global deduplication drives performance

Hugging Face leaderboard results

Key patterns:

1. More domain diversity improves average performance. The progression DC-1 (38.5) to DC-2 (38.4) to DC-5 (38.6) to DC-6 (40.0) shows that adding domains consistently lifts average accuracy once global deduplication has removed cross-source redundancy.

2. Global deduplication enables clean combination. All SlimPajama configurations except DC-4 outperform RedPajama-1.3B (38.0), which uses local deduplication only. The elimination of cross-source overlap means adding sources contributes genuinely new information.

3. Removing web crawl data hurts. DC-4 (no CommonCrawl/C4) scores lowest (37.6), demonstrating that web text provides essential breadth even when specialized sources are included.

4. Individual domains excel at specific tasks. DC-1 (CC only) achieves the highest ARC and MMLU scores. DC-4 leads on Winogrande. DC-5 leads on WSC273. No single combination dominates all tasks, reinforcing that diversity trades specialization for generalization.

5. Findings transfer to 7B scale. The best 1.3B configuration insights were applied to a 7B model trained with large batch sizes, achieving 63.4 average accuracy across the extended benchmark suite.

Training loss patterns

DC-6 (all sources) achieves the lowest training loss among SlimPajama configurations, consistent with the downstream results. DC-4 (no web crawl) shows the highest training loss, confirming that the large, diverse web crawl data is the most important single component.

Implications and limitations

The central finding is that diversity matters most after deduplication. When cross-source redundancy is removed, each additional source contributes genuinely new signal. Without global deduplication, adding sources may just increase redundancy without proportional benefit.

Limitations:

• Only seven fixed configurations are tested. No systematic search over continuous mixture proportions (contrast with DoReMi or Data Mixing Laws).
• The configurations are not independent: DC-6 includes all sources from DC-1 through DC-5, making it difficult to isolate the contribution of any single addition.
• Only 1.3B and 7B scales tested. Whether the diversity benefit continues scaling is unverified.
• English-only. Cross-lingual diversity effects are not studied.
• The paper is a technical report without formal peer review.

Reproducibility Details

Status: Highly Reproducible. All 1.3B models and datasets are publicly released under MIT license on HuggingFace.

Data

Models

1.3B parameter Cerebras-GPT architecture with ALiBi positional encoding and SwiGLU activation. All configurations trained on 330B tokens. 7B model trained with large batch-size (LBS) strategy on Cerebras 16x CS-2 cluster (80 PFLOP/s in bf16).

Hardware

Cerebras 16x CS-2 cluster, 80 PFLOP/s in bf16 mixed precision.

Artifacts

Citation

@article{shen2023slimpajamadc,
  title={SlimPajama-DC: Understanding Data Combinations for LLM Training},
  author={Shen, Zhiqiang and Tao, Tianhua and Ma, Liqun and Neiswanger, Willie and Liu, Zhengzhong and Wang, Hongyi and Tan, Bowen and Hestness, Joel and Vassilieva, Natalia and Soboleva, Daria and Xing, Eric},
  journal={arXiv preprint arXiv:2309.10818},
  year={2023}
}

ChemLLM is a Resource paper that delivers three interconnected artifacts: ChemData (a 7M-sample instruction tuning dataset for chemistry), ChemBench (a 4,100-question multiple-choice benchmark spanning nine chemistry tasks), and ChemLLM itself (a 7B-parameter language model fine-tuned on InternLM2-Base-7B). Together, these components form the first comprehensive framework for building and evaluating LLMs dedicated to the chemical domain. The primary contribution is not a novel architecture but rather the data curation pipeline, evaluation benchmark, and training methodology that converts structured chemical knowledge into dialogue-formatted instruction data.

Bridging Structured Chemical Databases and Conversational LLMs

While general-purpose LLMs like GPT-4 have shown promise on chemistry tasks, they are not specifically designed for the chemical domain. Several challenges motivate ChemLLM:

1. Structured data incompatibility: Most chemical information resides in structured databases (PubChem, ChEMBL, ChEBI, ZINC, USPTO) that are not naturally suited for training conversational language models. Using this data directly can degrade natural language processing capabilities.

2. Molecular notation understanding: Molecules are represented in specialized notations like SMILES, which differ from natural language and require explicit alignment during training.

3. Task diversity: Chemical tasks span name conversion, property prediction, molecular captioning, retrosynthesis, product prediction, yield prediction, and more. A uniform training pipeline must handle this diversity without task-specific adaptation.

4. Evaluation gaps: Existing chemical benchmarks (e.g., MoleculeNet) are designed for specialist models, not LLMs. Text-based evaluation metrics like BLEU and ROUGE are sensitive to output style rather than factual correctness, making them unreliable for scientific accuracy assessment.

Prior work focused on developing specialist models for individual downstream tasks while neglecting instruction-following and dialogue capabilities that are essential for broader reasoning and generalization.

Template-Based Instruction Construction from Structured Data

The core innovation is a systematic approach for converting structured chemical data into instruction-tuning format through two techniques:

Seed Template Prompt Technique

For each task type, the authors design a foundational seed template and use GPT-4 to generate variations that differ in expression but maintain semantic consistency. For each structured data entry, one template is randomly selected to create a single-turn dialogue sample. For example, converting IUPAC-to-SMILES entries:

• “Convert the IUPAC name [name] to its corresponding SMILES representation.”
• “What’s the SMILES notation for the chemical known as [name]?”
• “Show me the SMILES sequence for [name], please.”

Play as Playwrights Technique

To generate richer, multi-turn dialogues, the authors prompt GPT-4 with a chain-of-thought (CoT) style “script” construction method. GPT-4 is guided to create multi-turn exchanges that simulate expert discussions, smoothly transitioning between question and answer stages. An additional “answer masking” variant has the model inquire about supplementary chemical information before providing a final answer, simulating realistic expert reasoning.

Training Objective

The model is fine-tuned using LoRA with an autoregressive cross-entropy loss:

$$L_{CE} = -\sum_{c=1}^{M} y_{o,c} \log(p_{o,c})$$

where $M$ is the vocabulary size, $y_{o,c}$ is a binary indicator for whether observation $o$ belongs to class $c$, and $p_{o,c}$ is the predicted probability.

Two-Stage Training Pipeline and ChemBench Evaluation

Training Setup

ChemLLM uses a two-stage instruction tuning approach built on InternLM2-Base-7B:

Stage 1: Fine-tune on Multi-Corpus (1.7M Q&A pairs from Hugging Face) to enhance general linguistic capabilities, producing InternLM2-Chat-7B.

Stage 2: Fine-tune on a mixture of ChemData (7M entries) and Multi-Corpus, balancing domain-specific chemical expertise with general language ability.

Training details include:

• LoRA with rank 8, scale factor 16.0, dropout 0.1
• AdamW optimizer with initial learning rate $5.0 \times 10^{-5}$
• NEFTune noise injection (alpha = 5) to prevent overfitting
• Flash Attention-2 and KV Cache for efficiency
• ZeRO Stage-2 for parameter offloading
• Per-card batch size of 8 (total batch size 128)
• 1.06 epochs, 85,255 steps
• Training loss reduced from 1.4998 to 0.7158

ChemData Composition

ChemData spans three principal task categories with 7M instruction-tuning Q&A pairs:

Data sources include PubChem, ChEMBL, ChEBI, ZINC, USPTO, ORDerly, ChemRxiv, LibreTexts Chemistry, Wikipedia, and Wikidata.

ChemBench Design

ChemBench contains 4,100 multiple-choice questions across the same nine tasks as ChemData. The choice of multiple-choice format is deliberate: it minimizes the influence of output style and focuses evaluation on factual correctness, unlike BLEU/ROUGE-based evaluation. Wrong answers are generated by sampling nearby values (for prediction tasks) or using GPT-4 to create plausible distractors. Deduplication ensures no overlap between ChemData training entries and ChemBench questions.

ChemBench has been contributed to the OpenCompass evaluation platform.

Baselines

All evaluations use 5-shot prompting. Baselines include:

ChemLLM Matches GPT-4 on Chemical Tasks and Outperforms 7B Peers

Chemical Evaluation (ChemBench)

ChemLLM significantly outperforms general LLMs of similar scale and surpasses GPT-3.5 across all nine tasks. Compared to GPT-4, ChemLLM achieves higher scores on six of nine tasks, with the remaining three ranking just below GPT-4. LLaMA-2 scores near random chance (~25 per task), highlighting the difficulty of these tasks for models without chemical training.

Compared to InternLM2-Chat-7B (the Stage 1 model), ChemLLM shows substantial improvement, confirming the effectiveness of the Stage 2 chemical fine-tuning.

General Evaluation

ChemLLM outperforms all competing 7B models on MMLU, C-Eval, and GSM8K. On C-MHChem (Chinese middle and high school chemistry), ChemLLM scores 76.4, surpassing GPT-4. The authors note that chemical data fine-tuning may enhance reasoning capabilities due to the logical reasoning required in chemical problem-solving. ChemLLM also comprehensively surpasses InternLM2-Chat-7B on all four general benchmarks, indicating that chemical data does not harm general capabilities.

Qualitative Capabilities

The paper demonstrates qualitative performance on chemistry-related NLP tasks including:

• Chemical literature translation (English to Chinese and vice versa)
• Chemical poetry creation
• Information extraction from chemical text
• Text summarization of chemical research
• Reading comprehension on chemistry topics
• Named entity recognition for chemical entities
• Ethics and safety reasoning in chemical contexts

Limitations

The paper does not provide individual task-level scores in tabular form for ChemBench (only radar charts), making precise comparison difficult. Specific scores for each of the nine tasks across all baselines are not reported numerically. The evaluation is limited to 5-shot prompting without exploration of zero-shot or chain-of-thought prompting variants. The paper also does not discuss failure modes or systematic weaknesses of ChemLLM on particular task types.

Reproducibility Details

Data

Data sources for ChemData: PubChem, ChEMBL, ChEBI, ZINC, USPTO, ORDerly, ChemRxiv, LibreTexts Chemistry, Wikipedia, Wikidata.

Algorithms

• Two-stage instruction tuning (general then chemical)
• LoRA fine-tuning (rank 8, scale 16.0, dropout 0.1)
• Template-based instruction construction with GPT-4 for diversity
• Play as Playwrights CoT prompting for multi-turn dialogue generation
• NEFTune noise injection (alpha 5)
• DeepSpeed ZeRO++ for distributed training

Models

Evaluation

5-shot evaluation across all benchmarks. Multiple-choice format for ChemBench to minimize output style bias.

Hardware

• 2 machines, each with 8 NVIDIA A100 SMX GPUs
• 2 AMD EPYC 7742 64-Core CPUs per machine (256 threads each)
• SLURM cluster management
• BF16 mixed precision training
• Flash Attention-2 + KV Cache

Artifacts

Paper Information

Citation: Zhang, D., Liu, W., Tan, Q., Chen, J., Yan, H., Yan, Y., Li, J., Huang, W., Yue, X., Ouyang, W., Zhou, D., Zhang, S., Su, M., Zhong, H.-S., & Li, Y. (2024). ChemLLM: A Chemical Large Language Model. arXiv preprint arXiv:2402.06852.

@article{zhang2024chemllm,
  title={ChemLLM: A Chemical Large Language Model},
  author={Zhang, Di and Liu, Wei and Tan, Qian and Chen, Jingdan and Yan, Hang and Yan, Yuliang and Li, Jiatong and Huang, Weiran and Yue, Xiangyu and Ouyang, Wanli and Zhou, Dongzhan and Zhang, Shufei and Su, Mao and Zhong, Han-Sen and Li, Yuqiang},
  journal={arXiv preprint arXiv:2402.06852},
  year={2024}
}

This is a Resource paper. MoleculeNet provides a standardized benchmark suite for evaluating molecular machine learning methods. Its primary contribution is the curation of 17 public datasets spanning four categories of molecular properties, together with standardized evaluation metrics, multiple dataset splitting strategies, and open-source implementations of featurization and learning algorithms via the DeepChem library.

Why Molecular ML Needed a Unified Benchmark

Prior to MoleculeNet, algorithmic progress in molecular machine learning was difficult to measure. Individual papers benchmarked proposed methods on different datasets with different metrics, making cross-method comparison unreliable. Several factors make molecular ML particularly challenging:

1. Data scarcity: Molecular datasets are much smaller than those available for computer vision or NLP, since obtaining accurate chemical property measurements requires specialized instruments and expert supervision.
2. Heterogeneous outputs: Properties of interest range from quantum mechanical characteristics to macroscopic physiological effects on the human body.
3. Variable input structures: Molecules have arbitrary size, variable connectivity, and many possible 3D conformers, all of which must be encoded into fixed-length representations for conventional ML algorithms.
4. No standard evaluation protocol: Without prescribed metrics, splits, or data subsets, two methods using the same underlying database (e.g., PubChem) could be entirely incomparable.

Existing databases like PubChem, ChEMBL, and the Quantum Machine collections provided raw data but did not define evaluation protocols suitable for machine learning development. MoleculeNet bridges this gap, following the precedent set by ImageNet in computer vision and WordNet in NLP.

Core Design: Datasets, Splits, Metrics, and Featurizations

MoleculeNet is organized around four components: curated datasets, splitting methods, evaluation metrics, and molecular featurizations.

Datasets Across Four Property Categories

The benchmark includes 17 datasets covering over 700,000 compounds and more than 800 tasks. These are organized into four categories reflecting different levels of molecular properties:

Quantum mechanics datasets (QM7, QM7b, QM8, QM9) contain DFT-computed electronic properties for subsets of the GDB database. Physical chemistry datasets cover solubility (ESOL), hydration free energy (FreeSolv), and lipophilicity. Biophysics datasets include high-throughput screening results (PCBA, MUV), HIV inhibition activity, protein-ligand binding affinity (PDBbind), and BACE-1 inhibition. Physiology datasets cover blood-brain barrier penetration (BBBP), toxicity (Tox21, ToxCast), side effects (SIDER), and clinical trial toxicity (ClinTox).

Data Splitting Strategies

MoleculeNet implements four splitting methods, all using an 80/10/10 train/validation/test ratio:

• Random splitting: Standard random assignment to subsets.
• Scaffold splitting: Separates molecules by their 2D structural frameworks (Bemis-Murcko scaffolds), providing a harder generalization test since structurally different molecules appear in different subsets.
• Stratified splitting: Ensures each subset contains the full range of label values (used for QM7).
• Time splitting: Trains on older data and tests on newer data to mimic real-world development (used for PDBbind).

Evaluation Metrics

Regression tasks use MAE or RMSE depending on the dataset. Classification tasks use either ROC-AUC or PRC-AUC. The choice between ROC-AUC and PRC-AUC depends on class imbalance: PRC-AUC is recommended for datasets with positive rates below 2% (PCBA, MUV), since precision-recall curves better capture performance under extreme imbalance.

The false positive rate and precision are defined as:

$$ \text{FPR} = \frac{\text{false positive}}{\text{false positive} + \text{true negative}} $$

$$ \text{precision} = \frac{\text{true positive}}{\text{false positive} + \text{true positive}} $$

When positive samples form a small fraction of the data, false positives influence precision much more than FPR, making PRC-AUC more informative than ROC-AUC.

Featurization Methods

MoleculeNet implements six molecular featurization approaches:

1. ECFP (Extended-Connectivity Fingerprints): Fixed-length binary fingerprints capturing topological substructures via hashing.
2. Coulomb Matrix: Encodes nuclear charges and 3D coordinates through atomic self-energies and Coulomb repulsion:

$$ M_{IJ} = \begin{cases} 0.5 Z_{I}^{2.4} & \text{for } I = J \\ \frac{Z_{I} Z_{J}}{|\mathbf{R}_{I} - \mathbf{R}_{J}|} & \text{for } I \neq J \end{cases} $$

3. Grid Featurizer: Designed for PDBbind, incorporating both ligand and protein structural information including salt bridges, hydrogen bonds, and SPLIF fingerprints.
4. Symmetry Functions: Preserve rotational and permutation symmetry through radial and angular functions between atom pairs and triplets.
5. Graph Convolutions: Compute initial atom feature vectors and neighbor lists from molecular graphs.
6. Weave: Similar to graph convolutions but also computes pairwise atom features encoding bond properties, graph distance, and ring information.

Benchmarked Models and Experimental Setup

MoleculeNet benchmarks 12 learning algorithms divided into conventional methods and graph-based methods.

Conventional Methods

• Logistic Regression (classification only)
• Kernel SVM with radial basis function kernel
• Kernel Ridge Regression (KRR)
• Random Forests
• Gradient Boosting (XGBoost)
• Singletask/Multitask Networks: Fully connected networks with shared layers across tasks
• Bypass Networks: Multitask networks augmented with per-task “bypass” layers that directly connect inputs to outputs
• Influence Relevance Voting (IRV): Refined K-nearest neighbor classifiers using Jaccard-Tanimoto similarity:

$$ S(\vec{A}, \vec{B}) = \frac{A \cap B}{A \cup B} $$

Graph-Based Methods

• Graph Convolutional Models (GC): Extend circular fingerprints with learnable convolutions over molecular graphs.
• Weave Models: Update atom features using information from all other atoms and their pairwise features.
• Directed Acyclic Graph (DAG) Models: Define directed bonds toward a central atom and propagate features through the directed graph.
• Deep Tensor Neural Networks (DTNN): Use nuclear charges and distance matrices directly, updating atom embeddings based on pairwise physical distances.
• ANI-1: Learns transferable potentials using symmetry function features with atom-type-specific neural networks.
• Message Passing Neural Networks (MPNN): Generalized framework with edge-dependent message functions and set2set readout.

Experimental Protocol

Gaussian process hyperparameter optimization was applied to each dataset-model combination, followed by three independent runs with different random seeds. All results are reported as means with standard deviations. Variable training-size experiments were conducted on Tox21, FreeSolv, and QM7 to study data efficiency.

Key Findings Across Property Categories

Biophysics and Physiology

Graph convolutional and weave models showed strong performance on larger datasets with less overfitting than conventional methods. Graph-based models outperformed multitask networks at 30% training data compared to 90% on Tox21. However, for smaller single-task datasets (under 3,000 samples), kernel SVM and ensemble tree methods were more robust. On highly imbalanced datasets like MUV (0.20% positive rate), graph-based models struggled to control false positives.

Multitask training had a regularizing effect, reducing the gap between train and test scores compared to single-task models. Bypass networks consistently matched or exceeded vanilla multitask networks, confirming that per-task layers add explanatory power.

Physical Chemistry

Graph-based methods (GC, DAG, MPNN, Weave) provided significant improvements over single-task networks for predicting solubility, solvation energy, and lipophilicity. The best models achieved accuracy comparable to ab initio predictions (within 0.5 RMSE for ESOL, within 1.5 kcal/mol for FreeSolv). On FreeSolv, a weave model trained on approximately 200 samples matched the accuracy of alchemical free energy calculations.

Quantum Mechanics

Models incorporating 3D distance information (DTNN, MPNN, KRR with Coulomb matrix) substantially outperformed models using only topological features. DTNN and MPNN covered the best-performing models on 28 of 39 tasks across QM datasets. The choice of physics-aware featurization proved more important than the choice of learning algorithm for these tasks.

Summary of Best Performances

Graph-based models outperformed conventional methods on 11 of 17 datasets. Key results on the test set:

Conventional methods (KernelSVM, RF) still won on several smaller or scaffold-split datasets (HIV, BACE, MUV, PDBbind, BBBP, SIDER), highlighting that graph-based models are not universally superior, particularly under data scarcity or challenging splits.

Conclusions and Limitations

MoleculeNet demonstrated that learnable representations broadly offer the best performance for molecular machine learning. However, the authors identify several important caveats:

1. Data scarcity: Graph-based methods are not robust enough on complex tasks with limited training data.
2. Class imbalance: On heavily imbalanced classification datasets, conventional methods such as kernel SVM outperform learnable featurizations with respect to recall of positives.
3. Task-specific featurizations: For quantum mechanical and biophysical datasets, incorporating physics-aware features (Coulomb matrix, 3D coordinates) is more important than the choice of learning algorithm.
4. Data-driven physical chemistry: On FreeSolv, data-driven methods outperformed ab initio calculations with moderate data, suggesting data-driven approaches will become increasingly important as methods and datasets mature.

The authors express hope that MoleculeNet will stimulate algorithmic development similar to how ImageNet catalyzed breakthroughs in computer vision. Future directions include extending coverage to 3D protein structure prediction, DNA topological modeling, and other areas of molecular science.

Reproducibility Details

Data

All 17 datasets are publicly available and integrated into the DeepChem Python package. Users can load any dataset with a single library call.

Algorithms

All splitting methods (random, scaffold, stratified, time) and featurizations (ECFP, Coulomb matrix, grid, symmetry functions, graph convolutions, weave) are implemented in DeepChem. Hyperparameters were tuned via Gaussian process optimization. Three random seeds were used per experiment.

Models

All 12 models are implemented in DeepChem, built on Scikit-Learn and TensorFlow. No pretrained weights are provided; models are trained from scratch on each dataset.

Evaluation

Metrics include MAE, RMSE, ROC-AUC, and PRC-AUC as specified per dataset. Multi-task datasets report mean metric values across all tasks.

Hardware

The authors used Stanford’s Sherlock and Xstream GPU nodes. Specific GPU types and training times per model are provided in Table S1 of the supplementary material.

Artifacts

Paper Information

Citation: Wu, Z., Ramsundar, B., Feinberg, E. N., Gomes, J., Geniesse, C., Pappu, A. S., Leswing, K., & Pande, V. (2018). MoleculeNet: a benchmark for molecular machine learning. Chemical Science, 9(2), 513-530. https://doi.org/10.1039/c7sc02664a

@article{wu2018moleculenet,
  title={MoleculeNet: a benchmark for molecular machine learning},
  author={Wu, Zhenqin and Ramsundar, Bharath and Feinberg, Evan N. and Gomes, Joseph and Geniesse, Caleb and Pappu, Aneesh S. and Leswing, Karl and Pande, Vijay},
  journal={Chemical Science},
  volume={9},
  number={2},
  pages={513--530},
  year={2018},
  publisher={Royal Society of Chemistry},
  doi={10.1039/c7sc02664a}
}

DOCKSTRING is a Resource paper that delivers three integrated components for benchmarking machine learning models in drug discovery using molecular docking. The primary contributions are: (1) an open-source Python package wrapping AutoDock Vina for deterministic docking from SMILES strings, (2) a dataset of over 15 million docking scores and poses covering 260,000+ molecules docked against 58 medically relevant protein targets, and (3) a suite of benchmark tasks spanning regression, virtual screening, and de novo molecular design. The paper additionally provides baseline results across classical and deep learning methods.

Why Existing Molecular Benchmarks Fall Short

ML methods for drug discovery are frequently evaluated using simple physicochemical properties such as penalized logP or QED (quantitative estimate of druglikeness). These properties are computationally cheap and easy to optimize, but they do not depend on the interaction between a candidate compound and a protein target. As a result, strong performance on logP or QED benchmarks does not necessarily translate to strong performance on real drug design tasks.

Molecular docking offers a more realistic evaluation objective because docking scores depend on the 3D structure of the ligand-target complex. Docking is routinely used by medicinal chemists to estimate binding affinities during hit discovery and lead optimization. Several prior efforts attempted to bring docking into ML benchmarking, but each had limitations:

• VirtualFlow and DockStream require manually prepared target files and domain expertise.
• TDC and Cieplinski et al. provide SMILES-to-score wrappers but lack proper ligand protonation and randomness control, and cover very few targets (one and four, respectively).
• DUD-E is easily overfit by ML models that memorize actives vs. decoys.
• GuacaMol and MOSES rely on physicochemical properties or similarity functions that miss 3D structural subtleties.
• MoleculeNet compiles experimental datasets but does not support on-the-fly label computation needed for transfer learning or de novo design.

DOCKSTRING addresses all of these gaps: it standardizes the docking procedure, automates ligand and target preparation, controls randomness for reproducibility, and provides a large, diverse target set.

Core Innovation: Standardized End-to-End Docking Pipeline

The key innovation is a fully automated, deterministic docking pipeline that produces reproducible scores from a SMILES string in four lines of Python code. The pipeline consists of three stages:

Target Preparation. 57 of the 58 protein targets originate from the Directory of Useful Decoys Enhanced (DUD-E). PDB files are standardized with Open Babel, polar hydrogens are added, and conversion to PDBQT format is performed with AutoDock Tools. Search boxes are derived from crystallographic ligands with 12.5 A padding and a minimum side length of 30 A. The 58th target (DRD2, dopamine receptor D2) was prepared separately following the same protocol.

Ligand Preparation. Ligands are protonated at pH 7.4 with Open Babel, embedded into 3D conformations using the ETKDG algorithm in RDKit, refined with the MMFF94 force field, and assigned Gasteiger partial charges. Stereochemistry of determined stereocenters is maintained, while undetermined stereocenters are assigned randomly but consistently across runs.

Docking. AutoDock Vina runs with default exhaustiveness (8), up to 9 binding modes, and an energy range of 3 kcal/mol. The authors verified that fixing the random seed yields docking score variance of less than 0.1 kcal/mol across runs, making the pipeline fully deterministic.

The three de novo design objective functions incorporate a QED penalty to enforce druglikeness:

$$ f_{\text{F2}}(l) = s(l, \text{F2}) + 10(1 - \text{QED}(l)) $$

$$ f_{\text{PPAR}}(l) = \max_{t \in \text{PPAR}} s(l, t) + 10(1 - \text{QED}(l)) $$

$$ f_{\text{JAK2}}(l) = s(l, \text{JAK2}) - \min(s(l, \text{LCK}), -8.1) + 10(1 - \text{QED}(l)) $$

The F2 task optimizes binding to a single protease. The Promiscuous PPAR task requires strong binding to three nuclear receptors simultaneously. The Selective JAK2 task is adversarial, requiring strong JAK2 binding while avoiding LCK binding (two kinases with a score correlation of 0.80).

Experimental Setup: Regression, Virtual Screening, and De Novo Design

Dataset Construction

The dataset combines molecules from ExCAPE-DB (which curates PubChem and ChEMBL bioactivity assays). The authors selected all molecules with active labels against targets having at least 1,000 experimental actives, plus 150,000 inactive-only molecules. After discarding 1.8% of molecules that failed ligand preparation, the final dataset contains 260,155 compounds docked against 58 targets, producing over 15 million docking scores and poses. The dataset required over 500,000 CPU hours to generate.

Cluster analysis using DBSCAN (Jaccard distance threshold of 0.25 on RDKit fingerprints) found 52,000 clusters, and Bemis-Murcko scaffold decomposition identified 102,000 scaffolds, confirming high molecular diversity. Train/test splitting follows cluster labels to prevent data leakage.

Regression Baselines

Five targets of varying difficulty were selected: PARP1 (easy), F2 (easy-medium), KIT (medium), ESR2 (hard), and PGR (hard). Baselines include Ridge, Lasso, XGBoost, exact GP, sparse GP, MPNN, and Attentive FP.

Values are mean $R^2$ over three runs. Attentive FP achieves the best performance on every target but remains well below perfect prediction on the harder targets, confirming that docking score regression is a meaningful benchmark.

Virtual Screening Baselines

Models trained on PARP1, KIT, and PGR docking scores rank all molecules in ZINC20 (~1 billion compounds). The top 5,000 predictions are docked, and the enrichment factor (EF) is computed relative to a 0.1 percentile activity threshold.

The maximum possible EF is 1,000. Attentive FP substantially outperforms fingerprint similarity search (FSS) and Ridge regression across all targets.

De Novo Design Baselines

Four optimization methods were tested: SELFIES GA, Graph GA, GP-BO with UCB acquisition ($\beta = 10$), and GP-BO with expected improvement (EI), each with a budget of 5,000 objective function evaluations. Without QED penalties, all methods easily surpass the best training set molecules but produce large, lipophilic, undrug-like compounds. With QED penalties, the tasks become substantially harder: GP-BO with EI is the only method that finds 25 molecules better than the training set across all three tasks.

The Selective JAK2 task proved hardest due to the high correlation between JAK2 and LCK scores. Pose analysis of the top de novo molecule revealed a dual binding mode: type V inhibitor behavior in JAK2 (binding distant N- and C-terminal lobe regions) and type I behavior in LCK (hinge-binding), suggesting a plausible selectivity mechanism.

Key Findings and Limitations

Key findings:

1. Docking scores are substantially harder to predict than logP or QED, making them more suitable for benchmarking high-performing ML models. Graph neural networks (Attentive FP) achieve near-perfect $R^2$ on logP but only 0.63-0.91 on docking targets.
2. In-distribution regression difficulty does not necessarily predict out-of-distribution virtual screening difficulty. PARP1 is easiest for regression, but KIT is easiest for virtual screening.
3. Adding a QED penalty to de novo design objectives transforms trivially solvable tasks into meaningful benchmarks. The adversarial Selective JAK2 objective, which exploits correlated docking scores, may be an effective way to avoid docking score biases toward large and lipophilic molecules.
4. Docking scores from related protein targets are highly correlated, supporting the biological meaningfulness of the dataset and enabling multiobjective and transfer learning tasks.

Limitations acknowledged by the authors:

• Docking scores are approximate heuristics. They use static binding sites and force fields with limited calibration for certain metal ions. DOCKSTRING benchmarks should not substitute for rational drug design and experimental validation.
• The pipeline relies on AutoDock Vina specifically; other docking programs may produce different rankings.
• Top de novo molecules for F2 and Promiscuous PPAR contain conjugated ring structures uncommon in successful drugs.
• Platform support is primarily Linux, with noted scoring inconsistencies on macOS.

Future directions mentioned include multiobjective tasks (transfer learning, few-shot learning), improved objective functions for better pharmacokinetic properties and synthetic feasibility, and multifidelity optimization tasks combining docking with more expensive computational methods.

Reproducibility Details

Data

Algorithms

• Docking engine: AutoDock Vina with default exhaustiveness (8), up to 9 binding modes, energy range of 3 kcal/mol
• Ligand preparation: Open Babel (protonation at pH 7.4), RDKit ETKDG (3D embedding), MMFF94 (force field refinement), Gasteiger charges
• Regression models: Ridge, Lasso, XGBoost (hyperparameters via 20-configuration random search with 5-fold CV), exact GP and sparse GP (Tanimoto kernel on fingerprints), MPNN, Attentive FP (DeepChem defaults, 10 epochs)
• Optimization: Graph GA (population 250, offspring 25, mutation rate 0.01), SELFIES GA (same population/offspring settings), GP-BO with UCB ($\beta = 10$) or EI (batch size 5, 1000 offspring, 25 generations per iteration)

Evaluation

Hardware

The dataset required over 500,000 CPU hours to compute, using the University of Cambridge Research Computing Service (EPSRC and DiRAC funded). Per-target docking takes approximately 15 seconds on 8 CPUs.

Artifacts

Paper Information

Citation: García-Ortegón, M., Simm, G. N. C., Tripp, A. J., Hernández-Lobato, J. M., Bender, A., & Bacallado, S. (2022). DOCKSTRING: Easy Molecular Docking Yields Better Benchmarks for Ligand Design. Journal of Chemical Information and Modeling, 62(15), 3486-3502. https://doi.org/10.1021/acs.jcim.1c01334

Publication: Journal of Chemical Information and Modeling, 2022

Additional Resources:

• DOCKSTRING Project Page
• GitHub Repository

Citation

@article{garciaortegon2022dockstring,
  title={{DOCKSTRING}: Easy Molecular Docking Yields Better Benchmarks for Ligand Design},
  author={Garc{\'\i}a-Orteg{\'o}n, Miguel and Simm, Gregor N. C. and Tripp, Austin J. and Hern{\'a}ndez-Lobato, Jos{\'e} Miguel and Bender, Andreas and Bacallado, Sergio},
  journal={Journal of Chemical Information and Modeling},
  volume={62},
  number={15},
  pages={3486--3502},
  year={2022},
  publisher={American Chemical Society},
  doi={10.1021/acs.jcim.1c01334}
}

ChemBench is a Resource paper that introduces an automated benchmarking framework for evaluating the chemical knowledge and reasoning abilities of large language models against human expert chemists. The primary contribution is the benchmark corpus itself (2,788 question-answer pairs), the evaluation infrastructure, and the human baseline study that contextualizes model performance. The framework is designed to be extensible and can evaluate any system that returns text, including tool-augmented agents.

Why Chemistry Needs Its Own LLM Benchmark

Existing LLM benchmarks provide poor coverage of chemistry. BigBench contains only 2 of 204 tasks classified as chemistry-related, and the LM Eval Harness contains none. Developers of chemical language models often fall back on tabular property-prediction datasets (MoleculeNet, Therapeutic Data Commons, MatBench), which give a narrow view of chemical capabilities. Prior attempts at chemistry-specific benchmarks based on university entrance exams or automatic text mining have not gained wide acceptance because they cannot be used with black-box or tool-augmented systems, do not cover a broad range of topics and skills, or are not validated by domain experts.

At the same time, LLMs are increasingly used in chemistry: for property prediction, reaction optimization, materials generation, information extraction, and even autonomous experiment execution. Some users (students, general public) may rely on LLMs for safety-critical chemical questions without the expertise to evaluate outputs. Understanding where LLMs succeed and fail in chemistry is therefore both a scientific and a safety question.

ChemBench: Framework Design and Benchmark Corpus

ChemBench addresses these gaps with several design choices that distinguish it from prior work.

Diverse question corpus. The benchmark contains 2,788 question-answer pairs from multiple sources: 1,039 manually generated (from university exams, chemistry olympiads, textbooks, and novel questions) and 1,749 semi-automatically generated (from chemical databases covering GHS pictograms, daily allowed intakes, hazard statements, NMR peak counts, electron counts, IUPAC-SMILES conversions, oxidation states, and point groups). Questions span general, organic, inorganic, physical, analytical, and technical chemistry, among other topics.

Skill-based classification. Each question is annotated with the skills required to answer it: knowledge, reasoning, calculation, intuition, or combinations thereof. Questions are also classified by difficulty level (basic vs. advanced), enabling fine-grained analysis of model capabilities.

Both MCQ and open-ended formats. The corpus includes 2,544 multiple-choice and 244 open-ended questions, reflecting the reality that chemistry education and research involve more than multiple-choice testing.

Semantic annotation. Questions use tagged annotations for molecules ([START_SMILES]...[END_SMILES]), equations, units, and reactions. This allows models with special processing for scientific notation (e.g., Galactica) to handle these modalities appropriately, while remaining compatible with standard text-completion APIs.

Text-completion evaluation. ChemBench operates on text completions rather than raw logits, enabling evaluation of tool-augmented and agentic systems (not just bare models). Parsing uses multi-step regex followed by LLM-based extraction as a fallback.

ChemBench-Mini. A curated 236-question subset balances topic and skill diversity for fast, cost-effective routine evaluations. This subset was also used for the full human baseline study.

Evaluation Setup: Models, Human Experts, and Confidence

Models evaluated

The study evaluated a wide range of leading models, including both open-source and proprietary systems: o1-preview, GPT-4, Claude-3.5 (Sonnet), Llama-3.1-405B-Instruct, and others, as well as the agentic literature-search system PaperQA2. All models used greedy decoding (temperature 0) via API endpoints.

Human baseline

Nineteen chemistry experts participated through a custom web application (chembench.org). Volunteers included 2 post-postdoc researchers, 13 PhD students (with master’s degrees), and 1 bachelor’s holder. The analysis excluded anyone with fewer than 2 years of chemistry experience. For a subset of questions, volunteers were allowed to use external tools (web search, ChemDraw) but not LLMs or other people.

Confidence calibration

Selected top-performing models were prompted to estimate their confidence on a 1-5 ordinal scale (verbalized confidence estimates). This approach captures semantic uncertainty and works with models that do not expose logits.

Key Results: Where LLMs Outperform Chemists and Where They Fail

Overall performance

On ChemBench-Mini, the leading model (o1-preview) outperformed the best human expert by nearly a factor of two in overall accuracy. Many other models also exceeded average human performance. Llama-3.1-405B-Instruct achieved performance close to the leading proprietary models, showing that open-source models can be competitive in chemical settings.

Performance varies by topic

While models scored well on general and technical chemistry, they performed poorly on toxicity/safety and analytical chemistry. Predicting the number of NMR signals was particularly difficult (22% correct for o1-preview). This task requires reasoning about molecular symmetry from a SMILES string, which models struggle with compared to humans who can view molecular drawings.

Textbook questions vs. database-derived questions

Models performed better on textbook-inspired questions than on semi-automatically constructed tasks. For example, models could pass the German Chemical Prohibition Ordinance certification exam (71% for GPT-4, 61% for Claude-3.5 Sonnet) while human experts scored only 3% on the sampled subset. This suggests that good textbook question performance does not transfer to tasks requiring deeper reasoning or knowledge outside the training corpus.

Knowledge-intensive limitations

Models struggled with knowledge-intensive questions that required looking up facts in specialized databases (PubChem, Gestis). PaperQA2, which augments LLMs with literature search, could not compensate because the required knowledge lives in specialized databases rather than papers.

Chemical preference judgment

When asked to judge chemical preference (choosing between two molecules in an early virtual screening setting, following the Choung et al. dataset), model performance was often indistinguishable from random guessing, even for models that excelled at other ChemBench tasks. Human chemists showed reasonable inter-rater agreement on the same questions.

Confidence calibration is poor

For most models, verbalized confidence estimates did not correlate meaningfully with actual correctness. GPT-4 reported confidence of 1.0 for a correctly answered safety question but 4.0 for six incorrectly answered ones. Claude-3.5 Sonnet showed slightly better calibration on average but still produced misleading estimates in specific topic areas (e.g., GHS pictogram labeling: average confidence of 2.0 for correct answers vs. 1.83 for incorrect ones).

Scaling and molecular complexity

Model performance correlated with model size, consistent with observations in other domains. However, performance did not correlate with molecular complexity indicators, suggesting that models may rely on training data proximity rather than genuine structural reasoning.

Implications for Chemistry and LLM Development

The authors draw several conclusions from the ChemBench evaluation.

Chemistry education needs rethinking. Since LLMs already outperform average human chemists on many textbook-style questions, the value of rote memorization and problem-solving in chemistry curricula is diminishing. Critical reasoning and evaluation of model outputs become more important skills.

Breadth vs. depth matters. Model performance varies widely across topics and question types, even within a single topic. Aggregate scores can mask significant weaknesses in safety-critical areas.

Better human-model interaction is needed. Poor confidence calibration means users cannot trust models’ self-reported uncertainty. Developing better uncertainty estimation for chemical LLMs is an important direction.

Room for improvement through specialized data. Training on specialized chemical databases (rather than just papers) and integrating domain-specific tools could address the knowledge-intensive gaps identified by ChemBench.

Open science framework. ChemBench is designed for extensibility: new models can be added by contributors, and the leaderboard is publicly accessible. The use of a BigBench-compatible canary string helps prevent test set contamination in future training corpora.

Reproducibility Details

Data

All benchmark data is publicly available on GitHub and archived on Zenodo.

Algorithms

Evaluation uses greedy decoding (temperature 0) for all models. Parsing is multi-step: regex extraction of answer environments and enumeration letters/numbers, word-to-number conversion, and LLM-based fallback parsing (Claude-3.5 Sonnet). Confidence estimates are verbalized on an ordinal 1-5 scale.

Models

The paper evaluates multiple models including o1-preview, GPT-4, Claude-3.5 (Sonnet), Llama-3.1-405B-Instruct, Galactica, and PaperQA2. Model weights are not released (the contribution is the benchmark, not a model).

Evaluation

Hardware

Not applicable; the benchmark is designed for API-based evaluation. Cost context: Liang et al. report >US$10,000 for a single HELM evaluation, motivating ChemBench-Mini.

Artifacts

Paper Information

Citation: Mirza, A., Alampara, N., Kunchapu, S., Ríos-García, M., Emoekabu, B., Krishnan, A., … & Jablonka, K. M. (2025). A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists. Nature Chemistry, 17(7), 1027-1034. https://doi.org/10.1038/s41557-025-01815-x

@article{mirza2025chembench,
  title={A framework for evaluating the chemical knowledge and reasoning abilities of large language models against the expertise of chemists},
  author={Mirza, Adrian and Alampara, Nawaf and Kunchapu, Sreekanth and R{\'\i}os-Garc{\'\i}a, Marti{\~n}o and Emoekabu, Benedict and Krishnan, Aswanth and Gupta, Tanya and Schilling-Wilhelmi, Mara and Okereke, Macjonathan and Aneesh, Anagha and Asgari, Mehrdad and Eberhardt, Juliane and Elahi, Amir Mohammad and Elbeheiry, Hani M. and Gil, Mar{\'\i}a Victoria and Glaubitz, Christina and Greiner, Maximilian and Holick, Caroline T. and Hoffmann, Tim and Ibrahim, Abdelrahman and Klepsch, Lea C. and K{\"o}ster, Yannik and Kreth, Fabian Alexander and Meyer, Jakob and Miret, Santiago and Peschel, Jan Matthias and Ringleb, Michael and Roesner, Nicole C. and Schreiber, Johanna and Schubert, Ulrich S. and Stafast, Leanne M. and Wonanke, A. D. Dinga and Pieler, Michael and Schwaller, Philippe and Jablonka, Kevin Maik},
  journal={Nature Chemistry},
  volume={17},
  number={7},
  pages={1027--1034},
  year={2025},
  publisher={Springer Nature},
  doi={10.1038/s41557-025-01815-x}
}

Citation: Fan, S., Xie, Y., Cai, B., Xie, A., Liu, G., Qiao, M., Xing, J., & Nie, Z. (2025). OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discovery. arXiv preprint arXiv:2501.15415. https://doi.org/10.48550/arXiv.2501.15415

Publication: arXiv 2025

Additional Resources:

• Code and Dataset (GitHub)

Multi-Level Chemical Understanding (Method and Resource)

This is primarily a Methodological Paper ($\Psi_{\text{Method}}$) with a significant Resource ($\Psi_{\text{Resource}}$) contribution.

• Methodological: It proposes two novel architectures, DoubleCheck (an enhanced recognition model) and Mol-VL (an end-to-end vision-language model), to solve the newly formulated OCSU task.
• Resource: It constructs and releases Vis-CheBI20, the first large-scale dataset specifically designed for optical chemical structure understanding, containing 29.7K images and 117.7K image-text pairs.

The Motivation for OCSU Beyond Basic Graph Recognition

Existing methods for processing molecular images focus narrowly on Optical Chemical Structure Recognition (OCSR), which translates an image solely into a machine-readable graph or SMILES string. However, SMILES strings are not chemist-friendly and lack high-level semantic context.

• Gap: There is a lack of systems that can translate chemical diagrams into human-readable descriptions (e.g., functional groups, IUPAC names) alongside the graph structure.
• Goal: To enable Optical Chemical Structure Understanding (OCSU), bridging the gap between visual representations and both machine/chemist-readable descriptions to support drug discovery and property prediction.

Key Innovations: DoubleCheck, Mol-VL, and the Vis-CheBI20 Dataset

The paper introduces the OCSU task, enabling multi-level understanding (motif, molecule, and abstract levels). To solve this, it introduces two distinct paradigms:

1. DoubleCheck (OCSR-based): An enhancement to standard OCSR models (like MolScribe) that performs a “second look” at locally ambiguous atoms. It uses attentive feature enhancement to fuse global molecular features with local features from ambiguous regions.
2. Mol-VL (OCSR-free): An end-to-end Vision-Language Model (VLM) based on Qwen2-VL. It uses multi-task learning to directly generate text descriptions from molecular images without an intermediate SMILES step.
3. Vis-CheBI20 Dataset: A new benchmark specifically constructed for OCSU, deriving captions and functional group data from ChEBI-20 and PubChem.

Methodology and Experimental Evaluation

The authors evaluated both paradigms on Vis-CheBI20 and existing benchmarks (USPTO, ACS) across four subtasks:

1. Functional Group Caption: Retrieval/F1 score evaluation.
2. Molecule Description: Natural language generation metrics (BLEU, ROUGE, METEOR).
3. IUPAC Naming: Text generation metrics (BLEU, ROUGE).
4. SMILES Naming (OCSR): Exact matching accuracy ($Acc_s$).

Baselines:

• Task-Specific: MolScribe, MolVec, OSRA.
• LLM/VLM: Qwen2-VL, BioT5+, Mol-Instructions.
• Ablation: DoubleCheck vs. MolScribe backbone to test the “feature enhancement” mechanism.

Results and Conclusions: Paradigm Trade-Offs

• DoubleCheck Superiority: DoubleCheck outperformed MolScribe on OCSR tasks across all benchmarks. On USPTO, it achieved 92.85% $Acc_s$ (vs. 92.57%), and on the ACS dataset it showed a +3.12% gain on chiral molecules. On Vis-CheBI20, DoubleCheck improved over MolScribe by an average of 2.27% across all metrics.
• Paradigm Trade-offs:
  • Mol-VL (OCSR-free) excelled at semantic tasks like Functional Group Captioning, achieving 97.32% F1 (vs. 93.63% for DoubleCheck & RDKit and 89.60% for MolScribe & RDKit). It benefits from end-to-end learning of structural context.
  • DoubleCheck (OCSR-based) performed better on IUPAC naming recall and exact SMILES recovery, as explicit graph reconstruction is more precise for rigid nomenclature than VLM generation.
• Conclusion: Enhancing submodules improves OCSR-based paradigms, while end-to-end VLMs offer stronger semantic understanding but struggle with exact syntax generation (SMILES/IUPAC).

Reproducibility Details

Data

Vis-CheBI20 Dataset

• Source: Derived from ChEBI-20 and PubChem.
• Size: 29,700 molecular diagrams, 117,700 image-text pairs.
• Generation: Images generated from SMILES using RDKit to simulate real-world journal/patent styles.
• Splits (vary by task, see table below):

Algorithms

DoubleCheck (Attentive Feature Enhancement)

1. Ambiguity Detection: Uses atom prediction confidence to identify “ambiguous atoms”.
2. Masking: Applies a 2D Gaussian mask to the image centered on the ambiguous atom.
3. Local Encoding: A Swin-B encoder ($\Phi_l$) encodes the masked image region.
4. Fusion: Aligns local features ($\mathcal{F}_l$) with global features ($\mathcal{F}_g$) using a 2-layer MLP and fuses them via weighted summation.

$$ \begin{aligned} \mathcal{F}_e = \mathcal{F}_g + \text{MLP}(\mathcal{F}_g \oplus \hat{\mathcal{F}}_l) \cdot \hat{\mathcal{F}}_l \end{aligned} $$

5. Two-Stage Training:
   • Stage 1: Train atom/bond predictors (30 epochs).
   • Stage 2: Train alignment/fusion modules with random Gaussian mask noise (10 epochs).

Mol-VL (Multi-Task VLM)

• Prompting: System prompt: “You are working as an excellent assistant in chemistry…”
• Tokens: Uses <image> and </image> special tokens.
• Auxiliary Task: Functional group recognition (identifying highlighted groups) added to training to improve context learning.

Models

• DoubleCheck:
  • Backbone: MolScribe architecture.
  • Encoders: Swin-B for both global and local atom encoding.
• Mol-VL:
  • Base Model: Qwen2-VL (2B and 7B versions).
  • Vision Encoder: ViT with naive dynamic resolution and M-RoPE.

Evaluation

Key Metrics:

• SMILES: Exact Match Accuracy ($Acc_s$), Chiral Accuracy ($Acc_c$).
• Functional Groups: F1 Score (Information Retrieval task).
• Text Generation: BLEU-2/4, METEOR, ROUGE-L.

Selected Results:

Hardware

• DoubleCheck: Trained on 4 NVIDIA A100 GPUs for 4 days.
  • Max LR: 4e-4.
• Mol-VL: Trained on 4 NVIDIA A100 GPUs for 10 days.
  • Max LR: 1e-5, 50 epochs.

Artifacts

Limitations

The authors acknowledge several limitations:

• The long-tail distribution of functional groups in training data limits performance on uncommon chemical structures.
• Mol-VL struggles with exact syntax generation (SMILES and IUPAC) compared to explicit graph-reconstruction approaches.
• Vis-CheBI20 images are synthetically generated via RDKit, which may not fully capture the diversity of real-world journal and patent images.
• The authors note that OCSU technologies should be restricted to research purposes, as downstream molecule discovery applications could potentially generate harmful molecules.

Citation

@misc{fanOCSUOpticalChemical2025,
  title = {OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discovery},
  shorttitle = {OCSU},
  author = {Fan, Siqi and Xie, Yuguang and Cai, Bowen and Xie, Ailin and Liu, Gaochao and Qiao, Mu and Xing, Jie and Nie, Zaiqing},
  year = {2025},
  month = jan,
  number = {arXiv:2501.15415},
  eprint = {2501.15415},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2501.15415},
  archiveprefix = {arXiv}
}

This is a Resource and Benchmarking paper. It introduces Molecular Sets (MOSES), a platform designed to standardize the training, comparison, and evaluation of molecular generative models. It provides a standardized dataset, a suite of evaluation metrics, and a collection of baseline models to serve as reference points for the field.

Motivation: The Reproducibility Crisis in Generative Chemistry

Generative models are increasingly popular for drug discovery and material design, capable of exploring the vast chemical space ($10^{23}$ to $10^{80}$ compounds) more efficiently than traditional methods. However, the field faces a significant reproducibility crisis:

1. Lack of Standardization: There is no consensus on how to properly compare and rank the efficacy of different generative models.
2. Inconsistent Metrics: Different papers use different metrics or distinct implementations of the same metrics.
3. Data Variance: Models are often trained on different subsets of chemical databases (like ZINC), making direct comparison impossible.

MOSES aims to solve these issues by providing a unified “measuring stick” for distribution learning models in chemistry.

Core Innovation: Standardizing Chemical Distribution Learning

The core contribution is the standardization of the distribution learning definition for molecular generation. Why focus on distribution learning? Rule-based filters enforce strict boundaries like molecular weight limits. Distribution learning complements this by allowing chemists to apply implicit or soft restrictions. This ensures that generated molecules satisfy hard constraints and reflect complex chemical realities defined by the training distribution. These realities include the prevalence of certain substructures and the avoidance of unstable motifs.

MOSES specifically targets distribution learning by providing:

1. A Clean, Standardized Dataset: A specific subset of the ZINC Clean Leads collection with rigorous filtering.
2. Diverse Metrics: A comprehensive suite of metrics that measure validity alongside novelty, diversity (internal and external), chemical properties (properties distribution), and substructure similarity.
3. Open Source Platform: A Python library molsets that decouples the data and evaluation logic from the model implementation, ensuring everyone measures performance exactly the same way.

Experimental Setup and Baseline Generative Models

The authors benchmarked a wide variety of generative models against the MOSES dataset to establish baselines:

• Baselines: Character-level RNN (CharRNN), Variational Autoencoder (VAE), Adversarial Autoencoder (AAE), Junction Tree VAE (JTN-VAE), and LatentGAN.
• Non-Neural Baselines: HMM, n-gram models, and a combinatorial generator (randomly connecting fragments).
• Evaluation: Models were trained on the standard set and evaluated on:
  • Validity/Uniqueness: Can the model generate valid, non-duplicate SMILES? Uniqueness is measured at $k = 1{,}000$ and $k = 10{,}000$ samples.
  • Filters: What fraction of generated molecules pass the same medicinal chemistry and PAINS filters used for dataset construction?
  • Feature Distribution: Do generated molecules match the physicochemical properties of the training set? Evaluated using the Wasserstein-1 distance on 1D distributions of:
    • LogP: Octanol-water partition coefficient (lipophilicity).
    • SA: Synthetic Accessibility score (ease of synthesis).
    • QED: Quantitative Estimation of Drug-likeness.
    • MW: Molecular Weight.
  • Fréchet ChemNet Distance (FCD): Measures similarity in biological/chemical space using the penultimate-layer (second-to-last layer) activations of a pre-trained network (ChemNet).
  • Similarity to Nearest Neighbor (SNN): Measures the precision of generation by checking the closest match in the training set (Tanimoto similarity).

Key Findings and Metric Trade-offs

• CharRNN Performance: The simple character-level RNN (CharRNN) outperformed more complex models (like VAEs and GANs) on many metrics, achieving the best FCD scores ($0.073$).
• Metric Trade-offs: No single metric captures “quality.”
  • The Combinatorial Generator achieved 100% validity and high diversity. It struggled with distribution learning metrics (FCD), indicating it explores chemical space broadly without capturing natural distributions.
  • VAEs often achieve high Similarity to Nearest Neighbor (SNN) while exhibiting low novelty. The authors suggest this pattern may indicate overfitting to training set prototypes, though they treat this as a hypothesis rather than a proven mechanism.
• Implicit Constraints: A major finding was that neural models successfully learned implicit chemical rules (like avoiding PAINS structures) purely from the data distribution.
• Recommendation: The authors suggest using FCD/Test for general model ranking, while emphasizing the importance of checking specific metrics (validity, diversity) to diagnose model failure modes.
• Limitations of the Benchmark: MOSES focuses on distribution learning and uses FCD as a primary ranking metric. As the authors note, FCD captures multiple aspects of other metrics in a single number but does not give insights into specific issues, so more interpretable metrics are necessary for thorough investigation. The benchmark evaluates only 1D (SMILES) and 2D molecular features, without assessing 3D conformational properties.

Reproducibility Details

Data

The benchmark uses a curated subset of the ZINC Clean Leads collection.

• Source Size: ~4.6M molecules (4,591,276 after initial extraction).
• Final Size: 1,936,962 molecules.
• Splits: Train (1,584,664), Test (176,075), Scaffold Test (176,226).
  • Scaffold Test Split: This split is crucial for distinct generalization testing. It contains molecules whose Bemis-Murcko scaffolds are completely absent from the training and test sets. Evaluating on this split strictly tests a model’s ability to generate novel chemical structures (generalization).
• Filters Applied:
  • Molecular weight: 250 to 350 Da
  • Rotatable bonds: $\leq 7$
  • XlogP: $\leq 3.5$
  • Atom types: C, N, S, O, F, Cl, Br, H
  • No charged atoms or cycles > 8 atoms
  • Medicinal Chemistry Filters (MCF) and PAINS filters applied.

Evaluation Metrics

MOSES introduces a standard suite of metrics. Key definitions:

• Validity: Fraction of valid SMILES strings (via RDKit).
• Unique@k: Fraction of unique molecules in the first $k$ valid samples ($k = 1{,}000$ and $k = 10{,}000$).
• Filters: Fraction of generated molecules passing the MCF and PAINS filters used during dataset construction. High scores here indicate the model learned implicit chemical validity constraints from the data distribution.
• Novelty: Fraction of generated molecules not present in the training set.
• Internal Diversity (IntDiv): Average Tanimoto distance between generated molecules ($G$), useful for detecting mode collapse: $$ \text{IntDiv}_p(G) = 1 - \sqrt[p]{\frac{1}{|G|^2} \sum_{m_1, m_2 \in G} T(m_1, m_2)^p} $$
• Fragment Similarity (Frag): Cosine similarity of fragment frequency vectors (BRICS decomposition) between generated and test sets.
• Scaffold Similarity (Scaff): Cosine similarity of Bemis-Murcko scaffold frequency vectors between sets. Measures how well the model captures higher-level structural motifs.
• Similarity to Nearest Neighbor (SNN): The average Tanimoto similarity between a generated molecule’s fingerprint and its nearest neighbor in the reference set. This serves as a measure of precision; high SNN suggests the model produces molecules very similar to the training distribution, potentially indicating memorization if novelty is low. $$ \text{SNN}(G, R) = \frac{1}{|G|} \sum_{m_G \in G} \max_{m_R \in R} T(m_G, m_R) $$
• Fréchet ChemNet Distance (FCD): Fréchet distance between the Gaussian approximations (mean and covariance) of penultimate-layer activations from ChemNet. This measures how close the distribution of generated molecules is to the real distribution in chemical/biological space. The authors note that FCD correlates with other metrics. For example, if the generated structures are not diverse enough or the model produces too many duplicates, FCD will decrease because the variance is smaller. The authors suggest using FCD for hyperparameter tuning and final model selection. $$ \text{FCD}(G, R) = |\mu_G - \mu_R|^2 + \text{Tr}(\Sigma_G + \Sigma_R - 2(\Sigma_G \Sigma_R)^{1/2}) $$
• Properties Distribution (Wasserstein-1): The 1D Wasserstein-1 distance between the distributions of molecular properties (MW, LogP, SA, QED) in the generated and test sets.

Models & Baselines

The paper selects baselines to represent different theoretical approaches to distribution learning:

1. Explicit Density Models: Models where the probability mass function $P(x)$ can be computed analytically.
   • N-gram: Simple statistical models. They failed to generate valid molecules reliably due to limited long-range dependency modeling.
2. Implicit Density Models: Models that sample from the distribution without explicitly computing $P(x)$.
   • VAE/AAE: Optimizes a lower bound on the log-likelihood (ELBO) or uses adversarial training.
   • GANs (LatentGAN): Directly minimizes the distance between real and generated distributions via a discriminator.

Models are also distinguished by their data representation:

• String-based (SMILES): Models like CharRNN, VAE, and AAE treat molecules as SMILES strings. SMILES encodes a molecular graph by traversing a spanning tree in depth-first order, storing atom and edge tokens.
• Graph-based: JTN-VAE operates directly on molecular subgraphs (junction tree), ensuring chemical validity by construction but often requiring more complex training.

Key baselines implemented in PyTorch (hyperparameters are detailed in Supplementary Information 3 of the original paper):

• CharRNN: LSTM-based sequence model (3 layers, 768 hidden units). Trained with Adam ($lr = 10^{-3}$, batch size 64, 80 epochs, learning rate halved every 10 epochs).
• VAE: Encoder-decoder architectures (bidirectional GRU encoder, 3-layer GRU decoder with 512 hidden units) with KL regularization.
• AAE: Encoder (single layer bidirectional LSTM with 512 units) and decoder (2-layer LSTM with 512 units) initialized with adversarial formulation.
• LatentGAN: GAN (5-layer fully connected generator) trained on the latent space of a pre-trained heteroencoder.
• JTN-VAE: Tree-structured graph generation.

Code & Hardware Requirements

• Code Repository: Available at github.com/molecularsets/moses as well as the PyPI library molsets. The platform provides standard scripts (scripts/run.py to evaluate models end-to-end, and scripts/run_all_models.sh for multi-seed evaluations).
• Hardware: The repository supports GPU acceleration via nvidia-docker (defaulting to 10GB shared memory). However, specific training times and exact GPU models used by the authors for the baselines are not formally documented in the source text.
• Model Weights: Pre-trained model checkpoints are not natively pre-packaged as standalone downloads; practitioners are expected to re-train the default baselines using the provided scripts.

Artifacts

Paper Information

Citation: Polykovskiy, D., Zhebrak, A., Sanchez-Lengeling, B., Golovanov, S., Tatanov, O., Belyaev, S., Kurbanov, R., Artamonov, A., Aladinskiy, V., Veselov, M., Kadurin, A., Johansson, S., Chen, H., Nikolenko, S., Aspuru-Guzik, A., and Zhavoronkov, A. (2020). Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. Frontiers in Pharmacology, 11, 565644. https://doi.org/10.3389/fphar.2020.565644

Publication: Frontiers in Pharmacology, 2020

@article{polykovskiy2020moses,
  title={Molecular Sets (MOSES): A benchmarking platform for molecular generation models},
  author={Polykovskiy, Daniil and Zhebrak, Alexander and Sanchez-Lengeling, Benjamin and Golovanov, Sergey and Tatanov, Oktai and Belyaev, Stanislav and Kurbanov, Rauf and Artamonov, Aleksey and Aladinskiy, Vladimir and Veselov, Mark and Kadurin, Artur and Johansson, Simon and Chen, Hongming and Nikolenko, Sergey and Aspuru-Guzik, Al{\'a}n and Zhavoronkov, Alex},
  journal={Frontiers in Pharmacology},
  volume={11},
  pages={565644},
  year={2020},
  publisher={Frontiers},
  doi={10.3389/fphar.2020.565644}
}

PubMed-OCR is an OCR-centric corpus of scientific articles derived from PubMed Central Open Access PDFs. Each page image is annotated with Google Cloud Vision and released in a compact JSON schema with word-, line-, and paragraph-level bounding boxes. The corpus spans 209.5K articles (1.5M pages; ~1.3B words) and supports layout-aware modeling, coordinate-grounded QA, and evaluation of OCR-dependent pipelines. We analyze corpus characteristics (e.g., journal coverage and detected layout features) and discuss limitations, including reliance on a single OCR engine and heuristic line reconstruction. We release the data and schema to facilitate downstream research and invite extensions.

Key Contributions

• OCR-First Supervision: Unlike prior datasets for PubMed that align XML to PDFs, PubMed-OCR provides native OCR annotations (Google Cloud Vision), bypassing alignment errors and covering non-digital scanned pages.
• High-Density Annotation: At ~1.3B words across 1.5M pages, PubMed-OCR is far denser per page than comparable corpora like OCR-IDL: ~13x the word density (844 vs. 62.5 words/page) and ~6x the line density (106 vs. 17.5 lines/page), achieved despite drawing from fewer total pages.
• Multi-Level Bounding Boxes: Includes explicit word-, line-, and paragraph-level bounding boxes to support hierarchical document understanding and layout-aware modeling. We also hope that this leads to VQA datasets with grounded answers in document layout.
• Open Access & Reproducibility: Derived strictly from the redistributable PMCOA subset, releasing both the JSON annotations and original PDFs to ensure verifiable and reproducible research.

Technical Implementation

Corpus Construction

PubMed-OCR is built from PubMed Central Open Access (PMCOA) PDFs, chosen specifically because the PMCOA license permits redistribution of both the original documents and derived annotations. Each PDF is rendered to page images, then passed to the Google Cloud Vision (GCV) API. Each page produces a structured JSON annotation file capturing the detected text along with bounding box geometry at word, line, and paragraph levels.

JSON Annotation Schema

Each page annotation follows this compact schema. Bounding boxes are axis-aligned rectangles in [x1, y1, x2, y2] pixel coordinates. Words, lines, and paragraphs are stored as parallel flat lists under the text key:

{
  "text": {
    "words": [
      {"text": "Example", "box": [180, 746, 210, 786]}
    ],
    "lines": [
      {"text": "Example sentence", "box": [180, 746, 540, 786]}
    ],
    "paragraphs": [
      {"text": "Example sentence\nSecond line", "box": [180, 746, 540, 820]}
    ]
  },
  "image": "..."
}

Line Reconstruction

GCV returns word-level detections natively. Line and paragraph groupings are reconstructed using spatial heuristics: words are clustered into lines by vertical overlap and horizontal proximity, and paragraph grouping follows a similar process at a coarser scale. These heuristics work well for standard single-column scientific layouts but can fail on multi-column or irregularly structured pages (see Limitations).

Using the Dataset

The corpus spans 1.5M pages, so streaming is recommended for most use cases:

import json
from datasets import load_dataset

# Streaming is recommended for the full 1.5M-page corpus
ds = load_dataset("rootsautomation/pubmed-ocr", streaming=True, split="train")

# Inspect a page
page = next(iter(ds))
print(f"Article: {page['accession_id']},  Page: {page['page']}")

# Parse OCR annotations
ocr = json.loads(page["ocr_json"])
text = ocr["text"]

# Iterate over lines and words
for line in text["lines"][:5]:
    print(f"  Line: {line['text']}")
    print(f"  BBox: {line['box']}")

# Access individual word detections
for word in text["words"][:5]:
    print(f"  Word: {word['text']}, BBox: {word['box']}")

Full schema documentation is available on the HuggingFace dataset card.

Why This Matters

The lack of large-scale, high-quality OCR datasets with explicit geometric grounding has been a major bottleneck for training layout-aware models. By releasing PubMed-OCR, we provide the community with the dense, multi-level bounding box annotations necessary to build the next generation of document understanding systems. This dataset directly supports the development of models like GutenOCR, enabling them to learn precise token-to-pixel alignment and robust layout reasoning.

Limitations

• Single OCR engine: All annotations come from Google Cloud Vision. GCV’s error modes (handwriting, degraded scans, complex math, non-Latin scripts) propagate uncorrected into the dataset. Different OCR engines could yield different coverage patterns and error distributions.
• Heuristic line reconstruction: Spatial word-to-line clustering is approximate. Multi-column layouts, rotated text, or unusual page orientations may produce incorrect line groupings.
• PMCOA scope: Coverage is limited to the Open Access subset of PubMed Central. Commercial or subscription articles are excluded.

Citation

@misc{heidenreich2026pubmedocrpmcopenaccess,
      title={PubMed-OCR: PMC Open Access OCR Annotations},
      author={Hunter Heidenreich and Yosheb Getachew and Olivia Dinica and Ben Elliott},
      year={2026},
      eprint={2601.11425},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2601.11425},
}

Related Work

This dataset directly enables GutenOCR, a family of vision-language models trained on PubMed-OCR annotations to produce grounded OCR outputs with explicit bounding boxes.

For related work on document processing pipelines that consume OCR output, see LLMs for Page Stream Segmentation and Page Stream Segmentation with LLMs: Challenges and Applications.

This is primarily a Method paper with significant Resource contributions.

• Methodological Basis: The paper introduces a training pipeline (“mix-sourced distillation”) and domain-specific reinforcement learning to improve reasoning capabilities in chemical LLMs. It validates the approach through ablation studies across training stages.
• Resource Contribution: The authors constructed ChemFG, a 101 billion-token corpus annotated with “atomized” knowledge regarding functional groups and reaction centers.

Bridging the Chemical Reasoning Gap

Current chemical LLMs struggle to reason logically for two main reasons:

1. Shallow Domain Understanding: Models generally learn molecule-level properties directly, bypassing the intermediate “atomized” characteristics (e.g., functional groups) that ultimately dictate chemical behavior.
2. Specialized Reasoning Logic: Chemical logic differs fundamentally from math or code. Distilling reasoning from general teacher models like DeepSeek-R1 frequently fails because the teachers lack the domain intuition required to generate valid chemical rationales.

Atomized Knowledge and Mixed-Source Distillation

The authors introduce three structural innovations to solve the reasoning gap:

1. Atomized Knowledge Enhancement (ChemFG): A toolkit was built leveraging SMARTS notations to identify functional group changes during reactions. A critique of this approach is that it relies heavily on 2D cheminformatics abstractions, potentially missing deeper 3D stereochemical interactions.
2. Mix-Sourced Distillation: General models (DeepSeek-R1/o3-mini) are fed “pseudo-reasoning” prompts that include ground truth answers and functional group data. While this forces the teacher to generate high-quality rationales for the student to learn, it introduces a layer of hindsight bias into the generated reasoning chains. During inference, the student model lacks both the pre-calculated functional group metadata and the ground truth, forcing it to bridge an artificially steep generalization gap.
3. Chemical Reinforcement Learning: The intermediate model undergoes domain-specific reinforcement learning. The RL details are described in the paper’s Appendix D, with the authors citing the open-source DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) framework. The optimization relies on rule-based rewards (format adherence and canonicalized SMILES accuracy) across a variety of chemical tasks.

Benchmark Evaluation and Ablation Studies

The model was evaluated on comprehensive chemical benchmarks: SciKnowEval (19 tasks) and ChemEval (36 tasks).

• Baselines: Compared against similarly sized open models (Qwen2.5-14B-Instruct, Qwen3-14B), domain models (ChemLLM, MolInst), and frontier models (GPT-4o, DeepSeek-R1).
• Ablation: Evaluated across training stages (Base → ChemDFM-I → ChemDFM-R) to measure the specific impact of the instruction tuning versus the reasoning stages.
• Qualitative Analysis: The paper includes case studies demonstrating the model’s step-by-step chemical reasoning and its potential for human-AI collaboration (Sections 4.2 and 4.3).

Performance Outcomes and Numerical Limitations

• Performance vs. Baselines: ChemDFM-R outperforms similarly sized open models and domain models on molecule-centric and reaction-centric tasks, and surpasses the much larger DeepSeek-R1 on ChemEval (0.78 vs. 0.58 overall). It shows competitive results relative to o4-mini, though o4-mini leads on SciKnowEval (0.74 vs. 0.70).
• Reasoning Interactivity: The model generates readable rationales that allow users to catch structural errors or identify reaction mechanisms accurately. Section 4.3 of the paper demonstrates human-AI collaboration scenarios.
• Quantitative Limitations: The model struggles with tasks involving numerical prediction and calculation (e.g., yield extraction, molecular property calculation). The paper notes that all molecule-centric and reaction-centric tasks where ChemDFM-R falls short of Qwen2.5-14B-Instruct involve numerical reasoning.

Reproducibility Details

Data

The training data is constructed in three phases:

1. Domain Pre-training (ChemFG):

• Size: 101 billion tokens
• Composition:
  • 12M literature documents (79B tokens)
  • 30M molecules from PubChem/PubChemQC
  • 7M reactions from USPTO-FULL
• Augmentation: SMILES augmentation (10x) using R-SMILES
• Atomized Features: Annotated with a custom “Functional Group Identification Toolkit” that identifies 241 functional group types and tracks changes in reaction centers. Note: Data and toolkit are partially reproduced; while the toolkit (ChemFG-Tool) was open-sourced on GitHub, the 101 billion-token ChemFG dataset itself has not been publicly released.

2. Instruction Tuning:

• Sources: Molecule-centric (PubChem, MoleculeNet), Reaction-centric (USPTO), and Knowledge-centric (Exams, Literature QA) tasks
• Mixing: Mixed with general instruction data in a 1:2 ratio

3. Distillation Dataset:

• Sources:
  • ~70% ChemDFM-R instruction data
  • ~22% constructed pseudo-reasoning (functional group descriptions)
  • ~8% teacher rationales (from DeepSeek-R1/o3-mini)
• Mixing: Mixed with general data (including AM-Deepseek-R1-Distill-1.4M) in a 1:2 ratio

Algorithms

Functional Group Identification:

• Extends the thermo library’s SMARTS list
• For reactions, identifies “reacting functional groups” by finding reactants containing atoms involved in bond changes (reaction centers) that do not appear in the product

Mix-Sourced Distillation:

• Teacher models (DeepSeek-R1, o3-mini) are prompted with Question + Ground Truth + Functional Group Info to generate high-quality “Thoughts”
• These rationales are distilled into the student model using a supervised fine-tuning loss across target tokens $y_t$: $$ \mathcal{L}_{\text{SFT}} = - \sum_{t=1}^T \log P_\theta(y_t \mid x, y_{<t}) $$

Reinforcement Learning:

• Algorithm: The paper cites DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) as the RL framework; full details are in Appendix D of the paper. Note: While the underlying DAPO framework is open-source, the specific chemistry-oriented RL pipeline and environment used for ChemDFM-R has not been publicly released.
• Hyperparameters (from paper appendix): Learning rate 5e-7, rollout batch size 512, training batch size 128
• Rewards: The reward system applies rule-based constraints focusing on physical form and chemical validity. The total reward $R(y, y^*)$ for a generated response $y$ given target $y^*$ combines a format adherence reward ($R_{\text{format}}$) and an accuracy reward ($R_{\text{acc}}$) evaluated on canonicalized SMILES: $$ R(y, y^*) = R_{\text{format}}(y) + R_{\text{acc}}(\text{canonicalize}(y), \text{canonicalize}(y^*)) $$

Models

• Base Model: Qwen2.5-14B
• ChemDFM-I: Result of instruction tuning the domain-pretrained model for 2 epochs
• ChemDFM-R: Result of applying mix-sourced distillation (1 epoch) followed by RL on ChemDFM-I. Note: Model weights are publicly available on Hugging Face.

Hardware

Hardware and training time details are described in the paper’s appendices, which are not available in the extracted text. The details below are reported from the paper but could not be independently cross-verified against the main text:

• Compute: NVIDIA A800 Tensor Core GPUs
• Training Time: 30,840 GPU hours total (Domain Pretraining: 24,728 hours; Instruction Tuning: 3,785 hours; Distillation: 2,059 hours; Reinforcement Learning: 268 hours)

Evaluation

Benchmarks:

• SciKnowEval: 19 tasks (text-centric, molecule-centric, reaction-centric)
• ChemEval: 36 tasks, categorized similarly

Key Metrics: Accuracy, F1 Score, BLEU score (with PRS normalization for ChemEval)

Artifacts

Missing components: The 101B-token ChemFG pretraining dataset is not publicly released. The chemistry-oriented RL pipeline and training code are not open-sourced. The instruction tuning and distillation datasets are not available.

Paper Information

Citation: Zhao, Z., Chen, B., Wan, Z., Chen, L., Lin, X., Yu, S., Zhang, S., Ma, D., Zhu, Z., Zhang, D., Wang, H., Dai, Z., Wen, L., Chen, X., & Yu, K. (2025). ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledge. arXiv preprint arXiv:2507.21990. https://doi.org/10.48550/arXiv.2507.21990

Publication: arXiv 2025

@misc{zhao2025chemdfmr,
  title={ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledge},
  author={Zihan Zhao and Bo Chen and Ziping Wan and Lu Chen and Xuanze Lin and Shiyang Yu and Situo Zhang and Da Ma and Zichen Zhu and Danyang Zhang and Huayang Wang and Zhongyang Dai and Liyang Wen and Xin Chen and Kai Yu},
  year={2025},
  eprint={2507.21990},
  archivePrefix={arXiv},
  primaryClass={cs.CE},
  url={https://arxiv.org/abs/2507.21990}
}

This is primarily a Method paper ($\Psi_{\text{Method}}$), with a significant Resource component ($\Psi_{\text{Resource}}$).

It is a methodological investigation because it systematically evaluates a specific architecture (Transformers/RoBERTa) against established State-of-the-Art (SOTA) baselines like directed Message Passing Neural Networks (D-MPNNs) to determine “how well does this work?” in the chemical domain. It ablates dataset size, tokenization, and input representation.

It is also a resource paper as it introduces “PubChem-77M,” a curated dataset of 77 million SMILES strings designed to facilitate large-scale self-supervised pretraining for the community.

Overcoming Data Scarcity in Property Prediction

The primary motivation is data scarcity in molecular property prediction. Graph Neural Networks (GNNs) achieve strong performance on property prediction tasks when provided with sufficient labeled data. Generating these labels requires costly and time-consuming laboratory testing, leading to severe data scarcity in specialized chemical domains.

Massive quantities of unlabeled chemical structure data exist in the form of SMILES strings. Inspired by the success of Transformers in NLP, where self-supervised pretraining on large corpora yields strong transfer learning, the authors aim to use these unlabeled datasets to learn effective molecular representations. Additionally, Transformers benefit from a mature software ecosystem (HuggingFace) that offers efficiency advantages over GNNs.

Pretraining Scaling Laws and Novelty

Previous works applied Transformers to SMILES strings. This paper advances the field by systematically evaluating scaling laws and architectural components for this domain. Specifically:

• Scaling Analysis: It explicitly tests how pretraining dataset size (100K to 10M) impacts downstream performance.
• Tokenizer Comparison: It compares standard NLP Byte-Pair Encoding (BPE) against a chemically-aware “SmilesTokenizer”.
• Representation Comparison: It evaluates if the robust SELFIES string representation offers advantages over standard SMILES in a Transformer context.

Experimental Setup: Pretraining and Finetuning

The authors trained ChemBERTa (based on RoBERTa) using Masked Language Modeling (MLM) on subsets of the PubChem dataset. The core training objective minimizes the cross-entropy loss over a corrupted input where a subset of basic tokens, denoted by $\mathcal{M}$, are masked:

$$ \mathcal{L}_{\text{MLM}} = - \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \log P(x_i \mid x_{\setminus \mathcal{M}}; \theta) $$

where $x_i$ is the exact masked token, $x_{\setminus \mathcal{M}}$ is the corrupted SMILES context string, and $\theta$ represents the network parameters.

• Pretraining: Models were pretrained on dataset sizes of 100K, 250K, 1M, and 10M compounds.
• Baselines: Performance was compared against D-MPNN (Graph Neural Network), Random Forest (RF), and SVM using 2048-bit Morgan Fingerprints.
• Downstream Tasks: Finetuning was performed individually on small MoleculeNet classification tasks: BBBP (blood-brain barrier), ClinTox (clinical toxicity), HIV, and Tox21 (p53 stress-response). This poses a transfer learning challenge, as the model must adapt from pretraining on 10 million molecules to classifying datasets ranging from ~1.5K to ~41K examples.
• Ablations:
  • Tokenization: BPE vs. SmilesTokenizer on the 1M dataset, evaluated on Tox21.
  • Input: SMILES vs. SELFIES strings on the Tox21 task.

Results vs. Graph Neural Network Baselines

The main comparison between ChemBERTa (pretrained on 10M compounds) and Chemprop baselines on MoleculeNet tasks is summarized below (Table 1 from the paper):

• Scaling Improvements & Training Dynamics: Performance scales predictably with pretraining data size. Increasing data from 100K to 10M improved ROC-AUC by +0.110 and PRC-AUC by +0.059 on average across BBBP, ClinTox, and Tox21 (HIV was omitted due to resource constraints). Notably, researchers had to halt pretraining on the 10M subset after just 3 epochs due to overfitting, suggesting that simple 15% token masking might not provide a sufficiently difficult learning curvature for large-scale chemical representation.
• Performance Limits vs. GNNs: ChemBERTa generally performs below the D-MPNN baseline. On the Tox21 dataset, ChemBERTa-10M achieved a higher ROC-AUC (0.728) than D-MPNN (0.688); nonetheless, it recorded a substantially lower PRC-AUC (0.207 vs 0.429). This gap indicates that current Transformer iterations lack the explicit inductive biases of graph algorithms and struggle with the severe class imbalances typical of chemical datasets.
• Ablation Limitations (Tokenization & SELFIES): The authors’ ablation studies for tokenization (SmilesTokenizer narrowly beating BPE) and input representation (SELFIES performing comparably to SMILES) were evaluated exclusively on the single Tox21 task. Deriving broad architectural conclusions regarding “semantically-aware tokenization” or string robustness from an $N=1$ empirical evaluation is a significant limitation of the study. Broader benchmarking is required to validate these findings.
• Interpretability: Attention heads organically learn to track chemically relevant substructures (like specific functional groups and aromatic rings), mimicking the inductive biases of graph convolutions.

Reproducibility Details

Data

The authors curated a massive dataset for pretraining and utilized standard benchmarks for evaluation.

• Pretraining Data: PubChem-77M.
  • Source: 77 million unique SMILES from PubChem.
  • Preprocessing: Canonicalized and globally shuffled.
  • Subsets used: 100K, 250K, 1M, and 10M subsets.
  • Availability Note: The authors provided a direct link to the canonicalized 10M compound subset used for their largest experiments. Full reproducibility of the smaller (100K, 250K, 1M) or full 77M sets may require re-extracting from PubChem.
• Evaluation Data: MoleculeNet.
  • Tasks: BBBP (2,039), ClinTox (1,478), HIV (41,127), Tox21 (7,831).
  • Splitting: 80/10/10 train/valid/test split using a scaffold splitter to ensure chemical diversity between splits.

Algorithms

The core training methodology mirrors standard BERT/RoBERTa procedures adapted for chemical strings.

• Objective: Masked Language Modeling (MLM) with 15% token masking.
• Tokenization:
  • BPE: Byte-Pair Encoder (vocab size 52K).
  • SmilesTokenizer: Regex-based custom tokenizer available in DeepChem (documented here).
• Sequence Length: Maximum sequence length of 512 tokens.
• Finetuning: Appended a linear classification layer; backpropagated through the base model for up to 25 epochs with early stopping on ROC-AUC.

Models

• Architecture: RoBERTa (via HuggingFace).
  • Layers: 6
  • Attention Heads: 12 (72 distinct mechanisms total).
  • Implementation Note: The original training notebooks and scripts are maintained in the authors’ bert-loves-chemistry repository, alongside the primary downstream tasks integrated into DeepChem. A full Tox21 transfer learning tutorial has been incorporated into the DeepChem repository.
• Baselines (via Chemprop library):
  • D-MPNN: Directed Message Passing Neural Network with default hyperparameters.
  • RF/SVM: Scikit-learn Random Forest and SVM using 2048-bit Morgan fingerprints (RDKit).

Evaluation

Performance is measured using dual metrics to account for class imbalance common in toxicity datasets.

Hardware

• Compute: Single NVIDIA V100 GPU.
• Training Time: Approximately 48 hours for the 10M compound subset.
• Carbon Footprint: Estimated 17.1 kg $\text{CO}_2\text{eq}$ (offset by Google Cloud).

Artifacts

Reproducibility status: Partially Reproducible. Code and pre-trained models are available, and the 10M pretraining subset is downloadable. However, smaller subsets (100K, 250K, 1M) may need re-extraction from PubChem, and exact hyperparameter details for finetuning (learning rate, batch size) are not fully specified in the paper.

Paper Information

Citation: Chithrananda, S., Grand, G., & Ramsundar, B. (2020). ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction. arXiv preprint arXiv:2010.09885. https://doi.org/10.48550/arXiv.2010.09885

Publication: arXiv 2020 (Preprint)

Additional Resources:

• HuggingFace Model Hub (ChemBERTa-zinc-base-v1) - Additional pre-trained variations on PubChem & ZINC datasets are available on the author’s seyonec HF profile.
• bert-loves-chemistry GitHub Repository - Notebooks and scripts used for MLM pretraining and finetuning evaluations.

BibTeX

@misc{chithranandaChemBERTaLargeScaleSelfSupervised2020,
  title = {{{ChemBERTa}}: {{Large-Scale Self-Supervised Pretraining}} for {{Molecular Property Prediction}}},
  shorttitle = {{{ChemBERTa}}},
  author = {Chithrananda, Seyone and Grand, Gabriel and Ramsundar, Bharath},
  year = 2020,
  month = oct,
  number = {arXiv:2010.09885},
  eprint = {2010.09885},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2010.09885},
  urldate = {2025-12-24},
  archiveprefix = {arXiv}
}

This is primarily a Methodological paper ($\Psi_{\text{Method}}$) that introduces a novel pipeline (MERMaid) for extracting structured chemical data from unstructured PDF documents. It proposes a specific architecture combining fine-tuned vision models (VisualHeist) with vision-language models (DataRaider) and a retrieval-augmented generation system (KGWizard) to solve the problem of multimodal data ingestion.

Secondarily, it is a Resource paper ($\Psi_{\text{Resource}}$) as it releases the source code, prompts, and a new benchmark dataset (MERMaid-100) consisting of annotated reaction data across three chemical domains.

The Inaccessibility of Diagrammatic Reaction Data

• Data Inaccessibility: A significant volume of chemical knowledge currently resides in “print-optimized” PDF formats, specifically within graphical elements like figures, schemes, and tables, which resist standard text mining.
• Limitations of Prior Work: Existing tools (e.g., ChemDataExtractor, OpenChemIE) focus primarily on text, struggle with multimodal parsing, or lack the “contextual awareness” needed to interpret implicit information (e.g., “standard conditions” with modifications in optimization tables).
• Need for Structured Data: To enable self-driving laboratories and data-driven discovery, this unstructured literature must be converted into machine-actionable formats like knowledge graphs.

The MERMaid Pipeline: Vision Models and LLM RAG

• VisualHeist (Fine-tuned Segmentation): A custom fine-tuned model based on Microsoft’s Florence-2 that accurately segments figures, captions, and footnotes, even in messy supplementary materials.
• DataRaider (Context-Aware Extraction): A VLM-powered module (using GPT-4o) with a two-step prompt framework that performs “self-directed context completion.” It can infer missing reaction parameters from context and resolve footnote labels (e.g., linking “condition a” in a table to its footnote description).
• KGWizard (Schema-Adaptive Graph Construction): A text-to-graph engine that uses LLMs as higher-order functions to synthesize parsers dynamically. It employs Retrieval-Augmented Generation (RAG) to check for existing nodes during creation, implicitly resolving coreferences (e.g., unifying “MeCN” and “Acetonitrile”).
• Topic-Agnostic Design: MERMaid features a flexible design that works across three distinct domains: organic electrosynthesis, photocatalysis, and organic synthesis.

Benchmarking Segmentation and Extraction Accuracy

• Segmentation Benchmarking: The authors compared VisualHeist against OpenChemIE (LayoutParser) and PDFigCapX using a dataset of 121 PDFs from 5 publishers.
• End-to-End Extraction: Evaluated the full pipeline on MERMaid-100, a curated dataset of 100 articles across three domains (organic electrosynthesis, photocatalysis, organic synthesis).
  • Validating extraction of specific parameters (e.g., catalysts, solvents, yields) using “hard-match” accuracy.
• Knowledge Graph Construction: Automatically generated knowledge graphs for the three domains and assessed the structural integrity and coreference resolution accuracy.

End-to-End Extraction Performance

• Segmentation Results: VisualHeist achieved >93% F1 score across all document types (including pre-2000 papers and supplementary materials), outperforming OpenChemIE by 15-75% and PDFigCapX by 28-75% across all metrics.
• Extraction Accuracy: DataRaider achieved >92% accuracy for VLM-based parameter extraction and near-unity accuracy for domain-specific reaction parameters (e.g., anode, cathode, photocatalyst).
• Graph Building: KGWizard achieved 96% accuracy in node creation and coreference resolution.
• Overall Performance: The pipeline demonstrated an 87% end-to-end overall accuracy.
• Limitations: The architecture relies heavily on closed-weight models (GPT-4o) for reasoning and graph construction, which risks future reproducibility if API snapshots are deprecated. Additionally, the system remains vulnerable to cumulative error propagation from upstream OCR/OCSR tools like RxnScribe.
• Availability: The authors provide a modular, extensible framework that can be adapted to other scientific domains.

Reproducibility Details

Data

• Training Data (VisualHeist):
  • Dataset of 3,435 figures and 1,716 tables annotated from 3,518 PDF pages.
  • Includes main text, supplementary materials, and unformatted archive papers.
• Evaluation Data (MERMaid-100):
  • 100 PDF articles curated from three domains: organic electrosynthesis, photocatalysis, and organic synthesis.
  • Includes 104 image-caption/table-heading pairs relevant to reaction optimization.
  • Available for download at Zenodo (DOI: 10.5281/zenodo.14917752).

Algorithms

• Two-Step Prompt Framework (DataRaider):
  • Step 1: Generic base prompt + domain keys to extract “reaction dictionaries” and “footnote dictionaries”. Uses “fill-in-the-blank” inference for missing details.
  • Step 2: Safety check prompt where the VLM updates the reaction dictionary using the footnote dictionary to resolve entry-specific modifications.
• LLM-Synthesized Parsers (KGWizard):
  • Uses LLM as a function $g_{A,B}: A \times B \rightarrow (X \rightarrow Y)$ to generate Python code (parsers) dynamically based on input schema instructions.
• RAG for Coreference:
  • During graph construction, the system queries the existing database for matching values (e.g., “MeCN”) before creating new nodes to prevent duplication.
• Batching:
  • Articles processed in dynamic batch sizes (starting at 1, increasing to 30) to balance speed and redundancy checks.

Models

• VisualHeist: Fine-tuned Florence-2-large (Microsoft vision foundation model).
  • Hyperparameters: 12 epochs, learning rate $5 \times 10^{-6}$, batch size 4.
• DataRaider & KGWizard: GPT-4o (version gpt-4o-2024-08-06). Note: Requires an active OpenAI API key. The pipeline’s long-term reproducibility is currently tied to the continued availability of this specific closed-source endpoint.
• RxnScribe: Used for Optical Chemical Structure Recognition (OCSR) to convert reactant/product images to SMILES.

Evaluation

• Metrics:
  • Segmentation: Precision, Recall, F1, Accuracy.
  • Caption Extraction: Evaluated via Jaccard similarity, mapping predicted token sets $A$ and true token sets $B$ to a threshold condition: $$J(A, B) = \frac{|A \cap B|}{|A \cup B|} \ge 0.70$$
  • Data Extraction: Evaluated via Hard-Match accuracy, requiring exact correspondence between predicted sets ($\hat{Y}$) and ground-truth parameters ($Y$) for specific roles (e.g., anode vs. cathode): $$\text{HMA} = \frac{1}{|N|} \sum_{i=1}^{N} \mathbb{1}[y_i = \hat{y}_i]$$
• Baselines: OpenChemIE (LayoutParser + EasyOCR) and PDFigCapX.

Hardware

• Training (VisualHeist): 2x NVLINK Nvidia RTX A6000 GPUs (48GB VRAM) + Intel Xeon w7-2495X CPU (48 cores).
• DataRaider Evaluation: 13th Gen Intel Core i7-1360P CPU (12 cores).
• Inference Costs:
  • DataRaider: ~$0.051 per image.
  • KGWizard: ~$0.40 per JSON.
• Timing:
  • VisualHeist inference: ~4.5 seconds/image.
  • DataRaider inference: ~41.3 seconds/image.
  • KGWizard processing: ~110.6 seconds/file.

Paper Information

Citation: Leong, S. X., Pablo-García, S., Wong, B., & Aspuru-Guzik, A. (2025). MERMaid: Universal multimodal mining of chemical reactions from PDFs using vision-language models. Matter, 8(12), 102331. https://doi.org/10.1016/j.matt.2025.102331

Publication: Matter, 2025

Artifacts:

@article{leong2025mermaid,
  title={MERMaid: Universal multimodal mining of chemical reactions from PDFs using vision-language models},
  author={Leong, Shi Xuan and Pablo-Garc{\'i}a, Sergio and Wong, Brandon and Aspuru-Guzik, Al{\'a}n},
  journal={Matter},
  volume={8},
  number={12},
  pages={102331},
  year={2025},
  doi={10.1016/j.matt.2025.102331}
}

This is primarily a Methodological paper that proposes a novel neural architecture (MolGrapher), shifting the paradigm of Optical Chemical Structure Recognition (OCSR) from image captioning back to graph reconstruction. It also has a significant Resource component, releasing a synthetic data generation pipeline and a new large-scale benchmark (USPTO-30K) to address the scarcity of annotated real-world data.

2. Motivation

The automatic analysis of chemical literature is critical for accelerating drug and material discovery, but much of this information is locked in 2D images of molecular structures.

• Problem: Existing rule-based methods are rigid, while recent deep learning methods based on “image captioning” (predicting SMILES strings) struggle with complex molecules and fail to exploit the natural graph structure of molecules.
• Gap: There is a lack of diverse, annotated real-world training data, and captioning models suffer from “hallucinations” where they predict valid SMILES that do not match the image.

3. Novelty / Core Innovation

MolGrapher introduces a graph-based deep learning pipeline that explicitly models the molecule’s geometry and topology.

• Supergraph Concept: It first detects all atom keypoints and builds a “supergraph” of all plausible bonds.
• Hybrid Approach: It combines a ResNet-based keypoint detector with a Graph Neural Network (GNN) that classifies both atom nodes and bond nodes within the supergraph context. Both atoms and bonds are represented as nodes, with edges only connecting atom nodes to bond nodes.
• Synthetic Pipeline: A data generation pipeline that renders molecules with varying styles (fonts, bond widths) and augmentations (pepper patches, random lines, captions) to simulate real document noise.

At the core of the Keypoint Detector’s performance is the Weight-Adaptive Heatmap Regression (WAHR) loss. Since pixels without an atom drastically outnumber pixels containing an atom, WAHR loss is designed to counter the class imbalance. For ground-truth heatmap $y$ and prediction $p$:

$$ L_{WAHR}(p, y) = \sum_i \alpha_y (p_i - y_i)^2 $$

where $\alpha_y$ dynamically down-weights easily classified background pixels.

4. Methodology & Experiments

The authors evaluated MolGrapher against both rule-based (OSRA, MolVec) and deep learning baselines (DECIMER, Img2Mol, Image2Graph).

• Benchmarks: Evaluated on standard datasets: USPTO, Maybridge UoB, CLEF-2012, and JPO.
• New Benchmark: Introduced and tested on USPTO-30K, split into clean, abbreviated, and large molecule subsets.
• Ablations: Analyzed the impact of synthetic augmentations, keypoint loss functions, supergraph connectivity radius, and GNN layers.
• Robustness: Tested on perturbed images (rotations, shearing) to mimic scanned patent quality.

The GNN iteratively updates node embeddings through layers ${g^k}_{k \in [1, N]}$, where $e^{k+1} = g^k(e^k)$. Final predictions are obtained via two MLPs (one for atoms, one for bonds): $p_i = MLP_t(e_i^N)$, where $p_i \in \mathbb{R}^{C_t}$ contains the logits for atom or bond classes.

5. Results & Conclusions

MolGrapher achieved the highest accuracy among synthetic-only deep learning methods on most benchmarks tested.

• Accuracy: It achieved 91.5% accuracy on USPTO, outperforming all other synthetic-only deep learning methods including ChemGrapher (80.9%), Graph Generation (67.0%), and DECIMER 2.0 (61.0%).
• Large Molecules: It demonstrated superior scaling, correctly recognizing large molecules (USPTO-10K-L) where image captioning methods like Img2Mol failed completely (0.0% accuracy).
• Generalization: The method proved robust to image perturbations and style variations without requiring fine-tuning on real data. The paper acknowledges that MolGrapher cannot recognize Markush structures (depictions of sets of molecules with positional and frequency variation indicators).

Reproducibility Details

Data

The model relies on synthetic data for training due to the scarcity of annotated real-world images.

Algorithms

The pipeline consists of three distinct algorithmic stages:

1. Keypoint Detection:

   • Predicts a heatmap of atom locations using a CNN.
   • Thresholds heatmaps at the bottom 10th percentile and uses a $5\times5$ window for local maxima.
   • Uses Weight-Adaptive Heatmap Regression (WAHR) loss to handle class imbalance (background vs. atoms).

2. Supergraph Construction:

   • Connects every detected keypoint to neighbors within a radius of $3 \times$ the estimated bond length.
   • Prunes edges with no filled pixels or if obstructed by a third keypoint.
   • Keeps a maximum of 6 bond candidates per atom.

3. Superatom Recognition:

   • Detects “superatom” nodes (abbreviations like COOH).
   • Uses PP-OCR to transcribe the text at these node locations.

Models

The architecture utilizes standard backbones tailored for specific sub-tasks:

• Keypoint Detector: ResNet-18 backbone with $8\times$ dilation to preserve spatial resolution.
• Node Classifier: ResNet-50 backbone with $2\times$ dilation for extracting visual features at node locations.
• Graph Neural Network: A custom GNN that updates node embeddings based on visual features and neighborhood context. The initial node embedding combines the visual feature vector $v_i$ and a learnable type encoding $w_{t_i}$.
• Readout: MLPs classify nodes into atom types (e.g., C, O, N) and bond types (No Bond, Single, Double, Triple).

Evaluation

Accuracy is defined strictly: the predicted molecule must have an identical InChI string to the ground truth. Stereochemistry and Markush structures are excluded from evaluation.

Hardware

• GPUs: Trained on 3 NVIDIA A100 GPUs.
• Training Time: 20 epochs.
• Optimization: ADAM optimizer, learning rate 0.0001, decayed by 0.8 after 5000 iterations.
• Loss Weighting: Atom classifier loss weighted by 1; bond classifier loss weighted by 3.

Artifacts

Paper Information

Title: MolGrapher: Graph-based Visual Recognition of Chemical Structures

Authors: Lucas Morin, Martin Danelljan, Maria Isabel Agea, Ahmed Nassar, Valéry Weber, Ingmar Meijer, Peter Staar, Fisher Yu

Citation: Morin, L., Danelljan, M., Agea, M. I., Nassar, A., Weber, V., Meijer, I., Staar, P., & Yu, F. (2023). MolGrapher: Graph-based Visual Recognition of Chemical Structures. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 19552-19561.

Publication: ICCV 2023

Links:

• Paper
• GitHub Repository

@inproceedings{morinMolGrapherGraphbasedVisual2023,
  title = {{{MolGrapher}}: {{Graph-based Visual Recognition}} of {{Chemical Structures}}},
  shorttitle = {{{MolGrapher}}},
  booktitle = {Proceedings of the {{IEEE}}/{{CVF International Conference}} on {{Computer Vision}}},
  author = {Morin, Lucas and Danelljan, Martin and Agea, Maria Isabel and Nassar, Ahmed and Weber, Valéry and Meijer, Ingmar and Staar, Peter and Yu, Fisher},
  year = {2023},
  pages = {19552--19561},
  doi = {10.1109/ICCV51070.2023.01791},
  urldate = {2025-10-18},
  langid = {english}
}

This is primarily a Resource paper (Infrastructure Basis) with a significant Method component.

The primary contribution is DECIMER.ai, a fully open-source platform (web app and Python packages) for the entire chemical structure mining pipeline, filling a gap where most tools were proprietary or fragmented. It also contributes the RanDepict toolkit for massive synthetic data generation.

The secondary methodological contribution proposes and validates a specific deep learning architecture (EfficientNet-V2 encoder + Transformer decoder) that treats chemical structure recognition as an image-to-text translation task (SMILES generation).

The Scarcity of Machine-Readable Chemical Data

Data Scarcity: While the number of chemical publications is increasing, most chemical information is locked in non-machine-readable formats (images in PDFs) and is not available in public databases.

Limitations of Existing Tools: Prior OCSR (Optical Chemical Structure Recognition) tools were largely rule-based (fragile to noise) or proprietary.

Lack of Integration: There was no existing open-source system that combined segmentation (finding the molecule on a page), classification (confirming it is a molecule), and recognition (translating it to SMILES) into a single workflow.

DECIMER Architecture and Novel Image-to-SMILES Approach

Comprehensive Workflow: It is the first open-source platform to integrate segmentation (Mask R-CNN), classification (EfficientNet), and recognition (Transformer) into a unified pipeline.

Data-Driven Approach: Unlike tools like MolScribe which use intermediate graph representations and rules, DECIMER uses a purely data-driven “image-to-SMILES” translation approach without hard-coded chemical rules. The core recognition model operates as a sequence-to-sequence generator, mathematically formalizing the task as maximizing the conditional probability of a SMILES sequence given an image.

Massive Synthetic Training: The use of RanDepict to generate over 450 million synthetic images, covering diverse depiction styles and augmentations (including Markush structures), to train the model from scratch.

Benchmarking and Evaluation Methodology

Benchmarking: The system was tested against openly available tools (OSRA, MolVec, Imago, Img2Mol, SwinOCSR, MolScribe) on standard datasets: USPTO, UOB, CLEF, JPO, and a custom “Hand-drawn” dataset.

Robustness Testing: Performance was evaluated on both clean images and images with added distortions (rotation, shearing) to test the fragility of rule-based systems vs. DECIMER.

Markush Structure Analysis: Specific evaluation of the model’s ability to interpret Markush structures (generic structures with R-groups).

Comparison of Approaches: A direct comparison with MolScribe by training DECIMER on MolScribe’s smaller training set to isolate the impact of architecture vs. data volume.

Performance Outcomes and Key Findings

Comparative Performance: DECIMER Image Transformer consistently produced average Tanimoto similarities above 0.95 on in-domain test data and achieved competitive or leading results across external benchmarks, with extremely low rates of catastrophic failure. Tanimoto similarity is calculated based on molecular fingerprints $A$ and $B$ as: $$ T(A, B) = \frac{A \cdot B}{|A|^2 + |B|^2 - A \cdot B} $$

Data Volume Necessity: When trained on small datasets, MolScribe (graph/rule-based) outperformed DECIMER. DECIMER’s performance advantage relies heavily on its massive training scale (>400M images).

Robustness: The model showed no performance degradation on distorted images, unlike rule-based legacy tools.

Generalization: Despite having no hand-drawn images in the training set, the base model recognized 27% of hand-drawn structures perfectly (average Tanimoto 0.69), outperforming all alternative open tools. After fine-tuning with synthetic hand-drawn-like images from RanDepict, perfect predictions increased to 60% (average Tanimoto 0.89).

Reproducibility

Artifacts

Data

The models were trained on synthetic data generated from PubChem molecules.

Data Generation:

• Tool: RanDepict (uses CDK, RDKit, Indigo, PIKAChU)
• Augmentations: Rotation, shearing, noise, pixelation, curved arrows, text labels
• Format: Data saved as TFRecord files for TPU training

Algorithms

• SMILES Tokenization: Regex-based splitting (atoms, brackets, bonds). Added <start>, <end>, and padded with <pad>. <unk> used for unknown tokens.
• Markush Token Handling: To avoid ambiguity, digits following ‘R’ (e.g., R1) were replaced with unique non-digit characters during training to distinguish them from ring-closure numbers.
• Image Augmentation Pipeline: Custom RanDepict features (v1.1.4) were used to simulate “hand-drawn-like” styles based on ChemPIX’s implementation.

Models

The platform consists of three distinct models:

1. DECIMER Segmentation:

   • Architecture: Mask R-CNN (TensorFlow 2.10.0 implementation)
   • Purpose: Detects and cuts chemical structures from full PDF pages

2. DECIMER Image Classifier:

   • Architecture: EfficientNet-V1-B0
   • Input: 224x224 pixels
   • Training: Fine-tuned on ~10.9M images (balanced chemical/non-chemical)
   • Performance: AUC 0.99 on in-domain test set

3. DECIMER Image Transformer (OCSR Engine):

   • Encoder: EfficientNet-V2-M (CNN). Input size 512x512. 52M parameters
   • Decoder: Transformer. 4 encoder blocks, 4 decoder blocks, 8 attention heads. d_model=512, d_ff=2048. 59M parameters
   • Total Params: ~111 Million

Evaluation

• Primary Metric: Tanimoto Similarity (calculated on PubChem fingerprints of the predicted vs. ground truth SMILES)
• Secondary Metrics: Exact Match (Identity), BLEU score (for string similarity, esp. Markush)
• Failure Analysis: “Catastrophic failure” defined as Tanimoto similarity of 0 or invalid SMILES

Hardware

Training was performed on Google Cloud TPUs due to the massive dataset size.

• pubchem_1/pubchem_2: Trained on TPU v3-32 pod slice
• pubchem_3 (Final Model): Trained on TPU v3-256 pod slice
• Training Time:
  • Data generation (512x512): ~2 weeks on cluster (20 threads, 36 cores)
  • Model Training (EffNet-V2-M): 1 day and 7 hours per epoch on TPU v3-256

Paper Information

Citation: Rajan, K., Brinkhaus, H. O., Agea, M. I., Zielesny, A., & Steinbeck, C. (2023). DECIMER.ai: an open platform for automated optical chemical structure identification, segmentation and recognition in scientific publications. Nature Communications, 14(1), 5045. https://doi.org/10.1038/s41467-023-40782-0

Publication: Nature Communications 2023

Additional Resources:

• Web Application
• DECIMER Image Transformer GitHub
• RanDepict GitHub

@article{rajanDECIMERaiOpenPlatform2023,
  title = {DECIMER.ai: an open platform for automated optical chemical structure identification, segmentation and recognition in scientific publications},
  author = {Rajan, Kohulan and Brinkhaus, Henning Otto and Agea, M. Isabel and Zielesny, Achim and Steinbeck, Christoph},
  journal = {Nature Communications},
  volume = {14},
  number = {1},
  pages = {5045},
  year = {2023},
  doi = {10.1038/s41467-023-40782-0}
}

This is a Methodological Paper ($\Psi_{\text{Method}}$) with a significant Resource ($\Psi_{\text{Resource}}$) component.

• Method: The primary contribution is “ChemReco,” a specific deep learning pipeline (EfficientNet + Transformer) designed to solve the Optical Chemical Structure Recognition (OCSR) task for hand-drawn images. The authors conduct extensive ablation studies on architecture and data mixing ratios to validate performance.
• Resource: The authors explicitly state that “the primary focus of this paper is constructing datasets” due to the scarcity of hand-drawn molecular data. They introduce a comprehensive synthetic data generation pipeline involving RDKit modifications and image degradation to create training data.

Motivation: Digitizing Hand-Drawn Chemical Sketches

Hand-drawing is the most intuitive method for chemists and students to record molecular structures. However, digitizing these drawings into machine-readable formats (like SMILES) usually requires time-consuming manual entry or specialized software.

• Gap: Existing OCSR tools and rule-based methods often fail on hand-drawn sketches due to diverse writing styles, poor image quality, and the absence of labeled data.
• Application: Automated recognition enables efficient chemical research and allows for automatic grading in educational settings.

Core Innovation: Synthetic Pipeline and Hybrid Architecture

The paper introduces ChemReco, an end-to-end system for recognizing C-H-O structures. Key novelties include:

1. Synthetic Data Pipeline: A multi-stage generation method that modifies RDKit source code to randomize bond/angle parameters, followed by OpenCV-based augmentation, degradation, and background addition to simulate realistic hand-drawn artifacts.
2. Architectural Choice: The specific application of EfficientNet (encoder) combined with a Transformer (decoder) for this domain, which the authors demonstrate outperforms the more common ResNet+LSTM baselines.
3. Hybrid Training Strategy: Finding that a mix of 90% synthetic and 10% real data yields optimal performance, superior to using either dataset alone.

Methodology & Ablation Studies

The authors performed a series of ablation studies and comparisons:

• Synthesis Ablation: Evaluated the impact of each step in the generation pipeline (RDKit only $\rightarrow$ Augmentation $\rightarrow$ Degradation $\rightarrow$ Background) on validation loss and accuracy.
• Dataset Size Ablation: Tested model performance when trained on synthetic datasets ranging from 100,000 to 1,000,000 images.
• Real/Synthetic Ratio: Investigated the optimal mixing ratio of synthetic to real hand-drawn images (100:0, 90:10, 50:50, 10:90, 0:100), finding that the 90:10 ratio achieved 93.81% exact match, compared to 63.33% for synthetic-only and 65.83% for real-only.
• Architecture Comparison: Benchmarked four encoder-decoder combinations: ResNet vs. EfficientNet encoders paired with LSTM vs. Transformer decoders.
• Baseline Comparison: Compared results against a related study utilizing a CNN+LSTM framework.

Results & Interpretations

• Best Performance: The EfficientNet + Transformer model trained on a 90:10 synthetic-to-real ratio achieved a 96.90% Exact Match rate on the test set.
• Background Robustness: When training on synthetic data alone (no real images), the best accuracy on background-free test images was approximately 46% (using RDKit-aug-deg), while background test images reached approximately 53% (using RDKit-aug-bkg-deg). Adding random backgrounds during training helped prevent the model from overfitting to clean white backgrounds.
• Data Volume: Increasing the synthetic dataset size from 100k to 1M consistently improved accuracy (average exact match: 49.40% at 100k, 54.29% at 200k, 61.31% at 500k, 63.33% at 1M, all without real images in training).
• Encoder-Decoder Comparison (at 90:10 mix with 1M images):

• Superiority over Baselines: The model outperformed the cited CNN+LSTM baseline from ChemPix (93% vs 76% on the ChemPix test set).

Limitations

• Restricted atom types: The system only handles molecules composed of carbon, hydrogen, and oxygen (C-H-O), excluding nitrogen, sulfur, halogens, and other heteroatoms commonly found in organic chemistry.
• Structural complexity: Only structures with at most one ring are supported. Complex multi-ring systems and fused ring structures are not covered.
• Dataset availability: The real hand-drawn dataset (2,598 images) is not publicly released and is only available upon request from the corresponding author.
• Future directions: The authors suggest expanding to more heteroatoms, complex ring structures, and applications in automated grading of chemistry exams.

Reproducibility

The real hand-drawn dataset (2,598 images) is available upon request from the corresponding author, not publicly downloadable. The synthetic data generation pipeline is described in detail but relies on modified RDKit source code, which is included in the repository.

Data

The study utilizes a combination of collected SMILES data, real hand-drawn images, and generated synthetic images.

• Source Data: SMILES codes collected from PubChem, ZINC, GDB-11, and GDB-13. Filtered for C, H, O atoms and max 1 ring.
• Real Dataset: 670 selected SMILES codes drawn by multiple volunteers, totaling 2,598 images.
• Synthetic Dataset: Generated up to 1,000,000 images using the pipeline below.
• Training Mix: The optimal training set used 1 million images with a 90:10 ratio of synthetic to real images.

Algorithms

The Synthetic Image Generation Pipeline is critical for reproduction:

1. RDKit Modification: Modify source code to introduce random keys, character width, length, and bond angles.
2. Augmentation (OpenCV): Apply sequence: Resize ($p=0.5$), Blur ($p=0.4$), Erode/Dilate ($p=0.2$), Distort ($p=0.8$), Flip ($p=0.5$), Affine ($p=0.7$).
3. Degradation: Apply sequence: Salt+pepper noise ($p=0.1$), Contrast ($p=0.7$), Sharpness ($p=0.5$), Invert ($p=0.3$).
4. Background Addition: Random backgrounds are augmented (Crop, Distort, Flip) and added to the molecular image to prevent background overfitting.
5. Diffusion Enhancement: Stable Diffusion (v1-4) is used for image-to-image enhancement to better simulate hand-drawn styles (prompt: “A pencil sketch of [Formula]… without charge distribution”).

Models

The system uses an encoder-decoder architecture:

• Encoder: EfficientNet (pre-trained on ImageNet). The last layer is removed, and features are extracted into a Numpy array.
• Decoder: Transformer. Utilizes self-attention to generate the SMILES sequence. Chosen over LSTM for better handling of long-range dependencies.
• Output: Canonical SMILES string.

Evaluation

• Primary Metric: Exact Match (EM). A strict binary evaluation checking whether the complete generated SMILES perfectly replicates the target string.
• Other Metrics: Levenshtein Distance measures edit-level character proximity, while the Tanimoto coefficient evaluates structural similarity based on chemical fingerprints. Both were monitored during validation ablation runs.

Hardware

• CPU: Intel(R) Xeon(R) Gold 6130 (40 GB RAM).
• GPU: NVIDIA Tesla V100 (32 GB video memory).
• Framework: PyTorch 1.9.1.
• Training Configuration:
  • Optimizer: Adam (learning rate 1e-4).
  • Batch size: 32.
  • Epochs: 100.

Paper Information

Citation: Ouyang, H., Liu, W., Tao, J., et al. (2024). ChemReco: automated recognition of hand-drawn carbon-hydrogen-oxygen structures using deep learning. Scientific Reports, 14, 17126. https://doi.org/10.1038/s41598-024-67496-7

Publication: Scientific Reports 2024

Additional Resources:

• Official Code Repository

@article{ouyangChemRecoAutomatedRecognition2024,
  title = {ChemReco: Automated Recognition of Hand-Drawn Carbon--Hydrogen--Oxygen Structures Using Deep Learning},
  author = {Ouyang, Hengjie and Liu, Wei and Tao, Jiajun and Luo, Yanghong and Zhang, Wanjia and Zhou, Jiayu and Geng, Shuqi and Zhang, Chengpeng},
  journal = {Scientific Reports},
  volume = {14},
  number = {1},
  pages = {17126},
  year = {2024},
  publisher = {Nature Publishing Group},
  doi = {10.1038/s41598-024-67496-7}
}

The paper proposes an architecture (AtomLenz) and training framework (ProbKT* + Edit-Correction) to solve the problem of Optical Chemical Structure Recognition (OCSR) in data-sparse domains. It also releases a curated, relabeled dataset of hand-drawn molecules with atom-level bounding box annotations.

Overcoming Annotation Bottlenecks in OCSR

Optical Chemical Structure Recognition (OCSR) is critical for digitizing chemical literature and lab notes. However, existing methods face three main limitations:

1. Generalization Limits: They struggle with sparse or stylistically unique domains, such as hand-drawn images, where massive datasets for pretraining are unavailable.
2. Annotation Cost: “Atom-level” methods (which detect individual atoms and bonds) require expensive bounding box annotations, which are rarely available for real-world sketch data.
3. Lack of Interpretability/Localization: Pure “Image-to-SMILES” models (like DECIMER) work well but fail to localize the atoms or bonds in the original image, limiting human-in-the-loop review and mechanistic interpretability.

AtomLenz, ProbKT*, and Graph Edit-Correction

The core contribution is AtomLenz, an OCSR framework that achieves atom-level entity detection using only SMILES supervision on target domains. The authors construct an explicit object detection pipeline using Faster R-CNN trained via a composite multi-task loss. The objective aims to optimize a multi-class log loss $L_{cls}$ for predicted class $\hat{c}$ and a regression loss $L_{reg}$ for predicted bounding box coordinates $\hat{b}$:

$$ \mathcal{L} = L_{cls}(c, \hat{c}) + L_{reg}(b, \hat{b}) $$

To bridge the gap between image inputs and the weakly supervised SMILES labels, the system leverages:

• ProbKT (Probabilistic Knowledge Transfer):* Uses probabilistic logic and Hungarian matching to align predicted objects with the “ground truth” derived from the SMILES strings, enabling backpropagation without explicit bounding boxes.
• Graph Edit-Correction: Generates pseudo-labels by solving an optimization problem that finds the smallest edit on the predicted graph such that the corrected graph and the ground-truth SMILES graph become isomorphic, which forces fine-tuning on less frequent atom types. The combination of ProbKT* and Edit-Correction is abbreviated as EditKT*.
• ChemExpert: A chemically sound ensemble strategy that cascades predictions from multiple models (e.g., passing through DECIMER, then AtomLenz), halting at the first output that clears basic RDKit chemical validity checks.

Data Efficiency and Domain Adaptation Experiments

The authors evaluated the model specifically on domain adaptation and sample efficiency, treating hand-drawn molecules as the primary low-data target distribution:

• Pretraining: Initially trained on ~214k synthetic images from ChEMBL explicitly labeled with bounding boxes (generated via RDKit).
• Target Domain Adaptation: Fine-tuned on the Brinkhaus hand-drawn dataset (4,070 images) using purely SMILES supervision.
• Evaluation Sets:
  • Hand-drawn test set: 1,018 images.
  • ChemPix: 614 out-of-domain hand-drawn images.
  • Atom Localization set: 1,000 synthetic images to evaluate precise bounding box capabilities.
• Baselines: Compared against leading OCSR methods, including DECIMER (v2.2.0), Img2Mol, MolScribe, ChemGrapher, and OSRA.

State-of-the-Art Ensembles vs. Standalone Limitations

• SOTA Ensemble Performance: The ChemExpert module (combining AtomLenz and DECIMER) achieved state-of-the-art accuracy on both hand-drawn (63.5%) and ChemPix (51.8%) test sets.
• Data Efficiency under Bottleneck Regimes: AtomLenz effectively bypassed the massive data constraints of competing models. When all methods were retrained from scratch on the same 4,070-sample hand-drawn training set (enriched with atom-level annotations from EditKT*), AtomLenz achieved 33.8% exact accuracy, outperforming baselines like Img2Mol (0.0%), MolScribe (1.3%), and DECIMER (0.1%), illustrating its sample efficiency.
• Localization Success: The base framework achieved strong localization (mAP 0.801), a capability not provided by end-to-end transformers like DECIMER.
• Methodological Tradeoffs: While AtomLenz is highly sample efficient, its standalone performance when fine-tuned on the target domain (33.8% accuracy) underperforms fine-tuned models trained on larger datasets like DECIMER (62.2% accuracy). AtomLenz achieves state-of-the-art results primarily when deployed as part of the ChemExpert ensemble alongside DECIMER, since errors from the two approaches tend to occur on different samples, allowing them to complement each other.

Reproducibility Details

Artifacts

Data

The study utilizes a mix of large synthetic datasets and smaller curated hand-drawn datasets.

Algorithms

• Molecular Graph Constructor (Algorithm 1): A rule-based system to assemble the graph from detected objects:
  1. Filtering: Removes overlapping atom boxes (IoU threshold).
  2. Node Creation: Merges overlapping charge and stereocenter objects with their corresponding atom objects.
  3. Edge Creation: Iterates over bond objects; if a bond overlaps with exactly two atoms, an edge is added. If >2, it selects the most probable pair.
  4. Validation: Checks valency constraints; removes bonds iteratively if constraints are violated.
• Weakly Supervised Training:
  • ProbKT*: Uses Hungarian matching to align predicted objects with the “ground truth” implied by the SMILES string, allowing backpropagation without explicit boxes.
  • Graph Edit-Correction: Finds the smallest edit on the predicted graph such that the corrected and true SMILES graphs become isomorphic, then uses the correction to generate pseudo-labels for retraining.

Models

• Object Detection Backbone: Faster R-CNN.
  • Four distinct models are trained for different entity types: Atoms ($O^a$), Bonds ($O^b$), Charges ($O^c$), and Stereocenters ($O^s$).
  • Loss Function: Multi-task loss combining Multi-class Log Loss ($L_{cls}$) and Regression Loss ($L_{reg}$).
• ChemExpert: An ensemble wrapper that prioritizes models based on user preference (e.g., DECIMER first, then AtomLenz). It accepts the first prediction that passes RDKit chemical validity checks.

Evaluation

Primary metrics focused on structural correctness and localization accuracy.

Hardware

• Compute: Experiments utilized the Flemish Supercomputer Center (VSC) resources.
• Note: Specific GPU models (e.g., A100/V100) are not explicitly detailed in the text, but Faster R-CNN training is standard on consumer or enterprise GPUs.

Paper Information

Citation: Oldenhof, M., De Brouwer, E., Arany, Á., & Moreau, Y. (2024). Atom-Level Optical Chemical Structure Recognition with Limited Supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

Publication venue/year: CVPR 2024

Additional Resources:

• Official Repository
• Hand-drawn Dataset on Figshare

BibTeX:

@inproceedings{oldenhofAtomLevelOpticalChemical2024,
  title = {Atom-Level Optical Chemical Structure Recognition with Limited Supervision},
  author = {Oldenhof, Martijn and De Brouwer, Edward and Arany, {\'A}d{\'a}m and Moreau, Yves},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2024},
  eprint = {2404.01743},
  archiveprefix = {arXiv},
  primaryclass = {cs.CV}
}

This is a Methodological Paper with a significant Resource component.

• Method: It proposes a novel architecture (Swin Transformer backbone) and a specific loss function optimization (Focal Loss) for the task of Optical Chemical Structure Recognition (OCSR).
• Resource: It constructs a large-scale synthetic dataset of 5 million molecules, specifically designing it to cover complex cases like substituents and aromatic rings.

Motivation: Addressing Visual Context and Data Imbalance

• Problem: OCSR (converting images of chemical structures to SMILES) is difficult due to complex chemical patterns and long sequences. Existing deep learning methods (often CNN-based) struggle to achieve satisfactory recognition rates.
• Technical Gap: Standard CNN backbones (like ResNet or EfficientNet) focus on local feature extraction and miss global dependencies required for interpreting complex molecular diagrams.
• Data Imbalance: Chemical strings suffer from severe class imbalance (e.g., ‘C’ and ‘H’ are frequent; ‘Br’ or ‘Cl’ are rare), which causes standard Cross Entropy loss to underperform.

Core Innovation: Swin Transformers and Focal Loss

• Swin Transformer Backbone: SwinOCSR replaces the standard CNN backbone with a Swin Transformer, using shifted window attention to capture both local and global image features more effectively.
• Multi-label Focal Loss (MFL): The paper introduces a modified Focal Loss to OCSR, the first explicit attempt to address token imbalance in OCSR (per the authors). This penalizes the model for errors on rare tokens, addressing the “long-tail” distribution of chemical elements. The standard Focal Loss formulation heavily weights hard-to-classify examples: $$ \begin{aligned} FL(p_t) = -\alpha_t (1 - p_t)^\gamma \log(p_t) \\ \end{aligned} $$
• Structured Synthetic Dataset: Creation of a dataset explicitly balanced across four structural categories: Kekule rings, Aromatic rings, and their combinations with substituents.

Experimental Setup and Baselines

• Backbone Comparison: The authors benchmarked SwinOCSR against the backbones of leading competitors: ResNet-50 (used in Image2SMILES) and EfficientNet-B3 (used in DECIMER 1.0).
• Loss Function Ablation: They compared the performance of standard Cross Entropy (CE) loss against their proposed Multi-label Focal Loss (MFL).
• Category Stress Test: Performance was evaluated separately on molecules with/without substituents and with/without aromaticity to test robustness.
• Real-world Evaluation: The model was tested on 100 images manually extracted from the literature (with manually labeled SMILES), and separately on 100 CDK-generated images from those same SMILES, to measure the domain gap between synthetic and real-world data.

Results and Limitations

• Synthetic test set performance: With Multi-label Focal Loss (MFL), SwinOCSR achieved 98.58% accuracy on the synthetic test set, compared to 97.36% with standard CE loss. Both ResNet-50 (89.17%) and EfficientNet-B3 (86.70%) backbones scored lower when using CE loss (Table 3).
• Handling of long sequences: The model maintained high accuracy (94.76%) even on very long DeepSMILES strings (76-100 characters), indicating effective global feature extraction.
• Per-category results: Performance was consistent across molecule categories: Category 1 (Kekule, 98.20%), Category 2 (Aromatic, 98.46%), Category 3 (Kekule + Substituents, 98.76%), Category 4 (Aromatic + Substituents, 98.89%). The model performed slightly better on molecules with substituents and aromatic rings.
• Domain shift: While performance on synthetic data was strong, accuracy dropped to 25% on 100 real-world literature images. On 100 CDK-generated images from the same SMILES strings, accuracy was 94%, confirming that the gap stems from stylistic differences between CDK-rendered and real-world images. The authors attribute this to noise, low resolution, and variations such as condensed structural formulas and abbreviations.

Reproducibility Details

Data

• Source: The first 8.5 million structures from PubChem were downloaded, yielding ~6.9 million unique SMILES.
• Generation Pipeline:
  • Tools: CDK (Chemistry Development Kit) for image rendering; RDKit for SMILES canonicalization.
  • Augmentation: To ensure diversity, the dataset was split into 4 categories (1.25M each): (1) Kekule, (2) Aromatic, (3) Kekule + Substituents, (4) Aromatic + Substituents. Substituents were randomly added from a list of 224 common patent substituents.
  • Preprocessing: Images rendered as binary, resized to 224x224, and copied to 3 channels (RGB simulation).

Algorithms

• Loss Function: Multi-label Focal Loss (MFL). The single-label classification task was cast as multi-label to apply Focal Loss, using a sigmoid activation on logits.
• Optimization:
  • Optimizer: Adam with initial learning rate 5e-4.
  • Schedulers: Cosine decay for the Swin Transformer backbone; Step decay for the Transformer encoder/decoder.
  • Regularization: Dropout rate of 0.1.

Models

• Backbone (Encoder 1): Swin Transformer.
  • Patch size: $4 \times 4$.
  • Linear embedding dimension: 192.
  • Structure: 4 stages with Swin Transformer Blocks (Window MSA + Shifted Window MSA).
  • Output: Flattened patch sequence $S_b$.
• Transformer Encoder (Encoder 2): 6 standard Transformer encoder layers. Uses Positional Embedding + Multi-Head Attention + MLP.
• Transformer Decoder: 6 standard Transformer decoder layers. Uses Masked Multi-Head Attention (to prevent look-ahead) + Multi-Head Attention (connecting to encoder output $S_e$).
• Tokenization: DeepSMILES format used (syntactically more robust than SMILES). Vocabulary size: 76 tokens (76 unique characters found in dataset). Embedding dimension: 256.

Evaluation

• Metrics: Accuracy (Exact Match), Tanimoto Similarity (PubChem fingerprints), BLEU, ROUGE.

Hardware

• GPU: Trained on NVIDIA Tesla V100-PCIE.
• Training Time: 30 epochs.
• Batch Size: 256 images ($224 \times 224$ pixels).

Artifacts

Paper Information

Citation: Xu, Z., Li, J., Yang, Z. et al. (2022). SwinOCSR: end-to-end optical chemical structure recognition using a Swin Transformer. Journal of Cheminformatics, 14(41). https://doi.org/10.1186/s13321-022-00624-5

Publication: Journal of Cheminformatics 2022

Additional Resources:

• GitHub Repository

This is primarily a Resource paper ($\Psi_{\text{Resource}}$) with a strong Method component ($\Psi_{\text{Method}}$).

• Resource: It presents a complete software application (published as an “Application Note”) for Optical Chemical Structure Recognition (OCSR), including a graphical user interface (GUI) and a new curated “Real-World” dataset of 3,040 molecular images.
• Method: It proposes a novel “rule-free” pipeline that replaces traditional vectorization algorithms with deep learning object detection (YOLOv5) and segmentation models.

Motivation: Bottlenecks in Rule-Based Systems

• Legacy Backlog: Decades of scientific literature contain chemical structures only as 2D images (PDFs), which are not machine-readable.
• Limitations of Legacy Architecture: Existing tools (like OSRA, CLIDE, MolVec) rely on rule-based vectorization (interpreting vectors and nodes) which struggle with noise, low resolution, and complex drawing styles found in scanned documents.
• Deep Learning Gap: While deep learning (DL) has advanced computer vision, few practical, end-to-end DL tools existed for OCSR that could handle the full pipeline from PDF extraction to graph generation with high accuracy.

Core Innovation: Object Detection Paradigm for OCSR

• Object Detection Paradigm: MolMiner shifts away from the strategy of line-tracing (vectorization), opting to treat atoms and bonds directly as objects to be detected using YOLOv5. This allows it to “look once” at the image.
• End-to-End Pipeline: Integration of three specialized modules:
  1. MobileNetV2 for segmenting molecular figures from PDF pages.
  2. YOLOv5 for detecting chemical elements (atoms/bonds) as bounding boxes.
  3. EasyOCR for recognizing text labels and resolving abbreviations (supergroups) to full explicit structures.
• Synthetic Training Strategy: The authors bypassed manual labeling by building a data generation module that uses RDKit to create chemically valid images with perfect ground-truth annotations automatically.

Methodology: End-to-End Object Detection Pipeline

• Benchmarks: Evaluated on four standard OCSR datasets: USPTO (5,719 images), UOB (5,740 images), CLEF2012 (992 images), and JPO (450 images).
• New External Dataset: Collected and annotated a “Real-World” dataset of 3,040 images from 239 scientific papers to test generalization beyond synthetic benchmarks.
• Baselines: Compared against open-source tools: MolVec (v0.9.8), OSRA (v2.1.0), and Imago (v2.0).
• Qualitative Tests: Tested on difficult cases like hand-drawn molecules and large-sized scans (e.g., Palytoxin).

Results: Speed and Generalization Metrics

• Benchmark Performance: MolMiner outperformed open-source baselines on standard validation splits.
  • USPTO: 93% MCS accuracy (vs. 89% for MolVec, per Table 2). The commercial CLiDE Pro tool reports 93.8% on USPTO, slightly higher than MolMiner’s 93.3%.
  • Real-World Set: 87.8% MCS accuracy (vs. 50.1% for MolVec, 8.9% for OSRA, and 10.3% for Imago).
• Inference Velocity: The architecture allows for faster processing compared to CPU rule-based systems. On JPO (450 images), MolMiner finishes in under 1 minute versus 8-23 minutes for rule-based tools (Table 3).
• Robustness: Demonstrated ability to handle hand-drawn sketches and noisy scans, though limitations remain with crossing bonds, colorful backgrounds, crowded layout segmentation, and Markush structures.
• Software Release: Released as a free desktop application for Mac and Windows with a Ketcher-based editing plugin.

Reproducibility Details

Data

The system relies heavily on synthetic data for training, while evaluation uses both standard and novel real-world datasets.

Algorithms

• Data Generation:
  • Uses RDKit MolDraw2DSVG and CondenseMolAbbreviations to generate images and ground truth.
  • Augmentation: Rotation, line thinning/thickness variation, noise injection.
• Graph Construction:
  • A distance-based algorithm connects recognized “Atom” and “Bond” objects into a molecular graph.
  • Supergroup Parser: Matches detected text against a dictionary collected from RDKit, ChemAxon, and OSRA to resolve abbreviations (e.g., “Ph”, “Me”).
• Image Preprocessing:
  • Resizing: Images with max dim > 2560 are resized to 2560. Small images (< 640) resized to 640.
  • Padding: Images padded to nearest upper bound (640, 1280, 1920, 2560) with white background (255, 255, 255).
  • Dilation: For thick-line images, cv2.dilate (3x3 or 2x2 kernel) is applied to estimate median line width.

Models

The system is a cascade of three distinct deep learning models:

1. MolMiner-ImgDet (Page Segmentation):
   • Architecture: MobileNetV2.
   • Task: Semantic segmentation to identify and crop chemical figures from full PDF pages.
   • Classes: Background vs. Compound.
   • Performance: Recall 95.5%.
2. MolMiner-ImgRec (Structure Recognition):
   • Architecture: YOLOv5 (One-stage object detector). Selected over MaskRCNN/EfficientDet for speed/accuracy trade-off.
   • Task: Detects atoms and bonds as bounding boxes.
   • Labels:
     • Atoms: Si, N, Br, S, I, Cl, H, P, O, C, B, F, Text.
     • Bonds: Single, Double, Triple, Wedge, Dash, Wavy.
   • Performance: mAP@0.5 = 97.5%.
3. MolMiner-TextOCR (Character Recognition):
   • Architecture: EasyOCR (fine-tuned).
   • Task: Recognize specific characters in “Text” regions identified by YOLO (e.g., supergroups, complex labels).
   • Performance: ~96.4% accuracy.

Performance Evaluation & Accuracy Metrics

The paper argues that computing the Maximum Common Substructure (MCS) accuracy is superior to string comparisons of canonical identifiers like InChI or SMILES. The InChI string is heavily sensitive to slight canonicalization or tautomerization discrepancies (like differing aromaticity models). Therefore, for comparing structural isomorphism:

$$ \text{MCS_Accuracy} = \frac{|\text{Edges}_{\text{MCS}}| + |\text{Nodes}_{\text{MCS}}|}{|\text{Edges}_{\text{Ground_Truth}}| + |\text{Nodes}_{\text{Ground_Truth}}|} $$

Using this metric to evaluate bond- and atom-level recall directly measures OCR extraction fidelity.

Hardware

• Inference Hardware: Tested on Intel Xeon Gold 6230R CPU @ 2.10 GHz.
• Acceleration: Supports batch inference on GPU, which provides the reported speedups over rule-based CPU tools.
• Runtime: Under 1 minute on JPO (450 images), 7 minutes on USPTO (5,719 images), compared to 29-148 minutes for baseline tools on USPTO (Table 3).

Artifacts

Paper Information

Citation: Xu, Y., Xiao, J., Chou, C.-H., Zhang, J., Zhu, J., Hu, Q., Li, H., Han, N., Liu, B., Zhang, S., Han, J., Zhang, Z., Zhang, S., Zhang, W., Lai, L., & Pei, J. (2022). MolMiner: You only look once for chemical structure recognition. Journal of Chemical Information and Modeling, 62(22), 5321–5328. https://doi.org/10.1021/acs.jcim.2c00733

Publication: Journal of Chemical Information and Modeling (JCIM) 2022

Additional Resources:

• Github Repository
• Zenodo Dataset

@article{xuMolMinerYouOnly2022,
  title = {MolMiner: You only look once for chemical structure recognition},
  shorttitle = {MolMiner},
  author = {Xu, Youjun and Xiao, Jinchuan and Chou, Chia-Han and Zhang, Jianhang and Zhu, Jintao and Hu, Qiwan and Li, Hemin and Han, Ningsheng and Liu, Bingyu and Zhang, Shuaipeng and Han, Jinyu and Zhang, Zhen and Zhang, Shuhao and Zhang, Weilin and Lai, Luhua and Pei, Jianfeng},
  year = 2022,
  month = nov,
  journal = {Journal of Chemical Information and Modeling},
  volume = {62},
  number = {22},
  pages = {5321--5328},
  publisher = {American Chemical Society},
  issn = {1549-9596},
  doi = {10.1021/acs.jcim.2c00733},
}

This is primarily a Method paper with a significant Resource component.

• Method: It proposes a novel architectural framework (RCGD) and a new representation syntax (SSML) to solve the specific problem of handwritten chemical structure recognition.
• Resource: It introduces a new benchmark dataset, EDU-CHEMC, containing 50,000 handwritten images to address the lack of public data in this domain.

The Ambiguity of Handwritten Chemical Structures

Recognizing handwritten chemical structures is significantly harder than printed ones due to:

1. Inherent Ambiguity: Handwritten atoms and bonds vary greatly in appearance.
2. Projection Complexity: Converting 2D projected layouts (like Natta or Fischer projections) into linear strings is difficult.
3. Limitations of Existing Formats: Standard formats like SMILES require domain knowledge (valence rules) and have a high semantic gap with the visual image. They often fail to represent “invalid” structures commonly found in educational/student work.

Bridging the Semantic Gap with SSML and RCGD

The paper introduces two core contributions to bridge the semantic gap between image and markup:

1. Structure-Specific Markup Language (SSML): An extension of Chemfig that provides an unambiguous, visual-based graph representation. Unlike SMILES, it describes how to draw the molecule step-by-step, making it easier for models to learn visual alignments. It supports “reconnection marks” to handle cyclic structures explicitly.

2. Random Conditional Guided Decoder (RCGD): A decoder that treats recognition as a graph traversal problem. It introduces three novel mechanisms:

   • Conditional Attention Guidance: Uses branch angle directions to guide the attention mechanism, preventing the model from getting lost in complex structures.
   • Memory Classification: A module that explicitly stores and classifies “unexplored” branch points to handle ring closures (reconnections).
   • Path Selection: A training strategy that randomly samples traversal paths to prevent overfitting to a specific serialization order.

Experimental Setup and Baselines

Datasets:

• Mini-CASIA-CSDB (Printed): A subset of 97,309 printed molecular structure images, upscaled to $500 \times 500$ resolution.
• EDU-CHEMC (Handwritten): A new dataset of 52,987 images collected from educational settings (cameras, scanners, screens), including erroneous/non-existent structures.

Baselines:

• Compared against standard String Decoders (SD) (based on DenseWAP), tested with both SMILES and SSML on Mini-CASIA-CSDB and exclusively with SSML on EDU-CHEMC.
• Compared against BTTR and ABM (recent mathematical expression recognition models) adapted for the chemical structure task, both using SSML on EDU-CHEMC.
• On Mini-CASIA-CSDB, also compared against WYGIWYS (a SMILES-based string decoder at 300x300 resolution).

Ablation Studies:

• Evaluated the impact of removing Path Selection (PS) and Memory Classification (MC) mechanisms on EDU-CHEMC.
• Tested robustness to image rotation ($180^{\circ}$) on Mini-CASIA-CSDB.

Recognition Performance and Robustness

• Superiority of SSML: Models trained with SSML significantly outperformed those trained with SMILES (92.09% vs 81.89% EM on printed data) due to reduced semantic gap.
• Best Performance: RCGD achieved the highest Exact Match (EM) scores on both datasets:
  • Mini-CASIA-CSDB: 95.01% EM.
  • EDU-CHEMC: 62.86% EM.
• EDU-CHEMC Baselines: On the handwritten dataset, SD (DenseWAP) achieved 61.35% EM, outperforming both BTTR (58.21% EM) and ABM (58.78% EM). The authors note that BTTR and ABM’s reverse training mode, which helps in regular formula recognition, does not transfer well to graph-structured molecular data.
• Ablation Results (Table 5, EDU-CHEMC): Removing Path Selection alone dropped EM from 62.86% to 62.15%. Removing both Path Selection and Memory Classification dropped EM further to 60.31%, showing that memory classification has a larger impact.
• Robustness: RCGD showed minimal performance drop (0.85%) on rotated images compared to SMILES-based methods (10.36% drop). The SD with SSML dropped by 2.19%, confirming that SSML itself improves rotation invariance.
• Educational Utility: The method can recognize and reconstruct chemically invalid structures (e.g., a Carbon atom with 5 bonds), making it applicable for correcting and revising handwritten answers in chemistry education.

Reproducibility Details

Data

1. EDU-CHEMC (Handwritten)

• Total Size: 52,987 images.
• Splits: Training (48,998), Validation (999), Test (2,992).
• Characteristics: Real-world educational data, mixture of isolated molecules and reaction equations, includes invalid chemical structures.

2. Mini-CASIA-CSDB (Printed)

• Total Size: 97,309 images.
• Splits: Training (80,781), Validation (8,242), Test (8,286).
• Preprocessing: Original $300 \times 300$ images were upscaled to $500 \times 500$ RGB to resolve blurring issues.

Algorithms

1. SSML Generation

To convert a molecular graph to SSML:

1. Traverse: Start from the left-most atom.
2. Bonds/Atoms: Output atom text and bond format <bond>[:<angle>].
3. Branches: At branch points, use phantom symbols ( and ) to enclose branches, ordered by ascending bond angle.
4. Reconnections: Use ?[tag] and ?[tag, bond] to mark start/end of ring closures.

2. RCGD Specifics

• RCGD-SSML: Modified version of SSML for the decoder. Removes ( ) delimiters; adds \eob (end of branch). Maintains a dynamic Branch Angle Set ($M$).
• Path Selection: During training, when multiple branches exist in $M$, the model randomly selects one to traverse next. During inference, it uses beam search to score candidate paths.
• Loss Function: $$ \begin{aligned} L_{\text{total}} = L_{\text{ce}} + L_{\text{bc}} \end{aligned} $$
  • $L_{\text{ce}}$: Cross-entropy loss for character sequence generation.
  • $L_{\text{bc}}$: Multi-label classification loss for the memory module (predicting reconnection bond types for stored branch states).

Models

Encoder: DenseNet

• Structure: 3 dense blocks.
• Growth Rate: 24.
• Depth: 32 per block.
• Output: High-dimensional feature map $x \in \mathbb{R}^{d_x \times h \times w}$.

Decoder: GRU with Attention

• Hidden State Dimension: 256.
• Embedding Dimension: 256.
• Attention Projection: 128.
• Memory Classification Projection: 256.

Training Config:

• Optimizer: Adam.
• Learning Rate: 2e-4 with multi-step decay (gamma 0.5).
• Dropout: 15%.
• Strategy: Teacher-forcing used for validation selection.

Evaluation

Metrics:

• Exact Match (EM): Percentage of samples where the predicted graph structure perfectly matches the label. For SMILES, string comparison; for SSML, converted to graph for isomorphism check.
• Structure EM: Auxiliary metric for samples with mixed content (text + molecules), counting samples where all molecular structures are correct.

Artifacts:

Missing Components:

• No training or inference code is publicly released; only the dataset is available.
• Pre-trained model weights are not provided.

Paper Information

Citation: Hu, J., Wu, H., Chen, M., Liu, C., Wu, J., Yin, S., Yin, B., Yin, B., Liu, C., Du, J., & Dai, L. (2023). Handwritten Chemical Structure Image to Structure-Specific Markup Using Random Conditional Guided Decoder. Proceedings of the 31st ACM International Conference on Multimedia (pp. 8114-8124). https://doi.org/10.1145/3581783.3612573

Publication: ACM Multimedia 2023

Additional Resources:

• GitHub Repository / EDU-CHEMC Dataset

@inproceedings{huHandwrittenChemicalStructure2023,
  title = {Handwritten Chemical Structure Image to Structure-Specific Markup Using Random Conditional Guided Decoder},
  booktitle = {Proceedings of the 31st ACM International Conference on Multimedia},
  author = {Hu, Jinshui and Wu, Hao and Chen, Mingjun and Liu, Chenyu and Wu, Jiajia and Yin, Shi and Yin, Baocai and Yin, Bing and Liu, Cong and Du, Jun and Dai, Lirong},
  year = {2023},
  month = oct,
  pages = {8114--8124},
  publisher = {ACM},
  address = {Ottawa ON Canada},
  doi = {10.1145/3581783.3612573},
  isbn = {979-8-4007-0108-5}
}

This is primarily a methodological paper with a secondary resource contribution.

Method: It proposes a novel end-to-end deep learning architecture (Segmentation U-Net + Recognition Encoder-Decoder) to replace traditional rule-based optical chemical structure recognition (OCSR) systems.

Resource: It details a pipeline for generating large-scale synthetic datasets (images overlaying patent/journal backgrounds) necessary to train the deep learning models.

Motivation: Overcoming Brittle Rule-Based Systems

Existing tools for extracting chemical structures from literature (e.g., OSRA, CLIDE) rely on complex, handcrafted rules and heuristics (edge detection, vectorization). These systems suffer from:

1. Brittleness: They fail when image quality is low (low resolution, noise) or when artistic styles vary (wavy bonds, crossing lines).
2. Maintenance difficulty: Improvements require manual codification of new rules for every edge case, which is difficult to scale.
3. Data volume: The explosion of published life science papers (2000+ per day in Medline) creates a need for automated, robust curation tools that humans cannot match.

Core Innovation: End-to-End Pixel-to-SMILES Recognition

The authors present an end-to-end deep learning approach for this task that operates directly on raw pixels without explicit subcomponent recognition (e.g., detecting atoms and bonds separately). Key innovations include:

1. Pixel-to-SMILES: Treating structure recognition as an image captioning problem using an encoder-decoder architecture with attention, generating SMILES directly.
2. Low-Resolution Robustness: The model is trained on aggressively downsampled images (~60 dpi for segmentation, 256x256 for prediction), making it robust to poor quality and noisy inputs from legacy PDF extractions.
3. Implicit Superatom Handling: The model learns to recognize and generate sequences for superatoms (e.g., “OTBS”) contextually.

Experimental Setup and Large-Scale Synthetic Data

The authors validated their approach using a mix of large-scale synthetic training sets and real-world test sets:

1. Synthetic Generation: They created a segmentation dataset by overlaying USPTO molecules onto “whited-out” journal pages.
2. Ablation/Training: Metrics were tracked on Indigo (synthetic) and USPTO (real patent images) datasets.
3. External Validation:
   • Valko Dataset: A standard benchmark of 454 heterogeneous images from literature.
   • Proprietary Dataset: A collection of images from 47 articles and 5 patents to simulate real-world drug discovery curation.
4. Stress Testing: They analyzed performance distributions across molecular weight, heavy atom count, and rare elements (e.g., Uranium, Vanadium).

Results and Limitations in Complex Structures

• High Accuracy on Standard Sets: The model achieved 82% accuracy on the Indigo validation set and 77% on the USPTO validation set. No apparent overfitting was observed on the Indigo data (57M training examples), though some overfitting occurred on the smaller USPTO set (1.7M training examples).
• Real-World Viability: It achieved 83% accuracy on the proprietary internal test set, with validation and proprietary accuracies ranging from 77-83%, indicating the training sets reasonably approximate real drug discovery data.
• Segmentation Quality: Low segmentation error rates were observed: only 3.3% of the Valko dataset and 6.6% of the proprietary images failed to segment properly.
• Limitations on Complexity: Performance dropped to 41% on the Valko test set. Superatoms were the single largest contributor to prediction errors, with 21% of Valko samples containing one or more incorrectly predicted superatoms. Only 6.6% of total training images contained any superatom, limiting the model’s exposure.
• Stereochemistry Challenges: 60% of compounds with incorrectly predicted stereochemistry had explicit stereochemistry in both the ground truth and the prediction, but with wrong configurations assigned (e.g., predicting R instead of S). The model often correctly identified which atoms have stereocenters but assigned the wrong direction, suggesting the architecture may not incorporate sufficient spatial context for configuration assignment.

Reproducibility Details

Data

The authors utilized three primary sources for generating training data. All inputs were strictly downsampled to improve robustness.

Preprocessing:

• Segmentation Inputs: Grayscale, downsampled to ~60 dpi.
• Prediction Inputs: Resized to 256x256 such that bond lengths are approximately 3-12 pixels.
• Augmentation: Random affine transforms, brightness scaling, and binarization applied during training.

Algorithms

Segmentation Pipeline:

• Multi-scale Inference: Masks generated at resolutions from 30 to 60 dpi (3 dpi increments) and averaged for the final mask.
• Post-processing: Hough transform used to remove long straight lines (table borders). Mask blobs filtered by pixel count thresholds.

Prediction Pipeline:

• Sequence Generation: SMILES generated character-by-character via greedy decoding. During inference, predictions are made at several low resolutions and the sequence with the highest confidence (product of per-character softmax outputs) is returned.
• Attention-based Verification: Attention weights used to re-project predicted atoms back into 2D space to visually verify alignment with the input image.

Models

1. Segmentation Model (U-Net Variant):

• Architecture: U-Net style with skip connections.
• Input: 128x128x1 grayscale image.
• Layers: Alternating 3x3 Conv and 2x2 Max Pool.
• Activation: Parametric ReLU (pReLU).
• Parameters: ~380,000.

2. Structure Prediction Model (Encoder-Decoder):

• Encoder: CNN with 5x5 convolutions, 2x2 Max Pooling, pReLU. No pooling in first layers to preserve fine features.
• Decoder: 3 layers of GridLSTM cells.
• Attention: Soft/Global attention mechanism conditioned on the encoder state.
• Input: 256x256x1 image.
• Output: Sequence of characters (vocab size 65).
• Parameters: ~46.3 million.

Evaluation

Evaluation required an exact string match of the Canonical SMILES (including stereochemistry) to the ground truth.

Hardware

• Segmentation Training: 1 GPU, ~4 days (650k steps).
• Prediction Training: 8 NVIDIA Pascal GPUs, ~26 days (1M steps).
• Framework: TensorFlow.
• Optimizer: Adam.

Artifacts

No public code, pre-trained models, or generated datasets were released with this paper. The training pipeline relies on publicly available molecular databases (PubChem, USPTO) and open-source rendering tools (Indigo), but the specific training sets, model weights, and inference code remain unavailable.

Paper Information

Citation: Staker, J., Marshall, K., Abel, R., & McQuaw, C. (2019). Molecular Structure Extraction From Documents Using Deep Learning. Journal of Chemical Information and Modeling, 59(3), 1017-1029. https://doi.org/10.1021/acs.jcim.8b00669

Publication: Journal of Chemical Information and Modeling (JCIM) 2019

Additional Resources:

• Schrödinger Publication Page

@article{stakerMolecularStructureExtraction2019,
  title = {Molecular Structure Extraction From Documents Using Deep Learning},
  author = {Staker, Joshua and Marshall, Kyle and Abel, Robert and McQuaw, Carolyn},
  year = {2019},
  month = {feb},
  journal = {Journal of Chemical Information and Modeling},
  volume = {59},
  number = {3},
  pages = {1017--1029},
  doi = {10.1021/acs.jcim.8b00669},
  url = {https://doi.org/10.1021/acs.jcim.8b00669}
}

This is a Resource paper that introduces two complementary standards for representing chemical mixtures: the detailed Mixfile format for comprehensive mixture descriptions and the compact MInChI (Mixtures InChI) specification for canonical mixture identifiers.

The Missing Format for Complex Formulations

There is a fundamental gap in chemical informatics: current standards excel at representing pure individual molecules (SMILES, InChI, Molfile), but a corresponding standard for multi-component mixtures remains an open challenge. This is a major problem because real-world chemistry predominantly involves complex mixtures.

Everyday chemical work frequently involves:

• Reagents with specified purity (e.g., “$\geq$ 97% pure”)
• Solutions and formulations
• Complex mixtures like “hexanes” (which contains multiple isomers)
• Drug formulations with active ingredients and excipients

Without a machine-readable standard, chemists are forced to describe these mixtures in plain text that software cannot parse or analyze systematically. This creates barriers for automated safety analysis, inventory management, and data sharing.

Dual Design: Comprehensive Mixfiles and Canonical MInChIs

The authors propose a two-part solution:

1. Mixfile: A detailed, hierarchical JSON format that captures the complete composition of a mixture
2. MInChI: A compact, canonical string identifier derived from Mixfile data

This dual approach provides both comprehensive description (Mixfile) and simple identification (MInChI), similar to having both a detailed recipe and a short name for a dish.

What Makes a Good Mixture Format?

The authors identify three essential properties any mixture format must capture:

1. Compound: What molecules are present?
2. Quantity: How much of each component?
3. Hierarchy: How are components organized (e.g., mixtures-of-mixtures)?

The hierarchical aspect is crucial. Consider “hexanes”: it is a named mixture containing specific proportions of n-hexane, 2-methylpentane, 3-methylpentane, etc. A mixture format needs to represent both the individual isomers and the fact that they are grouped under the umbrella term “hexanes.”

Mixfile Format Details

Mixfile uses JSON as its foundation, making it both human-readable and easy to parse in modern programming languages. The core structure is a hierarchical tree where each component can contain:

• name: Component identifier
• molfile/smiles/inchi/formula: Molecular structure (molfile is the primary source of truth)
• quantity/units/relation/ratio: Concentration data with optional relation operators
• contents: Array of sub-components for hierarchical mixtures
• identifiers: Database IDs or URLs for additional information

Simple Example

A basic Mixfile might look like:

{
  "mixfileVersion": 0.01,
  "name": "Acetone, ≥99%",
  "contents": [
    {
      "name": "acetone",
      "smiles": "CC(=O)C",
      "quantity": 99,
      "units": "%",
      "relation": ">="
    }
  ]
}

Note that the paper specifies distinct fields for molecular structures: molfile (the primary source of truth), smiles, inchi, and formula. Concentration data uses separate quantity, units, and relation fields.

Complex Example: Mixture-of-Mixtures

For something like “ethyl acetate dissolved in hexanes,” the structure would be:

{
  "mixfileVersion": 0.01,
  "name": "Ethyl acetate in hexanes",
  "contents": [
    {
      "name": "ethyl acetate",
      "smiles": "CCOC(=O)C",
      "quantity": 10,
      "units": "%"
    },
    {
      "name": "hexanes",
      "contents": [
        {
          "name": "n-hexane",
          "smiles": "CCCCCC",
          "quantity": 60,
          "units": "%"
        },
        {
          "name": "2-methylpentane",
          "smiles": "CC(C)CCC",
          "quantity": 25,
          "units": "%"
        }
      ]
    }
  ]
}

This hierarchical structure captures the “recipe” of complex mixtures while remaining machine-readable.

MInChI: Canonical Mixture Identifiers

While Mixfiles provide comprehensive descriptions, simple identifiers are also needed for database storage and searching. This is where MInChI comes in.

A MInChI string is structured as:

MInChI=0.00.1S/<components>/n<indexing>/g<concentration>

• Header: Version information (0.00.1S in the paper’s specification)
• Components: Standard InChI for each unique molecule, sorted alphabetically by the InChI strings themselves, then concatenated with &
• Indexing (prefixed with /n): Hierarchical structure using curly braces {} for branches and & for adjacent nodes; uses 1-based integer indices referring to the sorted InChI list
• Concentration (prefixed with /g): Quantitative information for each component, with units converted to canonical codes

Why This Matters

MInChI strings enable simple database searches:

• Check if a specific component appears in any mixture
• Compare different formulations of the same product
• Identify similar mixtures based on string similarity

Validating the Standard Through Practical Tooling

The paper demonstrates the format’s capabilities through several practical applications and a proof-of-concept implementation:

Text Extraction Algorithm

The authors demonstrate a proof-of-concept algorithm that uses regular expressions and chemical name recognition to parse plain-text mixture descriptions into structured Mixfile data. The algorithm:

1. Applies regex rules to remove filler words and extract concentrations
2. Looks up cleaned names against a custom chemical database
3. Falls back to OPSIN for SMILES generation from chemical names
4. Generates 2D coordinates for molecular structures

Graphical Editor

An open-source editor provides:

• Tree-based interface for building and editing hierarchical structures
• Chemical structure sketching and editing
• Database lookup (e.g., PubChem integration)
• Automatic MInChI generation
• Import/export capabilities

Example Use Cases

The paper validates the format through real-world applications:

• Safety compliance: Automated hazard assessment based on concentration-dependent properties (e.g., solid osmium tetroxide vs. 1% aqueous solution)
• Inventory management: Precise, searchable laboratory records
• Data extraction: Parsing vendor catalogs and safety data sheets

Outcomes and Future Extensibility

The work successfully establishes the first standardized, machine-readable formats for chemical mixtures. Key achievements:

• Comprehensive representation: Mixfile captures component identity, quantity, and hierarchy
• Canonical identification: MInChI provides compact, searchable identifiers
• Practical tooling: Open-source editor and text extraction demonstrate feasibility
• Real-world validation: Format handles diverse use cases from safety to inventory

Limitations and Future Directions

The authors acknowledge areas for improvement:

• Machine learning improvements: Better text extraction using modern NLP techniques
• Extended coverage: Support for polymers, complex formulations, analytical results
• Community adoption: Integration with existing chemical databases and software

The hierarchical design makes Mixfile suitable for both “recipe” descriptions (how to make something) and analytical results (what was found). This flexibility should help drive adoption across different use cases in chemistry and materials science.

Reproducibility Details

Open Source Tooling & Data

While the central repository focusing on validating and establishing the MInChI standard is github.com/IUPAC/MInChI, the tools and datasets actually used to develop the paper’s proofs-of-concept are hosted elsewhere:

• Graphical Editor & App codebase: The Electron application and Mixfile handling codebase (console.js) can be found at github.com/cdd/mixtures.
• Text Extraction Data: The several thousand extracted mixture records generated through the text extraction method can be accessed inside the cdd/mixtures repository under reference/gathering.zip.

Artifacts

The paper was funded by NIH Grant 1R43TR002528-01. No specific hardware requirements are needed, as this is a format specification with lightweight tooling.

Algorithms

This section provides the specific algorithmic logic, schema definitions, and standardization rules needed to replicate the Mixfile parser or MInChI generator.

The Strict Mixfile JSON Schema

To implement the format, a parser must recognize these specific fields:

Root Structure:

{
  "mixfileVersion": 0.01,
  "header": {},
  "contents": []
}

Component Fields:

• name: string (required if no structure is provided)
• molfile: string (the primary source of truth for molecular structure)
• smiles, inchi, formula: derived/transient fields for convenience
• quantity: number OR [min, max] array for ranges
• units: string (must map to supported ontology)
• relation: string (e.g., ">", "~", ">=")
• ratio: array of two numbers [numerator, denominator]
• identifiers: database assignments (e.g., CASRN, PubChem)
• links: URLs relevant to the component
• contents: recursive array for hierarchical mixtures

MInChI Generation Algorithm

To generate MInChI=0.00.1S/..., the software must follow these steps:

1. Component Layer:

   • Calculate standard InChI for all structures in the mixture
   • Sort distinct InChIs alphabetically by the InChI string itself
   • Join with & to form the structure layer

2. Hierarchy & Concentration Layers:

   • Traverse the Mixfile tree recursively
   • Indexing: Use integer indices (1-based) referring to the sorted InChI list
   • Grouping: Use {} to denote hierarchy branches and & to separate nodes at the same level
   • Concentration: Convert all quantities to canonical unit codes and apply scaling factors

Unit Standardization Table

Replication requires mapping input units to canonical MInChI codes. The full table from the paper (Table 1) includes:

Text Extraction Logic

The paper defines a recursive procedure for parsing plain-text mixture descriptions:

1. Input: Raw text string (e.g., “2 M acetone in water”)
2. Rule Application: Apply RegEx rules in order:
   • Remove: Delete common filler words (“solution”, “in”)
   • Replace: Substitute known variations
   • Concentration: Extract quantities like “2 M”, “97%”
   • Branch: Split phrases like “A in B” into sub-nodes
3. Lookup: Check cleaned name against a custom table (handles cases like “xylenes” or specific structures)
4. OPSIN: If no lookup match, send to the OPSIN tool to generate SMILES from the chemical name
5. Embed: If structure found, generate 2D coordinates (Molfile) via RDKit

Paper Information

Citation: Clark, A. M., McEwen, L. R., Gedeck, P., & Bunin, B. A. (2019). Capturing mixture composition: an open machine-readable format for representing mixed substances. Journal of Cheminformatics, 11(1), 33. https://doi.org/10.1186/s13321-019-0357-4

Publication: Journal of Cheminformatics (2019)

@article{clark2019capturing,
  title={Capturing mixture composition: an open machine-readable format for representing mixed substances},
  author={Clark, Alex M and McEwen, Leah R and Gedeck, Peter and Bunin, Barry A},
  journal={Journal of cheminformatics},
  volume={11},
  number={1},
  pages={33},
  year={2019},
  publisher={BioMed Central}
}

Additional Resources:

• Official MInChI GitHub repository

Decades of chemical research, breakthroughs in medicine, and novel materials are archived in journals, patents, and textbooks. A huge portion of this knowledge is stored as images, a format inaccessible to standard computational tools. This imposes challenges for both data retrieval and leveraging modern computational tools to analyze and predict chemical properties, inefficiencies that compound across the literature: knowledge locked in image form is invisible to search, mining, and downstream model training.

This is the central challenge that Optical Chemical Structure Recognition (OCSR) aims to solve. At its heart, OCSR is to chemistry what OCR (Optical Character Recognition) is to text: a technology that teaches computers to extract chemical information directly from 2D diagrams of molecules. It’s the bridge between a picture of a molecule and a machine-readable format like SMILES (Simplified Molecular Input Line Entry System) that can be stored, searched, and used to power new discoveries.

Teaching a computer to read a chemical structure requires specialized techniques.

The Complexity of Chemical Graphs

Recognizing a molecule requires specialized techniques that extend standard Optical Character Recognition (OCR). A molecule is a graph: a collection of atoms (nodes) connected by bonds (edges).

(While this simplified view excludes complex structures like coordination compounds and polymers, it provides a highly effective starting point for this discussion.)

An OCSR system must overcome several hurdles:

• Varying Styles: Chemical drawings vary widely across publications. Bond lengths, angles, and fonts can differ dramatically from one document to another.

• Image Quality: Older documents might be scanned at low resolutions, containing noise, blur, or other artifacts that make interpretation difficult.

• Structural Complexity: From simple rings to sprawling polymers and complex Markush structures (common in patents to represent a whole family of related compounds), the variety is immense.

The Evolution of OCSR

The quest to automate this process has evolved significantly, moving from brittle, hand-coded systems to sophisticated AI that can learn from data.

Act 1: The Rule-Based Pioneers (OCR-1.0)

The first OCSR systems, developed in the early 1990s, represent what we can now call the “OCR-1.0” era. Tools like Kekulé, and later open-source solutions like OSRA and MolVec, operated like meticulous draftsmen. Their approach was methodical:

1. Vectorize the Image: Convert the pixel-based image into a collection of lines and shapes
2. Identify Components: Use a set of hard-coded rules to classify these components. “This thick line is a wedge bond.” “This group of pixels is the letter ‘O’.”
3. Reconstruct the Graph: Piece together the identified atoms and bonds into a coherent molecular graph

This rule-based approach was a real first step but brittle. It struggled with the messiness of real-world documents and was expensive to maintain because each new style or error required new rules.

Additionally, they were designed as interactive tools to assist human experts in digitizing chemical structures. There was always the assumption that a human would review and correct the output.

As a concrete case-study, consider the (reproduced) results from MolParser:

Table 2. Comparison of our method with existing OCSR models. We report the accuracy. We use bold to indicate the best performance and underline to denote the second-best performance. *: re-implemented results. †: results from original publications.

In this table, we see that the rule-based methods (OSRA, MolVec, Imago) perform reasonably well on cleaner datasets like USPTO and UoB but falter on more challenging ones like JPO and ColoredBG. Modern AI-based methods (MolGrapher, DECIMER, MolScribe, MolParser) improve most on the hardest benchmarks (like JPO and ColoredBG), especially when fine-tuned on real data, while the rule-based tools still do reasonably well on cleaner sets like USPTO and UoB.

Act 2: The AI Fork in the Road (2010s-2020s)

The rise of deep learning in the 2010s brought new paradigms that could learn from data. Here, the field split into two distinct paths.

Path A: The Rise of the Specialists (Graph-Based AI)

Some models replaced the hard-coded rules with AI components. Systems like MolGrapher and MolScribe use a two-stage process:

• Atom Detection: A neural network first identifies all the atoms in the image
• Bond Prediction: A second process then predicts the connections (bonds) between those atoms to form the final graph

These are highly specialized tools, trained specifically for the task of building a molecular graph.

Path B: The Rise of the Generalists (LVLMs)

Another, more direct method treats OCSR as an image captioning task. This approach aligns with the broader trend of Large Vision-Language Models (LVLMs): massive, general-purpose AIs like GPT-4V. Models like DECIMER and MolParser look at a molecular image and directly generate its textual representation, most commonly a SMILES string. This direct, end-to-end approach is powerful, though it requires enormous datasets to train effectively.

The Next Frontier: The OCR-2.0 Vision (2024+)

Recently, a proposal has emerged that charts a third path forward: OCR-2.0. This vision, proposed by Wei et al. in 2024, argues for a new class of models that combine the best of both worlds. An OCR-2.0 model should be:

1. End-to-End: A single, unified model that simplifies maintenance
2. Efficient & Low-Cost: A specialized, highly efficient perception engine. The paper argues that using a giant LVLM for a pure recognition task is often inefficient
3. Versatile: Capable of handling diverse artificial optical signals

The flagship model for this theory is GOT (General OCR Theory). It’s a single, unified model that can read an image and output structured text for a wide variety of inputs. It can translate a molecular diagram into a SMILES string, transcribe sheet music into musical notation, parse a bar chart into a data table, and describe a geometric shape using code.

This demonstrates that OCSR can be integrated into broader systems for processing human visual information. The same OCR-2.0 philosophy extends beyond chemistry: GutenOCR, for instance, applies grounded vision-language modeling to general document OCR, producing both text transcriptions and bounding-box outputs from a single model.

Pushing the Boundaries of Recognition

OCR-2.0 models like GOT push for breadth, and other state-of-the-art research deepens the depth of understanding for the uniquely complex task of chemical recognition.

Deepening Reasoning with a “Visual Chain of Thought”

The GTR-Mol-VLM model makes recognition more intelligent by mimicking how a person might analyze a complex diagram. The model traverses the molecule step-by-step, predicting an atom, then its bond, then the next atom, and so on. This “Visual Chain of Thought” improves accuracy, especially for complex molecules. It also faithfully recognizes abbreviations like “Ph” as single units, better representing the source image.

Deepening Application with “Visual Fingerprinting”

Subgrapher rethinks the end goal. Many applications (like searching a patent database) require only the identification of specific molecular features. Subgrapher detects key functional groups and backbones directly from the image and creates a visual fingerprint. This approach mirrors identifying a person by key features (“has glasses, a mustache”), making it well-suited to finding matches in a large set.

Why It Matters

The evolution of OCSR directly enables practical scientific advancements. This technology is a critical enabler for the future of science.

Searching Past Knowledge

OCSR digitizes decades of research from patents and journals, making it searchable and accessible for data mining. Imagine being able to search through every molecule ever published with a simple query. Or consider the practical impact: pharmaceutical companies can now automatically scan thousands of patent documents to ensure their new drug candidates don’t infringe existing intellectual property, a process that previously required substantial manual review by patent analysts.

Accelerating Drug Discovery

By extracting vast datasets of molecules, scientists can train AI models to predict drug efficacy and toxicity, speeding up the discovery pipeline. The more molecular data we can digitize, the better our predictive models become.

Building Universal Document Intelligence

OCSR contributes to building AI systems capable of processing complex human documents. A scientific paper is a mix of text, equations, charts, tables, and molecular diagrams. Unified OCR-2.0 models are the key to making all of this knowledge searchable holistically.

Looking Forward

The goal is a loop where scientific knowledge, regardless of how it is stored, can be fed back into systems that read, search, and reason over it.

From the rule-based systems of the 1990s to today’s models that read many printed diagrams reliably (though hard cases like low-quality scans and Markush structures remain open), OCSR has improved a great deal. As accuracy, efficiency, and breadth improve, more of the chemical literature becomes machine-readable.

This entire process begins with teaching a computer how to read a picture.

Citation: Fang, X., Wang, J., Cai, X., Chen, S., Yang, S., Tao, H., Wang, N., Yao, L., Zhang, L., & Ke, G. (2025). MolParser: End-to-end Visual Recognition of Molecule Structures in the Wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 24528-24538). https://doi.org/10.48550/arXiv.2411.11098

Publication: ICCV 2025

Additional Resources:

• MolParser-7M Dataset - 7M+ image-text pairs for OCSR
• MolParser-7M on HuggingFace - Dataset repository
• MolDet YOLO Detector - Object detection model for extracting molecular images from documents

Contribution: End-to-End OCSR and Real-World Resources

This is primarily a Method paper (see AI and Physical Sciences paper taxonomy), with a significant secondary contribution as a Resource paper.

Method contribution ($\Psi_{\text{Method}}$): The paper proposes a novel end-to-end architecture combining a Swin Transformer encoder with a BART decoder, and crucially introduces Extended SMILES (E-SMILES), a new syntactic extension to standard SMILES notation that enables representation of Markush structures, abstract rings, and variable attachment points found in patents. The work validates this method through extensive ablation studies, achieving the highest accuracy among tested OCSR systems on WildMol-10k (76.9%).

Resource contribution ($\Psi_{\text{Resource}}$): The paper introduces MolParser-7M, the largest OCSR dataset to date (7.7M image-text pairs), and WildMol, a challenging benchmark of 20,000 manually annotated real-world molecular images. The construction of these datasets through an active learning data engine with human-in-the-loop validation represents significant infrastructure that enables future OCSR research.

Motivation: Extracting Chemistry from Real-World Documents

The motivation stems from a practical problem in chemical informatics: vast amounts of chemical knowledge remain embedded in unstructured formats. Patents, research papers, and legacy documents depict molecular structures as images. This creates a barrier for large-scale data analysis and prevents Large Language Models from effectively understanding scientific literature in chemistry and drug discovery.

Existing OCSR methods struggle with real-world documents for two fundamental reasons:

1. Representational limitations: Standard SMILES notation cannot capture complex structural templates like Markush structures, which are ubiquitous in patents. These structures define entire families of compounds using variable R-groups and abstract patterns, making them essential for intellectual property but impossible to represent with conventional methods.
2. Data distribution mismatch: Real-world molecular images suffer from noise, inconsistent drawing styles, variable resolution, and interference from surrounding text. Models trained exclusively on clean, synthetically rendered molecules fail to generalize when applied to actual documents.

Novelty: E-SMILES and Human-in-the-Loop Data Engine

The novelty lies in a comprehensive system that addresses both representation and data quality challenges through four integrated contributions:

1. Extended SMILES (E-SMILES): A backward-compatible extension to the SMILES format that can represent complex structures previously inexpressible in standard chemical notations. E-SMILES uses a separator token <sep> to delineate the core molecular structure from supplementary annotations. These annotations employ XML-like tags to encode Markush structures, polymers, abstract rings, and other complex patterns. Critically, the core structure remains parseable by standard cheminformatics tools like RDKit, while the supplementary tags provide a structured, LLM-friendly format for capturing edge cases.

2. MolParser-7M Dataset: The largest publicly available OCSR dataset, containing over 7 million image-text pairs. What distinguishes this dataset is both its scale and its composition. It includes 400,000 “in-the-wild” samples (molecular images extracted from actual patents and scientific papers) and subsequently curated by human annotators. This real-world data addresses the distribution mismatch problem directly by exposing the model to the same noise, artifacts, and stylistic variations it encounters in production.

3. Human-in-the-Loop Data Engine: A systematic approach to collecting and annotating real-world training data. The pipeline begins with an object detection model that extracts molecular images from over a million PDF documents. An active learning algorithm then identifies the most informative samples (those where the current model struggles) for human annotation. The model pre-annotates these images, and human experts review and correct them. This creates an iterative improvement cycle: annotate, train, identify new challenging cases, repeat.

4. Efficient End-to-End Architecture: The model treats OCSR as an image captioning problem. A Swin-Transformer vision encoder extracts visual features, a simple MLP compresses them, and a BART decoder generates the E-SMILES string autoregressively. The model minimizes the standard negative log-likelihood of the target E-SMILES token sequence $y$ given the sequence history and input image $x$:

$$ \begin{aligned} \mathcal{L} = -\sum_{t=1}^{T} \log P(y_t \mid y_{<t}, x; \theta) \end{aligned} $$

The training strategy employs curriculum learning, starting with simple molecules and gradually introducing complexity and heavier data augmentation.

Experimental Setup: Two-Stage Training and Benchmarking

The evaluation focused on demonstrating that MolParser generalizes to real-world documents:

1. Two-Stage Training Protocol: The model underwent a systematic training process:

   • Pre-training: Initial training on millions of synthetic molecular images using curriculum learning. The curriculum progresses from simple molecules to complex structures while gradually increasing data augmentation intensity (blur, noise, perspective transforms).
   • Fine-tuning: Subsequent training on 400,000 curated real-world samples extracted from patents and papers. This fine-tuning phase is critical for adapting to the noise and stylistic variations of actual documents.

2. Benchmark Evaluation: The model was evaluated on multiple standard OCSR benchmarks to establish baseline performance on clean data. These benchmarks test recognition accuracy on well-formatted molecular diagrams.

3. Real-World Document Analysis: The critical test involved applying MolParser to molecular structures extracted directly from scientific documents. This evaluation measures the gap between synthetic benchmark performance and real-world applicability (the core problem the paper addresses).

4. Ablation Studies: Experiments isolating the contribution of each component:

   • The impact of real-world training data versus synthetic-only training
   • The effectiveness of curriculum learning versus standard training
   • The value of the human-in-the-loop annotation pipeline versus random sampling
   • The necessity of E-SMILES extensions for capturing complex structures

Outcomes and Empirical Findings

• Performance on Benchmarks: MolParser achieves competitive results on standard benchmarks and the best performance on real-world documents. On clean benchmarks like USPTO-10K, MolScribe (96.0%) slightly edges MolParser-Base (94.5%), but on WildMol-10k, MolParser-Base achieved 76.9% accuracy, significantly outperforming MolScribe (66.4%) and MolGrapher (45.5%). This gap validates the core hypothesis that training on actual document images is essential for practical deployment.

• Real-World Data is Critical: Models trained exclusively on synthetic data show substantial performance degradation when applied to real documents. The 400,000 in-the-wild training samples bridge this gap, demonstrating that data quality and distribution matching matter as much as model architecture. Ablation experiments showed that pretraining on MolParser-7M synthetic data alone achieved 51.9% accuracy on WildMol, while adding real-world fine-tuning raised this to 76.9%. Using the smaller MolGrapher-300k synthetic dataset without fine-tuning yielded only 22.4%.

• E-SMILES Enables Broader Coverage: The extended representation successfully captures molecular structures that were previously inexpressible, particularly Markush structures from patents. This expands the scope of what can be automatically extracted from chemical literature to include patent-style structural templates.

• Human-in-the-Loop Scales Efficiently: The active learning pipeline reduces annotation time by approximately 90% while maintaining high quality. This approach makes it feasible to curate large-scale, high-quality datasets for specialized domains where expert knowledge is expensive.

• Speed and Accuracy: The end-to-end architecture achieves both high accuracy and fast inference, making it practical for large-scale document processing. MolParser-Base processes 40 images per second on RTX 4090D, while the Tiny variant achieves 131 FPS. The direct image-to-text approach avoids the error accumulation of multi-stage pipelines.

• Downstream Applications: The Swin Transformer encoder, once trained on MolParser-7M, serves as an effective molecular fingerprint for property prediction. Paired with a simple two-layer MLP on MoleculeNet benchmarks, MolParser-pretrained features achieved an average ROC-AUC of 73.7% across five tasks, compared to 68.9% for ImageNet-pretrained Swin-T features. The authors also demonstrate chemical reaction parsing by feeding MolDet detections and MolParser E-SMILES into GPT-4o.

• Limitations: The authors acknowledge that molecular chirality is not yet fully exploited by the system. The E-SMILES format does not currently support dashed abstract rings, coordination bonds, special symbol Markush patterns, or replication of long structural segments. Additionally, scaling up the volume of real annotated training data could further improve performance.

The work establishes that practical OCSR requires more than architectural innovations. It demands careful attention to data quality, representation design, and the distribution mismatch between synthetic training data and real-world applications. The combination of E-SMILES, the MolParser-7M dataset, and the human-in-the-loop data engine provides a template for building reliable vision systems in scientific domains where clean training data is scarce but expert knowledge is available.

Artifacts

No official source code repository has been released. Model weights for MolParser itself are not publicly available as of the dataset release.

Reproducibility Details

Data

The training data is split into a massive synthetic pre-training set and a curated fine-tuning set.

Training Data Composition (MolParser-7M):

• Sources: 1.2M patents and scientific papers (PDF documents)
• Extraction: MolDet (YOLO11-based detector) identified ~20M molecular images, deduplicated to ~4M candidates
• Selection: Active learning ensemble (5-fold models) identified high-uncertainty samples for annotation
• Annotation: Human experts corrected model pre-annotations (90% time savings vs. from-scratch annotation)

Test Benchmarks:

Algorithms

Extended SMILES (E-SMILES) Encoding:

• Format: SMILES<sep>EXTENSION where <sep> separates core structure from supplementary annotations
• Extensions use XML-like tags:
  • <a>index:group</a> for substituents/variable groups (Markush structures)
  • <r> for groups connected at any ring position
  • <c> for abstract rings
  • <dum> for connection points
• Backward compatible: Core SMILES parseable by RDKit; extensions provide structured format for edge cases

Curriculum Learning Strategy:

• Phase 1: No augmentation, simple molecules (<60 tokens)
• Phase 2: Gradually increase augmentation intensity and sequence length
• Progressive complexity allows stable training on diverse molecular structures

Active Learning Data Selection:

1. Train 5 model folds on current dataset
2. Compute pairwise Tanimoto similarity of predictions on candidate images
3. Select samples with confidence scores 0.6-0.9 for human review (highest learning value)
4. Human experts correct model pre-annotations
5. Iteratively expand training set with hard samples

Data Augmentations:

• RandomAffine (rotation, scale, translation)
• JPEGCompress (compression artifacts)
• InverseColor (color inversion)
• SurroundingCharacters (text interference)
• RandomCircle (circular artifacts)
• ColorJitter (brightness, contrast variations)
• Downscale (resolution reduction)
• Bounds (boundary cropping variations)

Models

The architecture follows a standard Image Captioning (Encoder-Decoder) paradigm.

Architecture Specifications:

Training Configuration:

Curriculum Learning Schedule (Pre-training):

• Starts with simple molecules (<60 tokens, no augmentation)
• Gradually adds complexity and augmentation (blur, noise, perspective transforms)
• Enables stable learning across diverse molecular structures

Evaluation

Metrics: Exact match accuracy on predicted E-SMILES strings (molecule-level exact match)

Key Results:

Additional Performance:

• MolParser-Tiny: 131 FPS on RTX 4090D (66M params)
• Real-world vs. synthetic gap: Fine-tuning on MolParser-SFT-400k closed the performance gap between clean benchmarks and in-the-wild documents

Ablation Findings: