All of that structure is derived from a SMILES string.

Molecules are also drawn. A structure diagram carries the same topology a chemist reads directly off the page, in a form no language model ever sees. So I wanted to know whether MIST’s geometry is reachable from pixels, or whether it is a property of the string.

There are two standard ways to teach one model to land where another one does. You can push negatives apart with a contrastive loss, or you can skip the negatives and regress the target directly. I expected the predictive one to transfer better to real drawings, because a contrastive objective can exploit shortcuts that only exist inside a training batch. I wrote that down before running anything.

It held, then it reversed, and the mechanism I had named for it was wrong in both cases.

What I Built

One vision tower, one frozen text tower, and a single linear map between them.

Arm A optimizes the SigLIP sigmoid pairwise loss. Arm B regresses the frozen target in latent space with a smooth-L1 loss and no negatives, which is the I-JEPA move applied as an alignment head rather than as pretraining. Everything else is held identical: same molecules, same splits, same seed, same schedule, same evaluation, checked by an assertion before any table prints. Every number below is one seed.

Two details in the setup are load-bearing.

The pooling is MIST’s own. Their embedding is the last-layer hidden state at the first token, so anything else measures a different space. Loading the checkpoint with AutoModel also attaches a randomly initialized pooler unless you ask it not to, and pooler_output from that pooler returns plausible noise.

The contrastive labels come from molecular identity rather than batch position. Two rows in a batch can be depictions of the same molecule, and treating them as negatives because they sit at different indices would penalize the model for being right. That detail matters more than it looks, and I will come back to it.

This is not an OCSR contribution. Reading a molecule out of a picture is a mature field, and MolScribe, MolGrapher, DECIMER and Img2Mol all do it well. What is here is a controlled experiment about alignment objectives that happens to run on a domain with a real evaluation surface.

That domain has a catch worth stating early, because it shapes which number is the result. The training depictions are rendered from SMILES strings. A picture drawn from a string carries no information the string did not already have, so the in-distribution task is closer to inverting a renderer than to reading chemistry. Transfer to depictions that came from the world is the part that tests something. That is why the out-of-distribution slice is the headline and the in-distribution number is context.

I came to MIST from the tokenizer side. I had published a controlled comparison of BPE and Unigram-LM over chemistry SMILES a few weeks earlier, and MIST is a SMILES foundation model whose headline contribution is its own tokenizer. I was reading the paper closely for that reason and stayed for the embedding geometry.

What I Registered In Advance

Three predictions, committed before any training run existed:

1. Both arms will be near-ceiling on in-distribution synthetic retrieval, and that number on its own will mean almost nothing.
2. The arms will separate, if at all, on the WildMol-10k OOD slice and the scaffold-split linear probe. That gap is the result.
3. Arm B is expected to hold up better OOD, on the reasoning that regressing a fixed target cannot exploit in-batch shortcuts the way a contrastive objective can. This is a guess and is written down so it can be wrong in public.

I also registered what would count as a clean negative and ship as the result: both arms at or near chance on the OOD slice and the probe, with all baselines behaving correctly. That sentence turns out to matter more than the predictions do.

What Had To Pass First

Four gates run before any headline number is believed, because a comparison between two arms says nothing if the harness scoring both of them is broken.

The image-statistics floor is the one that earns its keep below. It is what a model scores using nothing but coarse pixel statistics, and any headline has to clear it.

A Perfect Model Scored Zero

Raw cosine against MIST measures almost nothing. Its embeddings occupy a narrow cone, where two unrelated molecules sit at cosine 0.517 and a model that ignores its input entirely and emits the training centroid scores 0.720. So every cosine in this project is centered, and each side is centered by its own training-split mean:

$$ \mathrm{cc} = \cos\left(\hat{\mathbf{u}} - \bar{\hat{\mathbf{u}}}_{\text{train}},\ \mathbf{y} - \bar{\mathbf{y}}_{\text{train}}\right) $$

Chance is zero. Both centroids come from the training split, since centering a held-out set by its own mean leaks that set’s geometry into its own score.

The version I shipped first subtracted the target centroid from both sides. That is valid only while the prediction sits at roughly the target’s scale. Arm B’s smooth-L1 pins it there. Arm A’s loss L2-normalizes both sides internally, so it constrains the output norm not at all, and weight decay shrinks it with nothing opposing.

Take an oracle predictor $\hat{\mathbf{u}} = s\mathbf{y}$, whose true cosine is exactly 1 for every $s > 0$:

A perfect model emitting unit-norm vectors scored below chance.

On the real 20K run, that defect scored a working contrastive model at +0.1088, inside noise of the +0.1058 image-statistics floor. Arm A’s head was emitting norm 6.35 against MIST’s 22.28. Under the corrected metric the same checkpoint scores +0.3052.

The uncomfortable part is what that number was. I had registered, in advance, that both arms landing at the floor with every baseline behaving correctly would be a clean negative and would ship as the result. The defect produced precisely that. Every exit criterion passed.

No test caught it. What caught it was asking what a perfect model would score under this metric, which is a question whose answer is knowable before you ask it.

Registering a prediction stops the goalposts moving once data exists. It does nothing at all about a defect that happens to confirm the prediction, and it makes that case more expensive rather than less, because the incentive to accept a number and stop looking is strongest exactly where you said the number would be.

There was a second defect of the same shape waiting. Had the contrastive labels come from batch position instead of molecular identity, the model would have been penalized for putting two depictions of one molecule in the same place. That depresses contrastive transfer and leaves predictive untouched, because only the contrastive loss reads other rows in the batch, which is prediction 3 arriving as an artifact of the label matrix.

The Data Ran Out

With the backbone frozen, I fitted a log-linear law to the closed-form ridge ceiling over 1K to 16K molecules. It described that range well, at r = 0.9906. Extrapolated, it said 1M molecules would reach +0.600.

Going from 796K to 1.6M depictions bought +0.0002.

The mechanism is in the features rather than the sample. On these images, 54.3% of DINOv2’s feature variance is depiction style rather than molecular identity. Two renders of the same molecule sit at cosine 0.857, and two different molecules sit at 0.809, which leaves very little room between “same structure” and “different structure” for a linear head to work with.

A linear head can reorganize what the backbone encodes. It cannot manufacture what the backbone discarded. That showed up directly in the probes, where alignment helped on the tasks whose property the frozen features already carried and did nothing at all on three of seven.

This is the second prediction I made in advance and missed, and it was the more expensive one, because I had been about to spend on data.

Unfreezing Reversed My Own Prediction

Letting the backbone move changes the picture. I swept two trainable modes rather than picking one: LoRA at rank 16 on the attention projections of all twelve blocks, and the top four blocks plus the final norm. The top-blocks mode won on every layer, and LoRA was also 1.84x slower per step despite training 16x fewer parameters, because backward cost tracks how deep the pass goes rather than how many parameters update.

The out-of-distribution gate is what makes this mean something, and it matters because an earlier lever failed it. Running the backbone at its native 518px instead of 224 improved every synthetic-image metric, several by multiple standard deviations, and moved the real-depiction metric by less than one. That lever was fitting RanDepict’s particular ink rather than learning chemistry. Unfreezing does not behave that way.

Then the registered prediction inverted.

The ordering was a property of the frozen bottleneck rather than of the objectives. Give the contrastive objective a backbone it can actually shape and it is the one that degrades less.

Had I only run the frozen half, I would have published a confirmed prediction with a mechanism attached, and the mechanism would have been wrong. The number was fine. The explanation underneath it was not, and nothing in the frozen regime could have told me.

Prediction 1 was wrong outright, which is worth recording since I registered it as the boring one. I expected in-distribution retrieval to sit near ceiling and carry no information. It came in at 0.0617 and 0.0310 against a 0.0156 chance floor, which is why it is reported throughout rather than waved off.

On one fixed checkpoint, the aligned image embedding out-probes MIST-direct on 6 of 7 tasks, losing only BBBP. Best-of-eight beats it on 7 of 7, and I am not quoting that as the result because it is selection over eight tries. The honest reading is that MIST bounds MIST-reconstruction rather than every downstream task where vision carries independent signal.

The Fairest Rule Picked the Worst Checkpoint

The two arms have losses on different scales, so selecting each on its own loss would compare two selection procedures rather than two objectives. I selected on centered cosine instead, the one metric both arms are measured by. Neutral by construction.

It chose epoch 1 of 24 for the contrastive arm, in both trainable modes.

I had written this down on the first day of the project, at the milestone where the two arms first ran side by side:

Selecting on it hands Arm A its worst retrieval checkpoint. Selecting on retrieval instead would reverse the ranking and would be equally defensible and equally partial.

And in the same entry, on why I was recording it at all:

Recording it now, at M3, because it is exactly the kind of thing that gets quietly resolved in whichever direction the first table happened to point.

I scoped that observation as measured outside the contrastive arm’s viable regime, which was correct at the time. At 20K molecules there is no batch size that is both a real contrastive batch and a small fraction of the dataset, so the arm was running at 256 while SigLIP’s own calibration assumes 16,384. Rebuilding at 1M fixed that, and the failure happened anyway, in the regime where it counted.

The obvious objection is that the constraint itself was the cause. Holding the learning rate, the schedule and the checkpoint rule identical across two objectives that optimize different things does not make a comparison fair, it handicaps whichever arm does not match the shared choice.

The repair would be to let each arm carry its own schedule, and gradient accumulation is the specific one that was on the table. I did not add it, and I still think that was right. Accumulation enlarges the contrastive arm’s negative pool and does nothing for the predictive one, so adding it converts a comparison of objectives into a comparison of training budgets. That is a second bias rather than a smaller one.

What I did not do was change the rule after seeing what it cost. Retrofitting a selection criterion once results exist is the specific thing pre-registration exists to prevent, so both epochs are reported instead.

Three shared choices ended up bending against the contrastive arm, all in the same direction. Weight decay at 0.05 against a scale-invariant loss, which is what produced the metric defect. Batch 256 against a calibration assuming 16,384. A selection metric that measures direction agreement, which is close to what the predictive arm optimizes directly.

It won the layer I had named in advance as the deciding one anyway, 0.4555 against 0.2847. So the measured gap is a floor rather than an estimate.

What I Would Not Trust Here

Three things would change a conclusion above, in order of how much.

The probe splits shift in composition. ESOL’s training split is 35% acyclic molecules against 0% of validation and test, and Tox21 is about 32%. Acyclic molecules have no Bemis-Murcko scaffold, so a scaffold split places them all in one group and that group is large enough to land in train. ESOL is the largest probe gain in this post, and it was scored on a held-out set with none of the molecules that make up a third of what it trained on.

One seed, and the sigmas are uneven. The only repeats in the project are a 5-seed frozen sweep at 20K. Arm A retrieval sigma runs .002 to .005 and BBBP .004 to .011. The retrieval orderings clear that by roughly two orders of magnitude and are safe. Individual probe cells do not clear it, and several rows show test scores above validation, which is what noise on a small scaffold-split test set looks like.

The predictive arm was truncated. It was still rising at epoch 24 in both modes, and 24 epochs was inherited from the frozen regime where it was enough. Its numbers are a lower bound, which cuts against this post’s own conclusion.

Conclusion: One Number Cannot Rank Two Objectives

The comparison never resolved to a winner in the way I expected when I started, and the reason is more interesting than a winner would have been. Under a frozen backbone the predictive arm holds up better, and that ordering is a fact about the bottleneck. Under a trainable one the contrastive arm wins out of distribution, and it does so while carrying every shared-procedure handicap.

Three of the four things above were numbers I initially believed. A metric that scored a perfect model below chance, a scaling law that was accurate everywhere I had fitted it, and a selection rule that was neutral by construction. Each looked fine, and each was checkable in advance by asking what a known answer should be.

The one that still bothers me is the metric, because it produced the exact result I had written down as shippable. Pre-registration protected me from moving the goalposts and did nothing about that.

Code, weights and data are public. The repository carries the full results, the harness gates and the pre-registration verbatim. The two aligned models are on Hugging Face as contrastive and predictive, and the 20K depiction set is published so the comparison can be re-run without standing up the rendering environment.

Two ways to align an image of a molecular structure diagram into the embedding space of a frozen chemistry language model, compared on one data and evaluation stack. A DINOv2 ViT-S/14-reg vision tower, a frozen MIST-28M text tower, and a 197,120-parameter linear map between them.

Arm A optimizes the SigLIP sigmoid pairwise loss. Arm B regresses the frozen target directly in latent space with no negatives. Everything else is held identical, and an assertion checks that before any comparison table prints.

The comparison is the deliverable. The models are how it was obtained.

Results

Retrieval R@1 against 64-molecule galleries of nearest-Tanimoto distractors, chance 0.0156. Out-of-distribution is WildMol-10k, real depictions extracted from patents and papers, never trained on.

With the backbone frozen, the ceiling is the representation. Fifty times more data moved a closed-form ridge ceiling by +0.020 against +0.088 predicted by a log-linear law fitted at small scale, and 54.3% of DINOv2’s feature variance on these images is depiction style rather than molecular identity.

Unfreezing moves retrieval 9.6x in-distribution and 11.1x out of distribution, and the pre-registered prediction reverses: the predictive arm transfers better under a frozen backbone and the contrastive arm wins outright once the backbone can move.

Features

• Pre-registered predictions, committed before any training run and quoted verbatim in the results with how each came out. One was wrong outright and one held then reversed.
• Four harness gates run before any headline number is believed: an oracle retrieval control, MIST against Tanimoto similarity, a random-embedding control, and a trivial image-statistics floor.
• Seven linear-probe points across five MoleculeNet datasets, on Bemis-Murcko scaffold splits committed as build artifacts so a probe number is comparable across clones.
• An out-of-distribution slice of real literature depictions, deduplicated against every training set by InChIKey with a positive control confirming the check detects injected leakage.
• Open weights and data, with an export path that verifies the published weights are the same function of pixels as the checkpoint they came from.

Usage

The released weights need timm and torch, and nothing from this repository:

import timm, torch

backbone = timm.create_model(
    "vit_small_patch14_reg4_dinov2.lvd142m", pretrained=False, num_classes=0, img_size=224
)
backbone.load_state_dict(torch.load("backbone.pt", weights_only=True))
head = torch.nn.Linear(384, 512)
head.load_state_dict(torch.load("projection_head.pt", weights_only=True))
backbone.eval(); head.eval()

with torch.no_grad():
    embedding = head(backbone(pixels))   # (batch, 512), in MIST space

Comparing images to each other works with the above alone. Comparing an image to a molecule additionally needs MIST to embed that molecule.

Weights are on Hugging Face as contrastive and predictive, and the 20K depiction set is published so the comparison can be re-run without the rendering environment.

Retrospective

Scoped to a couple of weekends, and complete rather than paused. There is no roadmap.

The parts worth keeping are methodological. A metric defect scored a working model at the pre-registered clean-negative floor with every exit criterion passing, and it was caught by asking what a perfect model would score rather than by a test. The arm-neutral checkpoint selection rule, chosen because it is the one scalar both arms are measured by, selected epoch 1 of 24 for the contrastive arm at a cost of 2.6x on in-distribution retrieval. That was reported rather than fixed, since changing a pre-registered rule after seeing results is what pre-registration exists to prevent.

Every v2 number is one seed. The only repeats in the project are a five-seed frozen sweep, and the retrieval orderings clear that noise floor by roughly two orders of magnitude while individual probe cells do not.

The code is Apache-2.0. The released weights carry research-only terms mirroring MIST’s, since they were trained to predict its embeddings.

Abstract

Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE’s principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open: the complete glyph base already covers every conformant molecule, so the learned pieces add compression rather than coverage, and a tiny alphabet under hard valence constraints could drive two frequency-based algorithms to converge. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces above the shared base never exceeds $0.161$, and at most $0.05$ once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into $29$–$41%$ more tokens; the arms largely agree on where to cut but not how deeply, so BPE’s segmentation is a strict coarsening of Unigram-LM’s on $80$–$99%$ of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale, past where embeddings remain learnable; only token-frequency imbalance attenuates in magnitude, shrinking with vocabulary size and most on the natural-products corpus, without closing. The subword algorithm is therefore a modeling decision, not a free default. We release all trained tokenizers and per-condition measurements.

Key Contributions & Results

• Near-disjoint vocabularies. Across all 22 matched conditions, cross-algorithm Jaccard overlap on the learned multi-glyph pieces never exceeds $0.161$, and drops to at most $0.05$ once weighted toward the high-frequency pieces a model updates most. The two algorithms share the glyph base and almost none of the subwords built on top of it.
• Same cuts, different depth. Unigram-LM segments held-out molecules into $29$–$41%$ more tokens. The two arms largely agree on where to cut but disagree on how deeply, so BPE’s segmentation is a strict coarsening of Unigram-LM’s on $80$–$99%$ of molecules. The disagreement is nesting, not conflict.
• The separation is structural. It holds across corpus typology, boundary policy, and vocabulary size, persisting even at $8\times$ the headline vocabulary, past the regime where token embeddings remain learnable. Only token-frequency imbalance attenuates in magnitude (shrinking with vocabulary size, most on the natural-products corpus) without closing.
• No language models trained. Every result is a property of the tokenizers and the corpora, so the comparison is exactly controlled. The claim is about vocabularies, not downstream task scores.
• Released artifacts. All trained tokenizers and per-condition measurements are archived for re-analysis on Zenodo.

Study Design

• Controlled grid. The comparison fixes a 165-token OpenSMILES glyph base, then trains BPE and Unigram-LM to matched target vocabulary sizes at the small scales where token embeddings stay learnable. Corpus, glyph base, and target size are held constant so any divergence is attributable to the algorithm.
• Three corpus typologies. Diverse, drug-like, and natural-products corpora span the chemistry a SMILES tokenizer is likely to meet, testing whether the divergence is a property of one chemical distribution or of the algorithms themselves.
• Both boundary policies. The grid runs under both pre-tokenization boundary policies, separating the effect of the subword algorithm from the effect of where pre-tokenization allows pieces to form.
• Mechanism diagnostics. The study reports chemistry-side measurements, previously unreported for SMILES, of three mechanism diagnostics carried from the natural-language literature (dead-zone surplus, scaffold fraction, and whole-pretoken absorption), alongside a structural-character and non-canonicity battery that localizes where the two arms diverge.

Interactive Demo

The live demo takes any molecule, aligns both algorithms on the shared glyph stream, and surfaces the paper’s three contrasts directly: membership (near-disjoint vocabularies), granularity (Unigram-LM stays near-atomic and emits more tokens), and compatibility (the disagreement is nesting, so BPE’s parse is usually a coarsening of Unigram-LM’s). It runs on free CPU: the tokenizers are re-implemented in pure Python and validated byte-faithful against the reference Smirk tokenizer.

Why This Matters

Chemical language models inherit BPE from natural language as an unexamined default. This study shows that the choice of subword algorithm changes the learned vocabulary itself, not merely the compression ratio: BPE and Unigram-LM hand a model different pieces to embed and different segment counts to attend over. That makes the tokenizer a modeling decision to be reported and controlled, the same way an architecture or an objective is, rather than a free default carried over without scrutiny. The released tokenizers and per-condition measurements let others re-analyze the comparison or extend it to downstream training.

Resources

• Paper (arXiv): Full technical report, also on ChemRxiv.
• Code (GitHub): The measurement and pipeline package, reproducible from SHA-pinned corpora.
• Interactive Demo: See both tokenizers carve up any molecule, in-browser.
• Artifacts (Zenodo): Trained tokenizers and full measurement deposits.

Citation

@misc{heidenreich2026neardisjoint,
  title        = {{Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES}},
  author       = {Heidenreich, Hunter},
  year         = {2026},
  eprint       = {2607.05691},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url          = {https://arxiv.org/abs/2607.05691}
}

Related Work

• Deconstructing Recurrence, Attention, and Gating: Related work isolating a single modeling choice under controlled conditions.

This paper is primarily a Resource contribution ($0.7 \Psi_{\text{Resource}}$) with a secondary Method component ($0.3 \Psi_{\text{Method}}$).

It establishes a new, independent benchmark dataset of 2,702 manually selected patent images to evaluate existing Optical Chemical Structure Recognition (OCSR) tools. The authors rigorously compare 8 different methods using this dataset to determine the state-of-the-art. The Resource contribution is evidenced by the creation of this curated benchmark, explicit evaluation metrics (exact connectivity table matching), and public release of datasets, processing scripts, and evaluation tools on Zenodo.

The secondary Method contribution comes through the development of “ChemIC,” a ResNet-50 image classifier designed to categorize images (Single vs. Multiple vs. Reaction) to enable a modular processing pipeline. However, this method serves to support the insights gained from the benchmarking resource.

Motivation: The Need for Realistic, Modality-Diverse Patent Benchmarks

Lack of Standardization: A universally accepted standard set of images for OCSR quality measurement is currently missing; existing tools are often evaluated on synthetic data or limited datasets.

Industrial Relevance: Patents contain diverse and “noisy” image modalities (Markush structures, salts, reactions, hand-drawn styles) that are critical for Freedom to Operate (FTO) and novelty checks in the pharmaceutical industry. These real-world complexities are often missing from existing benchmarks.

Modality Gaps: Different tools excel at different tasks (e.g., single molecules vs. reactions). Monolithic approaches frequently break down on complex patent documents, and there was minimal systematic understanding of which tools perform best for which image types.

Integration Needs: The authors aimed to identify tools to replace or augment their existing rule-based system (OSRA) within the SciWalker application, requiring a rigorous comparative study.

Core Innovation: A Curated Multi-Modality Dataset and Hybrid Classification Pipeline

Independent Benchmark: Creation of a manually curated test set of 2,702 images from real-world patents (WO, EP, US), specifically selected to include “problematic” edge cases like inorganic complexes, peptides, and Markush structures, providing a more realistic evaluation environment than synthetic datasets.

Comprehensive Comparison: Side-by-side evaluation of 8 open-access tools: DECIMER, ReactionDataExtractor, MolScribe, RxnScribe, SwinOCSR, OCMR, MolVec, and OSRA, using identical test conditions and evaluation criteria.

ChemIC Classifier: Implementation of a specialized image classifier (ResNet-50) to distinguish between single molecules, multiple molecules, reactions, and non-chemical images, facilitating a “hybrid” pipeline that routes images to the most appropriate tool.

Strict Evaluation Logic: Utilization of an exact match criterion for connectivity tables (ignoring partial similarity scores like Tanimoto) to reflect rigorous industrial requirements for novelty checking in patent applications.

Methodology: Exact-Match Evaluation Across Eight Open-Source Systems

Tool Selection: Installed and tested 8 tools: DECIMER v2.4.0, ReactionDataExtractor v2.0.0, MolScribe v1.1.1, RxnScribe v1.0, MolVec v0.9.8, OCMR, SwinOCSR, and OSRA v2.1.5.

Dataset Construction:

• Test Set: 2,702 patent images split into three “buckets”: A (Single structure - 1,454 images), B (Multiple structures - 661 images), C (Reactions - 481 images).
• Training Set (for ChemIC): 16,000 images from various sources (Patents, Im2Latex, etc.) split into 12,804 training, 1,604 validation, and 1,604 test images.

Evaluation Protocol:

• Calculated Precision, Recall, and F1 scores based on an exact connectivity table structure matching (rejecting Tanimoto similarity as industrially insufficient). The metrics follow standard formulations where true positives ($\text{TP}$) represent perfectly assembled structures: $$ \text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}} \qquad \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}} \qquad \text{F1} = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} $$
• Manual inspection by four chemists to verify predictions.
• Developed custom tools (ImageComparator and ExcelConstructor) to facilitate visual comparison and result aggregation.

Segmentation Test: Applied DECIMER segmentation to multi-structure images to see if splitting them before processing improved results, combining segmentation with MolScribe for final predictions.

Key Findings: Modality Specialization Outperforms Monolithic Approaches

Single Molecules: MolScribe achieved the highest performance (Precision: 87%, F1: 93%), followed closely by DECIMER (Precision: 84%, F1: 91%). These transformer-based approaches outperformed rule-based methods on single-structure images (e.g., MolScribe F1: 93% vs. OSRA F1: 78%).

Reactions: Evaluated on 103 randomly selected reaction images containing 284 total reactions, RxnScribe outperformed others (Recall: 97%, F1: 86%), demonstrating the value of specialized architectures for reaction diagrams. General-purpose tools struggled with reaction recognition.

Multiple Structures: Evaluated on 20 multi-structure images containing 146 single structures, all AI-based tools struggled. OSRA (rule-based) performed best here but still had low precision (58%). Combining DECIMER segmentation (with the expand option) with MolScribe on these same 20 images improved precision to 82% and F1 to 90%, showing that image segmentation as a preprocessing step can boost multi-structure performance.

Failures: Current tools fail on polymers, large oligomers, and complex Markush structures. Most tools (except MolVec) correctly recognize cis-trans and tetrahedral stereochemistry, but other forms (e.g., octahedral, axial, helical) are not recognized. None of the evaluated tools can reliably recognize dative/coordinate bonds in metal complexes, indicating gaps in training data coverage.

Classifier Utility: The ChemIC model achieved 99.62% accuracy on the test set, validating the feasibility of a modular pipeline where images are routed to the specific tool best suited for that modality. The authors estimate that a hybrid system (MolScribe + OSRA + RxnScribe) routed by ChemIC would achieve an average F1 of 80%, compared to 68% for OSRA alone across all modalities.

Reproducibility Details

Data

Algorithms

Scoring Logic:

• Single Molecules: Score = 1 if exact match of connectivity table (all atoms, valencies, bonds, superatom abbreviations, and charge correct), 0 otherwise. Stereochemistry correctness was not considered a scoring criterion. Tanimoto similarity explicitly rejected as too lenient.
• Reactions: Considered correct if at least one reactant and one product are correct and capture main features. Stoichiometry and conditions ignored.

Image Segmentation: Used DECIMER segmentation (with expand option) to split multi-structure images into single images before passing to MolScribe.

Models

Evaluation

Key Results on Single Structures (Bucket A - 400 random sample):

Key Results on Reactions (Bucket C):

Hardware

ChemIC Training: Trained on a machine with 40 Intel(R) Xeon(R) Gold 6226 CPUs. Training time approximately 6 hours for 100 epochs (early stopping at epoch 26).

Artifacts

Paper Information

Citation: Krasnov, A., Barnabas, S. J., Boehme, T., Boyer, S. K., & Weber, L. (2024). Comparing software tools for optical chemical structure recognition. Digital Discovery, 3(4), 681-693. https://doi.org/10.1039/D3DD00228D

Publication: Digital Discovery 2024

Additional Resources:

• Zenodo Repository (Code & Data)

@article{krasnovComparingSoftwareTools2024,
  title = {Comparing Software Tools for Optical Chemical Structure Recognition},
  author = {Krasnov, Aleksei and Barnabas, Shadrack J. and Boehme, Timo and Boyer, Stephen K. and Weber, Lutz},
  year = {2024},
  journal = {Digital Discovery},
  volume = {3},
  number = {4},
  pages = {681--693},
  publisher = {Royal Society of Chemistry},
  doi = {10.1039/D3DD00228D},
  langid = {english}
}

This is a Systematization paper ($\Psi_{\text{Systematization}}$). It organizes existing literature into two distinct evolutionary phases: Rule-based systems (1990s-2010s) and Machine Learning-based systems (2015-present). It synthesizes performance metrics across these paradigms to highlight the shift from simple classification to “image captioning” (sequence generation).

Justification: The paper focuses on “organizing and synthesizing existing literature” and answers the core question: “What do we know?” The dominant contribution is systematization based on several key indicators:

1. Survey Structure: The paper explicitly structures content by categorizing the field into two distinct historical and methodological groups: “Rule-based systems” and “ML-based systems”. It traces the “evolution of approaches from rule-based structure analyses to complex statistical models”, moving chronologically from early tools like OROCS and OSRA (1990s-2000s) to modern Deep Learning approaches like DECIMER and Vision Transformers.

2. Synthesis of Knowledge: The paper aggregates performance metrics from various distinct studies into unified comparison tables (Table 1 for rule-based and Table 2 for ML-based). It synthesizes technical details of different models, explaining how specific architectures (CNNs, LSTMs, Attention mechanisms) are applied to the specific problem of Optical Chemical Structure Recognition (OCSR).

3. Identification of Gaps: The authors dedicate specific sections to “Gaps of rule-based systems” and “Gaps of ML-based systems”. It concludes with recommendations for future development, such as the need for “standardized datasets” and specific improvements in image augmentation and evaluation metrics.

Motivation for Digitization in Cheminformatics

The primary motivation is the need to digitize vast amounts of chemical knowledge locked in non-digital formats (e.g., scanned PDFs, older textbooks). This is challenging because:

1. Representational Variety: A single chemical formula can be drawn in many visually distinct ways (e.g., different orientations, bond styles, fonts).
2. Legacy Data: Older documents contain noise, low resolution, and disconnected strokes that confuse standard computer vision models.
3. Lack of Standardization: There is no centralized database or standardized benchmark for evaluating OCSR performance, making comparison difficult.

Key Insights and the Paradigm Shift

The paper provides a structured comparison of the “evolution” of OCSR, specifically identifying the pivot point where the field moved from object detection to NLP-inspired sequence generation.

Key insights include:

• The Paradigm Shift: Identifying that OCSR has effectively become an “image captioning” problem where the “caption” is a SMILES or InChI string.
• Metric Critique: It critically analyzes the flaws in current evaluation metrics, noting that Levenshtein Distance (LD) is better than simple accuracy but still fails to capture semantic chemical severity (e.g., mistaking “F” for “S” is worse than a wrong digit).
• Hybrid Potential: Despite the dominance of ML, the authors argue that rule-based heuristics are still valuable for post-processing validation (e.g., checking element order, sequence structure, and formula correspondence).

Comparative Analysis of Rule-Based vs. ML Systems

As a review paper, it aggregates experimental results from primary sources. It compares:

• Rule-based systems: OSRA, chemoCR, Imago, Markov Logic OCSR, and various heuristic approaches.
• ML-based systems: DECIMER (multiple versions), MSE-DUDL, ICMDT (Image Captioning Model based on Deep Transformer-in-Transformer), and other BMS Kaggle competition solutions.

It contrasts these systems using:

• Datasets: BMS (synthetic, 4M images), PubChem (synthetic), U.S. Patents (real-world scanned).
• Metrics: Tanimoto similarity (structural overlap) and Levenshtein distance (string edit distance).

Outcomes, Critical Gaps, and Recommendations

1. Transformers are SOTA: Attention-based encoder-decoder models outperform CNN-RNN hybrids. DECIMER 1.0 achieved 96.47% Tanimoto $= 1.0$ on its test set using an EfficientNet-B3 encoder and Transformer decoder.
2. Data Hungry: Modern approaches require massive datasets (millions of images) and significant compute. DECIMER 1.0 trained on 39M images for 14 days on TPU, while the original DECIMER took 27 days on a single GPU. Rule-based systems required neither large data nor heavy compute but hit a performance ceiling.
3. Critical Gaps:
   • Super-atoms: Current models struggle with abbreviated super-atoms (e.g., “Ph”, “COOH”).
   • Stereochemistry: 3D information (wedges/dashes) is often lost or misinterpreted.
   • Resolution: Models are brittle to resolution changes; some require high-res, others fail if images aren’t downscaled.
4. Recommendation: Future systems should integrate “smart” pre-processing (denoising without cropping) and use domain-specific distance metrics. The authors also note that post-processing formula validation (checking element order, sequence structure, and formula correspondence) increases accuracy by around 5-6% on average. They suggest exploring Capsule Networks as an alternative to CNNs, since capsules add position invariance through routing-by-agreement rather than max-pooling.

Reproducibility

As a review paper, this work does not introduce original code, models, or datasets. The paper itself is open access via the Journal of Cheminformatics. This section summarizes the technical details of the systems reviewed.

Data

The review identifies the following key datasets used for training OCSR models:

Algorithms

The review highlights the progression of algorithms:

• Rule-Based: Hough transforms for bond detection, vectorization/skeletonization, and OCR for atom labels.
• Sequence Modeling:
  • Image Captioning: Encoder (CNN/ViT) → Decoder (RNN/Transformer).
  • Tokenization: Parsing InChI/SMILES into discrete tokens (e.g., splitting C13 into C, 13).
  • Beam Search: Used in inference (typical $k=15-20$) to find the most likely chemical string.

Models

Key architectures reviewed:

• DECIMER 1.0: Uses EfficientNet-B3 (Encoder) and Transformer (Decoder). Predicts SELFIES strings (more robust than SMILES).
• Swin Transformer: Often used in Kaggle ensembles as the visual encoder due to better handling of variable image sizes.
• Grid LSTM: Used in older deep learning approaches (MSE-DUDL) to capture spatial dependencies.

Evaluation

Metrics standard in the field:

• Levenshtein Distance (LD): Edit distance between predicted and ground truth strings. Lower is better. Formally, for two sequences $a$ and $b$ (e.g. SMILES strings) of lengths $|a|$ and $|b|$, the recursive distance $LD(a, b)$ is bounded from $0$ to $\max(|a|, |b|)$.
• Tanimoto Similarity: Measures overlap of molecular fingerprints ($0.0 - 1.0$). Higher is better. DECIMER 1.0 achieved a Tanimoto of 0.99 on PubChem data (Table 2). Calculated as: $$ \begin{aligned} T(A, B) = \frac{N_c}{N_a + N_b - N_c} \end{aligned} $$ where $N_a$ and $N_b$ are the number of bits set to 1 in fingerprints $A$ and $B$, and $N_c$ is the number of common bits set to 1.
• 1-1 Match Rate: Exact string matching (accuracy). For DECIMER 1.0, 96.47% of results achieved Tanimoto $= 1.0$.

Hardware

• Training Cost: High for SOTA. DECIMER 1.0 required ~14 days on TPU. The original DECIMER took ~27 days on a single NVIDIA GPU.
• Inference: Transformer models are heavy; rule-based systems run on standard CPUs but with lower accuracy.

Paper Information

Citation: Musazade, F., Jamalova, N., & Hasanov, J. (2022). Review of techniques and models used in optical chemical structure recognition in images and scanned documents. Journal of Cheminformatics, 14(1), 61. https://doi.org/10.1186/s13321-022-00642-3

Publication: Journal of Cheminformatics 2022

@article{musazadeReviewTechniquesModels2022,
  title = {Review of Techniques and Models Used in Optical Chemical Structure Recognition in Images and Scanned Documents},
  author = {Musazade, Fidan and Jamalova, Narmin and Hasanov, Jamaladdin},
  year = 2022,
  month = sep,
  journal = {Journal of Cheminformatics},
  volume = {14},
  number = {1},
  pages = {61},
  doi = {10.1186/s13321-022-00642-3}
}

This is primarily a Systematization paper ($0.7 \Psi_{\text{Systematization}}$) with a significant Resource component ($0.3 \Psi_{\text{Resource}}$).

It serves as a Systematization because it organizes nearly three decades of research in Optical Chemical Structure Recognition (OCSR), categorizing methods into rule-based systems (e.g., Kekulé, CLiDE, OSRA) and emerging machine-learning approaches (e.g., MSE-DUDL, Chemgrapher). It synthesizes information on 16 distinct tools, many of which are commercial or no longer available.

It acts as a Resource by defining a benchmark for the field. The authors evaluate the three available open-source tools (Imago, MolVec, OSRA) against four distinct datasets to establish baseline performance metrics for accuracy and speed.

Motivation: Digitizing Legacy Chemical Literature

A vast amount of chemical knowledge remains “hidden” in the primary scientific literature (printed or PDF), conveyed as 2D images. Because these depictions are not machine-readable, there is a “backlog of decades of chemical literature” that cannot be easily indexed or searched in open-access databases.

While Chemical Named Entity Recognition (NER) exists for text, translating graphical depictions into formats like SMILES or SDfiles requires specialized OCSR tools. The motivation is to enable the automated curation of this legacy data to feed public databases.

Core Innovations: Historical Taxonomy and Open Standards

The primary novelty is the comprehensive aggregation of the history of the field, which had not been thoroughly reviewed recently. It details the algorithmic evolution from the first work in 1990 to deep learning methods in 2019.

Specific contributions include:

• Historical Taxonomy: Classification of tools into rule-based vs. machine-learning, and open-source vs. commercial/unavailable.
• Open Source Benchmark: A comparative performance analysis of the only three open-source tools available at the time (Imago, MolVec, OSRA) on standardized datasets.
• Algorithmic Breakdown: Detailed summaries of the workflows for closed-source or lost tools (e.g., Kekulé, OROCS, ChemReader) based on their original publications.

Benchmarking Methodology and Open-Source Evaluation

The authors performed a benchmark study to evaluate the accuracy and speed of three open-source OCSR tools: MolVec (0.9.7), Imago (2.0), and OSRA (2.1.0).

They tested these tools on four datasets of varying quality and origin:

1. USPTO: 5,719 images from US patents (high quality).
2. UOB: 5,740 images from the University of Birmingham, published alongside MolRec.
3. CLEF 2012: 961 images from the CLEF-IP evaluation (well-segmented, clean).
4. JPO: 450 images from Japanese patents (low quality, noise, Japanese characters).

Evaluation metrics were:

• Accuracy: Percentage of perfectly recognized structures, mathematically defined as exact string matching between generated and reference standard InChI sequences $\text{Accuracy} = \frac{\text{Correct InChI Matches}}{\text{Total Images}}$ (verified by converting output to InChI strings and matching against reference InChIs).
• Speed: Total processing time for the dataset.

Results and General Conclusions

Benchmark Results (Table 2):

Key Observations:

• MolVec was the fastest tool, processing datasets significantly quicker than competitors (e.g., 28.65 min for USPTO vs. 145.04 min for OSRA).
• OSRA performed exceptionally well on clean, well-segmented data (94.90% on CLEF 2012) but was slower.
• Imago generally lagged in accuracy compared to the other two, particularly on the UOB dataset (63.54% vs. 88.39% for MolVec and 86.50% for OSRA).
• JPO Difficulty: All tools struggled with the noisy Japanese Patent Office dataset (accuracies ranged from 40.00% to 66.67%), highlighting issues with noise and non-standard labels.

General Conclusions:

• No “gold standard” tool existed (as of 2020) that solved all problems (page segmentation, R-groups, NLP integration).
• Rule-based approaches dominate the history of the field, but deep learning methods (MSE-DUDL, Chemgrapher) were emerging, though they were closed-source at the time of writing.
• There was a critical need for tools that could handle full-page recognition (combining segmentation and recognition).

Reproducibility Details

The authors provided sufficient detail to replicate the benchmarking study.

Artifacts

Data

The study used four public datasets. Images were converted to PNG (72 dpi) to ensure compatibility across all tools.

Algorithms

The paper does not propose a new algorithm but benchmarks existing ones. The execution commands for reproducibility were:

• Imago: Executed via command line without installation. ./imago_console -dir /image/directory/path
• MolVec: Executed as a JAR file. java -cp [dependencies] gov.nih.ncats.molvec.Main -dir [input_dir] -outDir [output_dir]
• OSRA: Installed via Conda (PyOSRA) due to compilation complexity. Required dictionaries for superatoms and spelling. osra -f sdf -a [superatom_dict] -l [spelling_dict] -w [output_file] [input_file]

Models

The specific versions of the open-source software tested were:

Evaluation

• Metric: Perfect structural match. The output SDfile/SMILES was converted to a Standard InChI string and compared to the ground truth InChI. Any deviation counted as a failure.
• Environment: Linux workstation (Ubuntu 20.04 LTS).

Hardware

The benchmark was performed on a high-end workstation to measure processing time.

• CPUs: 2x Intel Xeon Silver 4114 (40 threads total).
• RAM: 64 GB.
• Parallelization: MolVec had pre-implemented parallelization features that contributed to its speed.

Paper Information

Citation: Rajan, K., Brinkhaus, H. O., Zielesny, A., & Steinbeck, C. (2020). A review of optical chemical structure recognition tools. Journal of Cheminformatics, 12(1), 60. https://doi.org/10.1186/s13321-020-00465-0

Publication: Journal of Cheminformatics 2020

@article{rajanReviewOpticalChemical2020,
  title = {A Review of Optical Chemical Structure Recognition Tools},
  author = {Rajan, Kohulan and Brinkhaus, Henning Otto and Zielesny, Achim and Steinbeck, Christoph},
  year = 2020,
  month = oct,
  journal = {Journal of Cheminformatics},
  volume = {12},
  number = {1},
  pages = {60},
  issn = {1758-2946},
  doi = {10.1186/s13321-020-00465-0}
}

This is a Resource ($\Psi_{\text{Resource}}$) paper with a secondary contribution in Systematization ($\Psi_{\text{Systematization}}$).

It serves as an infrastructural foundation for the field by establishing the “yardstick” for chemical information retrieval. It defines three distinct tasks, curates the necessary datasets (text and image), and creates the evaluation metrics required to measure progress. Secondarily, it systematizes the field by analyzing 36 different runs from 9 research groups, categorizing the performance of various approaches against these new benchmarks.

Motivation: Bridging Text and Image Search in Chemistry

The primary motivation is to bridge the gap between distinct research communities (text mining and image understanding), which are both essential for chemical information retrieval but rarely interact. Professional searchers in chemistry rely heavily on non-textual information (structures), yet prior evaluation efforts lacked specific tasks to handle image data. The track aims to provide professional searchers with a clear understanding of the limits of current tools while stimulating research interest in both patent retrieval and chemical image recognition.

Novelty: The Image-to-Structure (I2S) Task

The core novelty is the introduction of the Image-to-Structure (I2S) task. While previous years provided image data, this was the first specific task requiring participants to translate a raster image of a molecule into a chemical structure file. Additionally, the Technology Survey (TS) task shifted its focus specifically to biomedical and pharmaceutical topics to investigate how general IR systems handle the high terminological diversity (synonyms, abbreviations) typical of biomedical patents.

Methodology: TREC 2011 Task Formulations

The organizers conducted a large-scale benchmarking campaign across three specific tasks:

1. Prior Art (PA) Task: A patent retrieval task using 1,000 topics distributed among the EPO, USPTO, and WIPO.
2. Technology Survey (TS) Task: An ad-hoc retrieval task focused on 6 specific biomedical/pharmaceutical information needs (e.g., “Tests for HCG hormone”).
3. Image-to-Structure (I2S) Task: A recognition task using 1,000 training images and 1,000 evaluation images from USPTO patents, where systems had to generate the correct chemical structure (MOL file).

A total of 9 groups submitted 36 runs across these tasks. Relevance judgments were performed using stratified sampling and a dual-evaluator system (junior and senior experts) for the TS task.

Outcomes: Task Achievements and Limitations

• Image-to-Structure Success: The new I2S task was the most successful task that year, with 5 participating groups submitting 11 runs. All participants recognized over 60% of the structures.
• Prior Art Saturation: Only 2 groups participated in the PA task. The organizers concluded that this task had reached its “final point,” having learned the extent to which relevant documents can be retrieved in one pass for chemical patent applications.
• Biomedical Complexity: Four teams submitted 14 runs for the TS task, which highlighted the complexity of biomedical queries. The use of specialized domain experts (senior evaluators) and students (junior evaluators) provided high-quality relevance data, though the small number of topics (6) limits broad generalization.

Reproducibility Details

The following details describe the benchmark environment established by the organizers, allowing for the replication of the evaluation.

Data

The track utilized a large collection of approximately 500GB of compressed text and image data.

Algorithms

The paper defines specific evaluation algorithms used to ground-truth the submissions:

• Stratified Sampling (TS): Pools were generated using the method from Yilmaz et al. (2008). The pool included the top 10 documents from all runs, 30% of the top 30, and 10% of the rest down to rank 1000.
• InChI Matching (I2S): Evaluation relied on generating Standard InChI Keys from both the ground truth MOL files and the participant submissions. Success was defined by exact string matching of these keys. This provided a relatively controversy-free measure of chemical identity.

Models

While the paper does not propose a single model, it evaluates several distinct approaches submitted by participants. Notable systems mentioned include:

• OSRA (SAIC-Frederik / NIH)
• ChemReader (University of Michigan)
• ChemOCR (Fraunhofer SCAI)
• UoB (University of Birmingham)
• GGA (GGA Software)

Evaluation

Performance was measured using standard IR metrics for text and exact matching for images.

Artifacts

Hardware

Specific hardware requirements for the participating systems are not detailed in this overview, but the dataset size (500GB) implies significant storage and I/O throughput requirements.

Paper Information

Citation: Lupu, M., Gurulingappa, H., Filippov, I., Zhao, J., Fluck, J., Zimmermann, M., Huang, J., & Tait, J. (2011). Overview of the TREC 2011 Chemical IR Track. In Proceedings of the Twentieth Text REtrieval Conference (TREC 2011).

Publication: Text REtrieval Conference (TREC) 2011

Resources:

• TREC 2011 Proceedings
• TREC 2011 Chemistry Track Data

@inproceedings{lupuOverviewTREC20112011,
  title = {Overview of the {{TREC}} 2011 {{Chemical IR Track}}},
  author = {Lupu, Mihai and Gurulingappa, Harsha and Filippov, Igor and Zhao, Jiashu and Fluck, Juliane and Zimmermann, Marc and Huang, Jimmy and Tait, John},
  year = {2011},
  booktitle = {Proceedings of the Twentieth Text REtrieval Conference (TREC 2011)},
  publisher = {NIST},
  abstract = {The third year of the Chemical IR evaluation track benefitted from the support of many more people interested in the domain, as shown by the number of co-authors of this overview paper. We continued the two tasks we had before, and introduced a new task focused on chemical image recognition. The objective is to gradually move towards systems really useful to the practitioners, and in chemistry, this involves both text and images. The track had a total of 9 groups participating, submitting a total of 36 runs.},
  langid = {english}
}

This is a Resource paper (benchmark infrastructure). It establishes a standardized test bed for the Intellectual Property (IP) Information Retrieval community by defining tasks, curating datasets (topics and relevance judgments), and establishing evaluation protocols. The paper does not propose a new method itself but aggregates and analyzes the performance of participant systems on these shared tasks.

Motivation for Standardized IP Information Retrieval

The volume of patent applications is increasing rapidly, necessitating automated methods to help patent experts find prior art and classify documents.

• Economic Impact: Thorough searches are critical due to the high economic value of granted patents.
• Complexity: Patent work-flows are specific; examiners need to find prior art for specific claims alongside whole documents, and often rely on non-textual data like flowcharts and chemical diagrams.
• Gap: Existing general IR tools are insufficient for the specific granularity (passages, images, structures) required in the IP domain.

Novel Multi-modal Tasks: Claims, Flowcharts, and Chemicals

The 2012 edition of the lab introduced three specific tasks targeting different modalities of patent data:

1. Passage Retrieval starting from Claims: Moving beyond document-level retrieval to identifying specific relevant passages based on claim text.
2. Flowchart Recognition: A new image analysis task requiring the extraction of structural information (nodes, edges, text) from patent images.
3. Chemical Structure Recognition: A dual task of segmenting molecular diagrams from full pages and recognizing them into structural files (MOL), specifically addressing the challenge of Markush structures in patents.

Benchmarking Setup and Evaluation

The “experiments” were the benchmarking tasks themselves, performed by participants (e.g., University of Birmingham, SAIC, TU Vienna).

• Passage Retrieval: Participants retrieved documents and passages for 105 test topics (sets of claims) from a corpus of 1.5 million patents. Performance was measured using PRES, Recall, and MAP at the document level, and AP/Precision at the passage level.
• Flowchart Recognition: Participants extracted graph structures from 100 test images. Evaluation compared the submitted graphs to ground truth using a distance metric based on the Maximum Common Subgraph (MCS).
• Chemical Structure:
  • Segmentation: Identifying bounding boxes of chemical structures in 30 multipage TIFF patents.
  • Recognition: Converting 865 “automatic” (standard MOL) and 95 “manual” (Markush/complex) diagrams into structure files.

Key Findings and Baseline Results

• Passage Retrieval: Approaches varied from two-step retrieval (document then passage) to full NLP techniques. Translation tools were universally used due to the multilingual corpus (English, German, French).
• Chemical Recognition: The best performing system (UoB, run uob-4) achieved 92% recall on total structures (886/960), with 96% on the automatic set and 57% on the manual set. SAIC achieved 83% total recall. The manual evaluation highlighted a critical need for standards extending MOL files to support Markush structures, which are common in patents but poorly supported by current tools.
• Flowchart Recognition: The evaluation was not completed at the time of writing the workshop notes. The evaluation required a combination of structural matching and edit-distance for text labels because OCR outputs rarely “hard-matched” the gold standard.

Chemical Structure Recognition Results

Segmentation (SAIC, best run using OSRA native rendering):

Recognition (automatic and manual sets):

Reproducibility Details

Data

The collection focuses on European Patent Office (EPO) and WIPO documents published up to 2002.

1. Passage Retrieval Data

• Corpus: >1.5 million XML patent documents (EP and WO sources).
• Training Set: 51 topics (sets of claims) with relevance judgments (18 DE, 21 EN, 12 FR).
• Test Set: 105 topics (35 per language).
• Topic Source: Extracted manually from search reports listing “X” or “Y” citations (highly relevant prior art).

2. Flowchart Data

• Format: Black and white TIFF images.
• Training Set: 50 images with textual graph representations.
• Test Set: 100 images.
• Ground Truth: A defined textual format describing nodes (NO), directed edges (DE), undirected edges (UE), and meta-data (MT).

3. Chemical Structure Data

• Segmentation: 30 patent files rendered as 300dpi monochrome multipage TIFFs.
• Recognition (Automatic Set): 865 diagram images fully representable in standard MOL format.
• Recognition (Manual Set): 95 diagram images containing Markush structures or variability not supported by standard MOL.

Algorithms

Ground Truth Generation:

• Qrels Generator: An in-house tool was used to manually map search report citations to specific XML passages (XPaths) for the passage retrieval task.
• McGregor Algorithm: Used for the flowchart evaluation to compute the Maximum Common Subgraph (MCS) between participant submissions and ground truth.

Evaluation

Passage Retrieval Metrics:

• Document Level: PRES (Patent Retrieval Evaluation Score), Recall, MAP. Cut-off at 100 documents.
• Passage Level: $AP(D)$ (Average Precision at document level) and $Precision(D)$ (Precision at document level), averaged across all relevant documents for a topic.

Flowchart Recognition Metric:

• Graph Distance ($d$): Defined quantitatively based on the Maximum Common Subgraph (MCS) between a target flowchart ($F_t$) and a submitted flowchart ($F_s$): $$ \begin{aligned} d(F_t, F_s) &= 1 - \frac{|mcs(F_t, F_s)|}{|F_t| + |F_s| - |mcs(F_t, F_s)|} \end{aligned} $$ where $|F|$ represents the size of the graph (nodes + edges).
• Levels: Evaluated at three levels: Basic (structure only), Intermediate (structure + node types), and Complete (structure + types + text labels).

Chemical Structure Metrics:

• Segmentation: Precision, Recall, and $F_1$ based on bounding box matches. A match is valid if borders are within a tolerance (0 to 55 pixels).
• Recognition:
  • Automatic: Comparison of InChI strings generated by Open Babel.
  • Manual: Visual comparison of images rendered by MarvinView.

Reproducibility

The CLEF-IP 2012 benchmark data was distributed to registered participants through the CLEF evaluation framework. The patent corpus is derived from the MAREC dataset (EPO and WIPO documents published until 2002). Evaluation tools for segmentation (bounding box comparison) and recognition (InChI comparison via Open Babel) were developed in-house by the organizers. The McGregor algorithm implementation for flowchart evaluation was also custom-built.

No public code repositories or pre-trained models are associated with this paper, as it is a benchmarking infrastructure paper. The evaluation protocols and data formats are fully described in the paper.

• Status: Partially Reproducible
• Missing components: The benchmark datasets were distributed to participants and are not hosted on a persistent public repository. The in-house evaluation tools (qrels generator, segmentation comparator, flowchart distance calculator) are not publicly released.

Paper Information

Citation: Piroi, F., Lupu, M., Hanbury, A., Magdy, W., Sexton, A. P., & Filippov, I. (2012). CLEF-IP 2012: Retrieval Experiments in the Intellectual Property Domain. CLEF 2012 Working Notes, CEUR Workshop Proceedings, Vol. 1178.

Publication: CLEF 2012 Working Notes (CEUR-WS Vol. 1178)

@inproceedings{piroi2012clefip,
  title={CLEF-IP 2012: Retrieval Experiments in the Intellectual Property Domain},
  author={Piroi, Florina and Lupu, Mihai and Hanbury, Allan and Magdy, Walid and Sexton, Alan P. and Filippov, Igor},
  booktitle={CLEF 2012 Working Notes},
  series={CEUR Workshop Proceedings},
  volume={1178},
  year={2012},
  publisher={CEUR-WS.org},
  url={https://ceur-ws.org/Vol-1178/CLEF2012wn-CLEFIP-PiroiEt2012.pdf}
}

Optical Chemical Structure Recognition (OCSR) aims to automatically extract machine-readable molecular representations (e.g., SMILES, InChI, mol files) from images of chemical structures. Methods have evolved from early rule-based systems to modern deep learning approaches.

This note organizes OCSR methods by their fundamental approach, providing a framework for understanding the landscape of techniques.

Common Limitations and Failure Modes

Regardless of the underlying paradigm, most OCSR systems struggle with a common set of challenges:

1. Stereochemistry: Ambiguous wedge/dash bonds, varying drawing conventions, and implicit stereocenters frequently lead to incorrect isomer generation.
2. Markush Structures: Generic structures with variable R-groups (common in patents) require complex subgraph mapping that sequence-based models often fail to capture.
3. Image Degradation: Artifacts, low resolution, skewed scans, and hand-drawn irregularities degrade the performance of both rule-based heuristics and CNN feature extractors.
4. Superatoms and Abbreviations: Textual abbreviations (e.g., “Ph”, “t-Bu”, “BoC”) embedded within the image require joint optical character recognition (OCR) and structural parsing.

Review & Survey Papers

Comprehensive surveys and systematization of knowledge papers that organize and synthesize the OCSR literature.

Deep Learning Methods

End-to-end neural network architectures that learn to map images directly to molecular representations.

Note on Paper Types: Papers listed below are primarily Method ($\Psi_{\text{Method}}$) papers focused on novel architectures and performance improvements. Some also have secondary Resource ($\Psi_{\text{Resource}}$) contributions through released tools or datasets. See the AI and Physical Sciences paper taxonomy for classification details.

Image-to-Sequence Paradigm

Treating chemical structure recognition as an image captioning task, these methods use encoder-decoder architectures (often with attention mechanisms) to generate sequential molecular representations like SMILES directly from pixels. Formally, given an image $I$, the model learns to sequentially output tokens $y_t$ to maximize the conditional probability: $$ p(Y|I) = \prod_{t=1}^{T} p(y_t | y_{<t}, I; \theta) $$ where $\theta$ represents the model parameters. This paradigm is powerful but can hallucinate chemically invalid structures if the decoder fails to learn chemical syntax rules.

Image-to-Graph Paradigm

Methods that explicitly construct molecular graphs as intermediate representations, identifying atoms as vertices $V$ and bonds as edges $E$ before converting to standard molecular formats. Graph approaches construct an adjacency matrix $A$ and feature vectors, effectively turning OCSR into a joint probability model over nodes, edges, and their spatial coordinates: $$ p(G|I) = \prod_{v \in V} p(v|I) \prod_{u,v \in V} p(e_{uv}|v_u, v_v, I) $$ This avoids hallucinating invalid character strings and explicitly grounds the predictions to the image space (via bounding boxes/segmentation), improving interpretability.

Image-to-Fingerprint Paradigm

Methods that bypass molecular graph reconstruction entirely, generating molecular fingerprints directly from images through functional group recognition and spatial analysis. These approaches prioritize retrieval and similarity search over exact structure reconstruction.

Image Classification and Filtering

Methods that classify chemical structure images for preprocessing purposes, such as detecting Markush structures or other problematic inputs that should be filtered before full OCSR processing.

Traditional Machine Learning Methods

Hybrid approaches combining classical machine learning algorithms (neural networks, SVMs, CRFs) with domain-specific heuristics and image processing. These methods (primarily from 1992-2014) used ML for specific subtasks like character recognition or symbol classification while relying on rule-based systems for chemical structure interpretation.

Rule-Based Methods

Classic approaches using heuristics, image processing, and domain-specific rules. While some systems use traditional OCR engines (which may contain ML components), the chemical structure recognition itself is purely algorithmic.

Note: The chemoCR systems use SVM-based OCR but employ rule-based topology-preserving vectorization for core structure reconstruction, placing them primarily in this category.

Core Methods

TREC 2011 Chemistry Track

The TREC 2011 Chemistry Track provided a standardized benchmark for comparing OCSR systems, introducing the novel Image-to-Structure task alongside Prior Art and Technology Survey tasks. Papers from this evaluation are grouped here.

CLEF 2012 Chemistry Track

The CLEF-IP 2012 benchmarking lab introduced three specific IR tasks in the intellectual property domain: claims-based passage retrieval, flowchart recognition, and chemical structure recognition. The chemical structure recognition task included both segmentation (identifying bounding boxes) and recognition (converting to MOL format) subtasks, with a particular focus on challenging Markush structures common in patents.

This is a personal working taxonomy, a lens I use to orient myself when reading papers, not a formal classification scheme. The categories are fuzzy in practice, and reasonable people will assign the same paper differently. The goal is clarity of thought, not consensus.

The taxonomy uses a superposition model where each paper is viewed as a linear combination of seven fundamental contribution types (basis vectors). Examples throughout draw primarily from chemistry and materials science, though the framework applies across physical sciences.

The framework helps answer: “What is this paper’s primary contribution?” by identifying rhetorical patterns and structural elements that signal different research paradigms.

Core Principle: The Superposition Model

All papers in this domain can be viewed as a superposition of fundamental contribution vectors. This is an analogy, not a formal mathematical claim: the seven types below are not linearly independent in the strict sense, and most papers project onto more than one.

Most papers exhibit a profile across multiple basis vectors, blending contribution types (e.g., Method + Theory). One vector usually provides the primary narrative thrust; secondary vectors supply the supporting evidence.

I tend to classify a paper by identifying its Primary Projection (the dominant contribution) and Secondary Projections (supporting work). When a paper seems to split evenly across two types, I look at which type’s indicators dominate the abstract and section headings, and I treat the secondary projection as context that supports the primary narrative, not a co-equal claim.

The Seven Basis Vectors ($\Psi$)

Assessment Guide: Rhetorical Indicators

These rhetorical patterns tend to signal which vector dominates. They are heuristics, not rules; the goal is to notice what the paper is primarily arguing for, not to tick boxes.

1. $\Psi_{\text{Method}}$: The Methodological Paper

Focuses on proposing a novel mechanism, architecture, or approximation (e.g., a new Transformer variant, a GNN with symmetry, a new DFT functional).

Rhetorical Indicators:

• Ablation Study: Authors systematically remove components of their system to prove their specific innovation drives the performance gain. In physics venues, this often appears as parameter sensitivity analysis or direct comparison to prior potentials and functionals rather than a classic ablation table.
• Baseline Comparison: A prominent table comparing the new method against the State-of-the-Art (SOTA)
• Pseudo-code: An explicit block detailing the algorithmic steps (e.g., for training, sampling, or inference)

Examples in the notes: Flow Matching for Generative Modeling (introduces a new continuous normalizing flow mechanism with ablations), MOFFlow (novel SE(3) flow matching architecture for metal-organic frameworks with SOTA comparisons).

2. $\Psi_{\text{Theory}}$: The Theoretical Paper

Focuses on mathematical guarantees, proofs, or derivations from first principles.

Rhetorical Indicators:

• Mathematical Proof Sections: Sections titled “Theorem 1,” “Proof of Equivariance,” or “Formal Bounds”
• Analysis of Limits/Capacity: Investigates the expressivity (e.g., comparing a GNN to the Weisfeiler-Lehman Test) or analyzes the geometry of the optimization landscape
• Generalization/OOD: Derives generalization bounds on test error or formally defines “chemical space coverage” for out-of-distribution (OOD) behavior
• Exact Constraints: Derives exact conditions that true physical functions (like the universal Density Functional) must satisfy

Examples in the notes: The Convexity Principle and Interacting Gases (formal theoretical analysis connecting flow-based generative models to thermodynamic principles).

3. $\Psi_{\text{Resource}}$: The Infrastructure Paper

Focuses on creating and sharing foundational tools for the community.

Rhetorical Indicators:

• Curation Description: Detailed steps on how data was generated, filtered, or curated (e.g., describing millions of CPU-hours of DFT calculations for a dataset like QM9). In physics contexts, this often takes the form of DFT protocol descriptions and computational settings rather than a formal “datasheet.”
• “Datasheets” and “Data Cards”: Inclusion of formal documentation detailing provenance, copyright, and potential biases in the data. In physics contexts, this often appears as detailed computational settings, DFT functional choices, and convergence criteria rather than a named datasheet document.
• Benchmark Definition: Argues that “Metric X on Dataset Y” is the correct proxy for progress in a specific scientific task

Examples in the notes: Molecular Sets (MOSES) (releases a benchmark dataset and evaluation suite for molecular generation), SELFIES 2023 (releases an updated molecular string representation standard with tooling and validation).

4. $\Psi_{\text{Systematization}}$: The Review Paper

Focuses on organizing and synthesizing existing literature.

Rhetorical Indicators:

• Survey Structure: Follows a linear, often chronological, progression or is grouped by architecture (e.g., Variational Autoencoders (VAEs), GANs, Diffusion models)
• Systematization of Knowledge (SoK): A higher-order contribution that proposes a new taxonomy or a unified framework to connect disparate concepts
• Citation Density: References a large fraction of the relevant prior literature; often includes a comparison table spanning many prior works across years or venues

Examples in the notes: Embedded Atom Method Review (1993) (organizes and contextualizes the body of work on EAM potentials into a coherent reference).

5. $\Psi_{\text{Position}}$: The Sociological Paper

Focuses on meta-science, arguing for a change in community norms, or critiquing systemic issues.

I see two subtypes come up often. The first is the roadmap/perspective paper: a constructive argument for where the field should focus, often written by senior researchers after a period of reflection. The second is the critique paper: a more adversarial argument that something the community is currently doing is wrong or counterproductive. Both share the same core signal (the argument itself is the contribution), but they have different tones and are often targeted at different audiences.

Rhetorical Indicators:

• Venue/Track: Often found in “Position Tracks” or called “Blue Sky” or “Forward Looking” papers
• Argumentative Tone: Uses qualitative or quantitative analysis (meta-analysis) to argue for a shift in how research is conducted or funded (e.g., a paper arguing that AI contracts the focus of science)
• Argument as Contribution: The paper presents no new experimental findings; the primary contribution is the argument itself

Examples in the notes: Fold Graciously (position paper arguing for a shift in how the protein folding community evaluates progress), SELFIES 2022 (makes a normative argument for adopting SELFIES over SMILES for robust molecular generation; the paper also has Method and Resource characteristics, but the argument-first framing and advocacy tone push it toward Position for me).

6. $\Psi_{\text{Discovery}}$: The Empirical Discovery Paper

Focuses on the discovery of novel scientific artifacts using AI/ML tools. The key criterion is experimental or higher-fidelity confirmation of the AI’s prediction (wet-lab synthesis, physical characterization, or a higher-fidelity simulation), rather than performance on a held-out test set.

Rhetorical Indicators:

• Structure: Follows a workflow: (1) Computational Screening (AI selects candidates), (2) Validation (wet-lab synthesis, physical characterization, or independent computational confirmation)
• Core Claim: The primary contribution is a new material, molecule, or physical finding, with the AI/ML part serving as the necessary first step
• Key Question: Does the AI’s prediction hold true against an independent ground truth (e.g., a physical experiment or a higher-fidelity simulation)?

Examples in the notes: Oxidation-Reduction Oscillations on Pt/SiO2 (1994) (validates a computationally-predicted dynamic phenomenon against physical experiment).

7. $\Psi_{\text{Application}}$: The Application Paper

Applies an existing method, architecture, or technique to a new scientific domain or task without introducing a new mechanism, dataset, or experimentally confirmed finding. The primary contribution is demonstrating feasibility or utility of transfer.

Rhetorical Indicators:

• “We Apply X to Y” framing: The abstract explicitly names an existing method and a new domain; the contribution is the connection, not the method itself.
• Benchmark-style Results: Performance reported on domain-specific tasks using existing metrics; baselines are often simple (classical methods, prior domain tools) rather than ML SOTA.
• Absence of novelty claims: The paper does not argue for a new architecture, does not release a new dataset, and does not report a validated experimental discovery.

Examples in the notes: Benchmarking LLMs for Molecular Property Prediction (evaluates GPT-3.5, GPT-4, and Llama-2 on six existing OGB molecular tasks without modifying the models), Maxsmi: SMILES Augmentation for Property Prediction (systematic evaluation of five augmentation strategies with existing CNN/RNN architectures), RNNs vs Transformers for Molecular Generation Tasks (empirical comparison of existing architectures on SMILES and SELFIES generation).

Edge Cases and Disambiguation

These are the pairs I find myself second-guessing most often. I’ve written down how I tend to resolve them, not as rules, but as a record of my reasoning.

Method vs. Discovery: The key question I ask is whether there is independent validation beyond a held-out test set. If the paper reports a wet-lab confirmation or a higher-fidelity simulation verifying the AI’s output, I lean toward Discovery: the finding is real and the AI was the tool. If the architecture is the thing being argued for, and validation is limited to benchmarks, I lean toward Method. One edge case I see often in computational chemistry is ML-predicts + DFT-validates: no wet lab, but DFT is genuinely independent of the ML model. I treat this as Discovery if the DFT result is the claimed contribution, and Method if the ML model’s performance against DFT is what’s being argued for.

Resource vs. Systematization: I think of this as artifact-first vs. interpretation-first. If the paper’s main output is something you can download and use (a benchmark, a curated dataset, a software library), it’s Resource. If the paper’s main output is a new way of thinking about a body of literature (a taxonomy, a conceptual unification), it’s Systematization. The presence of a clear benchmark definition with metrics tends to push toward Resource even when there is substantial review content.

Application vs. Method: The cleanest distinguisher here is whether anything new was designed. If an existing method is taken off the shelf and pointed at a new domain, that’s Application. If the domain’s requirements motivated a modification to the method (even a small one), then there may be a legitimate Method component. Application papers often read as feasibility demonstrations; Method papers read as capability expansions.

When I’m still stuck: I look at three things in roughly this order: the abstract’s final sentence (in my experience, especially in ML venues, authors tend to signal what they most want credit for in that last sentence), section heading vocabulary (presence of “theorem,” “algorithm,” “dataset,” “survey,” or “we find” is usually diagnostic), and venue track (position/perspective tracks, SoK tracks, and empirical tracks each carry strong prior signal). I treat the result as a best guess, not a verdict.

Applications

This framework has been useful to me in a few concrete ways:

• Organizing literature reviews: Knowing a paper’s vector profile tells me where it belongs in a review’s structure: whether it fits in the “methods” or “findings” or “community context” section, rather than forcing everything into chronological order.
• Understanding conference and journal acceptance criteria: Different venues weight vectors differently. Physics journals favor Discovery; ML venues favor Method and Theory; interdisciplinary venues like Nature or Science often weight Discovery heavily but also publish high-impact Position papers. Knowing this helps calibrate expectations.
• Identifying gaps in research portfolios: A collection that is heavy on Method but light on Resource or Discovery points to work that may lack empirical grounding or shared infrastructure. The vector breakdown makes these gaps visible.
• Recognizing different types of scientific contribution: Not everything that looks like a methods paper is one, and not everything that looks like a survey is one. The rhetorical patterns here help me notice when my first impression was wrong and reconsider what is actually being claimed.

On-Site Enforcement

Every note_type: paper entry on this site declares its Primary Projection through a single-valued paper_types frontmatter list. A schema validator rejects notes that omit this field or use values outside the seven canonical vectors, and the classification is surfaced as a Hugo taxonomy at /paper-types/. Browsing a vector landing page such as /paper-types/method/ or /paper-types/systematization/ is the fastest way to pull up every note that shares a contribution type.

Enforcement is deliberately single-select: it captures the Primary Projection only. Secondary vectors still exist in the text of each note, but the tag axis is chosen to discriminate between classes rather than to describe a paper exhaustively.