Image-to-Sequence OCSR: A Comparative Analysis

Overview

This note provides a comparative analysis of image-to-sequence methods for Optical Chemical Structure Recognition (OCSR). These methods treat molecular structure recognition as an image captioning task, using encoder-decoder architectures to generate sequential molecular representations (SMILES, SELFIES, InChI) directly from pixels.

For the full taxonomy of OCSR approaches including image-to-graph and rule-based methods, see the OCSR Methods taxonomy.

Architectural Evolution (2019-2025)

The field has undergone rapid architectural evolution, with clear generational shifts in both encoder and decoder design.

Timeline

Encoder Architectures

Decoder Architectures

Output Representation Comparison

The choice of molecular string representation significantly impacts model performance. Representations fall into three categories: core molecular formats for single structures, extended formats for molecular families and variable structures (primarily Markush structures in patents), and specialized representations optimizing for specific recognition challenges.

The Rajan et al. 2022 ablation study provides a comparison of core formats.

Core Molecular Formats

These represent specific, concrete molecular structures.

SMILES and Variants

SMILES remains the dominant format due to its compactness and highest accuracy on clean data. Standard SMILES uses single characters for ring closures and branches that may appear far apart in the sequence, creating learning challenges for sequence models.

DeepSMILES addresses these non-local syntax dependencies by modifying how branches and ring closures are encoded, making sequences more learnable for neural models. Despite this modification, DeepSMILES sequences are ~1.1× longer than standard SMILES (not shorter). The format offers partial validity improvements through regex-based tokenization with a compact 76-token vocabulary, providing a middle ground between SMILES accuracy and guaranteed validity.

SELFIES guarantees 100% valid molecules by design through a context-free grammar, eliminating invalid outputs entirely. This comes at the cost of ~1.5× longer sequences and a typical 2-5% accuracy drop compared to SMILES on exact-match metrics. The validity guarantee makes SELFIES particularly attractive for generative modeling applications.

InChI uses a layered canonical syntax fundamentally different from SMILES-based formats. While valuable for unique molecular identification, its complex multi-layer structure (formula, connectivity, stereochemistry, isotopes, etc.) and longer sequences make it less suitable for image-to-sequence learning, resulting in lower recognition accuracy.

Key Findings from Rajan et al. 2022

1. SMILES achieves highest exact-match accuracy on clean synthetic data
2. SELFIES guarantees 100% valid molecules but at cost of ~2-5% accuracy drop
3. InChI is problematic due to complex layered syntax and longer sequences
4. DeepSMILES offers middle ground with partial validity improvements through modified syntax

Extended Formats for Variable Structures

Markush structures represent families of molecules rather than single compounds, using variable groups (R1, R2, etc.) with textual definitions. They are ubiquitous in patent documents for intellectual property protection. Standard SMILES cannot represent these variable structures.

E-SMILES (Extended SMILES) maintains backward compatibility by using a <sep> token to separate core SMILES from XML-like annotations. Annotations encode Markush substituents (<a>index:group</a>), polymer structures (<p>polymer_info</p>), and abstract ring patterns (<r>abstract_ring</r>). The core structure remains parseable by standard RDKit.

CXSMILES optimizes representation by moving variable groups directly into the main SMILES string as special atoms with explicit atom indexing (e.g., C:1) to link to an extension block containing substituent tables. This handles both frequency variation and position variation in Markush structures.

Specialized Representations

These formats optimize for specific recognition challenges beyond standard single-molecule tasks.

RFL: Ring-Free Language

RFL fundamentally restructures molecular serialization through hierarchical ring decomposition, addressing a core challenge: standard 1D formats (SMILES, SSML) flatten complex 2D molecular graphs, losing explicit spatial relationships.

Mechanism: RFL decomposes molecules into three explicit components:

• Molecular Skeleton (𝒮): Main graph with rings “collapsed”
• Ring Structures (ℛ): Individual ring components stored separately
• Branch Information (ℱ): Connectivity between skeleton and rings

Technical approach:

1. Detect all non-nested rings using DFS
2. Calculate adjacency (γ) between rings based on shared edges
3. Merge isolated rings (γ=0) into SuperAtoms (single node placeholders)
4. Merge adjacent rings (γ>0) into SuperBonds (edge placeholders)
5. Progressive decoding: predict skeleton first, then conditionally decode rings using stored hidden states

Performance: RFL achieves SOTA results on both handwritten (95.38% EM) and printed (95.58% EM) structures, with particular strength on high-complexity molecules where standard baselines fail completely (0% → ~30% on hardest tier).

Note: RFL does not preserve original drawing orientation; it’s focused on computational efficiency through hierarchical decomposition.

SSML: Structure-Specific Markup Language

SSML is the primary orientation-preserving format in OCSR. Based on Chemfig (LaTeX chemical drawing package), it provides step-by-step drawing instructions rather than abstract molecular graphs.

Key characteristics:

• Describes how to draw the molecule, not just its graph structure
• Uses “reconnection marks” for cyclic structures
• Preserves branch angles and spatial relationships
• Significantly outperformed SMILES for handwritten recognition: 92.09% vs 81.89% EM (Hu et al. RCGD 2023)

Use case: Particularly valuable for hand-drawn structure recognition where visual alignment between image and reconstruction sequence aids model learning.

Training Data Comparison

Training data scale has grown dramatically, with a shift toward combining synthetic and real-world images.

Data Scale Evolution

Synthetic vs Real Data

Data Augmentation Strategies

Common augmentation techniques across methods:

Hardware and Compute Requirements

Hardware requirements span several orders of magnitude, from consumer GPUs to TPU pods.

Training Hardware Comparison

Inference Considerations

Few papers report inference speed consistently. Available data:

Accessibility Tiers

Benchmark Performance

Common Evaluation Datasets

Accuracy Comparison (Exact Match %)

Methods are roughly grouped by evaluation era; direct comparison is complicated by different test sets.

Note: Numbers are approximate and may vary by specific test split. See individual paper notes for precise figures.

Stereochemistry Recognition

Stereochemistry remains a persistent challenge across all methods:

Hand-Drawn Recognition

A distinct sub-lineage focuses on hand-drawn/sketched chemical structures.

Hand-Drawn vs Printed Trade-offs

• Hand-drawn methods sacrifice some accuracy on clean printed images
• Require specialized training data (synthetic hand-drawn simulation)
• Generally smaller training sets due to data collection difficulty
• Better suited for educational and lab notebook applications

Key Innovations by Method

Open Challenges

1. Stereochemistry: Consistent challenge across all methods; RL approaches (MolSight) show promise
2. Abbreviations/R-groups: E-SMILES and Markush-specific methods emerging
3. Real-world robustness: Gap between synthetic training and patent/paper images
4. Inference speed: Rarely reported; important for production deployment
5. Memory efficiency: Almost never documented; limits accessibility
6. Multi-molecule images: Most methods assume single isolated structure

References

Individual paper notes linked throughout. For the complete method listing, see the OCSR Methods taxonomy.