This is primarily a Resource ($\Psi_{\text{Resource}}$) paper, with secondary Method ($\Psi_{\text{Method}}$) contributions.

• Resource Basis: The core contribution is “ChemBERTa-3,” an open-source framework integrated into DeepChem that standardizes the pretraining and benchmarking of chemical foundation models. The authors focus heavily on infrastructure (AWS/Ray integration) and correcting benchmarking inconsistencies in the field.
• Method Basis: It trains models like “c3-MoLFormer” to reproduce and validate the infrastructure.

The Pretraining Scalability Challenge

• Scalability Challenges: Building robust molecular models is difficult due to the vast size of chemical space and the computational intensity of pretraining on large datasets.
• Proprietary Barriers: Many high-performing chemical foundation models (e.g., the full MoLFormer-XL) are partially closed-source or difficult to reproduce.
• Benchmarking Inconsistencies: There is a lack of systematic comparison between architectures (e.g., Graph vs. Transformer) using unified protocols. Specifically, previous comparisons relied on reported results that used differing scaffold splitting algorithms, making them inaccurate.

Unified Infrastructure & Standardized Benchmarking

• Unified Infrastructure: Integration of DeepChem with Ray for distributed, scalable pretraining and fine-tuning of both graph and transformer models.
• Standardized Benchmarking: Identification that MoLFormer’s scaffold splitting algorithm differs from the standard DeepChem/MoleculeNet splitter, and the subsequent standardization of these benchmarks for fair comparison.
• New DeepChem Tools: Introduction of the ModularTorchModel class for flexible loss computation and HuggingFaceModel wrappers to bridge ecosystems.

Benchmarking Transformers vs. Graph Models

• Architecture Comparison: Benchmarked Transformers (ChemBERTa, MoLFormer) against Graph models (GROVER, InfoGraph, InfoMax3D, DMPNN, GCN) and baselines (Random Forest).
• Pretraining Scale Disparity:
  • Transformers were pretrained on ZINC20 subsets ranging from 10M to 1.1B molecules (combining ZINC and PubChem).
  • Graph models were limited to 250K molecule subsets due to memory and computational overhead of message passing on large graphs. While this highlights the superior scalability of Transformer architectures, comparing a 1.1B-trained Transformer to a 250K-trained Graph model provides an unbalanced evaluation of architectural capacity.
• Reproducibility Validation: Trained “c3-MoLFormer” (a reproduction of MoLFormer) on 1.1B molecules using two distinct hardware setups: AWS spot instances (Ray) and a local HPC cluster.
• Scaffold Split Analysis: Compared performance metrics using “DeepChem scaffold splits” vs. “MoLFormer scaffold splits” to quantify the impact of data leakage/overlap.

Overcoming Scaffold Splitting Inconsistencies

• Scaling Transformers vs. Graphs: Transformer-based models are significantly easier to scale to large datasets than current graph-based approaches, though performance is comparable at small scales.
• Benchmarking sensitivity: MoLFormer’s reported superiority over baselines was partly inflated by its specific scaffold splitting method, which had higher structural overlap between train and test sets (yielding a lower Tanimoto distance, generally quantified via $1 - \frac{|A \cap B|}{|A \cup B|}$) than DeepChem splits. When standardized, baselines like DMPNN perform more competitively.
• Infrastructure Viability: The framework successfully replicated large-scale training (MoLFormer-1.1B) on both cloud and on-premise HPC, confirming reproducibility.
• Open Source Release: All code, configurations, and the c3-MoLFormer-1.1B model weights are released to facilitate future research.

Reproducibility Details

Data

• Pretraining:
  • Source: ZINC20 (1.4B compounds) and PubChem.
  • Scale: Subsets of 10M, 100M, and 1.1B (100% ZINC20 + 100% PubChem) were used for Transformers. Graph models used a 250K subset.
• Fine-tuning:
  • Suite: MoleculeNet.
  • Tasks: Classification (BACE, BBBP, Tox21, HIV, SIDER, ClinTox) and Regression (ESOL, FreeSolv, Lipophilicity, QM9).
  • Splits: Critical distinction made between “DeepChem scaffold splits” (80/10/10) and “MoLFormer scaffold splits” (which can be downloaded from https://ibm.ent.box.com/v/MoLFormer-data). The paper notes these algorithms differ.

Algorithms

• Framework: DeepChem integrated with Ray for distributed training. To recreate the environment, the repository relies on a nightly version of DeepChem (pip install --pre deepchem) and specific dependencies found within the requirements.txt. Pretraining scripts are available in the chemberta3_benchmarking/pretraining directory of the repository.
• Data Preparation: Featurization workflows (e.g., CircularFingerprint, RDKitConformer) are documented under chemberta3_benchmarking/data/data_preprocessing/ in the codebase.
• Modular Training: Uses ModularTorchModel to allow loss computation from intermediate values and flexible component connection.
• Training Brittleness:
  • Optimizer: Linear learning rate scheduler with warmup.
  • Instability Handling: The authors observed significant loss spikes during warmup. Their primary mitigation strategy involved checkpointing frequently and restarting from the last stable state upon a spike, highlighting a persistent brittleness in optimizing these large chemical foundation models.
  • Numerical Issues: Addressed NaN values by pretraining on a small dataset with low LR before scaling up.

Models

• ChemBERTa: RoBERTa-based architecture trained with Masked Language Modeling (MLM) and Multitask Regression (MTR). Specific model identifiers (e.g., DeepChem/ChemBERTa-100M-MLM) are hosted on Hugging Face so researchers can pull them directly via the transformers library. The core pretraining objective minimized the standard MLM loss: $$ \mathcal{L}{\text{MLM}} = - \frac{1}{|\mathcal{M}|} \sum{i \in \mathcal{M}} \log \hat{y}{i} $$ where $\mathcal{M}$ represents the set of masked SMILES token indices, and $\hat{y}{i}$ is the model’s predicted probability for the correct token given the corrupted sequence context.
• MoLFormer (c3-MoLFormer): Re-implementation of the MoLFormer architecture (Rotary embeddings, linear attention). Specific model identifiers (e.g., DeepChem/MoLFormer-c3-1.1B) are similarly available on Hugging Face.
  • Tokenizer: ibm/MoLFormer-XL-both-10pct tokenizer.
• Graph Models:
  • GROVER: Graph Transformer with node/edge/graph level self-supervision.
  • InfoGraph: Maximizes mutual information between graph-level and substructure representations.
  • InfoMax3D: Incorporates 3D conformer data (via RDKit ETKDGv2) into contrastive pretraining.
  • DMPNN: Directed Message Passing Neural Network (Chemprop variant).

Evaluation

• Metrics: ROC-AUC for classification; RMSE for regression (MAE for QM9).
• Baselines: Random Forest, GCN, DMPNN trained on fine-tuning splits only.
• Protocol: Three independent runs per configuration to report mean and range (not a confidence interval), with the exception of the compute-heavy QM9 dataset, which only received a single run. Benchmarking execution scripts (e.g., GCN, RF, DMPNN, ChemBERTa) are stored in the repo under chemberta3_benchmarking/models_benchmarking/ and contain the specific fine-tuning hyperparameters and optimizer configurations used for each downstream task.
• Key Results:
  • c3-MoLFormer-1.1B achieved ~0.848 ROC-AUC on BACE and ~0.900 on BBBP (using MoLFormer splits). This closely matches the original IBM MoLFormer metrics, validating the reproducibility of the open-source framework.
  • When constrained to the equivalent 250K subset, Graph models (InfoGraph, GROVER) performed comparably to Transformers, indicating that Transformer superiority in chemistry is largely driven by data scalability rather than an inherent architectural advantage at small scales.

Hardware

• Cloud (AWS):
  • Compute: 40 NVIDIA T4 GPUs (g4dn.12xlarge spot instances for pretraining, g4dn.2xlarge for benchmarking).
  • Cost: ~$4000 for MoLFormer 1.1B pretraining.
  • Time: ~10 days (260 hours) for 1.1B model pretraining.
  • Setup: Setup scripts for single-node and multi-node spot EC2 clusters are provided in the GitHub repository’s infra/ and spot/ folders.
• On-Premise HPC:
  • Compute: 16 nodes (AMD EPYC), each with 4 AMD MI300A APUs.
  • Environment: Ray multi-node multi-GPU framework.

Artifacts

Paper Information

Citation: Singh, R. et al. (2026). ChemBERTa-3: an open source training framework for chemical foundation models. Digital Discovery, 5, 662-685. https://doi.org/10.1039/D5DD00348B

Publication: Digital Discovery 2026

Additional Resources:

• ChemBERTa-3 GitHub Repository
• DeepChem Project
• DeepChem Hugging Face Models

@article{singhChemBERTa3OpenSource2026,
  author = {Singh, Riya and Barsainyan, Aryan Amit and Irfan, Rida and Amorin, Connor Joseph and He, Stewart and Davis, Tony and Thiagarajan, Arun and Sankaran, Shiva and Chithrananda, Seyone and Ahmad, Walid and Jones, Derek and McLoughlin, Kevin and Kim, Hyojin and Bhutani, Anoushka and Sathyanarayana, Shreyas Vinaya and Viswanathan, Venkat and Allen, Jonathan E. and Ramsundar, Bharath},
  title = {{{ChemBERTa-3}}: an open source training framework for chemical foundation models},
  journal = {Digital Discovery},
  year = {2026},
  volume = {5},
  pages = {662-685},
  publisher = {The Royal Society of Chemistry},
  doi = {10.1039/D5DD00348B},
  url = {https://doi.org/10.1039/D5DD00348B}
}

This is primarily a Methodological paper, as it proposes a specific neural architecture (GP-MoLFormer) and a novel fine-tuning algorithm (Pair-tuning) for molecular generation. It validates these contributions against standard baselines (e.g., JT-VAE, MolGen-7b).

It also contains a secondary Theoretical contribution by establishing an empirical scaling law that relates inference compute (generation size) to the novelty of the generated molecules.

Motivation: Data Scale and Prompt-Based Optimization

While large language models (LLMs) have transformed text generation, the impact of training data scale and memorization on molecular generative models remains under-explored. Specifically, there is a need to understand how training on billion-scale datasets affects the novelty of generated molecules and whether biases in public databases (like ZINC and PubChem) perpetuate memorization. Furthermore, existing optimization methods often require computationally expensive property predictors or reinforcement learning loops; there is a practical need for more efficient “prompt-based” optimization techniques.

Core Innovations: Architecture and Pair-Tuning

1. Architecture: The application of a linear-attention transformer decoder with Rotary Positional Embeddings (RoPE) to generative chemistry, allowing for efficient training on 1.1 billion SMILES.
2. Pair-Tuning: A novel, parameter-efficient fine-tuning method that uses property-ordered molecular pairs to learn “soft prompts” for optimization without updating the base model weights.
3. Scaling Analysis: An extensive empirical investigation mapping the trade-off between inference compute (up to 10B generations) and chemical novelty, fitting an exponential decay curve that demonstrates how novelty saturates as generation volume grows.

Experimental Methodology and Downstream Tasks

The authors evaluated GP-MoLFormer on three distinct tasks, though the comparisons highlight the difficulty of evaluating foundation models against classical baselines:

1. De Novo Generation: Comparing validity, uniqueness, and novelty against baselines (CharRNN, VAE, LIMO, MolGen-7b) on a held-out test set. Notably, this is an unequal comparison; most baselines were trained on the 1.6M molecule MOSES dataset, whereas GP-MoLFormer uses up to 1.1B molecules, meaning performance gains are heavily driven by data scale.
2. Scaffold-Constrained Decoration: Generating molecules from DRD2 active binder scaffolds and measuring the hit rate of active compounds against specialized scaffold decorators.
3. Property-Guided Optimization: Using Pair-tuning to optimize for Drug-likeness (QED), Penalized logP, and DRD2 binding activity, comparing the results to graph-based and reinforcement learning benchmarks.

Additionally, they performed a Scaling Study:

• Comparing models trained on raw (1.1B) vs. de-duplicated (650M) data.
• Generating up to 10 billion molecules to fit empirical scaling laws for novelty.

Key Findings and Scaling Laws

• Scale Driven Performance: GP-MoLFormer achieves high internal diversity and validity on generation metrics. However, its baseline novelty percentage (~32%) is considerably lower than classical models. The authors attribute this to the massive training scale forcing the model to heavily prioritize matching real-world molecule frequencies over pure exploration. GP-MoLFormer’s advantage in generation metrics over LLM-baselines like MolGen-7b likely stems heavily from its 10x larger training dataset rather than fundamental architectural superiority.
• Pair-Tuning Efficacy: The proposed pair-tuning method effectively optimizes properties (e.g., improving DRD2 activity scores) without requiring full model fine-tuning or external reward loops. While successful, the text-based generation yields ~94.5% validity during optimization, which lags behind graph and SELFIES-based baselines that guarantee 100% structural validity.
• Memorization vs. Novelty: Training on de-duplicated data (GP-MoLFormer-UNIQ) yields higher novelty (approx. 5-8% higher) than training on raw data, confirming that duplication bias in public databases leads directly to memorization.
• Inference Scaling Law: Novelty decays exponentially with generation size ($y = ae^{-bx}$), yet the model maintains generative capability (~16.7% novelty) even after generating an unprecedented 10 billion molecules.

Reproducibility Details

Data

• Sources: A combination of PubChem (111M SMILES) and ZINC (1B SMILES) databases. Downloading and pre-training instructions are located in the repository’s data/README.md.
• Preprocessing:
  • All SMILES were canonicalized using RDKit (no isomeric information).
  • GP-MoLFormer (Base): Trained on the full 1.1B dataset (includes duplicates).
  • GP-MoLFormer-UNIQ: Trained on a de-duplicated subset of 650M SMILES.
• Tokenization: Uses the tokenizer from Schwaller et al. (2019) with a vocabulary size of 2,362 tokens.
• Filtering: Sequences restricted to a maximum length of 202 tokens.

Algorithms

Pair-Tuning (Algorithm 1):

• Objective: Learn task-specific soft prompts $\phi_T$ to maximize the conditional probability of target molecule $b$ given a seed molecule $a$, where pair $(a, b)$ satisfies the property condition $b > a$. The base model parameters $\theta$ remain frozen.
• Prompt Structure: Autoregressive training optimizes the continuous embeddings of $n$ enhancement tokens against the cross-entropy loss of the target sequence: $$ \mathcal{L}(\phi_T) = - \sum_{i=1}^{|b|} \log P_{\theta}(b_i | \phi_T, a, b_{<i}) $$
• Hyperparameters: Trained for 1,000 epochs with a batch size of 35 and a fixed learning rate of $3 \times 10^{-2}$.
• Inference: The learned prompt $\phi_T$ and seed molecule $a$ are prepended as context, and candidates are sampled autoregressively until a termination token is produced.

Models

• Availability: The model trained on deduplicated data (GP-MoLFormer-UNIQ) is publicly available on Hugging Face. The full 1.1B base model is not explicitly hosted. The source code repository includes a disclosure that IBM will not maintain the code going forward.
• Architecture: Transformer decoder (~47M parameters: 12 layers, 12 heads, hidden size 768).
• Attention Mechanism: Combines Linear Attention (Generalized Random Feature map, $\phi$) with Rotary Positional Embeddings (RoPE). To avoid the quadratic complexity of standard attention while maintaining relative positional awareness, RoPE is applied to queries ($Q$) and keys ($K$) prior to the random feature mapping: $$ \text{Attention}(Q, K, V) = \frac{\sum_{n=1}^N \langle \phi(R_m q_m), \phi(R_n k_n) \rangle v_n}{\sum_{n=1}^N \langle \phi(R_m q_m), \phi(R_n k_n) \rangle} $$
• Inference Speed: ~3ms per forward pass on a single A100 GPU.

Evaluation

• Generation Quality Metrics: Validity, Uniqueness, Novelty (MOSES suite), Fréchet ChemNet Distance (FCD), Scaffold similarity (Scaf), and Similarity to Nearest Neighbor (SNN).
• MoLFormer-Based Metrics: The authors introduce Fréchet MoLFormer Distance (FMD) and MoLFormer-space IntDiv2 to measure distributional similarity using their own pre-trained continuous embeddings instead of standard fingerprints.
• Optimization Metrics: Penalized logP (calculated as $\text{logP} - \text{SA} - \text{max}(\text{maxrings}(size) - 6, 0)$), Drug-likeness (QED), and DRD2 activity scores.
• Scaling Metrics: Empirical fit for novelty decay: $y = ae^{-bx}$.

Hardware

• Compute: 16 x NVIDIA A100 (80 GB) GPUs across 2 nodes connected via EDR Infiniband.
• Training Time:
  • GP-MoLFormer (1.1B data): ~115 hours total (28.75 hours/epoch for 4 epochs).
  • GP-MoLFormer-UNIQ (650M data): ~80 hours total.
• Hyperparameters: Used a batch size of 1,600 molecules per GPU with a fixed learning rate of $1.6 \times 10^{-4}$ (scaled up to $8\times$ factor as GPUs increased).
• Optimization: Used distributed data-parallel training and adaptive bucketing by sequence length to handle scale.

Artifacts

The full 1.1B base model weights are not publicly hosted. The training data (PubChem and ZINC) is publicly available, and instructions for downloading and pre-processing are in the repository’s data/README.md.

Paper Information

Citation: Ross, J., Belgodere, B., Hoffman, S. C., Chenthamarakshan, V., Navratil, J., Mroueh, Y., & Das, P. (2025). GP-MoLFormer: A Foundation Model For Molecular Generation. Digital Discovery (2025). https://doi.org/10.1039/D5DD00122F

Publication: Digital Discovery (2025)

@article{ross2025gpmolformer,
  title={GP-MoLFormer: a foundation model for molecular generation},
  author={Ross, Jerret and Belgodere, Brian and Hoffman, Samuel C and Chenthamarakshan, Vijil and Navratil, Jiri and Mroueh, Youssef and Das, Payel},
  journal={Digital Discovery},
  year={2025},
  publisher={Royal Society of Chemistry},
  doi={10.1039/D5DD00122F}
}

This is primarily a Methodological paper with a secondary Resource contribution.

It fits the Method classification because it focuses on optimizing the architecture and pretraining pipeline for molecular transformers. The authors perform extensive ablation studies (varying dataset size from 5M to 77M, comparing MLM vs. MTR objectives) to determine “how well” these strategies work compared to baselines. The secondary Resource classification applies because they open-source the trained models and establish a benchmark on a massive 77M compound dataset.

Key methodological indicators:

• Baseline comparison: The paper explicitly compares ChemBERTa-2 against standard baselines (D-MPNN, Random Forest, GCN) and its predecessor (ChemBERTa-1) with prominent benchmark tables
• Ablation studies: Extensive experiments comparing multi-task and self-supervised pretraining by varying hyperparameters and pretraining dataset size
• Scaling analysis: Systematic investigation of whether larger datasets (up to 77M compounds) yield better performance

Motivations for Scaling Molecular Transformers

The authors aim to bridge the gap between NLP success stories (like GPT-3) and molecular machine learning by developing a “chemical foundation model”.

Key motivations:

• Label scarcity: Experimental labels for molecular properties are rare and expensive, but unlabeled SMILES strings are abundant
• Scaling hypothesis: Testing if scaling pretraining data (up to 77M compounds) yields consistent downstream improvements, similar to scaling laws in NLP
• Efficiency: Optimizing the pretraining process introduced in the original ChemBERTa by comparing self-supervised (MLM) and weakly supervised (MTR, using RDKit computed properties as labels) approaches

Novelty in Multi-Task Regression Objectives

Scale: Training on 77M unique SMILES from PubChem, which is one of the largest molecular pretraining datasets used to date (compared to 10M for ChemBERTa-1 or 18.7M for SMILES-BERT).

Pipeline optimization: A direct, controlled comparison of Masked Language Modeling (MLM) vs. Multi-Task Regression (MTR) pretraining objectives on identical datasets.

Proxy selection: The finding that MLM loss correlates well with MTR loss, allowing the cheaper MLM task to be used for hyperparameter tuning before running the expensive MTR pretraining.

Experimental Pretraining Setup on 77M Compounds

Pretraining Setup

Datasets: Subsets of PubChem containing 5M, 10M, and 77M unique SMILES.

Tasks:

• MLM: Masking 15% of tokens (following RoBERTa procedure). The model is optimized by minimizing the cross-entropy loss over the predicted masked tokens: $$ \mathcal{L}_{MLM} = -\sum_{i \in \mathcal{M}} \log P(x_i \mid \mathbf{x}_{\setminus \mathcal{M}}) $$ where $\mathcal{M}$ represents the set of masked token indices.
• MTR: Predicting 200 calculated molecular properties (via RDKit) simultaneously using a mean squared error objective: $$ \mathcal{L}_{MTR} = \frac{1}{200} \sum_{j=1}^{200} \frac{1}{N} \sum_{i=1}^{N} \left( \hat{y}_{ij} - y_{ij} \right)^2 $$ Continuous target labels $y_{ij}$ are mean-normalized prior to training to equilibrate the disparate scales of different chemical properties.

Hyperparameter search: Ran 50 random configurations on the 5M dataset; selected the top 5 to scale up to 10M and 77M.

Downstream Validation

Finetuning: Evaluated on 8 tasks from MoleculeNet (BACE, BBBP, ClinTox, Delaney, etc.) using scaffold splits (80/10/10).

Analysis: Used UMAP to visualize embeddings from MLM, MTR, and ECFP to check for clustering by label without finetuning.

Key Performance Outcomes and Scaling Realities

Highly competitive performance: ChemBERTa-2 outperforms the D-MPNN baseline (chemprop) on 6 out of 8 MoleculeNet tasks, though the margins demonstrate that task-specific baselines remain notably robust.

MTR superiority: Models pretrained on Multi-Task Regression (MTR) consistently perform better on downstream tasks than those pretrained on MLM on every finetuning task evaluated. MTR is substantially slower than MLM due to the larger input size from the 200-element label vector, but MLM loss serves as a reliable proxy for MTR loss, enabling cheaper architecture search before committing to full MTR pretraining.

Scaling laws versus downstream utility: Pretraining loss improved by 25-35% when increasing the dataset from 5M to 77M compounds. However, this improvement in pretraining loss does not uniformly transfer to downstream tasks. For MTR models, SR-p53 ROC-AUC decreases monotonically from 0.834 (5M) to 0.827 (10M) to 0.817 (77M), and Lipophilicity RMSE is worse at 77M (0.798) than at 5M (0.758), despite a dip at 10M (0.744). This variability in transfer challenges the assumption that pretraining improvements always yield downstream gains.

Transfer learning: The correlation between pretraining loss and downstream performance is task-dependent; it is strong for Lipophilicity but weaker for BACE classification.

Reproducibility Details

Data

The pretraining corpus is derived from PubChem.

Algorithms

Pretraining Objectives:

1. Masked Language Modeling (MLM): Follows RoBERTa procedure. Masks 15% of tokens. Max sequence length 512.
2. Multi-Task Regression (MTR): Predicting 200 RDKit properties. Labels are mean-normalized.

Tokenizer:

• Dictionary of common SMILES characters
• Maximum vocabulary size: 591 tokens

Optimization:

• Patience: Early stopping set to one pass through the dataset to ensure full coverage
• Hyperparameter search: Random search (50 configs) varying hidden size, attention heads, dropout, intermediate size, hidden layers, and learning rate. Note: The precise configuration of the winning models that were scaled to 77M is absent from the paper.

Models

• Architecture: Based on RoBERTa (HuggingFace implementation)
• Parameter scale: Models ranged between 5M and 46M parameters
• Selection: Top 5 configurations from the 5M-dataset random search were trained on the full 77M dataset
• Checkpoints: Pre-trained weights are hosted by DeepChem on Hugging Face. Direct links include DeepChem/ChemBERTa-77M-MTR and DeepChem/ChemBERTa-77M-MLM (Note: Model cards are currently empty).
• Code Reference: While the DeepChem repository is referenced for code, isolated training scripts tailored to recreate ChemBERTa-2’s exact pipeline are not separated from the generalized deepchem library tooling.

Evaluation

Benchmarks were performed on MoleculeNet using DeepChem.

Hardware

• Compute: AWS EC2 instances with Nvidia T4 GPUs
• Strategy: AWS Spot instances were used to reduce cost; implemented frequent checkpointing to handle interruptions.
• Note: For MTR, they wrote a custom data loader wrapper around HuggingFace’s text loader to handle CSV parsing efficiency, as the default CSV loader was a major bottleneck for the 200-element target vectors.

Paper Information

Citation: Ahmad, W., Simon, E., Chithrananda, S., Grand, G., & Ramsundar, B. (2022). ChemBERTa-2: Towards Chemical Foundation Models. arXiv preprint arXiv:2209.01712. https://doi.org/10.48550/arXiv.2209.01712

Publication: arXiv 2022 (Presented at 2021 ELLIS ML for Molecule Discovery Workshop)

Additional Resources:

• ChemBERTa-1 Paper

@misc{ahmadChemBERTa2ChemicalFoundation2022,
  title = {{{ChemBERTa-2}}: {{Towards Chemical Foundation Models}}},
  shorttitle = {{{ChemBERTa-2}}},
  author = {Ahmad, Walid and Simon, Elana and Chithrananda, Seyone and Grand, Gabriel and Ramsundar, Bharath},
  year = 2022,
  month = sep,
  number = {arXiv:2209.01712},
  eprint = {2209.01712},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2209.01712},
  urldate = {2025-12-25},
  archiveprefix = {arXiv}
}

This is a Methodological ($\Psi_{\text{Method}}$) paper. It proposes an architecture adaptation (Chemformer based on BART) and a specific pre-training strategy (“Combined” masking and augmentation). The paper validates this method by benchmarking against established models on multiple tasks, including direct synthesis, retrosynthesis, and molecular optimization. It also includes a secondary Resource ($\Psi_{\text{Resource}}$) contribution by making the pre-trained models and code available.

Motivation: Computational Bottlenecks in Cheminformatics

Existing Transformer models for cheminformatics are often developed for single applications and are computationally expensive to train from scratch. For example, training a Molecular Transformer for reaction prediction can take days, limiting hyperparameter exploration. Self-supervised pre-training (like BERT or T5) has significantly advanced NLP by reducing fine-tuning time and improving performance. In chemistry, applications have traditionally focused on task-specific datasets or encoder-only architectures, which perform poorly on sequence generation tasks. The authors aim to use transfer learning on a large unlabelled dataset to create a model that converges quickly and performs well across diverse sequence-to-sequence and discriminative tasks.

Core Innovation: BART Architecture and Combined Pre-training

The primary insight lies in the adaptation of the BART architecture for chemistry and the introduction of a “Combined” self-supervised pre-training task.

• Architecture: Chemformer uses the BART encoder-decoder structure, allowing it to handle both discriminative (property prediction) and generative (reaction prediction) tasks efficiently. This provides an alternative to encoder-only (BERT) or decoder-only (GPT) models.
• Combined Pre-training: The authors introduce a task that applies both Span Masking (randomly replacing tokens with <mask>) and SMILES Augmentation (permuting atom order) simultaneously. Formally, given a canonical SMILES sequence $x$, a corrupted sequence $\tilde{x} = \text{Mask}(\text{Augment}(x))$ is generated. The model is trained using an autoregressive cross-entropy loss to reconstruct the canonical sequence from the corrupted input: $$ \mathcal{L}_{\text{pre-train}} = -\sum_{t=1}^{|x|} \log P(x_t \mid x_{<t}, \tilde{x}) $$
• Tunable Augmentation: A downstream augmentation strategy is proposed where the probability of augmenting the input/output SMILES ($p_{aug}$) is a tunable hyperparameter, performed on-the-fly.

Experimental Setup and Pre-training Tasks

The authors pre-trained Chemformer on 100 million molecules from ZINC-15 and fine-tuned it on three distinct task types:

1. Seq2Seq Reaction Prediction:
   • Direct Synthesis: USPTO-MIT dataset (Mixed and Separated).
   • Retrosynthesis: USPTO-50K dataset.
2. Molecular Optimization: Generating molecules with improved properties (LogD, solubility, clearance) starting from ChEMBL matched molecular pairs.
3. Discriminative Tasks:
   • QSAR: Predicting properties (ESOL, FreeSolv, Lipophilicity) from MoleculeNet.
   • Bioactivity: Predicting pXC50 values for 133 genes using ExCAPE data.

Ablation studies compared three pre-training strategies (Masking, Augmentation, Combined) against a randomly initialized baseline.

Results, Trade-offs, and Conclusions

• Performance: Chemformer achieved competitive top-1 accuracy on USPTO-MIT (91.3% Mixed) and USPTO-50K (53.6-54.3%), outperforming the Augmented Transformer and graph-based models (GLN, GraphRetro).
• Convergence Speed: Pre-training significantly accelerated training; fine-tuning for just 20 epochs (30 mins) outperformed the previous baselines trained for significantly longer.
• Pre-training Tasks: The “Combined” task generally performed best for reaction prediction and bioactivity, while “Masking” was superior for molecular optimization.
• Augmentation Trade-off: The augmentation strategy improved top-1 accuracy but significantly degraded top-5/10 accuracy because beam search outputs became populated with augmented versions of the same molecule. This presents a considerable limitation for practical applications like retrosynthesis mapping, where retrieving a diverse set of candidate reactions is often critical.
• Discriminative Evaluation Caveats: Chemformer underperformed specialized baselines (like D-MPNN or MolBERT) on small discriminative datasets. The authors note that direct comparison is difficult: Chemformer was trained simultaneously on multiple subtasks (multi-task learning), while the literature baselines were trained and tuned on each subtask separately. Additionally, the Chemformer encoder uses fewer than 20M parameters compared to MolBERT’s approximately 85M, and Chemformer’s pre-training does not include molecular property objectives.
• Pre-training Data Scope: The 100M pre-training dataset from ZINC-15 was selected with constraints on molecular weight ($\le 500$ Da) and LogP ($\le 5$), focusing the learned representations on small, drug-like molecules.

Reproducibility Details

Data

Note: The primary GitHub repository for Chemformer was officially archived on February 11, 2026. Pre-trained weights and datasets used in the paper are still hosted externally on Box. Active development of Chemformer models has moved to the AiZynthModels repository.

The following datasets were used for pre-training and benchmarking.

Preprocessing:

• Tokenization: Regex-based tokenization (523 tokens total) derived from ChEMBL 27 canonical SMILES.
• Augmentation: SMILES enumeration (permuting atom order) used for pre-training and on-the-fly during fine-tuning ($p_{aug}=0.5$ for Seq2Seq, $p_{aug}=1.0$ for discriminative).

Algorithms

• Pre-training Tasks:
  1. Masking: Span masking (BART style).
  2. Augmentation: Input is a randomized SMILES; target is canonical SMILES.
  3. Combined: Input is augmented then masked; target is canonical SMILES.
• Optimization:
  • Optimizer: Adam ($\beta_1=0.9, \beta_2=0.999$).
  • Schedule: Linear warm-up (8000 steps) for pre-training; One-cycle schedule for fine-tuning.
• Inference: Beam search with width 10 for Seq2Seq tasks. Used molbart/inference_score.py and molbart/retrosynthesis/round_trip_inference.py for standard and round-trip validation.

Models

Two model sizes were trained. Both use the Pre-Norm Transformer layout with GELU activation.

Evaluation

Comparisons relied on Top-N accuracy for reaction tasks and validity metrics for optimization.

Hardware

• Compute: 4 NVIDIA V100 GPUs (batch size 128 per GPU).
• Training Time:
  • Pre-training: 2.5 days (Base) / 6 days (Large) for 1M steps.
  • Fine-tuning: ~20-40 epochs for reaction prediction (<12 hours).

Paper Information

Citation: Irwin, R., Dimitriadis, S., He, J., & Bjerrum, E. J. (2022). Chemformer: a pre-trained transformer for computational chemistry. Machine Learning: Science and Technology, 3(1), 015022. https://doi.org/10.1088/2632-2153/ac3ffb

Publication: Machine Learning: Science and Technology 2022

@article{irwinChemformerPretrainedTransformer2022,
  title = {Chemformer: A Pre-Trained Transformer for Computational Chemistry},
  shorttitle = {Chemformer},
  author = {Irwin, Ross and Dimitriadis, Spyridon and He, Jiazhen and Bjerrum, Esben Jannik},
  year = 2022,
  month = jan,
  journal = {Machine Learning: Science and Technology},
  volume = {3},
  number = {1},
  pages = {015022},
  publisher = {IOP Publishing},
  issn = {2632-2153},
  doi = {10.1088/2632-2153/ac3ffb}
}

This is primarily a Method paper. It adapts a specific architecture (Transformer) to a specific task (InChI-to-IUPAC translation) and evaluates its performance against both machine learning and commercial baselines. It also has a secondary Resource contribution, as the trained model and scripts are released as open-source software.

Motivation: The Bottleneck in Algorithmic IUPAC Nomenclature

Generating correct IUPAC names is difficult due to the comprehensive but complex rules defined by the International Union of Pure and Applied Chemistry. Commercial software generates names from structures but remains closed-source with opaque methodologies and frequent inter-package disagreements. Open identifiers like InChI and SMILES lack direct human readability. This creates a need for an open, automated method to generate informative IUPAC names from standard identifiers like InChI, which are ubiquitous in online chemical databases.

Novelty: Treating Chemical Translation as a Character-Level Sequence

The key novelty is treating chemical nomenclature translation as a character-level sequence-to-sequence problem using a Transformer architecture, specifically using InChI as the source language.

• Standard Neural Machine Translation (NMT) uses sub-word tokenization. This model processes InChI and predicts IUPAC names character-by-character.
• It demonstrates that character-level tokenization outperforms byte-pair encoding or unigram models for this specific chemical task.
• It uses InChI’s standardization to avoid the canonicalization issues inherent in SMILES-based approaches.
• The attention mechanism allows the decoder to align specific parts of the generated IUPAC name with corresponding structural features in the source InChI string, operating via the standard scaled dot-product attention: $$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

Methodology & Experimental Validation

• Training: The model was trained on 10 million InChI/IUPAC pairs sampled from PubChem using a character-level objective. The model is supervised using categorical cross-entropy loss across the vocabulary of characters: $$ \mathcal{L} = -\sum_{i=1}^{N} y_i \log(\hat{y}_i) $$
• Ablation Studies: The authors experimentally validated architecture choices, finding that LSTM models and sub-word tokenization (BPE) performed worse than the Transformer with character tokenization. They also optimized dropout rates.
• Performance Benchmarking: The model was evaluated on a held-out test set of 200,000 samples. Performance was quantified primarily by Whole-Name Accuracy and Normalized Edit Distance (based on the Damerau-Levenshtein distance, scaled by the maximum string length).
• Commercial Comparison: The authors compared their model against four major commercial packages (ACD/I-Labs, ChemAxon, Mestrelab, and PubChem’s Lexichem). However, this evaluation used a highly limited test set of only 100 molecules, restricting the statistical confidence of the external baseline.
• Error Analysis: They analyzed performance across different chemical classes (organics, charged species, macrocycles, inorganics) and visualized attention coefficients to interpret model focus.

Key Results and the Inorganic Challenge

• High Accuracy on Organics: The model achieved 91% whole-name accuracy on the test set, performing particularly well on organic compounds.
• Comparable to Commercial Tools: On the limited 100-molecule benchmark, the edit distance between the model’s predictions and commercial packages (15-23%) was similar to the variation found between the commercial packages themselves (16-21%).
• Limitations on Inorganics: The model performed poorly on inorganic (14% accuracy) and organometallic compounds (20% accuracy). This is attributed to inherent data limitations in the standard InChI format (which deliberately disconnects metal atoms from their ligands) and low training data coverage for those classes.
• Character-Level Superiority: Character-level tokenization was found to be essential; byte-pair encoding reduced accuracy significantly.

Reproducibility Details

Data

The dataset was derived from PubChem’s public FTP server (CID-SMILES.gz and CID-IUPAC.gz).

Algorithms

• Framework: OpenNMT-py 2.0.0 (using PyTorch). Training scripts and vocabularies are available as supplementary files to the original publication. Pre-trained model weights are hosted on Zenodo.
• Architecture Type: Transformer Encoder-Decoder.
• Optimization: ADAM optimizer ($\beta_1=0.9, \beta_2=0.998$).
• Learning Rate: Linear warmup over 8000 steps to 0.0005, then decayed by inverse square root of iteration.
• Regularization:
  • Dropout: 0.1 (applied to dense and attentional layers).
  • Label Smoothing: Magnitude 0.1.
• Training Strategy: Teacher forcing used for both training and validation.
• Gradient Accumulation: Gradients accumulated over 4 batches before updating parameters.
• Inference: Beam search with width 10 and length penalty 1.0.

Models

• Structure: 6 layers in encoder, 6 layers in decoder.
• Attention: 8 heads per attention sub-layer.
• Dimensions:
  • Feed-forward hidden state size: 2048.
  • Embedding vector length: 512.
• Initialization: Glorot’s method.
• Position: Positional encoding added to word vectors.

Evaluation

Metrics reported include Whole-Name Accuracy (percentage of exact matches) and Normalized Edit Distance (Damerau-Levenshtein, scale 0-1).

Hardware

• GPU: Tesla K80.
• Training Time: 7 days.
• Throughput: ~6000 tokens/sec (InChI) and ~3800 tokens/sec (IUPAC).
• Batch Size: 4096 tokens (approx. 30 compounds).

Artifacts

Paper Information

Citation: Handsel, J., Matthews, B., Knight, N. J., & Coles, S. J. (2021). Translating the InChI: Adapting Neural Machine Translation to Predict IUPAC Names from a Chemical Identifier. Journal of Cheminformatics, 13(1), 79. https://doi.org/10.1186/s13321-021-00535-x

Publication: Journal of Cheminformatics 2021

@article{handselTranslatingInChIAdapting2021a,
  title = {Translating the {{InChI}}: Adapting Neural Machine Translation to Predict {{IUPAC}} Names from a Chemical Identifier},
  shorttitle = {Translating the {{InChI}}},
  author = {Handsel, Jennifer and Matthews, Brian and Knight, Nicola J. and Coles, Simon J.},
  year = 2021,
  month = oct,
  journal = {Journal of Cheminformatics},
  volume = {13},
  number = {1},
  pages = {79},
  issn = {1758-2946},
  doi = {10.1186/s13321-021-00535-x},
  urldate = {2025-12-20},
  abstract = {We present a sequence-to-sequence machine learning model for predicting the IUPAC name of a chemical from its standard International Chemical Identifier (InChI). The model uses two stacks of transformers in an encoder-decoder architecture, a setup similar to the neural networks used in state-of-the-art machine translation. Unlike neural machine translation, which usually tokenizes input and output into words or sub-words, our model processes the InChI and predicts the IUPAC name character by character. The model was trained on a dataset of 10 million InChI/IUPAC name pairs freely downloaded from the National Library of Medicine's online PubChem service. Training took seven days on a Tesla K80 GPU, and the model achieved a test set accuracy of 91\%. The model performed particularly well on organics, with the exception of macrocycles, and was comparable to commercial IUPAC name generation software. The predictions were less accurate for inorganic and organometallic compounds. This can be explained by inherent limitations of standard InChI for representing inorganics, as well as low coverage in the training data.},
  langid = {english},
  keywords = {Attention,GPU,InChI,IUPAC,seq2seq,Transformer}
}

This is primarily a Method paper with significant elements of Position.

• Method: The authors propose a specific neural architecture (Transformer with custom tokenization) and a verification pipeline (round-trip check) to solve the SMILES $\leftrightarrow$ IUPAC translation task. They rigorously benchmark this against rule-based baselines (OPSIN).
• Position: The authors explicitly argue for a paradigm shift, suggesting that “heavy” neural architectures should replace complex, costly rule-based legacy systems even for “exact” algorithmic tasks.

The Cost of Rule-Based Chemical Naming

• Complexity of Naming: Generating IUPAC names manually is error-prone and requires deep algorithmic knowledge.
• Lack of Open Source Tools: While open-source tools exist for Name-to-Structure (e.g., OPSIN), there were no open-source tools for the inverse “Structure-to-Name” conversion at the time of writing.
• Cost of Development: Developing rule-based converters “from scratch” is prohibitively expensive and time-consuming compared to training a neural model on existing data.

Struct2IUPAC Core Innovation

• Struct2IUPAC: The first effective open-source neural model for converting SMILES to IUPAC names, treating chemical translation as a Neural Machine Translation (NMT) problem.
• Verification Loop: A novel inference pipeline that generates multiple candidates via beam search and validates them using a reverse converter (OPSIN) to ensure the generated name maps back to the original structure.
• Custom Tokenization: A manually curated rule-based tokenizer for IUPAC names that handles specific chemical suffixes, prefixes, and stereochemical markers.

Experimental Setup and Stress Testing

• Accuracy Benchmarking: The models were tested on a held-out subset of 100,000 molecules from PubChem. The authors measured accuracy across different beam sizes (1, 3, 5).
• Comparison to Rules: The neural IUPAC2Struct model was compared directly against the rule-based OPSIN tool.
• Stress Testing:
  • Sequence Length: Evaluated performance across varying token lengths, identifying a “sweet spot” (10-60 tokens) and failure modes for very short (e.g., methane) or long molecules.
  • Stereochemistry: Tested on “stereo-dense” compounds. The authors define a “stereo-density” index ($I$) as the ratio of stereocenters ($S$) to total tokens ($N$): $$I = \frac{S}{N}$$ They observed a performance drop for these dense molecules, though the model still handled many stereocenters robustly.
  • Tautomers: Verified the model’s ability to handle different tautomeric forms (e.g., Guanine and Uracil variants).
• Latency Analysis: Benchmarked inference speeds on CPU vs. GPU relative to output sequence length.

Benchmarks and Outcomes

• High Accuracy: The Struct2IUPAC model achieved 98.9% accuracy (Beam 5 with verification). The reverse model (IUPAC2Struct) achieved 99.1%, comparable to OPSIN’s 99.4%.
• Distribution Modeling vs. Intuition: The authors claim the model infers “chemical logic,” because it correctly generates multiple valid IUPAC names for single molecules where naming ambiguity exists (e.g., parent group selection). However, this more likely reflects the Transformer successfully modeling the high-frequency conditional probability distribution of synonymous names present in the PubChem training data, rather than learning intrinsic chemical rules.
• Production Readiness: Inference on GPU takes less than 0.5 seconds even for long names, making it viable for production use.
• Paradigm Shift: The authors conclude that neural networks are a viable, cost-effective alternative to developing rule-based algorithms for legacy notation conversion.

Reproducibility Details

Data

The study utilized the PubChem database.

• Cleaning: Molecules that could not be processed by RDKit were removed. Molecules containing tokens not in the tokenizer (e.g., aromatic selenium) were excluded.
• Availability: A subset of 100,000 test molecules is available on GitHub (data/test_100000.csv) and Zenodo. The full train/test splits are not explicitly provided.

Algorithms

• Tokenization:
  • SMILES: Character-based tokenization.
  • IUPAC: Custom rule-based tokenizer splitting suffixes (-one, -al), prefixes (-oxy, -di), and special symbols ((, ), R(S)).
• Verification Step:
  1. Generate $N$ names using Beam Search ($N=5$).
  2. Reverse translate the candidate name using OPSIN.
  3. Check if the OPSIN structure matches the original input SMILES.
  4. Display the first verified match; otherwise, report failure.

Models

• Architecture: Standard Transformer with 6 encoder layers and 6 decoder layers.
• Hyperparameters:
  • Attention Heads: 8
  • Attention Dimension ($d_{\text{model}}$): 512
  • Feed-Forward Dimension ($d_{\text{ff}}$): 2048
• Training Objective: The models were trained using standard autoregressive cross-entropy loss over the target token sequence $y$ given the input string $x$: $$\mathcal{L} = - \sum_{t=1}^{T} \log P(y_t \mid y_{<t}, x)$$
• Training: Two separate models were trained: Struct2IUPAC (SMILES $\to$ IUPAC) and IUPAC2Struct (IUPAC $\to$ SMILES).
• Availability: Code for model architecture is provided in the GitHub repository. Pre-trained weights for the IUPAC2Struct model are available, but the Struct2IUPAC model weights are not publicly released, meaning researchers would need to retrain that model on their own PubChem data to reproduce those results.

Evaluation

Evaluation was performed on a random subset of 100,000 molecules from the test set.

• Robustness: Accuracy drops significantly for augmented (non-canonical) SMILES (37.16%) and stereo-enriched compounds (66.52%).

Hardware

• Training Infrastructure: 4 $\times$ Tesla V100 GPUs and 36 CPUs.
• Training Time: Approximately 10 days under full load.
• Inference Speed: <0.5s per molecule on GPU; scale is linear with output token length.

Artifacts

Paper Information

Citation: Krasnov, L., Khokhlov, I., Fedorov, M. V., & Sosnin, S. (2021). Transformer-based artificial neural networks for the conversion between chemical notations. Scientific Reports, 11(1), 14798. https://doi.org/10.1038/s41598-021-94082-y

Publication: Scientific Reports 2021

@article{krasnovTransformerbasedArtificialNeural2021a,
  title = {Transformer-Based Artificial Neural Networks for the Conversion between Chemical Notations},
  author = {Krasnov, Lev and Khokhlov, Ivan and Fedorov, Maxim V. and Sosnin, Sergey},
  year = 2021,
  month = jul,
  journal = {Scientific Reports},
  volume = {11},
  number = {1},
  pages = {14798},
  publisher = {Nature Publishing Group},
  doi = {10.1038/s41598-021-94082-y}
}

Additional Resources:

• GitHub Repository
• Web Demo

This is primarily a Method paper, with a strong secondary contribution as a Resource paper.

• Method: It proposes a neural machine translation (NMT) architecture to approximate the complex, rule-based algorithm of IUPAC naming, treating it as a language translation task.
• Resource: It provides an open-source tool and trained models to the community, addressing a gap where such functionality was previously limited to proprietary software.

Motivation: Democratizing IUPAC Nomenclature

The International Union of Pure and Applied Chemistry (IUPAC) naming scheme is universally accepted but algorithmically complex. Generating these names correctly is challenging for humans, and automated generation is largely missing from major open-source toolkits like CDK, RDKit, or Open Babel. While reliable commercial tools exist (e.g., ChemAxon’s molconvert), there was a lack of open-source alternatives for the scientific community. STOUT aims to fill this gap using a data-driven approach.

Core Innovation: Sequence-to-Sequence Naming

• Language Translation Approach: The authors treat chemical representations (SMILES/SELFIES) and IUPAC names as two different languages, applying Neural Machine Translation (NMT) to translate between them.
• Use of SELFIES: The work establishes SELFIES (Self-Referencing Embedded Strings) as a robust choice over SMILES for deep learning tokenization in this specific task, capitalizing on its syntactic robustness.
• Hardware Acceleration: The paper benchmarks GPU versus TPU training and highlights the practical necessity of Tensor Processing Units (TPUs) for training large-scale chemical language models, reducing training time by an order of magnitude.

Methodology & Translation Validation

• Data Scale: The model was trained on datasets of 30 million and 60 million molecules derived from PubChem.
• Hardware Benchmarking: Training efficiency was compared between an nVidia Tesla V100 GPU and Google TPU v3-8/v3-32 units.
• Bidirectional Translation: The system was tested on two distinct tasks:
  1. Forward: SELFIES → IUPAC names
  2. Reverse: IUPAC names → SELFIES
• Validation: Performance was evaluated on a held-out test set of 2.2 million molecules.

Translation Accuracy & Hardware Scaling

• High Accuracy: The model achieved an average BLEU score of ~90% and a Tanimoto similarity index > 0.9 for both translation directions.
• Generalization: Even when predictions were textually mismatched (low BLEU score), the underlying chemical structures often remained highly similar (high Tanimoto similarity), suggesting the system captures fundamental chemical semantics rather than merely memorizing strings.
• Impact of Data Size: Expanding training from 30 million to 60 million molecules yielded consistent performance gains without saturating.
• Hardware Necessity: Training on TPUs proved up to 54 times faster than a standard GPU baseline (Tesla V100), making scaling highly computationally tractable.

Reproducibility

Data

The dataset was curated from PubChem (111 million molecules). Note that the specific 30M and 60M subsets are not directly linked in the publication repository, which means a user would have to reconstruct the filtering process.

Preprocessing & Filtering:

• Explicit hydrogens removed; converted to canonical SMILES.
• Filtering Rules: MW < 1500 Da, no counter ions, limited element set (C, H, O, N, P, S, F, Cl, Br, I, Se, B), no hydrogen isotopes, 3-40 bonds, no charged groups.
• Ground Truth Generation: ChemAxon’s molconvert (Marvin Suite 20.15) was used to generate target IUPAC names for training.
• Representation: All SMILES were converted to SELFIES for training.

Algorithms

• Tokenization:
  • SELFIES: Split iteratively by brackets [ and ].
  • IUPAC: Split via punctuation ((, ), {, }, [, ], -, ., ,) and a discrete set of sub-word chemical morphemes (e.g., methyl, benzene, fluoro).
  • Padding: SELFIES padded to 48 tokens; IUPAC padded to 78 tokens. “Start” and “End” sequence markers append each chain.
• Optimization: Adam optimizer instantiated with a learning rate of $0.0005$.
• Objective Function: Sparse categorical cross-entropy, assessing prediction probabilities for token $i$ over vocabulary $V$: $$ \mathcal{L} = -\sum_{i=1}^{V} y_i \log(\hat{y}_i) $$

Models

• Architecture: Encoder-Decoder sequence-to-sequence network with Bahdanau attention mechanism context weighting.
• Components:
  • Encoder/Decoder: Recurrent Neural Networks (RNN) constructed using Gated Recurrent Units (GRU).
  • Attention: Bahdanau (additive) soft attention, which calculates alignment scores to softly weight encoder hidden states natively: $$ e_{tj} = v_a^\top \tanh(W_a s_{t-1} + U_a h_j) $$
  • Embedding: Decoder output passes through a continuous embedding layer before concatenating with the attention context vector.
• Implementation: Python 3 backend using TensorFlow 2.3.0. Note: The linked GitHub repository currently defaults to the STOUT V2.0 transformer models, so researchers aiming to reproduce this specific V1 RNN paper should reference the older tag/commit history.

Evaluation

Metrics heavily emphasize both linguistic accuracy and cheminformatic structural correctness:

Hardware

Comparison of hardware efficiency for training large chemical language models:

Paper Information

Citation: Rajan, K., Zielesny, A., & Steinbeck, C. (2021). STOUT: SMILES to IUPAC names using neural machine translation. Journal of Cheminformatics, 13(1), 34. https://doi.org/10.1186/s13321-021-00512-4

Publication: Journal of Cheminformatics 2021

@article{rajanSTOUTSMILESIUPAC2021,
  title = {STOUT: SMILES to IUPAC Names Using Neural Machine Translation},
  shorttitle = {STOUT},
  author = {Rajan, Kohulan and Zielesny, Achim and Steinbeck, Christoph},
  year = {2021},
  month = apr,
  journal = {Journal of Cheminformatics},
  volume = {13},
  number = {1},
  pages = {34},
  issn = {1758-2946},
  doi = {10.1186/s13321-021-00512-4},
  urldate = {2025-09-22},
  abstract = {Chemical compounds can be identified through a graphical depiction, a suitable string representation, or a chemical name. A universally accepted naming scheme for chemistry was established by the International Union of Pure and Applied Chemistry (IUPAC) based on a set of rules. Due to the complexity of this ruleset a correct chemical name assignment remains challenging for human beings and there are only a few rule-based cheminformatics toolkits available that support this task in an automated manner. Here we present STOUT (SMILES-TO-IUPAC-name translator), a deep-learning neural machine translation approach to generate the IUPAC name for a given molecule from its SMILES string as well as the reverse translation, i.e. predicting the SMILES string from the IUPAC name. In both cases, the system is able to predict with an average BLEU score of about 90% and a Tanimoto similarity index of more than 0.9. Also incorrect predictions show a remarkable similarity between true and predicted compounds.},
  langid = {english},
  keywords = {Attention mechanism,Chemical language,Deep neural network,DeepSMILES,IUPAC names,Neural machine translation,Recurrent neural network,SELFIES,SMILES}
}

Additional Resources:

• GitHub Repository
• STOUT V2.0 Note
• Struct2IUPAC Note
• HandSEL Note (InChI to IUPAC)

Method (Primary) / Resource (Secondary)

This paper presents a Methodological contribution by developing and validating a Transformer-based neural machine translation model (STOUT V2) for bidirectional chemical nomenclature (SMILES $\leftrightarrow$ IUPAC). It systematically compares this new architecture against previous RNN-based baselines (STOUT V1) and performs ablation studies on tokenization strategies.

It also serves as a significant Resource contribution by generating a massive training dataset of nearly 1 billion SMILES-IUPAC pairs (curated via commercial Lexichem software) and releasing the resulting models and code as open-source tools for chemical naming.

The Need for Robust Open-Source IUPAC Nomenclature Rules

Assigning systematic IUPAC names to chemical structures requires adherence to complex rules, challenging human consistency. Deterministic, rule-based software options like OpenEye Lexichem and ChemAxon are reliable commercial solutions. Existing open-source tools like OPSIN focus on parsing names to structures.

The previous version of STOUT (V1), based on RNNs/GRUs, achieved ~90% BLEU accuracy, with known limitations in capturing long-distance dependencies required for stereochemistry handling. This work uses the sequence-learning capabilities of Transformers combined with large-scale datasets to create a competitive open-source IUPAC naming tool.

Architectural Shift and Billion-Scale Training

The primary advancements over previous iterations address both architecture and dataset scale:

1. Architecture Shift: Moving from an RNN-based Seq2Seq model to a Transformer-based architecture (4 layers, 8 heads), which captures intricate chemical patterns better than GRUs.
2. Billion-Scale Training: Training on a dataset of nearly 1 billion molecules (combining PubChem and ZINC15), significantly larger than the 60 million used for STOUT V1.
3. Tokenization Strategy: Determining that character-wise tokenization for IUPAC names is superior to word-wise tokenization in terms of both accuracy and training efficiency (15% faster).

Experimental Validation and Scaling Limits

The authors conducted three primary experiments to validate bidirectional translation (SMILES $\rightarrow$ IUPAC and IUPAC $\rightarrow$ SMILES):

• Experiment 1 (Optimization): Assessed the impact of dataset size (1M vs 10M vs 50M) and tokenization strategy on SMILES-to-IUPAC performance.
• Experiment 2 (Scaling): Trained models on 110 million PubChem molecules for both forward and reverse translation tasks to test performance on longer sequences.
• Experiment 3 (Generalization): Trained on the full ~1 billion dataset (PubChem + ZINC15) for both translation directions.
• External Validation: Benchmarked against an external dataset from ChEBI (1,485 molecules) and ChEMBL34 to test generalization to unseen data.

Evaluation Metrics:

• Textual Accuracy: BLEU scores (1-4) and Exact String Match.
• Chemical Validity: Retranslation of generated names back to SMILES using OPSIN, followed by Tanimoto similarity checks (PubChem fingerprints) against the original input.

Translation Accuracy and Structural Validity

• Superior Performance: STOUT V2 achieved an average BLEU score of 0.99 (vs 0.94 for V1). While exact string matches varied by experiment (83-89%), the model notably achieved a perfect BLEU score (1.0) on 97.49% of a specific test set where STOUT V1 only reached 66.65%.
• Structural Validity (“Near Misses”): When the generated name differed from the ground truth string, the re-generated structure often remained chemically valid. The model maintained an average Tanimoto similarity $T(A,B)$ of 0.68 for these divergent names between bit-vector fingerprints $A$ and $B$, roughly defined as: $$ T(A,B) = \frac{\sum (A \cap B)}{\sum (A \cup B)} $$ Critique: Note that an average Tanimoto coefficient of 0.68 typically suggests moderate structural similarity/drift, not an almost-identical “near miss” (which would be $>0.85$). This implies the model constructs chemically related but structurally distinct outputs when it fails exact string matching.
• Tokenization: Character-level splitting for IUPAC names outperformed word-level splitting and was more computationally efficient.
• Data Imbalance & Generalization: The model’s drop in performance for sequences >600 characters highlights a systemic issue in open chemical databases: long, highly complex SMILES strings are significantly underrepresented. Even billion-scale training datasets are still bound by the chemical diversity of their source material.
• Limitations:
  • Preferred Names (PINs): The model mimics Lexichem’s naming conventions, generating valid IUPAC names distinct from strict Preferred IUPAC Names (PINs).
  • Sequence Length: Performance degrades for very long SMILES (>600 characters) due to scarcity in the training data.
  • Algorithmic Distillation Bottleneck: Because the 1 billion training pairs were generated entirely by OpenEye’s Lexichem, STOUT V2 acts as a knowledge distillation of that specific commercial algorithm. The model learns Lexichem’s heuristic mapping, specific dialects, and potential systematic errors, rather than deriving true nomenclature rules from first principles.

Reproducibility Details

Data

The training data was derived from PubChem and ZINC15. Ground truth IUPAC names were generated using OpenEye Lexichem TK 2.8.1 to ensure consistency.

Preprocessing:

• SMILES: Canonicalized, isomeric, and kekulized using RDKit (v2023.03.1).
• Formatting: Converted to TFRecord format in 100 MB chunks for TPU efficiency.

Algorithms

• SMILES Tokenization: Regex-based splitting. Atoms (e.g., “Cl”, “Au”), bonds, brackets, and digits are separate tokens.
• IUPAC Tokenization: Character-wise split was selected as the optimal strategy (treating every character as a token).
• Optimization: Adam optimizer with a custom learning rate scheduler based on model dimensions.
• Loss Function: Trained to minimize the Sparse Categorical Cross-Entropy $L$, masking padding tokens. For a correctly predicted target class $t$ alongside probabilities $p_i$, the masked loss is represented mathematically as: $$ L = - \sum_{i=1}^{m} m_i y_{i} \log(p_{i}) $$ where $m_i$ masks padded positions.
• Code Availability: The main STOUT V2 repository contains the inference package. The training pipeline/instructions (originally linked to a separate repo that is currently a 404) can still be found within the Zenodo archive release.

Models

The model follows the standard Transformer architecture from “Attention is All You Need” (Vaswani et al.).

• Architecture: 4 Transformer layers (encoder/decoder stack).
• Attention: Multi-head attention with 8 heads.
• Dimensions: Embedding size ($d_{model}$) = 512; Feed-forward dimension ($d_{ff}$) = 2048.
• Regularization: Dropout rate of 0.1.
• Context Window: Max input length (SMILES) = 600; Max output length (IUPAC) = 700-1000.
• Weights: Model weights for forward and reverse architectures are available via Zenodo (v3).

Evaluation

Evaluation focused on both string similarity and chemical structural integrity.

Artifacts

Hardware

Training was conducted entirely on Google Cloud Platform (GCP) TPUs.

• STOUT V1: Trained on TPU v3-8.
• STOUT V2: Trained on TPU v4-128 pod slices (128 nodes).
• Large Scale (Exp 3): Trained on TPU v4-256 pod slice (256 nodes).
• Training Time: Average of 15 hours and 2 minutes per epoch for the 1 billion dataset.
• Framework: TensorFlow 2.15.0-pjrt with Keras.

Paper Information

Citation: Rajan, K., Zielesny, A., & Steinbeck, C. (2024). STOUT V2.0: SMILES to IUPAC name conversion using transformer models. Journal of Cheminformatics, 16(146). https://doi.org/10.1186/s13321-024-00941-x

Publication: Journal of Cheminformatics 2024

@article{rajanSTOUTV20SMILES2024,
  title = {{{STOUT V2}}.0: {{SMILES}} to {{IUPAC}} Name Conversion Using Transformer Models},
  shorttitle = {{{STOUT V2}}.0},
  author = {Rajan, Kohulan and Zielesny, Achim and Steinbeck, Christoph},
  year = 2024,
  month = dec,
  journal = {Journal of Cheminformatics},
  volume = {16},
  number = {1},
  pages = {146},
  issn = {1758-2946},
  doi = {10.1186/s13321-024-00941-x}
}

Additional Resources:

• Web Application (Includes Ketcher drawing, bulk submission, and DECIMER integration)
• DECIMER Project
• STOUT V1 Note
• Zenodo Archive (Code Snapshot)

This paper represents a $\Psi_{\text{Method}}$ projection that proposes a novel architectural pipeline for indexing and searching chemical literature. The framework unifies text, molecular diagrams, and structured reaction records. It also contains a secondary $\Psi_{\text{Resource}}$ projection, providing a functional demonstration tool and curating a specific benchmark dataset for Suzuki coupling reactions.

The Gap in Passage-Level Chemical Retrieval

Scientific literature documents chemical reactions through a combination of text and visual diagrams. Textual descriptions detail parameters like yield and operational temperature, whereas diagrams graphically model these structural transformations. Existing tools such as SciFinder or Reaxys perform document-level or individual compound retrieval. They fail to explicitly link molecular figures to localized textual descriptions. This structure prevents researchers from directly extracting a corresponding reaction diagram alongside the exact textual protocol. Researchers require passage-level retrieval of synthesis protocols to efficiently access complete reaction conditions.

Core Innovation: Unified Multimodal Indexing

The core methodological innovation is a multimodal passage-level indexing and linking pipeline.

• Unified Indexing: The framework processes text and diagrams in parallel and directly links them into a single index structure. This architecture supports search queries utilizing raw text, discrete SMILES strings, or multimodal combinations.
• Compound-Passage Linking: The mechanism applies conflict-resolution logic linking chemical diagrams to specific text citations using two parallel heuristics:
  1. Token-based Alignment: Matching parsed diagram labels against documented text strings (e.g., “compound 5”) using normalized Levenshtein distance.
  2. Fingerprint-based Alignment: Matching chemical structures against generated SMILES strings via structural Tanimoto Similarity.
• ReactionMiner Integration: The pipeline parses and incorporates formatted reaction records (reactants, products, catalysts, quantitative yields) directly derived from segmented text passages.

Methodology & Expert Evaluation

The authors evaluated the system utilizing a chemical case study targeting specific synthesis domains alongside qualitative expert assessment.

• Dataset: Evaluators processed a corpus of 7 research manuscripts and 6 supplementary data documents detailing Suzuki coupling reactions.
• Volume: The resulting index processed 1,282 extracted passages (indexing 538), extracted 383 unique SMILES, and logged 219 parsed reactions.
• Qualitative Evaluation: Practicing structural chemists developed real-world queries (such as cross-referencing the conceptual “Burke group” alongside an explicit structural SMARTS pattern) to gauge retrieval capability.

Key Findings & System Limitations

• Diagram-to-Text Linking: The pipeline accurately paired visual molecular diagrams with structurally derived text details, permitting testers to navigate directly from a molecule query card to the exact origin passage within the source PDF.
• Contextual Insight Extraction: Specialized chemists found the parsed reaction representations (yield metrics, isolated catalysts) functionally pragmatic as high-level extractive summaries.
• Extrapolative Retrieval: The architecture permitted the effective retrieval of targeted chemical derivatives (such as benzo[b]thiophen-2-ylboronic acid) via structurally related input queries (dibenzothiophene).

The system evaluation highlights several architectural restrictions:

• Domain-Restricted Validation: The initial validation is entirely qualitative and bounded to the specific subclass of Suzuki coupling reactions. The evaluation omits standardized quantitative retrieval baselines (e.g., MAP, NDCG) and lacks systematic ablation data for the fusion scoring mechanism.
• Algorithmic Transparency: The multimodal query routing mechanism does not clearly indicate the dominant retrieval feature. This hides whether keyword text or structural similarity actually drove the final result placement. This ambiguity limits operator control.
• Optical Processing Brittleness: The embedded vision inference and primitive parsing pipelines display inherent fragility, producing intermittent failures when associating text passages with correctly parsed molecular diagrams.
• Metadata Logging Incompleteness: Practicing chemists requested additional structured metadata targets (such as specific molar equivalents and parameterized mol% values) to successfully bridge the extracted data stream directly into digital electronic lab notebooks.

Reproducibility

Artifacts

Data

• Source: The corpus features 7 primary research papers and 6 auxiliary supplementary information documents focusing on Suzuki coupling reactions, sourced from practicing chemists at UIUC. This evaluation dataset is strictly internal and not publicly available.
• Preprocessing:
  • Engineers convert source PDFs to full-page raster images.
  • The system extracts localized graphical layout and raw text via PyTesseract.
  • The pipeline segments valid passage chunks emphasizing reaction-related sentences utilizing product-indicative lexicons and topic modeling.

Algorithms

• Diagram Extraction: A YOLOv8 model identifies and segments molecular regions within structured PDF pages.
• Diagram Parsing: The architecture relies on ChemScraper to infer structural semantics from raw diagrams:
  • Born-digital PDFs: SymbolScraper extracts vector lines and polygons directly from bounding box definitions.
  • Raster images: The system employs the Line Segment Detector (LSD) and watershed bounding algorithms to isolate native geometric primitives.
• Text Entity Extraction: The framework deploys ChemDataExtractor 2.0 to extract explicit molecular aliases. A translation layer maps these entities to string representations via OPSIN.
• Linking Logic (Fusion Score):
  • Text Link: The algorithm calculates a normalized Levenshtein ratio connecting visual diagram labels against proximal text mentions based on calculated edit distance.
  • Structure Link: The algorithm computes the discrete Tanimoto Similarity between generated 2048-bit Morgan fingerprints extracted from localized visual diagram features and baseline text SMILES queries: $$ T(A, B) = \frac{A \cdot B}{|A|^{2} + |B|^{2} - A \cdot B} $$ where $A$ and $B$ represent the boolean bit vectors of the respective fingerprint pairs.
  • Conflict Resolution Protocol: The system fuses structural geometry bounds and discrete textual tokenization metrics, prioritizing the ranking sequence that yields a higher terminal similarity score. During final retrieval, the candidate subset is systematically re-ranked leveraging the hybrid calculation of the BM25 explicit metric and the localized count of exact SMILES pattern hits.

Models

• Reaction Extraction Parameters: The engineers configure a LLaMA-3.1-8b model fine-tuned entirely via LoRA targeting custom tokens representing reaction entities (compounds, reagents, thermal inputs) directly pulled from text sub-chunks. Exact prompt constraints, the fine-tuning dataset, and specific LoRA hyperparameters are omitted from the source text.
• Diagram Processing Bounds: The codebase incorporates a segmentation-aware multi-task neural network topology built into ChemScraper to execute low-level raster image parsing tasks.

Evaluation

• Search Engine Base: The authors implemented their indexing framework scaling atop PyTerrier.
• Text Feature Ranking: The metric utilizes standalone BM25 bounds mapping keyword-similarity.
• Structure Feature Operations: The topology operates RDKit bindings powering substructure coordinate mapping logic and exact molecular similarity searches.
• Multimodal Fusion Processing:
  • The algorithm filters out terminal candidates mapping initial structural properties (SMILES queries) against the document-wide lexical properties (BM25 scores).
  • The final fusion routing assigns the strongest positive weight to retrieved passages that accumulate dense local clusters of structurally exact verified SMILES patterns.

Hardware

• Compute Infrastructure: The hardware and parameter requirements to host the multi-stage vision extractors (YOLOv8, ChemScraper) alongside a local 8B LLM are entirely unspecified in the paper.

Paper Information

Citation: Shah, A. K., et al. (2025). Multimodal Search in Chemical Documents and Reactions. arXiv preprint arXiv:2502.16865. https://doi.org/10.48550/arXiv.2502.16865

Publication: SIGIR ‘25 (Demo Track), 2025

@misc{shahMultimodalSearchChemical2025,
  title = {Multimodal {{Search}} in {{Chemical Documents}} and {{Reactions}}},
  author = {Shah, Ayush Kumar and Dey, Abhisek and Luo, Leo and Amador, Bryan and Philippy, Patrick and Zhong, Ming and Ouyang, Siru and Friday, David Mark and Bianchi, David and Jackson, Nick and Zanibbi, Richard and Han, Jiawei},
  year = 2025,
  month = feb,
  number = {arXiv:2502.16865},
  eprint = {2502.16865},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2502.16865},
  archiveprefix = {arXiv}
}

Additional Resources:

• Online Demo (Note: While the landing page advertises the system as open-source, the exact repository URL and installation prerequisites are omitted from the official manuscript.)

This is primarily a Method paper with a significant Resource contribution.

Method: The paper proposes a novel “Cross-modal Dialogue Foundation Model” architecture that aligns five distinct chemical modalities (2D graphs, 3D conformations, images, MS2 spectra, IR spectra) to a single LLM decoder using separate encoders and projection modules. It establishes strong baseline performance across multiple modalities compared against current generalist models.

Resource: The paper addresses the scarcity of multimodal chemical data by constructing a 7.6M instruction-tuning dataset. This dataset is largely synthesized from seed SMILES strings using approximate calculations (MMFF94, CFM-ID, Chemprop-IR) and specialist model predictions.

Bridging Experimental Data and LLMs

Existing chemical AI models generally fall into two distinct categories. Task-specific specialist models achieve high accuracy on singular objectives, such as property prediction or molecular generation, but require strict formatting and lack conversational flexibility. Conversely, early chemical large language models provide natural language interaction but are restricted to text and SMILES strings. ChemDFM-X addresses this gap by enabling large multimodal models to process the experimental characterization data (MS2 and IR spectra) and visual data routinely used in practical chemistry workflows.

Synthetic Data Scaling for Modality Alignment

The core novelty lies in the “Any-to-Text” alignment strategy via synthetic data scaling:

1. Comprehensive Modality Support: ChemDFM-X incorporates experimental characterization data (MS2 and IR spectra) alongside 2D graphs, 3D conformations, and images. The data representations are formally defined mathematically rather than as raw pixels:

   • Molecular Graph: An undirected graph $G = (\textbf{V}, \textbf{E})$ with atom set $\textbf{V}$ and bond set $\textbf{E}$.
   • Molecular Conformation: An undirected graph $G = (\textbf{V}’, \textbf{E})$ storing spatial coordinates: $\textbf{v}_i = (x_i, y_i, z_i, a_i)$.
   • MS2 Spectrum: Treated as a point sequence of discrete mass-to-charge ratios and intensities, tokenized via a discrete codebook: $\textbf{M} = ((r_1, I_1), (r_2, I_2), \dots, (r_n, I_n))$.
   • IR Spectrum: Treated as a dense sequence of continuous wave lengths and absorption intensities, directly reshaped for feature extraction: $\textbf{R} = ((w_1, t_1), (w_2, t_2), \dots, (w_l, t_l))$.

   The authors trained new Sequence Transformer encoders from scratch for the MS2 and IR modalities since suitable pre-trained models did not exist.

2. Synthetic Data Generation Pipeline: The authors generated a 7.6M sample dataset by starting with 1.3M seed SMILES and using “approximate calculations” to generate missing modalities:

   • 3D conformations via MMFF94 force field optimization
   • MS2 spectra via CFM-ID 4.0 (Competitive Fragmentation Modeling)
   • IR spectra via Chemprop-IR (Message Passing Neural Network)

3. Cross-Modal Synergy: The model demonstrates that training on reaction images improves recognition performance by leveraging semantic chemical knowledge (reaction rules) to correct visual recognition errors, an emergent capability from multimodal training.

Multimodal Benchmarking with ChemLLMBench

The model was evaluated using a customized version of ChemLLMBench and MoleculeNet across three modality categories:

1. Structural Modalities (2D Graphs & 3D Conformations):

   • Molecule recognition and captioning
   • Property prediction (MoleculeNet: BACE, BBBP, ClinTox, HIV, Tox21)
   • Compared against specialist models (Mole-BERT, Uni-Mol, MolXPT, MolCA) and generalist models (3D-MoLM, ChemDFM, ChemLLM)

2. Visual Modalities (Images):

   • Single molecule image recognition
   • Reaction image recognition
   • Compared against GPT-4O, Gemini 1.5 Pro, Qwen-VL, LLaVA, and specialist models MolNextr and MolScribe

3. Characterization Modalities (MS2 & IR Spectra):

   • Spectral analysis tasks (identifying molecules from spectra)
   • Contextualized spectral interpretation (combining spectra with reaction context)
   • Novel evaluation requiring integration of spectroscopic data with reaction knowledge

Cross-Modal Synergy and Generalist Performance

Key Findings:

1. Leading Generalist Performance: ChemDFM-X establishes a new benchmark among existing generalist models (such as 3D-MOLM and ChemLLM), achieving performance metrics that match dedicated specialist models across several multimodal tasks.

2. Failure of General LMMs: General vision models (GPT-4O, Gemini 1.5 Pro, Qwen-VL, LLaVA, InternLM-XComposer2, DocOwl) failed significantly on chemical image recognition tasks (0% accuracy for most models on molecule and reaction recognition, Table 9), demonstrating that chemical domain knowledge cannot be assumed from general pre-training.

3. Cross-Modal Error Correction: In reaction image recognition, ChemDFM-X achieved higher accuracy (53.0%) than on single molecules (46.0%) (Table 9). The authors conclude the model uses its internal knowledge of chemical reaction rules to correct recognition errors in the visual modality, an emergent capability from multimodal training.

4. Reliance on Reaction Context for Spectra: In zero-shot scenarios, ChemDFM-X essentially fails at pure spectral recognition (achieving 0% and 1% top-1 accuracy on MS2 and IR spectra alone, Table 11). However, when SMILES-based reaction context is included, performance rises to 45% (MS2) and 64% (IR) on the reaction prediction task, and 29% (MS2) and 60% (IR) on retrosynthesis (Table 11). This indicates the model uses spectral data as a soft prior to constrain textual deductions. Furthermore, the paper compares ChemDFM-X’s spectral identification performance exclusively against text-only LLMs that cannot process spectra, omitting comparisons against established specialist tools.

5. Surrogate Distillation Trade-offs: Because the spectral training data relies entirely on outputs from CFM-ID 4.0 and Chemprop-IR, ChemDFM-X effectively distills these surrogate models. Any inherent predictive biases or inaccuracies from these underlying tools are permanently embedded in the new ChemDFM-X encoders.

Main Conclusion: The “separate encoders + unified decoder” architecture with synthetic data generation enables effective multimodal chemical understanding, bridging the gap between specialist and generalist AI systems for chemistry.

Reproducibility Details

Data

The authors constructed a 7.6M sample instruction-tuning dataset derived from 1.3M seed SMILES (sourced from PubChem and USPTO). Note: The final 7.6M multimodal tuning dataset itself isn’t publicly available.

Generation Pipeline:

Data Augmentation:

• Molecule images augmented with “handwritten” style using the ChemPix pipeline
• Multiple rendering styles (RDKit default, Indigo clean)
• Spectra generated at multiple energy levels (10eV, 20eV, 40eV for MS2)

Algorithms

Architecture: “Separate Encoders + Unified Decoder”

Code Availability: The authors have only released inference logic. The cross-modal projection training and synthetic data-generation scripts are closed.

Modality Alignment:

• Each modality has a dedicated encoder (frozen pre-trained models where available)
• For graph, conformation, MS2, and IR modalities: 2-layer MLP projector (Linear, GELU, Linear) maps encoder features to LLM input space
• For images: H-Reducer module compresses image tokens by factor of $n=8$ to handle high-resolution chemical images, then projects to LLM input space
• All projected features are concatenated and fed to the unified LLM decoder

Models

Base LLM:

• ChemDFM (13B): LLaMA-based model pre-trained on chemical text and SMILES

Modality Encoders:

Design Rationale: MS2 and IR encoders trained from scratch as Sequence Transformers treating spectral peaks as token sequences, since no suitable pre-trained models exist for chemical spectra.

Evaluation

Metrics:

• Accuracy (Acc) for recognition tasks
• BLEU-2/4 and METEOR for captioning tasks
• AUC-ROC for property prediction (classification)

Code Availability: The adapted code for evaluating on ChemLLMBench and their custom spectral recognition tasks is closed-source.

Benchmarks:

• ChemLLMBench: Adapted for multimodal inputs across molecule captioning, property prediction, and reaction understanding
• MoleculeNet: Standard molecular property prediction tasks (BACE, BBBP, ClinTox, HIV, Tox21)
• USPTO: Reaction prediction and retrosynthesis tasks
• Custom Spectral Tasks: Novel evaluations requiring spectral interpretation

Hardware

Note: The type and quantity of GPUs used, along with the total training wall-time, were not published.

Training Configuration:

• Total Batch Size: 256
• Epochs: 3
• Optimizer: AdamW

Modality-Specific Learning Rates (Peak):

Note: Different learning rates reflect the varying degrees of domain adaptation required. Images (general CLIP) need more adaptation than graphs (chemical Mole-BERT).

Artifacts

Paper Information

Citation: Zhao, Z., Chen, B., Li, J., Chen, L., Wen, L., Wang, P., Zhu, Z., Zhang, D., Wan, Z., Li, Y., Dai, Z., Chen, X., & Yu, K. (2024). ChemDFM-X: Towards Large Multimodal Model for Chemistry. Science China Information Sciences, 67(12), 220109. https://doi.org/10.1007/s11432-024-4243-0

Publication: Science China Information Sciences, December 2024

Additional Resources:

• arXiv Version
• Code Repository

@article{zhaoChemDFMXLargeMultimodal2024,
  title = {{{ChemDFM-X}}: {{Towards Large Multimodal Model}} for {{Chemistry}}},
  author = {Zhao, Zihan and Chen, Bo and Li, Jingpiao and Chen, Lu and Wen, Liyang and Wang, Pengyu and Zhu, Zichen and Zhang, Danyang and Wan, Ziping and Li, Yansi and Dai, Zhongyang and Chen, Xin and Yu, Kai},
  year = {2024},
  month = dec,
  journal = {Science China Information Sciences},
  volume = {67},
  number = {12},
  pages = {220109},
  doi = {10.1007/s11432-024-4243-0},
  archiveprefix = {arXiv},
  eprint = {2409.13194},
  primaryclass = {cs.LG}
}

Citation: Chang, Q., Chen, M., Pi, C., et al. (2024). RFL: Simplifying Chemical Structure Recognition with Ring-Free Language. arXiv preprint arXiv:2412.07594. https://doi.org/10.48550/arXiv.2412.07594

Publication: arXiv preprint (December 2024)

Additional Resources:

• Official Code Repository

Methodological Contribution

This is a Methodological paper ($\Psi_{\text{Method}}$). It introduces a novel representation system (Ring-Free Language) and a specialized neural architecture (Molecular Skeleton Decoder) designed to solve specific limitations in converting 2D images to 1D chemical strings. The paper validates this method through direct comparison with state-of-the-art baselines and ablation studies.

Motivation: Limitations of 1D Serialization

Current Optical Chemical Structure Recognition (OCSR) methods typically rely on “unstructured modeling,” where 2D molecular graphs are flattened into 1D strings like SMILES or SSML. While simple, these linear formats struggle to explicitly capture complex spatial relationships, particularly in molecules with multiple rings and branches. End-to-end models often fail to “understand” the graph structure when forced to predict these implicit 1D sequences, leading to error accumulation in complex scenarios.

Innovation: Ring-Free Language (RFL) and Molecular Skeleton Decoder (MSD)

The authors propose two primary contributions to decouple spatial complexity:

1. Ring-Free Language (RFL): A divide-and-conquer representation that splits a molecular graph $G$ into three explicit components: a molecular skeleton $\mathcal{S}$, individual ring structures $\mathcal{R}$, and branch information $\mathcal{F}$. This allows rings to be collapsed into “SuperAtoms” or “SuperBonds” during initial parsing.
2. Molecular Skeleton Decoder (MSD): A hierarchical architecture that progressively predicts the skeleton first, then the individual rings (using SuperAtom features as conditions), and finally classifies the branch connections.

Methodology and Experiments

The method was evaluated on both handwritten and printed chemical structures against two baselines: DenseWAP (Zhang et al. 2018) and RCGD (Hu et al. 2023).

• Datasets:
  • EDU-CHEMC: ~49k handwritten samples (challenging, diverse styles)
  • Mini-CASIA-CSDB: ~89k printed samples (from ChEMBL)
  • Synthetic Complexity Dataset: A custom split of ChEMBL data grouped by structural complexity (atoms + bonds + rings) to test generalization
• Ablation Studies: Tested the necessity of the MSD architecture vs. standard decoders and the impact of the [conn] token for filtering branch candidates

Outcomes and Conclusions

• SOTA Performance: The proposed method (MSD-RCGD) outperformed the previous state-of-the-art (RCGD) on both handwritten (95.38% Exact Match) and printed (95.58% Exact Match) datasets.
• Universal Improvement: Applying MSD/RFL to an older baseline (DenseWAP) improved its accuracy significantly (e.g., from 87.6% to 81.9% on complex sets), proving the method is model-agnostic.
• Complexity Handling: The method showed robust generalization on unseen high-complexity molecules, where standard 1D baselines failed completely (0% accuracy for standard DenseWAP vs. ~30% for MSD version on the hardest tier).

Reproducibility Details

Data

The authors utilized one handwritten and one printed dataset, plus a synthetic set for stress-testing complexity.

Algorithms

RFL Splitting (Encoding):

1. Detect Rings: Use DFS to find all non-nested rings $\mathcal{R}$.
2. Determine Adjacency ($\gamma$): Calculate shared edges between rings.
3. Merge:
   • If $\gamma(r_i) = 0$ (isolated), merge ring into a SuperAtom node.
   • If $\gamma(r_i) > 0$ (adjacent), merge ring into a SuperBond edge.
4. Update: Record connection info in $\mathcal{F}$ and remove ring details from the main graph to form Skeleton $\mathcal{S}$.

MSD Decoding:

• Hierarchical Prediction: The model predicts the Skeleton $\mathcal{S}$ first.
• Contextual Ring Prediction: When a SuperAtom/Bond token is predicted, its hidden state $f^s$ is stored. After the skeleton is finished, $f^s$ is used as a condition to autoregressively decode the specific ring structure.
• Token [conn]: A special token separates connected ring bonds from unconnected ones to sparsify the branch classification task.

Models

The architecture follows a standard Image-to-Sequence pattern but with a forked decoder.

• Encoder: DenseNet (Growth rate=24, Depth=32 per block)
• Decoder (MSD):
  • Core: GRU with Attention (Hidden dim=256, Embedding dim=256, Dropout=0.15)
  • Skeleton Module: Autoregressively predicts sequence tokens. Uses Maxout activation.
  • Branch Module: A binary classifier (MLP) taking concatenated features of skeleton bonds $f_{bs}$ and ring bonds $f_{br}$ to predict connectivity matrix $\mathcal{F}$.
• Loss Function: $\mathcal{L} = \lambda_1 \mathcal{L}_{ce} + \lambda_2 \mathcal{L}_{cls}$ (where $\lambda=1$)

Evaluation

Metrics focus on exact image reconstruction and structural validity.

Hardware

• Compute: 4 x NVIDIA Tesla V100 (32GB RAM)
• Training Configuration:
  • Batch size: 8 (Handwritten), 32 (Printed)
  • Epochs: 50
  • Optimizer: Adam ($lr=2\times10^{-4}$, decayed by 0.5 via MultiStepLR)

Citation

@misc{changRFLSimplifyingChemical2025,
  title = {RFL: Simplifying Chemical Structure Recognition with Ring-Free Language},
  shorttitle = {RFL},
  author = {Chang, Qikai and Chen, Mingjun and Pi, Changpeng and Hu, Pengfei and Zhang, Zhenrong and Ma, Jiefeng and Du, Jun and Yin, Baocai and Hu, Jinshui},
  year = {2024},
  month = dec,
  number = {arXiv:2412.07594},
  eprint = {2412.07594},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2412.07594},
  archiveprefix = {arXiv}
}

Citation: Zhang, W., Wang, X., Feng, B., & Liu, W. (2025). MolSight: Optical Chemical Structure Recognition with SMILES Pretraining, Multi-Granularity Learning and Reinforcement Learning. arXiv preprint arXiv:2511.17300. https://doi.org/10.48550/arXiv.2511.17300

Publication: arXiv 2025

Additional Resources:

• Official Repository

Contribution: A Framework for Optical Chemical Structure Recognition

This is primarily a Method paper. It proposes a novel three-stage training framework (Pretraining → Fine-tuning → RL Post-training) to improve Optical Chemical Structure Recognition (OCSR). Specifically, it introduces the use of Group Relative Policy Optimization (GRPO) to solve non-differentiable chemical validity issues.

It also has a Resource component, as the authors construct and release Stereo-200k, a dataset specifically designed to train models on challenging stereoisomeric molecules.

Motivation: Resolving Stereochemical Cues

Existing OCSR systems struggle to accurately recognize stereochemical information (e.g., chirality, geometric isomerism) because the visual cues distinguishing stereoisomers (such as wedge and dash bonds) are subtle. Current methods often fail to capture the geometric relationships required to distinguish molecules with identical connectivity but different spatial arrangements. Accurate recognition is critical for downstream tasks like drug discovery where stereochemistry determines pharmacological effects.

Core Innovations: GRPO and Multi-Granularity Learning

MolSight introduces three key technical innovations:

1. Reinforcement Learning for OCSR: It is the first OCSR system to incorporate RL (specifically GRPO) to directly optimize for chemical semantic correctness.
2. Multi-Granularity Learning: It employs auxiliary heads for chemical bond classification and atom localization. Unlike previous approaches that optimize these jointly, MolSight decouples the coordinate head to prevent interference with SMILES generation.
3. SMILES-M Notation: A lightweight extension to SMILES to handle Markush structures (common in patents) without significant sequence length increase.

Experimental Methodology

The authors evaluated MolSight using a rigorous mix of real and synthetic benchmarks:

• Baselines: Compared against rule-based (OSRA, MolVec) and deep learning methods (MolScribe, MolGrapher, DECIMER).
• Benchmarks: Evaluated on real-world datasets (USPTO, Maybridge UoB, CLEF-2012, JPO) and synthetic datasets (Staker, ChemDraw, Indigo, Stereo-2K).
• Ablation Studies: Tested the impact of the bond head, coordinate head, and RL stages separately.
• Transfer Learning: Assessed the quality of learned representations by using the frozen encoder for molecular property prediction on MoleculeNet.

Results and Conclusions

• SOTA Performance: MolSight achieved 85.1% stereochemical accuracy on the USPTO dataset, significantly outperforming the previous SOTA (MolScribe) which achieved 69.0%.
• RL Effectiveness: Reinforcement learning post-training specifically improved performance on stereoisomers, raising Tanimoto similarity and exact match rates on the Stereo-2k test set.
• Robustness: The model maintained high performance even on perturbed (rotated/sheared) and low-resolution images, outperforming rule-based methods significantly in these scenarios.

Reproducibility Details

Data

The training pipeline uses three distinct data sources:

1. Pre-training: MolParser-7M. Contains diverse images but requires the SMILES-M extension to handle Markush structures.
2. Fine-tuning: PubChem-1M and USPTO-680K. Used for multi-granularity learning with bond and coordinate labels.
3. RL Post-training: Stereo-200k. A self-collected dataset from the first 2M compounds in PubChem, filtered for chirality (’@’) and cis-trans isomerism (’/’, ‘\’). It uses 5 different RDKit drawing styles to ensure robustness.

Algorithms

• Reinforcement Learning: Uses GRPO (Group Relative Policy Optimization).
  • Reward Function: A linear combination of Tanimoto similarity and Stereochemical exact match. $$ R = w_t \cdot \text{Tanimoto} + w_s \cdot \text{ExactMatch} $$ where $w_t=0.4$ and $w_s=0.6$.
  • Sampling: Samples 4 completions per image with temperature 1.0 during RL training.
• Auxiliary Tasks:
  • Bond Classification: Concatenates hidden states of two atom queries to predict bond type via MLP.
  • Atom Localization: Treated as a classification task (SimCC) but optimized using Maximum Likelihood Estimation (MLE) to account for uncertainty.

Models

• Architecture: Encoder-Decoder Transformer.
  • Encoder: EfficientViT-L1 (~53M params), chosen for linear attention efficiency.
  • Decoder: 6-layer Transformer with RoPE, SwiGLU, and RMSNorm. Randomly initialized (no LLM weights) due to vocabulary mismatch.
  • Coordinate Head: Separated from the main decoder. It adds 2 extra Transformer layers to process atom queries before prediction to improve accuracy.
• Parameter Tuning:
  • Stage 3 (RL) uses LoRA (Rank=8, Alpha=16) to optimize the decoder.

Evaluation

• Metrics:
  • Exact Match: String identity for standard SMILES.
  • Tanimoto Coefficient: Fingerprint similarity for chemical semantics.
  • OKS (Object Keypoint Similarity): Used specifically for evaluating atom localization accuracy.
• Perturbation: Robustness tested with random rotations [-5°, 5°] and xy-shearing [-0.1, 0.1].

Hardware

• Compute: Training and inference performed on a single node.
• Processors: Intel Xeon Silver 4210R CPU.
• Accelerators: 4x NVIDIA GeForce RTX 3090/4090 GPUs.
• Hyperparameters:
  • Stage 1: Batch size 512, LR $4 \x10^{-4}$.
  • Stage 2: Batch size 256, Bond head LR $4 \x10^{-4}$, Coord head LR $4 \x10^{-5}$.
  • Stage 3 (RL): Batch size 64, Base LR $1 \x10^{-4}$.

Citation

@misc{zhang2025molsight,
      title={MolSight: Optical Chemical Structure Recognition with SMILES Pretraining, Multi-Granularity Learning and Reinforcement Learning},
      author={Wenrui Zhang and Xinggang Wang and Bin Feng and Wenyu Liu},
      year={2025},
      eprint={2511.17300},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.17300},
}

Citation: Morin, L., Weber, V., Nassar, A., Meijer, G. I., Van Gool, L., Li, Y., & Staar, P. (2025). MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14505-14515. https://doi.org/10.1109/CVPR52734.2025.01352

Publication: CVPR 2025

Additional Resources:

• GitHub Repository

Overcoming Unimodal Limitations for Markush Structures

The automated analysis of chemical literature, particularly patents, is critical for drug discovery and material science. A major bottleneck is the extraction of Markush structures, which are complex chemical templates that represent families of molecules using a core backbone image and textual variable definitions. Existing methods are limited because they either rely solely on images (OCSR) and miss the textual context, or focus solely on text and miss the structural backbone. This creates a practical need for a unified, multi-modal approach that jointly interprets visual and textual data to accurately extract these structures for prior-art search and database construction. This paper proposes a Method and introduces a new Resource (M2S dataset) to bridge this gap.

MarkushGrapher: The Multi-Modal Architecture

The core innovation is MarkushGrapher, a multi-modal architecture that jointly encodes image, text, and layout information. Key contributions include:

• Dual-Encoder Architecture: Combines a Vision-Text-Layout (VTL) encoder (based on UDOP) with a specialized, pre-trained Optical Chemical Structure Recognition (OCSR) encoder (MolScribe). Let $E_{\text{VTL}}$ represent the combined sequence embedding and $E_{\text{OCSR}}$ represent the domain-specific visual embeddings.
• Joint Recognition: The model autoregressively generates a sequential graph representation (Optimized CXSMILES) and a substituent table simultaneously. It leverages cross-modal dependencies, allowing text to clarify ambiguous visual details like bond types.
• Synthetic Data Pipeline: A comprehensive pipeline generates realistic synthetic Markush structures (images and text) from PubChem data, overcoming the lack of labeled training data.
• Optimized Representation: A compacted version of CXSMILES moves variable groups into the SMILES string and adds explicit atom indexing to handle complex “frequency” and “position” variation indicators.

Experimental Validation on the New M2S Benchmark

The authors validated their approach using the following setup:

• Baselines: Compared against image-only chemistry models (DECIMER, MolScribe) and general-purpose multi-modal models (Uni-SMART, GPT-4o, Pixtral, Llama-3.2).
• Datasets: Evaluated on three benchmarks:
  1. MarkushGrapher-Synthetic: 1,000 generated samples.
  2. M2S: A new benchmark of 103 manually annotated real-world patent images.
  3. USPTO-Markush: 74 Markush backbone images from USPTO patents.
• Ablation Studies: Analyzed the impact of the OCSR encoder, late fusion strategies, and the optimized CXSMILES format.

Reaching State-of-the-Art Exact Matches

• Performance Limits: MarkushGrapher significantly outperformed all baselines. On the M2S benchmark, it achieved a 38% Exact Match on CXSMILES (compared to 21% for MolScribe) and 29% Exact Match on tables.
• Complex Feature Handling: The model demonstrates a unique capability to reliably recognize complex Markush features like frequency variation (‘Sg’) and position variation (’m’) indicators. The alternative baselines scored near zero on these specific sub-tasks.
• Cross-Modal Reasoning: Qualitative analysis verified the model can correctly infer visual details (such as bond order) that appear ambiguous in the image but become apparent with the text description.
• Robustness: The model generalizes well to real-world data despite being trained purely on synthetic data iterations.

Reproducibility Details

Data

Training Data

• Source: Synthetic dataset generated from PubChem SMILES.
• Size: 210,000 synthetic images.
• Pipeline:
  1. Selection: Sampled SMILES from PubChem based on substructure diversity.
  2. Augmentation: SMILES augmented to artificial CXSMILES using RDKit (inserting variable groups, frequency indicators).
  3. Rendering: Images rendered using Chemistry Development Kit (CDK) with randomized drawing parameters (font, bond width, spacing).
  4. Text Generation: Textual definitions generated using manual templates extracted from patents; 10% were paraphrased using Mistral-7B-Instruct-v0.3 to increase diversity.
  5. OCR: Bounding boxes extracted via a custom SVG parser aligned with MOL files.

Evaluation Data

• M2S Dataset: 103 images from USPTO, EPO, and WIPO patents (1999-2023), manually annotated with CXSMILES and substituent tables.
• USPTO-Markush: 74 images from USPTO patents (2010-2016).
• MarkushGrapher-Synthetic: 1,000 samples generated via the pipeline.

Algorithms

• Optimized CXSMILES:
  • Compression: Variable groups moved from the extension block to the main SMILES string as special atoms to reduce sequence length.
  • Indexing: Atom indices appended to each atom (e.g., C:1) to explicitly link the graph to the extension block (crucial for m and Sg sections).
  • Vocabulary: Specific tokens used for atoms and bonds.
• Augmentation: Standard image augmentations (shift, scale, blur, pepper noise, random lines) and OCR text augmentations (character substitution/insertion/deletion).

Models

• Architecture: Encoder-Decoder Transformer.
  • VTL Encoder: T5-large encoder (initialized from UDOP) that processes image patches, text tokens, and layout (bounding boxes).
  • OCSR Encoder: Vision encoder from MolScribe (Swin Transformer), frozen during training.
  • Text Decoder: T5-large decoder.
• Fusion Strategy: Late Fusion. The core multi-modal alignment combines the textual layout features with specialized chemical vision explicitly. The fused representation $H_{\text{fused}}$ relies on the VTL output $e_1$ concatenated with the projected OCSR output $e_2$ before decoding: $$ H_{\text{fused}} = [e_1 \oplus \text{Proj}(e_2)] $$
• Parameters: 831M total (744M trainable).

Evaluation

Metrics:

• CXSMILES Exact Match (EM): Requires perfect match of SMILES string, variable groups, m sections, and Sg sections (ignoring stereochemistry).
• Tanimoto Score: Similarity of RDKit DayLight fingerprints (Markush features removed).
• Table Exact Match: All variable groups and substituents must match.
• Table F1-Score: Aggregated recall and precision of substituents per variable group.

Hardware

• Compute: Trained on a single NVIDIA H100 GPU.
• Training Config: 10 epochs, batch size of 10, learning rate 5e-4, 100 warmup steps, weight decay 1e-3.

Citation

@inproceedings{morinMarkushGrapherJointVisual2025,
  title = {MarkushGrapher: Joint Visual and Textual Recognition of Markush Structures},
  shorttitle = {MarkushGrapher},
  booktitle = {2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  author = {Morin, Lucas and Weber, Valéry and Nassar, Ahmed and Meijer, Gerhard Ingmar and Van Gool, Luc and Li, Yawei and Staar, Peter},
  year = {2025},
  month = jun,
  pages = {14505--14515},
  doi = {10.1109/CVPR52734.2025.01352}
}

Citation: Li, D., Xu, X., Pan, J., Gao, W., & Zhang, S. (2024). Image2InChI: Automated Molecular Optical Image Recognition. Journal of Chemical Information and Modeling, 64(9), 3640-3649. https://doi.org/10.1021/acs.jcim.3c02082

Publication: Journal of Chemical Information and Modeling (JCIM) 2024

Additional Resources:

• BMS Dataset (Kaggle)

Note: These notes are based on the Abstract and Supporting Information files only.

Image2InChI as a Methodological Innovation

This is a Methodological Paper ($\Psi_{\text{Method}}$). It proposes a specific new deep learning architecture (“Image2InChI”) to solve the task of Optical Chemical Structure Recognition (OCSR). The rhetorical focus is on engineering a system that outperforms baselines on specific metrics (InChI accuracy, MCS accuracy) and providing a valuable reference for future algorithmic work.

Bottlenecks in Chemical Literature Digitization

The accurate digitization of chemical literature is a bottleneck in AI-driven drug discovery. Chemical structures in patents and papers exist as optical images (pixels), but machine learning models require machine-readable string representations (like InChI or SMILES). Efficiently and automatically bridging this gap is a prerequisite for large-scale data mining in chemistry.

Hierarchical SwinTransformer and Attention Integration

The core novelty is the Image2InChI architecture, which integrates:

1. Improved SwinTransformer Encoder: Uses a hierarchical vision transformer to capture image features.
2. Feature Fusion with Attention: A novel network designed to integrate image patch features with InChI prediction steps.
3. End-to-End InChI Prediction: The architecture frames the problem as a direct image-to-sequence translation targeting InChI strings directly, diverging from techniques predicting independent graph components. The model is optimized using a standard Cross-Entropy Loss over the token vocabulary: $$ \mathcal{L}{\text{CE}} = - \sum{t=1}^{T} \log P(y_t \mid y_{<t}, \mathbf{X}) $$ where $\mathbf{X}$ represents the input image features, $y_t$ is the predicted token, and $T$ is the sequence length.

Benchmarking on the BMS Dataset

• Benchmark Validation: The model was trained and tested on the BMS1000 (Bristol-Myers Squibb) dataset from a Kaggle competition.
• Ablation/Comparative Analysis: The authors compared their method against other models in the supplement.
• Preprocessing Validation: They justified their choice of denoising algorithms (8-neighborhood vs. Gaussian/Mean) to ensure preservation of bond lines while removing “spiky point noise”.

High InChI Recognition Metrics

• High Accuracy: The model achieved 99.8% InChI accuracy, 94.8% Maximum Common Substructure (MCS) accuracy, and 96.2% Longest Common Subsequence (LCS) accuracy on the benchmarked dataset. It remains to be seen how well these models generalize to heavily degraded real-world patent images.
• Effective Denoising: The authors concluded that eight-neighborhood filtering is superior to mean or Gaussian filtering for this specific domain because it removes isolated noise points without blurring the fine edges of chemical bonds.
• Open Source: The authors committed to releasing the code to facilitate transparency and further research.

Reproducibility Details

Data

The primary dataset used is the BMS (Bristol-Myers Squibb) Dataset.

Other Datasets: The authors also utilized JPO (Japanese Patent Office), CLEF (CLEF-IP 2012), UOB (MolrecUOB), and USPTO datasets for broader benchmarking.

Preprocessing Pipeline:

1. Denoising: Eight-neighborhood filtering (threshold < 4 non-white pixels) is used to remove salt-and-pepper noise while preserving bond lines. Mean and Gaussian filtering were rejected due to blurring.
2. Sequence Padding:
   • Analysis showed max InChI length < 270.
   • Fixed sequence length set to 300.
   • Tokens: <sos> (190), <eos> (191), <pad> (192) used for padding/framing.
3. Numerization: Characters are mapped to integers based on a fixed vocabulary (e.g., ‘C’ -> 178, ‘H’ -> 182).

Algorithms

Eight-Neighborhood Filtering (Denoising):

Pseudocode logic:

• Iterate through every pixel.
• Count non-white neighbors in the 3x3 grid (8 neighbors).
• If count < threshold (default 4), treat as noise and remove.

InChI Tokenization:

• InChI strings are split into character arrays.
• Example: Vitamin C InChI=1S/C6H8O6... becomes [<sos>, C, 6, H, 8, O, 6, ..., <eos>, <pad>...].
• Mapped to integer tensor for model input.

Models

Architecture: Image2InChI

• Encoder: Improved SwinTransformer (Hierarchical Vision Transformer).
• Decoder: Transformer Decoder with patch embedding.
• Fusion: A novel “feature fusion network with attention” integrates the visual tokens with the sequence generation process.
• Framework: PyTorch 1.8.1.

Evaluation

Metrics:

• InChI Acc: Exact match accuracy of the predicted InChI string (Reported: 99.8%).
• MCS Acc: Maximum Common Substructure accuracy (structural similarity) (Reported: 94.8%).
• LCS Acc: Longest Common Subsequence accuracy (string similarity) (Reported: 96.2%).
• Morgan FP: Morgan Fingerprint similarity (Reported: 94.1%).

Hardware

Citation

@article{li2024image2inchi,
  title={Image2InChI: Automated Molecular Optical Image Recognition},
  author={Li, Da-zhou and Xu, Xin and Pan, Jia-heng and Gao, Wei and Zhang, Shi-rui},
  journal={Journal of Chemical Information and Modeling},
  volume={64},
  number={9},
  pages={3640--3649},
  year={2024},
  publisher={American Chemical Society},
  doi={10.1021/acs.jcim.3c02082}
}

This paper is a combination of Method (primary) and Resource (secondary).

It is primarily a Method paper because it proposes ChemVLM, a novel multimodal architecture specifically tailored for the chemical domain, utilizing a “ViT-MLP-LLM” framework. The authors introduce a specific two-stage training strategy to align visual features with chemical text representations.

Secondarily, it is a Resource paper as it introduces a comprehensive suite of three new datasets: ChemOCR, MMCR-Bench, and MMChemBench, developed to rigorously evaluate multimodal capabilities in chemistry, covering OCR, reasoning, and property prediction.

Bridging the Visual Gap in Chemical LLMs

The primary motivation is the limitation of existing models in handling the multimodal nature of chemistry.

• Visual Data Gap: Chemical tasks heavily rely on visual information (molecular structures, reactions) which purely text-based chemical LLMs cannot process.
• Limitations of Generalist Models: General multimodal models (like GPT-4V or LLaVA) lack specialized chemical domain knowledge, leading to hallucinations or misinterpretations.
• Inadequacy of OCR Tools: Traditional chemical OCR tools (like MolScribe) excel at modality conversion (Image-to-SMILES) but fail at complex reasoning tasks.

Domain-Specific Data Curation and Benchmarking

• Data-Driven Alignment: The underlying “ViT-MLP-LLM” framework is standard in multimodal modeling, paralleling architectures like LLaVA. The core innovation here is the rigorous creation of a bilingual multimodal dataset spanning hand-drawn molecules, reactions, and exam questions augmented with style transfers. The training data pipeline heavily relies on generating synthetic variance using tools like RanDepict and RDKit to introduce distortions, rotations, and handwritten styles, alongside GPT-4 generated prompts to ensure linguistic diversity.
• Model Integration: ChemVLM merges InternViT-6B (a large-scale vision transformer) with ChemLLM-20B (a chemical language model). Visual features $X_v$ are mapped into the linguistic embedding space via an MLP projector, producing aligned token sequences alongside text instructions $X_q$. The joint multimodal sequence is trained using standard autoregressive next-token prediction: $$ \mathcal{L} = -\sum_{i} \log P(y_i \mid X_v, X_q, y_{<i}) $$
• Three Custom Benchmarks: The authors introduce tailored benchmarks to assess distinct competencies:
  • ChemOCR: For image-to-SMILES conversion.
  • MMCR-Bench: College entrance exam questions testing complex logical reasoning.
  • MMChemBench: For molecule captioning and zero-shot property prediction.

Evaluating Chemical OCR and Reasoning

The authors benchmarked ChemVLM against both open-source (LLaVA, Qwen-VL, InternVL) and proprietary (GPT-4V) models across three primary domains:

1. Chemical OCR: Evaluated on 1,000 image-text pairs from ChemOCR. The primary metric is the Tanimoto similarity between the Morgan fingerprints of the generated structure ($A$) and the ground-truth SMILES ($B$): $$ T(A, B) = \frac{|A \cap B|}{|A| + |B| - |A \cap B|} $$ They report both the average Tanimoto similarity and the strict exact-match rate (Tanimoto@1.0).
2. Multimodal Chemical Reasoning (MMCR): Tested on MMCR-Bench (1,000 exam questions), ScienceQA, and CMMU. Performance was scored based on accuracy for multiple-choice and fill-in-the-blank questions.
3. Multimodal Molecule Understanding: Evaluated on MMChemBench for molecule captioning and property prediction.
4. Text-Only Reasoning: Tested on SciBench, a text-only benchmark for university-level science, to ensure the model retains fundamental linguistic reasoning.
5. Generalization: Tested on non-chemistry subjects within the CMMU framework (Biology, Physics, Math) to assess cross-domain competence.

Performance Gains and Existing Limitations

• Multimodal Reasoning Leadership: ChemVLM achieved state-of-the-art results on MMCR-Bench (41.7%), surpassing generalist models like GPT-4V (40.1%). However, scoring for portions of these benchmarks relied heavily on an LLM-as-a-judge (the Qwen-max API), which can introduce bias as LLM evaluators often favor structural characteristics and verbosity produced by similar autoregressive models. Furthermore, the model was fine-tuned on 200,000 exam questions and tested on MMCR-Bench (also derived from Chinese college entrance exams). While the authors state the data was deduplicated, the potential for data leakage remains a significant unaddressed confounder.
• Superior Understanding: In molecule captioning and prediction, ChemVLM showed significant improvements over general baseline models, scoring 80.9% on prediction compared to GPT-4V’s 38.6%. This is a natural consequence of testing a custom-trained model on domain-specific benchmarks.
• OCR Capabilities vs. Dedicated Tools: ChemVLM outperformed generalist MLLMs in chemical structure recognition, achieving an average Tanimoto similarity of 71.0% (vs. GPT-4V’s 15.0%). However, it remains significantly inferior to pure structural OCR tools like MolScribe in strict modality conversion tasks, only achieving an exact structural match (Tanimoto@1.0) of 42.9% compared to MolScribe’s 89.1%.
• Textual Retention and Generalization Claims: The authors claim the diverse training strategy imparts broad scientific reasoning, pointing to performance retention on non-chemistry subjects (Biology, Physics, Math) and strong results on the purely textual SciBench benchmark. However, this cross-domain generalization highly likely stems from the underlying base model (ChemLLM-20B/InternLM2) or the inclusion of 1.3 million “General” visual QA pairs in their training blend, rather than emergent general scientific skills originating purely from learning chemistry representations.

Reproducibility Details

Data

The training and evaluation data relied on a mix of open-source repositories and custom curation. Many of the curated datasets have been formally released by the authors on Hugging Face (di-zhang-fdu/chemvlm-sft-datasets).

Preprocessing: Images were augmented using RanDepict for style variation. Text data (SMILES) was validated and cleaned. Prompts were diversified using GPT-4 to generate different linguistic styles.

Algorithms

• Architecture: “ViT-MLP-LLM” structure.
  • Vision Encoder: InternViT-6B, processing images at $448 \times 448$ resolution. Images are segmented into tiles (max 12).
  • Projector: Multi-Layer Perceptron (MLP) initialized randomly to map visual features to text embedding space.
  • LLM: ChemLLM-20B, a domain-specific model.
• Training Strategy: Two-stage supervised fine-tuning.
  1. Modal Alignment: Freeze LLM and base Vision Encoder weights. Train only the randomly initialized MLP projector and LoRA layers (rank 32) of the Vision Encoder. Uses diverse multimodal data.
  2. Supervised Fine-Tuning (SFT): Keep LLM and Vision Encoder base weights frozen, but add LoRA (rank 16) to the LLM and retain LoRA (rank 32) on the Vision Encoder. The MLP projector is fully trained. Data includes specialized chemistry and general corpora.
• Optimization:
  • Optimizer: AdamW
  • Context Length: 2048 tokens
  • Chat Template: InternLM2 dialogue schema

Models

• ChemVLM-26B: The primary model released. It combines the 6B parameter vision encoder and the 20B parameter language model. Weights are fully available at AI4Chem/ChemVLM-26B-1-2. An 8B version is also available.
• Baselines: Comparisons were made against GPT-4V, Qwen-VL-Chat, LLaVA-v1.5-13B, InternVL-v1.5, and Yi-VL-Plus.

Evaluation

Performance was measured across three distinct task types. Exact evaluation scripts have been released in the official repository.

Hardware

• Training Compute: Training utilized 16 NVIDIA A100 (80GB) GPUs.
• Configuration:
  • Batch size: 4 (per GPU, resulting in an effective global batch size of 256)
  • Gradient Accumulation: 4 iterations
  • Precision: Deepspeed bfloat16 (bf16) with ZeRO-3 offloading strategy
  • Framework: Training runs on the InternVL-v1.5 codebase rather than standalone scripts.
• Inference Compute: Evaluating the 26B model requires at least one 80GB A100 GPU (with Flash Attention + bfloat16). The 8B variant requires a GPU with at least 48GB of VRAM.

Artifacts

Paper Information

Citation: Li, J., et al. (2025). ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area. Proceedings of the AAAI Conference on Artificial Intelligence, 39(1), 415-423. https://doi.org/10.1609/aaai.v39i1.32020

Publication: AAAI 2025

@inproceedings{li2025chemvlm,
  title={ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area},
  author={Li, Junxian and Zhang, Di and Wang, Xunzhi and Hao, Zeying and Lei, Jingdi and Tan, Qian and Zhou, Cai and Liu, Wei and Yang, Yaotian and Xiong, Xinrui and Wang, Weiyun and Chen, Zhe and Wang, Wenhai and Li, Wei and Su, Mao and Zhang, Shufei and Ouyang, Wanli and Li, Yuqiang and Zhou, Dongzhan},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={39},
  number={1},
  pages={415--423},
  year={2025},
  url={https://doi.org/10.1609/aaai.v39i1.32020},
  doi={10.1609/aaai.v39i1.32020}
}

Additional Resources:

• Official Repository

Citation: Rajan, K., Steinbeck, C., & Zielesny, A. (2022). Performance of chemical structure string representations for chemical image recognition using transformers. Digital Discovery, 1(2), 84-90. https://doi.org/10.1039/D1DD00013F

Publication: Digital Discovery 2022

Additional Resources:

• ChemRxiv Preprint (PDF)
• Official Code Repository
• Data on Zenodo
• Related work: DECIMER, DECIMER 1.0, IMG2SMI

Methodological Focus and Resource Contributions

This is a Methodological Paper ($\Psi_{\text{Method}}$) with a secondary contribution as a Resource Paper ($\Psi_{\text{Resource}}$).

It functions as a systematic ablation study, keeping the model architecture (EfficientNet-B3 + Transformer) constant while varying the input/output representation (SMILES, DeepSMILES, SELFIES, InChI) to determine which format yields the best performance for Optical Chemical Structure Recognition (OCSR). It also contributes large-scale benchmarking datasets derived from ChEMBL and PubChem.

The Syntax Challenge in Chemical Image Recognition

Optical Chemical Structure Recognition (OCSR) is essential for extracting chemical information buried in scientific literature and patents. While deep learning offers a promising alternative to rule-based approaches, neural networks struggle with the syntax of standard chemical representations like SMILES. Specifically, the tokenization of SMILES strings (where ring closures and branches are marked by single characters potentially far apart in the sequence) creates learning difficulties for sequence-to-sequence models. Newer representations like DeepSMILES and SELFIES were developed to address these syntax issues, but their comparative performance in image-to-text tasks had not been rigorously benchmarked.

Isolating String Representation Variables

The core novelty is the comparative isolation of the string representation variable in an OCSR context. Previous approaches often selected a representation (usually SMILES) without validating if it was optimal for the learning task. This study specifically tests the hypothesis that syntax-robust representations (like SELFIES) improve deep learning performance compared to standard SMILES. It provides empirical evidence on the trade-off between validity (guaranteed by SELFIES) and accuracy (highest with SMILES).

Large-Scale Image-to-Text Translation Experiments

The authors performed a large-scale image-to-text translation experiment:

• Task: Converting 2D chemical structure images into text strings.
• Data:
  • ChEMBL: ~1.6M molecules, split into two datasets (with and without stereochemistry).
  • PubChem: ~3M molecules, split similarly, to test performance scaling with data size.
• Representations: The same chemical structures were converted into four formats: SMILES, DeepSMILES, SELFIES, and InChI.
• Metric: The models were evaluated on:
  • Validity: Can the predicted string be decoded back to a molecule?
  • Exact Match: Is the predicted string identical to the ground truth?
  • Tanimoto Similarity: How chemically similar is the prediction to the ground truth (using PubChem fingerprints)? The similarity $\mathcal{T}$ between two molecular fingerprints $A$ and $B$ is calculated as: $$ \mathcal{T}(A, B) = \frac{A \cdot B}{||A||^2 + ||B||^2 - A \cdot B} $$

Comparative Performance and Validity Trade-offs

• SMILES is the most accurate: Contrary to the hypothesis that syntax-robust formats would learn better, SMILES consistently achieved the highest exact match accuracy (up to 88.62% on PubChem data) and average Tanimoto similarity (0.98). This is likely due to SMILES having shorter string lengths and fewer unique tokens compared to SELFIES.
• SELFIES guarantees validity: While slightly less accurate in direct translation, SELFIES achieved 100% structural validity (every prediction could be decoded), whereas SMILES predictions occasionally contained syntax errors.
• InChI is unsuitable: InChI performed significantly worse (approx. 64% exact match) due to extreme string lengths (up to 273 tokens).
• Stereochemistry adds difficulty: Including stereochemistry reduced accuracy across all representations due to increased token count and visual complexity.
• Recommendation: Use SMILES for maximum accuracy; use SELFIES if generating valid structures is the priority (e.g., generative tasks).

Reproducibility Details

Data

The study used curated subsets from ChEMBL and PubChem. Images were generated synthetically.

Image Generation:

• Tool: CDK Structure Diagram Generator (SDG).
• Specs: $300 \times 300$ pixels, rotated by random angles ($0-360^{\circ}$), saved as 8-bit PNG.

Algorithms

Tokenization Rules (Critical for replication):

• SELFIES: Split at every ][ (e.g., [C][N] $\rightarrow$ [C], [N]).
• SMILES / DeepSMILES: Regex-based splitting:
  • Every heavy atom (e.g., C, N).
  • Every bracket ( and ).
  • Every bond symbol = and #.
  • Every single-digit number.
  • Everything inside square brackets [] is kept as a single token.
• InChI: The prefix InChI=1S/ was treated as a single token and removed during training, then re-added for evaluation.

Models

The model follows the DECIMER architecture.

• Encoder: EfficientNet-B3 (pre-trained with “Noisy Student” weights).
  • Output: Image feature vectors of shape $10 \times 10 \times 1536$.
• Decoder: Transformer (similar to the “Base” model from Attention Is All You Need).
  • Layers: 4 encoder-decoder layers.
  • Attention Heads: 8.
  • Dimension ($d_{\text{model}}$): 512.
  • Feed-forward ($d_{\text{ff}}$): 2048.
  • Dropout: 10%.
• Loss: Sparse categorical cross-entropy.
• Optimizer: Adam with custom learning rate scheduler.

Evaluation

Metrics were calculated after converting all predictions back to standard SMILES.

Hardware

• Training: Google Cloud TPUs (v3-8).
• Format: Data converted to TFRecords (128 image/text pairs per record) for TPU efficiency.
• Batch Size: 1024.

Citation

@article{rajanPerformanceChemicalStructure2022,
  title = {Performance of Chemical Structure String Representations for Chemical Image Recognition Using Transformers},
  author = {Rajan, Kohulan and Steinbeck, Christoph and Zielesny, Achim},
  year = 2022,
  journal = {Digital Discovery},
  volume = {1},
  number = {2},
  pages = {84--90},
  publisher = {Royal Society of Chemistry},
  doi = {10.1039/D1DD00013F}
}

Citation: Yi, J., Wu, C., Zhang, X., Xiao, X., Qiu, Y., Zhao, W., Hou, T., & Cao, D. (2022). MICER: a pre-trained encoder-decoder architecture for molecular image captioning. Bioinformatics, 38(19), 4562-4572. https://doi.org/10.1093/bioinformatics/btac545

Publication: Bioinformatics 2022

Additional Resources:

• GitHub Repository

MICER’s Contribution to Optical Structure Recognition

This is a Method paper according to the AI for Physical Sciences taxonomy. It proposes MICER, an encoder-decoder architecture that integrates transfer learning (fine-tuning pre-trained models) and attention mechanisms for Optical Chemical Structure Recognition (OCSR). The study includes rigorous benchmarking comparing MICER against three rule-based tools (OSRA, MolVec, Imago) and existing deep learning methods (DECIMER). The authors conduct extensive factor comparison experiments to isolate the effects of stereochemistry, molecular complexity, data volume, and encoder backbone choices.

The Challenge of Generalizing in OCSR

Chemical structures in scientific literature are valuable for drug discovery, but they are locked in image formats that are difficult to mine automatically. Traditional OCSR tools (like OSRA) rely on hand-crafted rules and expert knowledge. They are brittle, struggle with stylistic variations, and have low generalization ability. While deep learning has been applied (e.g., DECIMER), previous attempts often used frozen pre-trained feature extractors (without fine-tuning) or failed to fully exploit transfer learning, leading to suboptimal performance. The goal of this work is to build an end-to-end “image captioning” system that translates molecular images directly into SMILES strings without intermediate segmentation steps.

Integrating Fine-Tuning and Attention for Chemistry

The core novelty lies in the specific architectural integration of transfer learning with fine-tuning for the chemical domain. Unlike DECIMER, which used a frozen network, MICER fine-tunes a pre-trained ResNet on molecular images. This allows the encoder to adapt from general object recognition to specific chemical feature extraction.

The model incorporates an attention mechanism into the LSTM decoder, allowing the model to focus on specific image regions (atoms and bonds) when generating each character of the SMILES string. The paper explicitly analyzes “intrinsic features” of molecular data (stereochemistry, complexity) to guide the design of a robust training dataset, combining multiple chemical toolkits (Indigo, RDKit) to generate diverse styles.

Experimental Setup and Ablation Studies

The authors performed two types of experiments: Factor Comparison (ablations) and Benchmarking.

Factor Comparisons: They evaluated how performance is affected by:

• Stereochemistry (SI): Comparing models trained on data with and without stereochemical information.
• Molecular Complexity (MC): Analyzing performance across 5 molecular weight intervals.
• Data Volume (DV): Training on datasets ranging from 0.64 million to 10 million images.
• Pre-trained Models (PTMs): Comparing 8 different backbones (e.g., ResNet, VGG, Inception, MobileNet) versus a base CNN.

Benchmarking:

• Baselines: OSRA, MolVec, Imago (rule-based); Base CNN, DECIMER (deep learning).
• Datasets: Four test sets (100k images each, except UOB): Uni-style, Multi-style, Noisy, and Real-world (UOB dataset).
• Metrics: Sequence Accuracy (Exact Match), Levenshtein Distance (ALD), and Tanimoto Similarity (Fingerprint match).

Results and Core Insights

MICER achieved 97.54% Sequence Accuracy on uni-style data and 82.33% on the real-world UOB dataset, significantly outperforming the next best method (OSRA scored approximately 23% on uni-style; DECIMER scored roughly 21% on UOB). ResNet101 was identified as the most effective encoder (87.58% accuracy in preliminary tests), outperforming deeper (DenseNet) or lighter (MobileNet) networks. Performance saturates around 6 million training samples. Stereochemical information drops accuracy by nearly 6%, indicating wedge and dash bonds are harder to recognize. Visualizing attention maps showed the model correctly attends to specific atoms (e.g., focusing on ‘S’ or ‘Cl’ pixels) when generating the corresponding character.

Reproducibility Details

Data

The training data was curated from the ZINC20 database.

Preprocessing:

• Filtering: Removed organometallics, mixtures, and invalid molecules.
• Standardization: SMILES were canonicalized and de-duplicated.
• Generation: Images generated using Indigo and RDKit toolkits to vary styles.

Dataset Size:

• Total: 10 million images selected for the final model.
• Composition: 6 million “default style” (Indigo) + 4 million “multi-style” (Indigo + RDKit).
• Splits: 8:1:1 ratio for Training/Validation/Test.

Vocabulary: A token dictionary of 39 characters + special tokens: [pad], [sos], [eos], [0]-[9], [C], [1], [c], [O], [N], [n], [F], [H], [o], [S], [s], [B], [r], [I], [i], [P], [p], (, ), [, ], @, =, #, /, -, +, \, %. Note that two-letter atoms like ‘Br’ are tokenized as distinct characters [B], [r].

Algorithms

• Tokenization: Character-level tokenization (not atom-level); the model learns to assemble ‘C’ and ’l’ into ‘Cl’.
• Attention Mechanism: Uses a soft attention mechanism where the decoder calculates an attention score between the encoder’s feature map ($8 \times 8 \times 512$) and the current hidden vector. Formula: $$ \begin{aligned} \text{att_score} &= \text{softmax}(L_a(\tanh(L_f(F) + L_b(b_t)))) \end{aligned} $$
• Training Configuration:
  • Loss Function: Cross-entropy loss
  • Optimizer: Adam optimizer
  • Learning Rate: 2e-5
  • Batch Size: 256
  • Epochs: 15

Models

Encoder:

• Backbone: Pre-trained ResNet101 (trained on ImageNet).
• Modifications: The final layer is removed to output a Feature Map of size $8 \times 8 \times 512$.
• Flattening: Reshaped to a $64 \times 512$ feature matrix for the decoder.

Decoder:

• Type: Long Short-Term Memory (LSTM) with Attention.
• Dropout: 0.3 applied to minimize overfitting.

The encoder uses a pilot block, max-pool, and 4 layers of convolutional blocks (CB), feeding into the attention LSTM.

Evaluation

Metrics:

• SA (Sequence Accuracy): Strict exact match of SMILES strings.
• ALD (Average Levenshtein Distance): Edit distance for character-level error analysis.
• AMFTS / MFTS@1.0: Tanimoto similarity of ECFP4 fingerprints to measure structural similarity.

Test Sets:

• Uni-style: 100,000 images (Indigo default).
• Multi-style: 100,000 images (>10 styles).
• Noisy: 100,000 images with noise added.
• UOB: 5,575 real-world images from literature.

Hardware

• Compute: 4 x NVIDIA Tesla V100 GPUs
• Training Time: Approximately 42 hours for the final model

Citation

@article{yiMICERPretrainedEncoder2022,
  title = {{{MICER}}: A Pre-Trained Encoder--Decoder Architecture for Molecular Image Captioning},
  shorttitle = {{{MICER}}},
  author = {Yi, Jiacai and Wu, Chengkun and Zhang, Xiaochen and Xiao, Xinyi and Qiu, Yanlong and Zhao, Wentao and Hou, Tingjun and Cao, Dongsheng},
  year = {2022},
  month = sep,
  journal = {Bioinformatics},
  volume = {38},
  number = {19},
  pages = {4562--4572},
  issn = {1367-4811},
  doi = {10.1093/bioinformatics/btac545}
}

Citation: Khokhlov, I., Krasnov, L., Fedorov, M. V., & Sosnin, S. (2022). Image2SMILES: Transformer-Based Molecular Optical Recognition Engine. Chemistry-Methods, 2(1), e202100069. https://doi.org/10.1002/cmtd.202100069

Publication: Chemistry-Methods 2022

Additional Resources:

• Official Code (Data Generator)
• Syntelly Demo

Contribution: Image2SMILES as a Method and Resource

This is primarily a Method paper with a significant Resource component.

• Method: It proposes a specific neural architecture (ResNet backbone and Transformer Decoder) to solve the Optical Chemical Structure Recognition (OCSR) task, answering “How well does this work?” with extensive benchmarks against rule-based systems like OSRA.
• Resource: A core contribution is the “Generate and Train!” paradigm, where the authors release a comprehensive synthetic data generator to overcome the lack of labeled training data in the field.

Motivation: Bottlenecks in Recognizing Trapped Chemical Structures

Retrieving chemical structure data from legacy scientific literature is a major bottleneck in cheminformatics.

• Problem: Chemical structures are often “trapped” in image formats (PDFs, scans). Manual extraction is slow, and existing rule-based tools (e.g., OSRA) are brittle when facing diverse drawing styles, “Markush” structures (templates), or visual contamination.
• Gap: Deep learning approaches require massive datasets, but no large-scale annotated dataset of chemical figures exists.
• Goal: To create a robust, data-driven recognition engine that can handle the messiness of real-world chemical publications (e.g., text overlays, arrows, partial overlaps).

Core Innovation: The “Generate and Train!” Pipeline and FG-SMILES

• “Generate and Train!” Paradigm: The authors assert that architecture is secondary to data simulation. They developed an advanced augmentation pipeline that simulates geometry (rotation, bonds) alongside specific chemical drawing artifacts like “Markush” variables ($R_1$, $R_2$), functional group abbreviations (e.g., -OMe, -Ph), and visual “contamination” (stray text, arrows).
• FG-SMILES: A modified SMILES syntax designed to handle functional groups and Markush templates as single tokens (pseudo-atoms), allowing the model to predict generalized scaffolds.
• Encoder-Free Architecture: The authors found that a standard Transformer Encoder was unnecessary. They feed the flattened feature map from a ResNet backbone directly into the Transformer Decoder, which improved performance.

Methodology and Benchmarking Against OSRA

• Training: The model was trained on 10 million synthetically generated images derived from PubChem structures, selected via a complexity-biased sampling algorithm.
• Validation (Synthetic): Evaluated on a hold-out set of 1M synthetic images.
• Validation (Real World):
  • Dataset A: 332 manually cropped structures from 10 specific articles, excluding reaction schemes.
  • Dataset B: 296 structures systematically extracted from Journal of Organic Chemistry (one paper per issue from 2020) to reduce selection bias.
• Comparison: Benchmarked against OSRA (v2.11), a widely used rule-based OCSR tool.

Results: High-Precision Extraction and Key Limitations

• Performance:
  • Synthetic: 90.7% exact match accuracy.
  • Real Data (Dataset A): Image2SMILES achieved 79.2% accuracy compared to OSRA’s 62.1%.
  • Real Data (Dataset B): Image2SMILES achieved 62.5% accuracy compared to OSRA’s 24.0%.
• Confidence Correlation: There is a strong correlation between the model’s confidence score and prediction validity. Thresholding at 0.995 yields 99.85% accuracy while ignoring 22% of data, enabling high-precision automated pipelines.
• Key Failures: The model struggles with functional groups absent from its training dictionary (e.g., $\text{NMe}_2$, Ms), confusion of R-group indices ($R’$ vs $R_1$), and explicit hydrogens rendered as groups.

Reproducibility Details

Data

• Source: A subset of 10 million molecules sampled from PubChem.
• Selection Logic: Bias towards complex/rare structures using a “Full Coefficient” (FC) probability metric based on molecule size and ring/atom rarity.
  • Formula: $BC=0.1+1.2\left(\frac{n_{\max}-n}{n_{\max}}\right)^{3}$ where $n_{\max}=60$.
• Generation: Uses RDKit for rendering with augmentations: rotation, font size, line thickness, whitespace, and CoordGen (20% probability).
• Contamination: “Visual noise” is stochastically added, including parts of other structures, labels, and arrows cropped from real documents.
• Target Format: FG-SMILES (Functional Group SMILES). Replaces common functional groups with pseudo-atoms (e.g., [Me], [Ph], [NO2]) and supports variable R-group positions using a v token.

Algorithms

• Contamination Augmentation: A dedicated algorithm simulates visual noise (arrows, text) touching or overlapping the main molecule to force robustness.
• Functional Group Resolution: An algorithm identifies overlapping functional group templates (SMARTS) and resolves them to prevent nested group conflicts (e.g., resolving Methyl vs Methoxy).
• Markush Support: Stochastic replacement of substituents with R-group labels ($R_1$, $R’$, etc.) based on a defined probability table (e.g., $P(R)=0.2$, $P(R_1)=0.15$).

Models

• Architecture: “Image-to-Sequence” hybrid model.
  • Backbone: ResNet-50, but with the last two residual blocks removed. Output shape: $512 \times 48 \times 48$.
  • Neck: No Transformer Encoder. CNN features are flattened and passed directly to the Decoder.
  • Decoder: Standard Transformer Decoder (6 layers).
• Input: Images resized to $384 \times 384 \times 3$.
• Output: Sequence of FG-SMILES tokens.

Evaluation

• Metric: Binary “Exact Match” (valid/invalid).
  • Strict criteria: Stereo and R-group indices must match exactly (e.g., $R’$ vs $R_1$ is a failure).
• Datasets:
  • Internal: 5% random split of generated data (500k samples).
  • External (Dataset A & B): Manually cropped real-world images from specified journals.

Hardware

• Training: 4 $\times$ Nvidia V100 GPUs + 36 CPU cores.
• Duration: ~2 weeks for training (5 epochs, ~63 hours/epoch). Data generation took 3 days on 80 CPUs.
• Optimizer: RAdam with learning rate $3 \cdot 10^{-4}$.

Citation

@article{khokhlovImage2SMILESTransformerBasedMolecular2022,
  title = {Image2SMILES: Transformer-Based Molecular Optical Recognition Engine},
  shorttitle = {Image2SMILES},
  author = {Khokhlov, Ivan and Krasnov, Lev and Fedorov, Maxim V. and Sosnin, Sergey},
  year = {2022},
  journal = {Chemistry-Methods},
  volume = {2},
  number = {1},
  pages = {e202100069},
  issn = {2628-9725},
  doi = {10.1002/cmtd.202100069},
  url = {https://chemistry-europe.onlinelibrary.wiley.com/doi/10.1002/cmtd.202100069}
}

Citation: Yoo, S., Kwon, O., & Lee, H. (2022). Image-to-Graph Transformers for Chemical Structure Recognition. arXiv preprint arXiv:2202.09580. https://doi.org/10.48550/arXiv.2202.09580

Publication: arXiv 2022

Contribution and Taxonomic Classification

This is a Method paper. It proposes a novel deep learning architecture designed to extract molecular structures from images by directly predicting the graph topology. The paper validates this approach through ablation studies (comparing ResNet-only baselines to the Transformer-augmented model) and extensive benchmarking against existing tools.

The Challenge with SMILES and Non-Atomic Symbols

• Handling Abbreviations: Chemical structures in scientific literature often use non-atomic symbols (superatoms like “R” or “Ph”) to reduce complexity. Standard tools that generate SMILES strings fail here because SMILES syntax does not support arbitrary non-atomic symbols.
• Robustness to Style: Existing rule-based tools are brittle to the diverse drawing styles found in literature.
• Data Utilization: Pixel-wise graph recognition tools (like ChemGrapher) require expensive pixel-level labeling. An end-to-end approach can utilize massive amounts of image-molecule pairs (like USPTO data) without needing exact coordinate labels.

The Image-to-Graph (I2G) Architecture

The core novelty is the Image-to-Graph (I2G) architecture that bypasses string representations entirely:

• Hybrid Encoder: Combines a ResNet backbone (for locality) with a Transformer encoder (for global context), allowing the model to capture relationships between atoms that are far apart in the image.
• Graph Decoder (GRAT): A modified Transformer decoder that generates the graph auto-regressively. It uses feature-wise transformations to modulate attention weights based on edge information (bond types).
• Coordinate-Aware Training: The model is forced to predict the exact 2D coordinates of atoms in the source image, which significantly boosts accuracy by grounding the decoder’s attention.

Experimental Setup and Baselines

• Baselines: The model was compared against OSRA (rule-based), MolVec (rule-based), and ChemGrapher (deep learning pixel-wise).
• Benchmarks: Evaluated on four standard datasets: UoB, USPTO, CLEF, and JPO. Images were converted to PDF and back to simulate degradation.
• Large Molecule Test: A custom dataset (OLED) was created from 12 journal papers (434 images) to test performance on larger, more complex structures (average 52.8 atoms).
• Ablations: The authors tested the impact of the Transformer encoder, auxiliary losses, and coordinate prediction.

Empirical Results and Robustness

• SOTA Performance: The proposed model outperformed existing models with a 17.1% relative improvement on benchmark datasets.
• Robustness: On large molecules (OLED dataset), it achieved a 12.8% relative improvement.
• Data Scaling: Adding real-world USPTO data to the synthetic training set improved performance by 20.5%, demonstrating the model’s ability to learn from noisy, unlabeled coordinates.
• Handling Superatoms: The model successfully recognized pseudo-atoms (e.g., $R_1$, $R_2$) as distinct nodes, whereas SMILES-based tools collapsed them into generic “Any” atoms.

Reproducibility Details

Data

The authors used a combination of synthetic and real-world data for training.

Preprocessing:

• Input resolution is fixed at $800 x800$ pixels.
• Images are virtually split into a $25 x25$ grid (625 patches total), where each patch is $32 x32$ pixels.

Algorithms

Encoder Logic:

• Grid Serialization: The $25 x25$ grid is flattened into a 1D sequence. 2D position information is concatenated to ResNet features before the Transformer.
• Auxiliary Losses: To aid convergence, classifiers on the encoder predict three things per patch: (1) number of atoms, (2) characters in atom labels, and (3) edge-sharing neighbors. These losses decrease to zero during training.

Decoder Logic:

• Auto-regressive Generation: At step $t$, the decoder generates a new node and connects it to existing nodes.
• Attention Modulation: Attention weights are transformed using bond information: $$ \begin{aligned} \text{Att}(Q, K, V) = \text{softmax} \left( \frac{QK^T}{\sqrt{d}} \odot \gamma(e_{ij}) + \beta(e_{ij}) \right) V \end{aligned} $$ where $e_{ij}$ is the edge type.
• Coordinate Prediction: The decoder outputs coordinates for each atom, which acts as a mechanism to track attention history.

Models

• Image Encoder: ResNet-34 backbone followed by a Transformer encoder.
• Graph Decoder: A “Graph-Aware Transformer” (GRAT) that outputs nodes (atom labels, coordinates) and edges (bond types).

Evaluation

Metrics focus on structural identity, as standard string matching (SMILES) is insufficient for graphs with superatoms.

Citation: Li, Y., Chen, G., & Li, X. (2022). Automated Recognition of Chemical Molecule Images Based on an Improved TNT Model. Applied Sciences, 12(2), 680. https://doi.org/10.3390/app12020680

Publication: MDPI Applied Sciences 2022

Additional Resources:

• Kaggle Competition: BMS Molecular Translation

Contribution: Image-to-Text Translation for Chemical Structures

This is a Method paper.

It proposes a novel neural network architecture, the Image Captioning Model based on Deep TNT (ICMDT), to solve the specific problem of “molecular translation” (image-to-text). The classification is supported by the following rhetorical indicators:

• Novel Mechanism: It introduces the “Deep TNT block” to improve upon the existing TNT architecture by fusing features at three levels (pixel, small patch, large patch).
• Baseline Comparison: The authors explicitly compare their model against four other architectures (CNN+RNN and CNN+Transformer variants).
• Ablation Study: Section 4.3 is dedicated to ablating specific components (position encoding, patch fusion) to prove their contribution to the performance gain.

Motivation: Digitizing Historical Chemical Literature

The primary motivation is to speed up chemical research by digitizing historical chemical literature.

• Problem: Historical sources often contain corrupted or noisy images, making automated recognition difficult.
• Gap: Existing models like the standard TNT (Transformer in Transformer) function primarily as encoders for classification and fail to effectively integrate local pixel-level information required for precise structure generation.
• Goal: To build a dependable generative model that can accurately translate these noisy images into InChI (International Chemical Identifier) text strings.

Novelty: Multi-Level Feature Fusion with Deep TNT

The core contribution is the Deep TNT block and the resulting ICMDT architecture.

• Deep TNT Block: The Deep TNT block expands upon standard local and global modeling by stacking three transformer blocks to process information at three granularities:
  1. Internal Transformer: Processes pixel embeddings.
  2. Middle Transformer: Processes small patch embeddings.
  3. Exterior Transformer: Processes large patch embeddings.
• Multi-level Fusion: The model fuses pixel-level features into small patches, and small patches into large patches, allowing for finer integration of local details.
• Position Encoding: A specific strategy of applying shared position encodings to small patches and pixels, while using a learnable 1D encoding for large patches.

Methodology: Benchmarking on the BMS Dataset

The authors evaluated the model on the Bristol-Myers Squibb Molecular Translation dataset.

• Baselines: They constructed four comparative models:
  • EfficientNetb0 + RNN (Bi-LSTM)
  • ResNet50d + RNN (Bi-LSTM)
  • EfficientNetb0 + Transformer
  • ResNet101d + Transformer
• Ablation: They tested the impact of removing the large patch position encoding (ICMDT*) and reverting the encoder to a standard TNT-S (TNTD).
• Pre-processing Study: They experimented with denoising ratios and cropping strategies.

Results & Conclusions: State-of-the-Art InChI Translation

• Performance: ICMDT achieved the lowest Levenshtein distance (0.69) compared to the best baseline (1.45 for ResNet101d+Transformer).
• Convergence: The model converged significantly faster than the baselines, outperforming others as early as epoch 6.7.
• Ablation Results: The full Deep TNT block reduced error by nearly half compared to the standard TNT encoder (0.69 vs 1.29 Levenshtein distance).
• Limitations: The model struggles with stereochemical layers (e.g., identifying clockwise neighbors or +/- signs) compared to non-stereochemical layers.
• Future Work: Integrating full object detection to predict atom/bond coordinates to better resolve 3D stereochemical information.

Reproducibility Details

Data

The experiments used the Bristol-Myers Squibb Molecular Translation dataset from Kaggle.

Pre-processing Strategy:

• Effective: Padding resizing (reshaping to square, padding with border pixels).
• Ineffective: Smart cropping (removing white borders degraded performance).
• Augmentation: GaussNoise, Blur, RandomRotate90, and PepperNoise ($SNR=0.996$).
• Denoising: Best results found by mixing denoised and original data (Ratio 2:13) during training.

Algorithms

• Optimizer: Lookahead ($\alpha=0.5, k=5$) and RAdam ($\beta_1=0.9, \beta_2=0.99$).
• Loss Function: Anti-Focal loss ($\gamma=0.5$) combined with Label Smoothing. While standard Focal Loss adds a modulating factor $(1-p_t)^\gamma$ to cross-entropy to focus on hard negatives, Anti-Focal Loss modifies this to reduce the disparity between training and inference distributions: $$ \text{Anti-Focal Loss}(p_t) = - (1 + p_t)^\gamma \log(p_t) $$
• Training Schedule:
  • Initial resolution: $224 \times 224$
  • Fine-tuning: Resolution $384 \times 384$ for labels $>150$ length.
  • Batch size: Dynamic, increasing from 16 to 1024.
• Inference Strategy:
  • Beam Search ($k=16$ initially, $k=64$ if failing InChI validation).
  • Test Time Augmentation (TTA): Rotations of $90^\circ$.
  • Ensemble: Step-wise logit ensemble and voting based on Levenshtein distance scores.

Models

ICMDT Architecture:

• Encoder (Deep TNT):
  • Internal Block: Hidden size 160, Heads 4, MLP act GELU.
  • Middle Block: Hidden size 640, Heads 10.
  • Exterior Block: Hidden size 2560, Heads 16.
  • Patch Sizes: Small patch $16 \times 16$, Pixel patch $4 \times 4$.
• Decoder (Vanilla Transformer):
  • Dimensions: 5120 (Hidden), 1024 (FFN).
  • Depth: 3 layers, Heads: 8.
  • Vocab Size: 193 (InChI tokens).

Evaluation

Metric: Levenshtein Distance (measures single-character edit operations between generated and ground truth InChI strings).

Citation

@article{liAutomatedRecognitionChemical2022,
  title = {Automated {{Recognition}} of {{Chemical Molecule Images Based}} on an {{Improved TNT Model}}},
  author = {Li, Yanchi and Chen, Guanyu and Li, Xiang},
  year = 2022,
  month = jan,
  journal = {Applied Sciences},
  volume = {12},
  number = {2},
  pages = {680},
  publisher = {Multidisciplinary Digital Publishing Institute},
  issn = {2076-3417},
  doi = {10.3390/app12020680}
}

Citation: Sundaramoorthy, C., Kelvin, L. Z., Sarin, M., & Gupta, S. (2021). End-to-End Attention-based Image Captioning. arXiv preprint arXiv:2104.14721. https://doi.org/10.48550/arXiv.2104.14721

Publication: arXiv 2021 (preprint)

Note: This is an arXiv preprint and has not undergone formal peer review.

Methodological Contribution

This is a Methodological Paper. It proposes a novel architectural approach to molecular image translation by replacing the standard CNN encoder with a Vision Transformer (ViT). The authors validate this method through comparative benchmarking against standard CNN+RNN baselines (e.g., ResNet+LSTM) and provide optimizations for inference speed.

Motivation and Problem Statement

The core problem addressed is existing molecular translation methods (extracting chemical structure from images into computer-readable InChI format) rely heavily on rule-based systems or CNN+RNN architectures. These current approaches often underperform when handling noisy images (common in scanned old journals) or images with few distinguishable features. There is a significant need in drug discovery to digitize and analyze legacy experimental data locked in image format within scientific publications.

Core Innovations: End-to-End ViT Encoder

The primary contribution is the use of a completely convolution-free Vision Transformer (ViT) as the encoder, allowing the model to utilize long-range dependencies among image patches from the very beginning via self-attention: $$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$ The architecture is a pure Transformer (Encoder-Decoder), treating the molecular image similarly to a sequence of tokens (patches). Furthermore, the authors implement a specific caching strategy for the decoder to avoid recomputing embeddings for previously decoded tokens, reducing the time complexity of the decoding step.

Experimental Setup and Baselines

The model was compared against standard CNN + RNN and ResNet (18, 34, 50) + LSTM with attention. Ablation studies were conducted varying the number of transformer layers (3, 6, 12, 24) and image resolution (224x224 vs 384x384). The model trained on a large combined dataset, including Bristol Myers Squibb data, SMILES, GDB-13, and synthetically augmented images containing noise and artifacts. Performance was evaluated using the Levenshtein distance metric, which computes the minimum number of single-character edits to transform the predicted string into the ground truth.

Performance Outcomes and Capabilities

The proposed 24-layer ViT model (input size 384) achieved the lowest Levenshtein distance of 6.95, significantly outperforming the ResNet50+LSTM baseline (7.49) and the standard CNN+RNN (103.7). Increasing the number of layers had a strong positive impact, with the 24-layer model becoming competitive with current approaches. The model demonstrated high robustness on datasets with low distinguishable features and noise. The proposed caching optimization reduced the decoding time complexity per timestep from $O(MN^2 + N^3)$ to $O(MN + N^2)$.

Reproducibility Details

Data

The model was trained on a combined dataset randomly split into 70% training, 10% test, and 20% validation.

Algorithms

• Training Objective: Cross-entropy loss minimization.
• Inference Decoding: Autoregressive decoding predicting the next character of the InChI string.
• Positional Encoding: Standard sine and cosine functions of different frequencies.
• Optimization:
  • Caching: Caches the output of each layer during decoding to avoid recomputing embeddings for already decoded tokens.
  • JIT: PyTorch JIT compiler used for graph optimization.
  • Self-Critical Training: Finetuning performed using self-critical sequence training (SCST).

Models

• Encoder (Vision Transformer):
  • Input: Flattened 2D patches of the image. Patch size: $16 \x16$.
  • Projection: Trainable linear projection to latent vector size $D$.
  • Structure: Alternating layers of Multi-Head Self-Attention (MHSA) and MLP blocks.
• Decoder (Vanilla Transformer):
  • Input: Tokenized InChI string + sinusoidal positional embedding.
  • Vocabulary: 275 tokens (including <SOS>, <PAD>, <EOS>).
• Hyperparameters (Best Model):
  • Image Size: $384 \x384$.
  • Layers: 24.
  • Feature Dimension: 512.
  • Attention Heads: 12.
  • Optimizer: Adam.
  • Learning Rate: $3 \x10^{-5}$ (decayed by 0.5 in last 2 epochs).
  • Batch Size: Varied [64-512].

Evaluation

• Primary Metric: Levenshtein Distance (lower is better).

Hardware

• System: 70GB GPU system.
• Framework: PyTorch and PyTorch Lightning.

Citation

@misc{sundaramoorthyEndtoEndAttentionbasedImage2021,
  title = {End-to-{{End Attention-based Image Captioning}}},
  author = {Sundaramoorthy, Carola and Kelvin, Lin Ziwen and Sarin, Mahak and Gupta, Shubham},
  year = 2021,
  month = apr,
  number = {arXiv:2104.14721},
  eprint = {2104.14721},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2104.14721},
  archiveprefix = {arXiv}
}

Citation: Rajan, K., Zielesny, A. & Steinbeck, C. (2021). DECIMER 1.0: deep learning for chemical image recognition using transformers. Journal of Cheminformatics, 13(1), 61. https://doi.org/10.1186/s13321-021-00538-8

Publication: Journal of Cheminformatics 2021

Additional Resources:

• GitHub Repository
• DECIMER Project Page

Evaluating the Contribution: A Methodological Shift

Method (Dominant) with strong Resource elements.

This is primarily a Method paper because it proposes a specific architectural evolution. It replaces CNN-RNN/Encoder-Decoder models with a Transformer-based network to solve the problem of image-to-structure translation. It validates this methodological shift through rigorous ablation studies comparing feature extractors (InceptionV3 vs. EfficientNet) and decoder architectures.

It also serves as a Resource contribution by releasing the open-source software, trained models, and describing the curation of a massive synthetic training dataset (>35 million molecules).

Motivation: Unlocking Locked Chemical Knowledge

• Data Inaccessibility: A vast amount of chemical knowledge (pre-1990s) is locked in printed or scanned literature and is not machine-readable.
• Manual Bottlenecks: Manual curation and extraction of this data is tedious, slow, and error-prone.
• Limitations of Prior Tools: Existing Optical Chemical Structure Recognition (OCSR) tools are often rule-based or struggle with the noise and variability of full-page scanned articles. Previous deep learning attempts were not publicly accessible or robust enough.

Key Innovation: Transformer-Based Molecular Translation

• Transformer Architecture: Shifts from the standard CNN-RNN (Encoder-Decoder) approach to a Transformer-based decoder, significantly improving accuracy.
• EfficientNet Backbone: Replaces the standard InceptionV3 feature extractor with EfficientNet-B3, which improved feature extraction quality for chemical images.
• SELFIES Representation: Utilizes SELFIES (SELF-referencing Embedded Strings) as the target output. This guarantees 100% robust molecular strings and eliminates the “invalid SMILES” problem common in generative models.
• Massive Scaling: Trains on a synthetic dataset of 35-39 million molecules, demonstrating that scaling data size directly correlates with improved model performance.

Methodology and Experimental Validation

• Feature Extractor Ablation: Compared InceptionV3 vs. EfficientNet-B3 (and B7) on a 1-million molecule subset to determine the optimal image encoder.
• Architecture Comparison: Benchmarked the Encoder-Decoder (CNN+RNN) against the Transformer model using Tanimoto similarity metrics. The structural similarity between predicted and ground truth molecules was measured via Tanimoto similarity over molecular fingerprints: $$ T(\mathbf{A}, \mathbf{B}) = \frac{\mathbf{A} \cdot \mathbf{B}}{|\mathbf{A}|^2 + |\mathbf{B}|^2 - \mathbf{A} \cdot \mathbf{B}} $$
• Data Scaling: Evaluated performance across increasing training set sizes (1M, 10M, 15M, 35M) to observe scaling laws.
• Stereochemistry & Ions: Tested the model’s ability to handle complex stereochemical information and charged groups (ions), creating separate datasets for these tasks.
• Augmentation Robustness: Evaluated the model on augmented images (blur, noise, varying contrast) to simulate real-world scanned document conditions.

Results: Superior Accuracy and Scaling Limit Observations

• Superior Architecture: The Transformer model with EfficientNet-B3 features significantly outperformed the Encoder-Decoder baseline. On the 1M dataset, the Transformer achieved 74.57% exact matches (Tanimoto 1.0) compared to only 7.03% for the Encoder-Decoder.
• High Accuracy at Scale: With the full 35-million molecule training set (Dataset 1), the model achieved a Tanimoto 1.0 score of 96.47% and an average Tanimoto similarity of 0.99.
• Isomorphism: 99.75% of predictions with a Tanimoto score of 1.0 were confirmed to be structurally isomorphic to the ground truth (checked via InChI).
• Stereochemistry Costs: Including stereochemistry and ions increased the token count and difficulty, resulting in slightly lower accuracy (~89.87% exact match on Dataset 2).
• Hardware Efficiency: Training on TPUs (v3-8) was ~4x faster than Nvidia V100 GPUs, reducing training time for the largest models from months to under 14 days.

Reproducibility Details

Data

The authors generated synthetic data from PubChem.

Algorithms

• Text Representation: SELFIES used to avoid invalid intermediate strings. Tokenized via Keras tokenizer.
  • Dataset 1 Tokens: 27 unique tokens. Max length 47.
  • Dataset 2/3 Tokens: 61 unique tokens (due to stereo/ion tokens).
• Augmentation: Implemented using imgaug python package. Random application of:
  • Gaussian/Average Blur, Additive Gaussian Noise, Salt & Pepper, Coarse Dropout, Gamma Contrast, Sharpen, Brightness.
• Optimization: Adam optimizer with a custom learning rate scheduler (following the “Attention is all you need” paper).

Models

The final architecture is an Image-to-SELFIES Transformer.

• Encoder (Feature Extractor):
  • EfficientNet-B3 (pre-trained on Noisy-student).
  • Input: $299 \times 299 \times 3$ images (normalized -1 to 1).
  • Output Feature Vector: $10 \times 10 \times 1536$.
• Decoder (Transformer):
  • 4 Encoder-Decoder layers.
  • 8 Parallel Attention Heads.
  • Dimension size: 512.
  • Feed-forward size: 2048.
  • Dropout: 0.1.

Evaluation

Evaluation was performed on a held-out test set (10% of total data) selected via RDKit MaxMin algorithm for diversity.

Hardware

• Training Hardware: TPU v3-8 (Google Cloud). TPU v3-32 was tested but v3-8 was chosen for cost-effectiveness.
• Comparison Hardware: Nvidia Tesla V100 (32GB GPU).
• Performance:
  • TPU v3-8 was ~4x faster than V100 GPU.
  • 1 Million molecule model convergence: ~8.5 hours on TPU vs ~30 hours on GPU.
  • Largest model (35M) took < 14 days on TPU.

Citation

@article{rajanDECIMER10Deep2021,
  title = {DECIMER 1.0: Deep Learning for Chemical Image Recognition Using Transformers},
  shorttitle = {DECIMER 1.0},
  author = {Rajan, Kohulan and Zielesny, Achim and Steinbeck, Christoph},
  year = {2021},
  month = {dec},
  journal = {Journal of Cheminformatics},
  volume = {13},
  number = {1},
  pages = {61},
  issn = {1758-2946},
  doi = {10.1186/s13321-021-00538-8},
  url = {https://doi.org/10.1186/s13321-021-00538-8}
}

Citation: Weir, H., Thompson, K., Woodward, A., Choi, B., Braun, A., & Martínez, T. J. (2021). ChemPix: Automated Recognition of Hand-Drawn Hydrocarbon Structures Using Deep Learning. Chemical Science, 12(31), 10622-10633. https://doi.org/10.1039/D1SC02957F

Publication: Chemical Science 2021

Additional Resources:

• GitHub Repository

Paper Classification and Core Contribution

This is primarily a Method paper, with a secondary contribution as a Resource paper.

The paper’s core contribution is the ChemPix architecture and training strategy using neural image captioning (CNN-LSTM) to convert hand-drawn chemical structures to SMILES. The extensive ablation studies on synthetic data generation (augmentation, degradation, backgrounds) and ensemble learning strategies confirm the methodological focus. The secondary resource contribution includes releasing a curated dataset of hand-drawn hydrocarbons and code for generating synthetic training data.

The Structural Input Bottleneck in Computational Chemistry

Inputting molecular structures into computational chemistry software for quantum calculations is often a bottleneck, requiring domain expertise and cumbersome manual entry in drawing software. While optical chemical structure recognition (OCSR) tools exist, they typically struggle with the noise and variability of hand-drawn sketches. There is a practical need for a tool that allows chemists to simply photograph a hand-drawn sketch and immediately convert it into a machine-readable format (SMILES), making computational workflows more accessible.

CNN-LSTM Image Captioning and Robust Synthetic Generalization

1. Image Captioning Paradigm: The authors treat the problem as neural image captioning, using an encoder-decoder (CNN-LSTM) framework to “translate” an image directly to a SMILES string. This avoids the complexity of explicit atom/bond detection and graph assembly.
2. Synthetic Data Engineering: The paper introduces a rigorous synthetic data generation pipeline that transforms clean RDKit-generated images into “pseudo-hand-drawn” images via randomized backgrounds, degradation, and heavy augmentation. This allows the model to achieve >50% accuracy on real hand-drawn data without ever seeing it during pre-training.
3. Ensemble Uncertainty Estimation: The method utilizes a “committee” (ensemble) of networks to improve accuracy and estimate confidence based on vote agreement, providing users with reliability indicators for predictions.

Extensive Ablation and Real-World Evaluation

1. Ablation Studies on Data Pipeline: The authors trained models on datasets generated at different stages of the pipeline (Clean RDKit $\rightarrow$ Augmented $\rightarrow$ Backgrounds $\rightarrow$ Degraded) to quantify the value of each transformation in bridging the synthetic-to-real domain gap.
2. Sample Size Scaling: They analyzed performance scaling by training on synthetic dataset sizes ranging from 50,000 to 500,000 images to understand data requirements.
3. Real-world Validation: The model was evaluated on a held-out test set of hand-drawn images collected via a custom web app, providing genuine out-of-distribution testing.
4. Fine-tuning Experiments: Comparisons of “zero-shot” performance (training only on synthetic data) versus fine-tuning with a small fraction of real hand-drawn data to assess the value of limited real-world supervision.

State-of-the-Art Hand-Drawn OCSR Performance

1. Pipeline Efficacy: Augmentation and image degradation were the most critical factors for generalization, improving accuracy from ~8% to nearly 50% on hand-drawn data. Adding backgrounds had a surprisingly negligible effect compared to degradation.

2. State-of-the-Art Performance: The final ensemble model achieved 76% accuracy (top-1) and 86% accuracy (top-3) on the hand-drawn test set, demonstrating practical viability for real-world use.

3. Synthetic Generalization: A model trained on 500,000 synthetic images achieved >50% accuracy on real hand-drawn data without any fine-tuning, validating the synthetic data generation strategy as a viable alternative to expensive manual labeling.

4. Ensemble Benefits: The voting committee approach improved accuracy and provided interpretable uncertainty estimates through vote distributions.

Reproducibility Details

Data

The study relies on two primary data sources: a massive synthetic dataset generated procedurally and a smaller collected dataset of real drawings.

Data Generation Parameters:

• Augmentations: Rotation, Resize ($200-300px$), Blur, Dilate, Erode, Aspect Ratio, Affine transform ($\pm 20px$), Contrast, Quantize, Sharpness
• Backgrounds: Randomly translated $\pm 100$ pixels and reflected

Algorithms

Ensemble Voting
A committee of networks casts votes for the predicted SMILES string. The final prediction is the one with the highest vote count. Validity of SMILES is checked using RDKit.

Beam Search
Used in the decoding layer with a beam width of $k=5$ to explore multiple potential SMILES strings. It approximates the sequence $\mathbf{\hat{y}}$ that maximizes the joint probability:

$$ \mathbf{\hat{y}} = \arg\max_{\mathbf{y}} \sum_{t=1}^{T} \log P(y_t \mid y_{<t}, \mathbf{x}) $$

Optimization:

• Optimizer: Adam

• Learning Rate: $1 \x10^{-4}$

• Batch Size: 20

• Loss Function: Cross-entropy loss across the sequence of $T$ tokens, computed as:

  $$ \mathcal{L} = -\sum_{t=1}^{T} \log P(y_t \mid y_{<t}, \mathbf{x}) $$

  where $\mathbf{x}$ is the image representation and $y_t$ is the predicted SMILES character. This is calculated as perplexity for validation.

Models

The architecture is a standard image captioning model (Show, Attend and Tell style) adapted for chemical structures.

Encoder (CNN):

• Input: 256x256 images (implicit from scaling operations)
• Structure: 4 blocks of Conv2D + MaxPool
  • Block 1: 64 filters, (3,3) kernel
  • Block 2: 128 filters, (3,3) kernel
  • Block 3: 256 filters, (3,3) kernel
  • Block 4: 512 filters, (3,3) kernel
• Activation: ReLU throughout

Decoder (LSTM):

• Hidden Units: 512
• Embedding Dimension: 80
• Attention: Mechanism with intermediary vector dimension of 512

Evaluation

• Primary Metric: Exact SMILES match accuracy (character-by-character identity between predicted and ground truth SMILES)
• Perplexity: Used for saving model checkpoints (minimizing uncertainty)
• Top-k Accuracy: Reported for $k=1$ (76%) and $k=3$ (86%)

Citation

@article{weir2021chempix,
  title={ChemPix: Automated Recognition of Hand-Drawn Hydrocarbon Structures Using Deep Learning},
  author={Weir, Hayley and Thompson, Keiran and Woodward, Amelia and Choi, Benjamin and Braun, Augustin and Mart{\'i}nez, Todd J.},
  journal={Chemical Science},
  volume={12},
  number={31},
  pages={10622--10633},
  year={2021},
  publisher={Royal Society of Chemistry},
  doi={10.1039/D1SC02957F}
}

Method. The paper proposes a novel architectural framework (ABC-Net) for Optical Chemical Structure Recognition (OCSR). It reformulates the problem from image captioning (sequence generation) to keypoint estimation (pixel-wise detection), backed by ablation studies on noise and comparative benchmarks against state-of-the-art tools.

Motivation for Keypoint-Based OCSR

• Inefficiency of Rule-Based Methods: Traditional tools (OSRA, MolVec) rely on hand-coded rules that are brittle, require domain expertise, and fail to handle the wide variance in molecular drawing styles.
• Data Inefficiency of Captioning Models: Recent Deep Learning approaches (like DECIMER, Img2mol) treat OCSR as image captioning (Image-to-SMILES). This is data-inefficient because canonical SMILES require learning traversal orders, necessitating millions of training examples.
• Goal: To create a scalable, data-efficient model that predicts graph structures directly by detecting atomic/bond primitives.

ABC-Net’s Divide-and-Conquer Architecture

• Divide-and-Conquer Strategy: ABC-Net breaks the problem down into detecting atom centers and bond centers as independent keypoints.
• Keypoint Estimation: It leverages a Fully Convolutional Network (FCN) to generate heatmaps for object centers. This is inspired by computer vision techniques like CornerNet and CenterNet.
• Angle-Based Bond Detection: To handle overlapping bonds, the model classifies bond angles into 60 distinct bins ($0-360°$) at detected bond centers, allowing separation of intersecting bonds.
• Implicit Hydrogen Prediction: The model explicitly predicts the number of implicit hydrogens for aromatic heteroatoms to resolve ambiguity in dearomatization.

Experimental Setup and Synthetic Data

• Dataset Construction: Synthetic dataset of 100,000 molecules from ChEMBL, rendered using two different engines (RDKit and Indigo) to ensure style diversity.
• Baselines: Compared against two rule-based methods (MolVec, OSRA) and one deep learning method (Img2mol).
• Robustness Testing: Evaluated on the external UOB dataset (real-world images) and synthetic images with varying levels of salt-and-pepper noise (up to $p=0.6$).
• Data Efficiency: Analyzed performance scaling with training set size (10k to 160k images).

Results, Generalization, and Noise Robustness

• Superior Accuracy: ABC-Net achieved 94-98% accuracy on synthetic test sets, significantly outperforming MolVec (<50%), OSRA (~61%), and Img2mol (~89-93%).
• Generalization: On the external UOB benchmark, ABC-Net achieved >95% accuracy, whereas the deep learning baseline (Img2mol) dropped to 78.2%, indicating better generalization.
• Data Efficiency: The model reached ~95% performance with only 80,000 training images, proving it requires orders of magnitude less data than captioning-based models (which often use millions).
• Noise Robustness: Performance remained stable (<2% drop) with noise levels up to $p=0.1$. Even at extreme noise ($p=0.6$), Tanimoto similarity remained high, suggesting the model recovers most substructures even when exact matches fail.

Reproducibility Details

Artifacts

Reproducibility Status: Partially Reproducible. The code is provided, but key components like the pre-trained weights, exact training environment dependencies, and the generated synthetic datasets are missing from the open-source release, making exact reproduction difficult.

Data

The authors constructed a synthetic dataset because labeled pixel-wise OCSR data is unavailable.

• Source: ChEMBL database
• Filtering: Excluded molecules with >50 non-H atoms or rare atom types/charges (<1000 occurrences).
• Sampling: 100,000 unique SMILES selected such that every atom type/charge appears in at least 1,000 compounds.
• Generation: Images generated via RDKit and Indigo libraries.
  • Augmentation: Varied bond thickness, label mode, orientation, and aromaticity markers.
  • Resolution: $512 \times 512$ pixels.
  • Noise: Salt-and-pepper noise added during training ($P$ = prob of background flip, $Q = 50P$).

Algorithms

1. Keypoint Detection (Heatmaps)

• Down-sampling: Input $512 \times 512$ → Output $102 \times 102$ (stride 5).

• Label Softening: To handle discretization error, ground truth peaks are set to 1, first-order neighbors to 0.95, others to 0.

• Loss Function: Penalty-reduced pixel-wise binary focal loss (variants of CornerNet loss). The loss formulation is given as:

  $$ L_{det} = - \frac{1}{N} \sum_{c,x,y} \begin{cases} (1 - \hat{Y}_{c,x,y})^{\alpha} \log(\hat{Y}_{c,x,y}) & \text{if } Y_{c,x,y} = 1 \\ (1 - Y_{c,x,y})^{\beta} (\hat{Y}_{c,x,y})^{\alpha} \log(1 - \hat{Y}_{c,x,y}) & \text{otherwise} \end{cases} $$

  • $\alpha=2$ (focal parameter), $\beta=4$ (penalty reduction).
  • Weight balancing: Classes <10% frequency weighted 10x.

2. Bond Direction Classification

• Angle Binning: $360°$ divided into 60 intervals.
• Inference: A bond is detected if the angle probability is a local maximum and exceeds a threshold.
• Non-Maximum Suppression (NMS): Required for opposite angles (e.g., $30°$ and $210°$) representing the same non-stereo bond.

3. Multi-Task Weighting

• Uses Kendall’s uncertainty weighting to balance 8 different loss terms (atom det, bond det, atom type, charge, H-count, bond angle, bond type, bond length).

Models

Architecture: ABC-Net (Custom U-Net / FCN)

• Input: $512 \times 512 \times 1$ (Grayscale).
• Contracting Path: 5 steps. Each step has conv-blocks + $2 \times 2$ MaxPool.
• Expansive Path: 3 steps. Transpose-Conv upsampling + Concatenation (Skip Connections).
• Heads: Separate $1 \times 1$ convs for each task map (Atom Heatmap, Bond Heatmap, Property Maps).
• Output Dimensions:
  • Heatmaps: $(1, 102, 102)$
  • Bond Angles: $(60, 102, 102)$
• Pre-trained Weights: Stated as available in paper, but missing from the public GitHub repository.

Evaluation

Metrics:

• Detection: Precision & Recall (Object detection level).
• Regression: Mean Absolute Error (MAE) for bond lengths.
• Structure Recovery:
  • Accuracy: Exact SMILES match rate.
  • Tanimoto: ECFP similarity (fingerprint overlap).

Hardware

• Training Configuration: 15 epochs, Batch size 64.
• Optimization: Adam Optimizer. LR $2.5 \times 10^{-4}$ (first 5 epochs) → $2.5 \times 10^{-5}$ (last 10).
• Compute: “High-Performance Computing Center” mentioned, specific GPU model not listed, but method described as “efficient” on GPU.

Paper Information

Citation: Zhang, X.-C., Yi, J.-C., Yang, G.-P., Wu, C.-K., Hou, T.-J., & Cao, D.-S. (2022). ABC-Net: A divide-and-conquer based deep learning architecture for SMILES recognition from molecular images. Briefings in Bioinformatics, 23(2), bbac033. https://doi.org/10.1093/bib/bbac033

Publication: Briefings in Bioinformatics 2022

Additional Resources:

• GitHub Repository

Citation

@article{zhangABCNetDivideandconquerBased2022,
  title = {ABC-Net: A Divide-and-Conquer Based Deep Learning Architecture for {SMILES} Recognition from Molecular Images},
  author = {Zhang, Xiao-Chen and Yi, Jia-Cai and Yang, Guo-Ping and Wu, Cheng-Kun and Hou, Ting-Jun and Cao, Dong-Sheng},
  journal = {Briefings in Bioinformatics},
  volume = {23},
  number = {2},
  pages = {bbac033},
  year = {2022},
  publisher = {Oxford University Press},
  doi = {10.1093/bib/bbac033}
}

Citation: Clevert, D.-A., Le, T., Winter, R., & Montanari, F. (2021). Img2Mol - accurate SMILES recognition from molecular graphical depictions. Chemical Science, 12(42), 14174-14181. https://doi.org/10.1039/D1SC01839F

Publication: Chemical Science (2021)

Additional Resources:

• GitHub Repository
• Paper on Royal Society of Chemistry

Method Classification

This is a method paper that introduces Img2Mol, a deep learning system for Optical Chemical Structure Recognition (OCSR). The work focuses on building a fast, accurate, and robust system for converting molecular structure depictions into machine-readable SMILES strings.

Systematization and Motivation

Vast amounts of chemical knowledge exist only as images in scientific literature and patents, making this data inaccessible for computational analysis, database searches, or machine learning pipelines. Manually extracting this information is slow and error-prone, creating a bottleneck for drug discovery and chemical research.

While rule-based OCSR systems like OSRA, MolVec, and Imago exist, they are brittle. Small variations in drawing style or image quality can cause them to fail. The authors argue that a deep learning approach, trained on diverse synthetic data, can generalize better across different depiction styles and handle the messiness of real-world images more reliably.

Two-Stage Architecture and Core Novelty

The novelty lies in a two-stage architecture that separates perception from decoding, combined with aggressive data augmentation to ensure robustness. The key contributions are:

1. Two-Stage Architecture with CDDD Embeddings

Img2Mol uses an intermediate representation to predict SMILES from pixels. A custom CNN encoder maps the input image to a 512-dimensional Continuous and Data-Driven Molecular Descriptor (CDDD) embedding - a pre-trained, learned molecular representation that smoothly captures chemical similarity. A pre-trained decoder then converts this CDDD vector into the final canonical SMILES string.

This two-stage design has several advantages:

• The CDDD space is continuous and chemically meaningful, so nearby embeddings correspond to structurally similar molecules. This makes the regression task easier than learning discrete token sequences directly.
• The decoder is pre-trained and fixed, so the CNN only needs to learn the image → CDDD mapping. This decouples the visual recognition problem from the sequence generation problem.
• CDDD embeddings naturally enforce chemical validity constraints, reducing the risk of generating nonsensical structures.

2. Extensive Data Augmentation for Robustness

The model was trained on 11.1 million unique molecules from ChEMBL and PubChem, but the critical insight is how the training images were generated. To expose the CNN to maximum variation in depiction styles, the authors:

• Used three different cheminformatics libraries (RDKit, OEChem, Indigo) to render images, each with its own drawing conventions
• Applied wide-ranging augmentations: varying bond thickness, font size, rotation, resolution (190-2500 pixels), and other stylistic parameters
• Over-sampled larger molecules to improve performance on complex structures, which are underrepresented in chemical databases

This ensures the network rarely sees the same depiction of a molecule twice, forcing it to learn invariant features.

3. Fast Inference

Because the architecture is a simple CNN followed by a fixed decoder, inference is very fast - especially compared to rule-based systems that rely on iterative graph construction algorithms. This makes Img2Mol practical for large-scale document mining.

Experimental Validation and Benchmarks

The evaluation focused on demonstrating that Img2Mol is more accurate, robust, and generalizable than existing rule-based systems:

1. Benchmark Comparisons: Img2Mol was tested on several standard OCSR benchmarks - USPTO (patent images), University of Birmingham (UoB), and CLEF datasets - against three open-source baselines: OSRA, MolVec, and Imago. Notably, no deep learning baselines were available at the time for comparison.

2. Resolution and Molecular Size Analysis: The initial model, Img2Mol(no aug.), was evaluated across different image resolutions and molecule sizes (measured by number of atoms) to understand failure modes. This revealed that:

   • Performance degraded for molecules with >35 atoms
   • Very high-resolution images lost detail when downscaled to the fixed input size
   • Low-resolution images (where rule-based methods failed completely) were handled well

3. Data Augmentation Ablation: A final model, Img2Mol, was trained with the full augmentation pipeline (wider resolution range, over-sampling of large molecules). Performance was compared to the initial version to quantify the effect of augmentation.

4. Depiction Library Robustness: The model was tested on images generated by each of the three rendering libraries separately to confirm that training on diverse styles improved generalization.

5. Generalization Tests: Img2Mol was evaluated on real-world patent images from the STAKER dataset, which were not synthetically generated. This tested whether the model could transfer from synthetic training data to real documents.

6. Hand-Drawn Molecule Recognition: As an exploratory test, the authors evaluated performance on hand-drawn molecular structures - a task the model was never trained for - to see if the learned features could generalize to completely different visual styles.

7. Speed Benchmarking: Inference time was measured and compared to rule-based baselines to demonstrate the practical efficiency of the approach.

Results, Conclusions, and Limitations

• Substantial Performance Gains: Img2Mol outperformed all three rule-based baselines on nearly every benchmark. Accuracy was measured both as exact SMILES match and as Tanimoto similarity (a chemical fingerprint-based metric that measures structural similarity). Even when Img2Mol didn’t predict the exact molecule, it often predicted a chemically similar one.

• Robustness Across Conditions: The full Img2Mol model (with aggressive augmentation) showed consistent performance across all image resolutions and molecule sizes. In contrast, rule-based systems were “brittle” - performance dropped sharply with minor perturbations to image quality or style.

• Depiction Library Invariance: Img2Mol’s performance was stable across all three rendering libraries (RDKit, OEChem, Indigo), validating the multi-library training strategy. Rule-based methods struggled particularly with RDKit-generated images.

• Strong Generalization to Real-World Data: Despite being trained exclusively on synthetic images, Img2Mol performed well on real patent images from the STAKER dataset. This suggests the augmentation strategy successfully captured the diversity of real-world depictions.

• Overfitting in Baselines: Rule-based methods performed surprisingly well on older benchmarks (USPTO, UoB, CLEF) but failed on newer datasets (Img2Mol’s test set, STAKER). This suggests they may be implicitly tuned to specific drawing conventions in legacy datasets.

• Limited Hand-Drawn Recognition: Img2Mol could recognize simple hand-drawn structures but struggled with complex or large molecules. This is unsurprising given the lack of hand-drawn data in training, but it highlights a potential avenue for future work.

• Speed Advantage: Img2Mol was substantially faster than rule-based competitors, especially on high-resolution images. This makes it practical for large-scale literature mining.

The work establishes that deep learning can outperform traditional rule-based OCSR systems when combined with a principled two-stage architecture and comprehensive data augmentation. The CDDD embedding acts as a bridge between visual perception and chemical structure, providing a chemically meaningful intermediate representation that improves both accuracy and robustness. The focus on synthetic data diversity proves to be an effective strategy for generalizing to real-world documents.

Reproducibility Details

Models

Architecture: Custom 8-layer Convolutional Neural Network (CNN) encoder

• Input: $224 \times 224$ pixel grayscale images
• Backbone Structure: 8 convolutional layers organized into 3 stacks, followed by 3 fully connected layers
  • Stack 1: 3 Conv layers ($7 \times 7$ filters, stride 3, padding 4) + Max Pooling
  • Stack 2: 2 Conv layers + Max Pooling
  • Stack 3: 3 Conv layers + Max Pooling
  • Head: 3 fully connected layers
• Output: 512-dimensional CDDD embedding vector

Decoder: Pre-trained CDDD decoder (from Winter et al.) - fixed during training, not updated

Algorithms

Loss Function: Mean Squared Error (MSE) regression minimizing the distance between the predicted and true embeddings:

$$ l(d) = l(\text{cddd}_{\text{true}} - \text{cddd}_{\text{predicted}}) $$

Optimizer: AdamW with initial learning rate $10^{-4}$

Training Schedule:

• Batch size: 256
• Training duration: 300 epochs
• Plateau scheduler: Multiplies learning rate by 0.7 if validation loss plateaus for 10 epochs
• Early stopping: Triggered if no improvement in validation loss for 50 epochs

Noise Tolerance: The decoder requires the CNN to predict embeddings with noise level $\sigma \le 0.15$ to achieve >90% accuracy

Data

Training Data: 11.1 million unique molecules from ChEMBL and PubChem

Splits: Approximately 50,000 examples each for validation and test sets

Synthetic Image Generation:

• Three cheminformatics libraries: RDKit, OEChem, and Indigo
• Augmentations: Resolution (190-2500 pixels), rotation, bond thickness, font size
• Salt stripping: Keep only the largest fragment
• Over-sampling: Larger molecules (>35 atoms) over-sampled to improve performance

Evaluation

Metrics:

• Exact SMILES match accuracy
• Tanimoto similarity (chemical fingerprint-based structural similarity)

Benchmarks:

• USPTO (patent images)
• University of Birmingham (UoB)
• CLEF dataset
• STAKER (real-world patent images)
• Hand-drawn molecular structures (exploratory)

Baselines: OSRA, MolVec, Imago (rule-based systems)

Hardware

⚠️ Unspecified in paper or supplementary materials. Inference speed reported as ~4 minutes for 5000 images; training hardware (GPU model, count) is undocumented.

Known Limitations

Molecular Size: Performance degrades for molecules with >35 atoms. This is partly a property of the CDDD latent space itself - for larger molecules, the “volume of decodable latent space” shrinks, making the decoder more sensitive to small noise perturbations in the predicted embedding.

Citation: Rajan, K., Zielesny, A. & Steinbeck, C. (2020). DECIMER: towards deep learning for chemical image recognition. Journal of Cheminformatics, 12(65). https://doi.org/10.1186/s13321-020-00469-w

Publication: Journal of Cheminformatics 2020

Additional Resources:

• Official GitHub Repository
• DECIMER Image-to-SMILES Repository

Contribution: Method for Optical Chemical Entity Recognition

This is primarily a Method ($\Psi_{\text{Method}}$) paper with a strong Resource ($\Psi_{\text{Resource}}$) component.

• Method: It proposes a novel architecture (DECIMER) that repurposes “show-and-tell” image captioning networks for Optical Chemical Entity Recognition (OCER), providing an alternative to traditional rule-based segmentation pipelines.
• Resource: It establishes a framework for generating massively scalable synthetic training data using open-source cheminformatics tools (CDK) and databases (PubChem), circumventing the scarcity of manually annotated chemical images.

Motivation: Brittleness of Heuristic Pipelines

The extraction of chemical structures from scientific literature (OCER) is critical for populating open-access databases. Traditional OCER systems (like OSRA or CLiDE) rely on complex multi-step pipelines involving vectorization, character recognition, and graph compilation. These systems are brittle and incorporating new structural features requires laborious engineering. Inspired by deep reinforcement learning systems like AlphaGo Zero, the authors sought to formulate an end-to-end deep learning approach that learns directly from data with minimal prior assumptions.

Novelty: Image Captioning for Molecular Graphs

• Image-to-Text Formulation: The paper frames chemical structure recognition as an image captioning problem, translating a bitmap image directly into a SMILES string using an encoder-decoder network. This bypasses explicit segmentation of atoms and bonds entirely.
• Synthetic Data Strategy: The authors generate synthetic images from PubChem using the CDK Structure Diagram Generator, scaling the dataset size to 15 million.
• Robust String Representations: The study performs key ablation experiments on string representations, comparing standard SMILES against DeepSMILES to evaluate how syntactic validity affects the network’s learning capability.

Experimental Setup and Validation Strategies

• Data Scaling: Models were trained on dataset sizes ranging from 54,000 to 15 million synthetic images to observe empirical scaling laws regarding accuracy and compute time.
• Representation Comparison: The authors compared the validity of predicted strings and recognition accuracy when training on SMILES versus DeepSMILES. The cross-entropy loss formulation for sequence generation can be represented as: $$ \mathcal{L} = -\sum_{t=1}^{T} \log P(y_t \mid y_{<t}, \mathbf{x}) $$ where $\mathbf{x}$ is the image representation and $y_t$ are the tokens of the SMILES/DeepSMILES string.
• Metric Evaluation: Performance was measured using Validity (syntactic correctness) and Tanimoto Similarity $T$, computed on molecular fingerprints to capture partial correctness even if the exact string prediction failed: $$ T(A, B) = \frac{|A \cap B|}{|A| + |B| - |A \cap B|} $$

Results and Critical Conclusions

• Data Representation: DeepSMILES proved superior to standard SMILES for training stability and output validity. Preliminary tests suggested SELFIES performs even better (0.78 Tanimoto vs 0.53 for DeepSMILES at 6M images).
• Scaling Behavior: Accuracy improves linearly with dataset size. The authors extrapolate that near-perfect detection would require training on 50 to 100 million structures.
• Current Limitations: At the reported training scale (up to 15M), the model does not yet rival traditional heuristic approaches, but the learning curve suggests it is a viable trajectory given sufficient compute and data.

Reproducibility Details

Data

The training data is synthetic, generated using the Chemistry Development Kit (CDK) Structure Diagram Generator (SDG) based on molecules from PubChem.

Curation Rules (applied to PubChem data):

• Molecular weight < 1500 Daltons.
• Elements restricted to: C, H, O, N, P, S, F, Cl, Br, I, Se, B.
• No counter ions or charged groups.
• No isotopes (e.g., D, T).
• Bond count between 5 and 40.
• SMILES length < 40 characters.
• Implicit hydrogens only (except in functional groups).

Preprocessing:

• Images: Generated as 299x299 bitmaps to match Inception V3 input requirements.
• Augmentation: One random rotation applied per molecule; no noise or blurring added in this iteration.

Algorithms

• Architecture: "Show, Attend and Tell" (Attention-based Image Captioning).
• Optimization: Adam optimizer with learning rate 0.0005.
• Loss Function: Sparse Categorical Crossentropy.
• Training Loop: Trained for 25 epochs per model. Batch size of 640 images.

Models

The network is implemented in TensorFlow 2.0.

• Encoder: Inception V3 (Convolutional NN), used unaltered. Extracts feature vectors saved as NumPy arrays.
• Decoder: Gated Recurrent Unit (GRU) based Recurrent Neural Network (RNN) with soft attention mechanism.
• Embeddings: Image embedding dimension size of 600.

Evaluation

The primary metric is Tanimoto similarity (Jaccard index) on PubChem fingerprints, which is robust for measuring structural similarity even when exact identity is not reached.

Hardware

Training was performed on a single node.

• GPU: 1x NVIDIA Tesla V100.
• CPU: 2x Intel Xeon Gold 6230.
• RAM: 384 GB.
• Compute Time:
  • Linear scaling with data size.
  • 15 million structures took ~27 days (64,909s per epoch).
  • Projected time for 100M structures: ~4 months on single GPU.

Citation

@article{rajanDECIMERDeepLearning2020,
  title = {{{DECIMER}}: Towards Deep Learning for Chemical Image Recognition},
  shorttitle = {{{DECIMER}}},
  author = {Rajan, Kohulan and Zielesny, Achim and Steinbeck, Christoph},
  year = 2020,
  month = dec,
  journal = {Journal of Cheminformatics},
  volume = {12},
  number = {1},
  pages = {65},
  issn = {1758-2946},
  doi = {10.1186/s13321-020-00469-w}
}

Citation: Oldenhof, M., Arany, A., Moreau, Y., & Simm, J. (2020). ChemGrapher: Optical Graph Recognition of Chemical Compounds by Deep Learning. Journal of Chemical Information and Modeling, 60(10), 4506-4517. https://doi.org/10.1021/acs.jcim.0c00459

Publication: Journal of Chemical Information and Modeling 2020 (arXiv preprint Feb 2020)

Additional Resources:

• arXiv Page

Classifying the Methodology

This is a Method paper. It proposes a novel deep learning architecture and a specific graph-reconstruction algorithm to solve the problem of Optical Chemical Structure Recognition (OCSR). It validates this method by comparing it against the existing standard tool (OSRA), demonstrating superior performance on specific technical challenges like stereochemistry.

The OCR Stereochemistry Challenge

Chemical knowledge is frequently locked in static images within scientific publications. Extracting this structure into machine-readable formats (graphs, SMILES) is essential for drug discovery and database querying. Existing tools, such as OSRA, rely on optical character recognition (OCR) and expert systems or hand-coded rules. These tools struggle with bond multiplicity and stereochemical information, often missing atoms or misinterpreting 3D cues (wedges and dashes). A machine learning approach allows for improvement via data scaling.

Decoupled Semantic Segmentation and Classification Pipeline

The core novelty is the segmentation-classification pipeline which decouples object detection from type assignment:

1. Semantic Segmentation: The model first predicts pixel-wise maps for atoms, bonds, and charges using a modified U-Net/dilated convolution architecture.
2. Graph Building Algorithm: A specific algorithm iterates over the segmentation maps to generate candidate locations for atoms and bonds.
3. Refinement via Classification: Dedicated classification networks take cutouts of the original image combined with the segmentation mask to verify and classify each candidate (e.g., distinguishing a single bond from a double bond, or a wedge from a dash).

Additionally, the authors developed a novel method for synthetic data generation by modifying the source code of RDKit to output pixel-wise labels during the image drawing process. This solves the lack of labeled training data.

Evaluating Synthetics and Benchmarks

• Synthetic Benchmarking: The authors generated test sets in 3 different stylistic variations. For each style, they tested on both stereo (complex 3D information) and non-stereo compounds.
• Baseline Comparison: They compared the error rates of ChemGrapher against OSRA (Optical Structure Recognition Application).
• Ablation Study: They analyzed the F1 scores of the segmentation networks versus the classification networks independently to understand where errors propagated.
• Real-world Case Study: They manually curated 61 images cut from journal articles to test performance on real, non-synthetic data.

Advancements Over OSRA

• Superior Accuracy: ChemGrapher consistently achieved lower error rates than OSRA across all synthetic styles, particularly for stereochemical information (wedge and dash bonds).
• Component Performance: The classification networks showed significantly higher F1 scores than the segmentation networks. This suggests the two-stage approach allows the classifier to correct segmentation noise.
• Real-world Viability: In the manual case study, ChemGrapher correctly predicted 46 of 61 images, compared to 42 of 61 for OSRA.
• Limitations: The model struggles with thick bond lines and non-carbon atom labels in real-world images due to the bias in the synthetic training data.

Reproducibility Details

Data

The authors created a custom synthetic dataset using ChEMBL and RDKit, as no pixel-wise labeled dataset existed.

Algorithms

Algorithm 1: Graph Building

1. Segment: Apply segmentation network $s(x)$ to get maps $S^a$ (atoms), $S^b$ (bonds), $S^c$ (charges).
2. Atom Candidates: Identify candidate blobs in $S^a$.
3. Classify Atoms: For each candidate, crop the input image and segmentation map. Feed to $c_A$ and $c_C$ to predict Atom Type and Charge. Add to Vertex set $V$ if valid.
4. Bond Candidates: Generate all pairs of nodes in $V$ within $2 \times$ bond length distance.
5. Classify Bonds: For each pair, create a candidate mask (two rectangles meeting in the middle to encode directionality). Feed to $c_B$ to predict Bond Type (single, double, wedge, etc.). Add to Edge set $E$.

Models

The pipeline uses four distinct Convolutional Neural Networks (CNNs).

1. Semantic Segmentation Network ($s$)

• Architecture: 8-layer fully convolutional network (Dense Prediction Convolutional Network).
• Kernels: $3\times3$ for all layers.
• Dilation: Uses dilated convolutions to expand receptive field without losing resolution. Dilations: 1, 2, 4, 8, 8, 4, 2, 1.
• Input: Binary B/W image.
• Output: Multi-channel probability maps for Atom Types ($S^a$), Bond Types ($S^b$), and Charges ($S^c$).

2. Classification Networks ($c_A, c_B, c_C$)

• Purpose: Refines predictions on small image patches.
• Architecture: 5 convolutional layers followed by a MaxPool and 2 Fully Connected layers.
  • Layer 1: Depthwise separable convolution.
  • Layers 2-4: Dilated convolutions (factors 2, 4, 8).
  • Layer 5: Standard convolution + MaxPool ($124 \x124$).
• Inputs:
  • Crop of the binary image ($x^{cut}$).
  • Crop of the segmentation map ($S^{cut}$).
  • “Highlight” mask ($h_L$) indicating the specific candidate location (e.g., a dot for atoms, two rectangles for bonds).

Evaluation

• Metric: F1 Score for individual network performance (segmentation pixels and classification accuracy).
• Metric: Error Rate (percentage of incorrect graphs) for overall system. A graph is “incorrect” if there is at least one mistake in atoms or bonds.
• Baselines: Compared against OSRA (v2.1.0 implied by context of standard tools).

Hardware

• GPU: Training and inference performed on a single NVIDIA Titan Xp.

Citation

@article{oldenhof2020chemgrapher,
  title={ChemGrapher: Optical Graph Recognition of Chemical Compounds by Deep Learning},
  author={Oldenhof, Martijn and Arany, Adam and Moreau, Yves and Simm, Jaak},
  journal={Journal of Chemical Information and Modeling},
  volume={60},
  number={10},
  pages={4506--4517},
  year={2020},
  publisher={ACS Publications},
  doi={10.1021/acs.jcim.0c00459}
}

Citation: Hong, C., Du, X., & Zhang, L. (2015). Research on Chemical Expression Images Recognition. Proceedings of the 2015 Joint International Mechanical, Electronic and Information Technology Conference, 267-271. https://doi.org/10.2991/jimet-15.2015.50

Publication: JIMET 2015 (Atlantis Press)

Additional Resources:

• JSME Editor (used for visualization)

Contribution: New OCSR Workflow for Adhesion and Wedge Bonds

Method. The paper proposes a novel algorithmic pipeline (OCSR) for recognizing 2D organic chemical structures from images. It validates this method by comparing it against an existing tool (OSRA) using a quantitative metric (Tanimoto Coefficient) on a test set of 200 images.

Motivation: Challenges with Connecting Symbols and Stereochemistry

A vast amount of chemical structural information exists in scientific literature (PDFs/images) that is not machine-readable. Manually converting these images to formats like InChI or CML is labor-intensive. Existing tools face challenges with:

1. Adhesion: Poor separation when chemical symbols touch or overlap with bonds.
2. Stereochemistry: Incomplete identification of “real” (solid) and “virtual” (dashed/hashed) wedge bonds.

Core Innovation: Vector-Based Separation and Stereochemical Logic

The authors propose a specific OCSR (Optical Chemical Structure Recognition) workflow with two key technical improvements:

1. Vector-based Separation: The method vectorizes the image (using Potrace) to extract straight lines and curves, allowing better separation of “adhesive” chemical symbols (like H, N, O attached to bonds).
2. Stereochemical Logic: Specific rules for identifying wedge bonds:
   • Virtual (Dashed) Wedges: Identified by grouping connected domains and checking linear correlation of their center points.
   • Real (Solid) Wedges: Identified after thinning by analyzing linear correlation and width variance of line segments.

Methodology & Experimental Setup

• Dataset: 200 chemical structure images collected from the network.

• Baselines: Compared against OSRA (Optical Structure Recognition Application), a free online tool.

• Metric: Tanimoto Coefficient, measuring the similarity of the set of recognized bonds and symbols against the ground truth. The similarity $T(A, B)$ is defined as:

  $$ T(A, B) = \frac{|A \cap B|}{|A| + |B| - |A \cap B|} $$

Results & Conclusions

• Performance: The proposed OCSR method achieved higher recognition rates than OSRA.
  • Exact Match (100%): OCSR achieved 90.0% vs. OSRA’s 82.2%.
  • High Similarity (>85%): OCSR recognized 157 structures vs. OSRA’s 114.
• Limitations: The paper notes that “real wedge” and “virtual wedge” identification was a primary focus, but general recognition effectiveness still “has room for improvement”.

Reproducibility Details

Data

The study used a custom collection of images, not a standard benchmark.

Algorithms

The recognition pipeline follows these specific steps:

1. Preprocessing:
   • Grayscale: via cvCvtColor (OpenCV).
   • Binarization: via Otsu’s method.
2. Isolated Symbol Removal:
   • Identifies connected domains with aspect ratios in [0.8, 3.0].
   • Recognizes them using OCR (GOCR, OCRAD, Tesseract) and removes them from the image.
3. Virtual Wedge Recognition:
   • Groups small connected domains (points/clumps).
   • Calculates linear correlation of center points; if collinear, treats as a dashed bond.
4. Vectorization & Thinning:
   • Thinning: Rosenfeld algorithm (optimized) to reduce lines to single pixel width.
   • Vectorization: Uses Potrace to convert pixels to vector segments.
   • Merging: Combines split vector segments based on angle thresholds to form long straight lines.
5. Adhesive Symbol Separation:
   • Identifies curves (short segments after vectorization) attached to long lines.
   • Separates these domains and re-runs OCR.
6. “Super Atom” Merging:
   • Merges adjacent vertical/horizontal symbols (e.g., “HO”, “CH3”) based on distance thresholds between bounding boxes.

Models

The system relies on off-the-shelf OCR tools for character recognition; no custom ML models were trained.

• OCR Engines: GOCR, OCRAD, TESSERACT.
• Visualization: JSME (JavaScript Molecule Editor) used to render output strings.

Evaluation

Hardware

• Requirements: Unspecified; runs on standard CPU architecture (implied by use of standard libraries like OpenCV and Potrace).

Citation

@inproceedings{hongResearchChemicalExpression2015,
  title = {Research on {{Chemical Expression Images Recognition}}},
  booktitle = {Proceedings of the 2015 {{Joint International Mechanical}}, {{Electronic}} and {{Information Technology Conference}}},
  author = {Hong, Chen and Du, Xiaoping and Zhang, Lu},
  year = {2015},
  publisher = {Atlantis Press},
  address = {Chongqing, China},
  doi = {10.2991/jimet-15.2015.50},
  isbn = {978-94-6252-129-2}
}

Citation: Campos, D., & Ji, H. (2021). IMG2SMI: Translating Molecular Structure Images to Simplified Molecular-input Line-entry System (No. arXiv:2109.04202). arXiv. https://doi.org/10.48550/arXiv.2109.04202

Publication: arXiv preprint (2021)

Additional Resources:

• Paper on arXiv

Contributions & Taxonomy

This is both a Method and Resource paper:

• Method: It adapts standard image captioning architectures (encoder-decoder) to the domain of Optical Chemical Structure Recognition (OCSR), treating molecule recognition as a translation task.
• Resource: It introduces MOLCAP, a large-scale dataset of 81 million molecules aggregated from public chemical databases, addressing the data scarcity that previously hindered deep learning approaches to OCSR.

The Bottleneck in Chemical Literature Translation

Chemical literature is “full of recipes written in a language computers cannot understand” because molecules are depicted as 2D images. This creates a fundamental bottleneck:

• The Problem: Chemists must manually redraw molecular structures to search for related compounds or reactions. This is slow, error-prone, and makes large-scale literature mining impossible.
• Existing Tools: Legacy systems like OSRA (Optical Structure Recognition Application) rely on handcrafted rules and often require human correction, making them unfit for unsupervised, high-throughput processing.
• The Goal: An automated system that can translate structure images directly to machine-readable strings (SMILES/SELFIES) without human supervision, enabling large-scale knowledge extraction from decades of chemistry literature and patents.

Core Innovation: SELFIES and Image Captioning

The core novelty is demonstrating that how you represent the output text is as important as the model architecture itself. Key contributions:

1. Image Captioning Framework: Applies modern encoder-decoder architectures (ResNet-101 + Transformer) to OCSR, treating it as an image-to-text translation problem with a standard cross-entropy loss objective over the generation sequence: $$ \mathcal{L} = -\sum\limits_{t=1}^{T} \log P(y_t \mid y_1, \ldots, y_{t-1}, x) $$

2. SELFIES as Target Representation: The key mechanism relies on using SELFIES (Self-Referencing Embedded Strings) as the output format. SELFIES is based on a formal grammar where every possible string corresponds to a valid molecule, eliminating the syntactic invalidity problems (unmatched parentheses, invalid characters) that plague SMILES generation.

3. MOLCAP Dataset: Created a comprehensive dataset of 81 million unique molecules from PubChem, ChEMBL, GDB13, and other sources. Generated 256x256 pixel images using RDKit for 1 million training samples and 5,000 validation samples.

4. Task-Specific Evaluation: Demonstrated that traditional NLP metrics (BLEU) are poor indicators of scientific utility. Introduced evaluation based on molecular fingerprints (MACCS, RDK, Morgan) and Tanimoto similarity: $$ T(a, b) = \frac{c}{a + b - c} $$ where $c$ is the number of common fingerprint bits, and $a$ and $b$ are the number of set bits in each respective molecule’s fingerprint. This formulation reliably measures functional chemical similarity.

Experimental Setup and Ablation Studies

The evaluation focused on comparing IMG2SMI to existing systems and identifying which design choices matter most:

1. Baseline Comparisons: Benchmarked against OSRA (rule-based system) and DECIMER (first deep learning approach) on the MOLCAP dataset to establish whether modern architectures could surpass traditional methods.

2. Ablation Studies: Extensive ablations isolating key factors:

   • Decoder Architecture: Transformer vs. RNN/LSTM decoders
   • Encoder Fine-tuning: Fine-tuned vs. frozen pre-trained ResNet weights
   • Output Representation: SELFIES vs. character-level SMILES vs. BPE-tokenized SMILES (the most critical ablation)

3. Metric Analysis: Systematic comparison of evaluation metrics including BLEU, ROUGE, Levenshtein distance, exact match accuracy, and molecular fingerprint-based similarity measures.

Results, Findings, and Limitations

Performance Gains:

• 163% improvement over OSRA on MACCS Tanimoto similarity.
• Approximately 10x improvement on ROUGE scores.
• Average Tanimoto similarity exceeds 0.85 (functionally similar molecules even when not exact matches).

Key Findings:

• SELFIES is Critical: Using SELFIES yields 99.4% valid molecules, compared to only ~2% validity for character-level SMILES. This robustness is essential for practical deployment.
• Architecture Matters: Transformer decoder significantly outperforms RNN/LSTM approaches. Fine-tuning the ResNet encoder (vs. frozen weights) yields substantial performance gains (e.g., MACCS FTS: 0.76 to 0.94).
• Metric Insights: BLEU is a poor metric for this task. Molecular fingerprint-based Tanimoto similarity is most informative because it measures functional chemical similarity.

Limitations:

• Low Exact Match: Only 7.24% exact matches. The model captures the overarching functional groups and structure but misses fine details like exact double bond placement.
• Complexity Bias: Trained on large molecules (average length >40 tokens), so it performs poorly on very simple structures where OSRA still excels.

Conclusion: The work establishes that modern architectures combined with robust molecular representations (SELFIES) can significantly outperform traditional rule-based systems. The system is already useful for literature mining where functional similarity is more important than exact matches, though low exact match accuracy and poor performance on simple molecules indicate clear directions for future work.

Reproducibility Details

Models

Architecture: Image captioning system based on DETR (Detection Transformer) framework.

Visual Encoder:

• Backbone: ResNet-101 pre-trained on ImageNet
• Feature Extraction: 4th layer extraction (convolutions only)
• Output: 2048-dimensional dense feature vector

Caption Decoder:

• Type: Transformer encoder-decoder
• Layers: 3 stacked encoder layers, 3 stacked decoder layers
• Attention Heads: 8
• Hidden Dimensions: 2048 (feed-forward networks)
• Dropout: 0.1
• Layer Normalization: 1e-12

Training Configuration:

• Optimizer: AdamW
• Learning Rate: 5e-5 (selected after sweep from 1e-4 to 1e-6)
• Weight Decay: 1e-4
• Batch Size: 32
• Epochs: 5
• Codebase: Built on open-source DETR implementation

Data

MOLCAP Dataset:

Preprocessing:

• Images generated using RDKit at 256x256 resolution
• Molecules converted to canonical representations
• SELFIES tokenization for model output

Evaluation

Primary Metrics:

Secondary Metrics (shown to be less informative for chemical tasks):

• BLEU, ROUGE (better suited for natural language)
• Levenshtein distance (doesn’t capture chemical similarity)

Hardware

• GPU: Single NVIDIA GeForce RTX 2080 Ti
• Training Time: ~5 hours per epoch, approximately 24 hours total for 5 epochs
• Memory: Sufficient for batch size 32 with ResNet-101 + Transformer architecture

Citation

@article{campos2021img2smi,
  title={IMG2SMI: Translating Molecular Structure Images to Simplified Molecular-input Line-entry System},
  author={Campos, Daniel and Ji, Heng},
  journal={arXiv preprint arXiv:2109.04202},
  year={2021},
  doi={10.48550/arXiv.2109.04202}
}

In computational chemistry and AI drug discovery, visualization pipelines are often brittle; breaking on edge cases or failing silently when processing millions of molecules for training data.

I built molecular-string-renderer to treat molecular visualization as a strict software engineering problem. It is a highly configurable, fault-tolerant wrapper around RDKit that standardizes the conversion of text-based chemical representations (SMILES, InChI, SELFIES) into high-fidelity raster and vector graphics.

Features

This library differentiates itself from standard plotting scripts through strict architectural patterns designed for reliability:

1. Strategy Pattern for SVG Generation

RDKit’s vector rendering can sometimes fail on complex molecular topologies. I implemented a Hybrid Strategy that ensures the pipeline never crashes during batch processing:

• Vector Strategy: Attempts to generate a true, scalable vector graphic.
• Raster Fallback: If the vector engine fails, the system automatically renders a high-res PNG and embeds it transparently into the SVG container.

2. Native Generative AI Support

With the rise of Large Language Models in chemistry, SELFIES (Self-Referencing Embedded Strings) has become a standard output format. This library handles SELFIES natively, managing the decoding and sanitization lifecycle internally so that ML training loops can simply “pass strings and get images.”

3. Strict Configuration Contracts

The library uses Pydantic models (RenderConfig, ParserConfig, OutputConfig) to enforce strict data contracts. This ensures that visualization parameters are validated before any heavy computation begins, preventing runtime errors deep in a batch job.

Usage

The library provides a simple Python API for rendering single molecules or batches of molecules from various string formats.

Results

• Type Safety: The codebase runs with strict mypy settings, ensuring type safety across the entire pipeline.
• Grid Auto-Fitting: Implemented smart layout algorithms that automatically adjust grid dimensions based on the input batch size.
• Format Agnostic: Decouples the parsing logic (SMILES vs. MolBlock vs. SELFIES) from the rendering logic, making it trivial to add support for new proprietary formats.

Why This Matters

For AI research, infrastructure must be bulletproof. When generating 100,000 images for a diffusion model or visualizing the latent space of a chemical LLM, the pipeline must handle edge cases gracefully. This tool bridges the gap between academic scripts and production-grade data pipelines.

Related Work

• Visualizing SMILES and SELFIES Strings: walkthrough of the visualization pipeline this library implements
• Isomer Dataset Generation: related project generating molecular datasets using SMILES/SELFIES representations

Citation: Weininger, D. (1988). SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences, 28(1), 31-36. https://doi.org/10.1021/ci00057a005

Publication: Journal of Chemical Information and Computer Sciences, 1988

Additional Resources:

• SMILES notation overview - Modern usage summary
• Converting SMILES to 2D images - Practical visualization tutorial

Core Contribution: A String-Based Molecular Notation

This is a Method paper that introduces a novel notation system for representing chemical structures as text strings. It establishes the encoding rules and input conventions for SMILES (Simplified Molecular Input Line Entry System), while explicitly deferring the canonicalization algorithm to subsequent papers in the series.

The Computational Complexity of Chemical Information in the 1980s

As computers became central to chemical information processing in the 1980s, the field faced a fundamental problem: existing line notations were either too complex for chemists to use practically or too limited for computational applications. Previous systems required extensive training to write correctly and were prone to errors.

The goal was ambitious: create a system that could represent any molecule as a simple text string, making it both human-readable and machine-efficient. This would enable compact database storage, fast processing, and easy exchange between software systems.

Separating Input Rules from Canonicalization

Weininger’s key insight was to separate the problem into two parts: create simple, flexible rules that chemists could easily learn for input, while deferring to the computer the complex task of generating a unique, canonical representation. This division of labor made SMILES both practical and powerful.

The specific innovations include:

1. Simple input rules - Chemists could write molecules intuitively (e.g., CCO or OCC for ethanol)
2. Ring closure notation - Breaking one bond and marking ends with matching digits
3. Implicit hydrogens - Automatic calculation based on standard valences keeps strings compact
4. Algorithmic aromaticity detection - Automatic recognition of aromatic systems from Kekulé structures
5. Human-readable output - Unlike binary formats, SMILES strings are readable and debuggable

Important scope note: This first paper in the series establishes the input syntax and encoding rules. The canonicalization algorithm (how to generate unique SMILES) is explicitly stated as the subject of following papers: “specification of isomerisms, substructures, and unique SMILES generation are the subjects of following papers.”

Demonstrating Notation Rules Across Molecular Classes

The paper is primarily a specification document establishing notation rules. The methodology is demonstrated through worked examples showing how to encode various molecular structures:

• Basic molecules: Ethane (CC), ethylene (C=C), acetylene (C#C)
• Branches: Isobutyric acid (CC(C)C(=O)O)
• Rings: Cyclohexane (C1CCCCC1), benzene (c1ccccc1)
• Aromatic systems: Tropone (O=c1ccccc1), quinone (showing exocyclic bond effects)
• Complex structures: Morphine (40 characters vs 1000-2000 for connection tables)
• Edge cases: Salts, isotopes, charged species, tautomers

Performance comparisons are mentioned qualitatively: SMILES processing was approximately 100 times faster than traditional connection table methods on the hardware of the era (1988), with dramatic reductions in storage space.

Performance and Practical Viability

The paper successfully establishes SMILES as a practical notation system with several key outcomes:

Practical benefits:

• Compactness: 40 characters for morphine vs 1000-2000 for connection tables
• Speed: ~100x faster processing than traditional methods
• Accessibility: Simple enough for chemists to learn without extensive training
• Machine-friendly: Efficient parsing and string-based operations

Design principles validated:

• Separating user input from canonical representation makes the system both usable and rigorous
• Implicit hydrogens reduce string length without loss of information
• Ring closure notation with digit markers is more intuitive than complex graph syntax
• Automatic aromaticity detection handles most cases correctly

Acknowledged limitations:

• Canonicalization algorithm not included in this paper
• Stereochemistry handling deferred to subsequent papers
• Some edge cases (like unusual valence states) require explicit specification

The paper concludes by positioning SMILES as a foundation for database storage, substructure searching, and chemical informatics applications - a vision that proved accurate as SMILES became one of the most widely used molecular representations in computational chemistry.

Reproducibility Details

To implement the method described in this paper, the following look-up tables and algorithms are required. Note: These details are critical for replication but are often glossed over in high-level summaries.

1. The Valence Look-Up Table

To calculate implicit hydrogens, the system assumes the “lowest normal valence” greater than or equal to the explicit bond count. The paper explicitly defines these valences:

Example: For sulfur in $\text{H}_2\text{SO}_4$, the explicit bond count is 6 (four oxygens), so the system uses valence 6 with zero implicit hydrogens. Without knowing S allows valence 6, the algorithm would fail.

2. Explicit Hydrogen Requirements

The paper lists exactly three cases where hydrogen must be explicitly specified:

1. Hydrogen connected to zero atoms (rare edge case)
2. Hydrogen connected to other hydrogen (molecular hydrogen, $\text{H}_2$, written as [H][H])
3. Hydrogen connected to more than one atom (bridging hydrogens)
4. Isotopic hydrogen (deuterium [2H], tritium [3H])

For all other cases, hydrogens are implicit and calculated from the valence table.

3. Ring Closure Notation

Standard SMILES supports single digits 1-9 for ring closures. For rings numbered 10 and higher, the notation requires a percent sign prefix:

• Ring closures 1-9: C1CCCCC1
• Ring closures 10+: C%10CCCCC%10, C2%13%24 (ring 2, ring 13, ring 24)

Without this rule, a parser would fail on large polycyclic structures.

4. Aromaticity Detection Algorithm

The system uses an algorithmic version of Hückel’s Rule ($4N+2$ π-electrons). The “excess electron” count for the aromatic system is determined by these rules:

Carbon contribution:

• C in aromatic ring: Contributes 1 electron
• C double-bonded to exocyclic electronegative atom (e.g., $\text{C}=\text{O}$ in quinone): Contributes 0 electrons (the carbon “loses” its electron to the oxygen)

Heteroatom contribution:

• O, S in ring: Contributes 2 electrons (lone pair)
• N in ring: Contributes 1 electron (pyridine-like) or 2 electrons (pyrrole-like, must have explicit hydrogen [nH])

Charge effects:

• Positive charge: Reduces electron count by 1
• Negative charge: Increases electron count by 1

Critical example - Quinone:

O=C1C=CC(=O)C=C1

Quinone has 6 carbons in the ring, but the two carbons bonded to exocyclic oxygens contribute 0 electrons each. The four remaining carbons contribute 4 electrons total (not 6), so quinone is not aromatic by this algorithm. This exocyclic bond rule is essential for correct aromaticity detection.

Aromatic ring test:

1. All atoms must be sp² hybridized
2. Count excess electrons using the rules above
3. Calculate whether the system complies with Hückel’s parity rule constraint: $$ \text{Excess Electrons} \equiv 2 \pmod 4 \iff \text{Excess Electrons} = 4N + 2 $$ If the electron count satisfies this property for some integer $N$, the ring is determined to be aromatic.

Encoding Rules Reference

The following sections provide a detailed reference for the six fundamental SMILES encoding rules. These are the rules a user would apply when writing SMILES strings.

1. Atoms

Atoms use their standard chemical symbols. Elements in the “organic subset” (B, C, N, O, P, S, F, Cl, Br, I) can be written directly when they have their most common valence - so C automatically means a carbon with enough implicit hydrogens to satisfy its valence.

Everything else goes in square brackets: [Au] for gold, [NH4+] for ammonium ion, or [13C] for carbon-13. Aromatic atoms get lowercase letters: c for aromatic carbon in benzene.

2. Bonds

Bond notation is straightforward:

• - for single bonds (usually omitted)
• = for double bonds
• # for triple bonds
• : for aromatic bonds (also usually omitted)

So CC and C-C both represent ethane, while C=C is ethylene.

3. Branches

Branches use parentheses, just like in mathematical expressions. Isobutyric acid becomes CC(C)C(=O)O - the main chain is CC C(=O)O with a methyl (C) branch.

4. Rings

This is where SMILES gets clever. You break one bond and mark both ends with the same digit. Cyclohexane becomes C1CCCCC1 - the 1 connects the first and last carbon, closing the ring.

You can reuse digits for different rings in the same molecule, making complex structures manageable.

5. Disconnected Parts

Salts and other disconnected structures use periods. Sodium phenoxide: [Na+].[O-]c1ccccc1. The order doesn’t matter - you’re just listing the separate components.

6. Aromaticity

Aromatic rings can be written directly with lowercase letters. Benzoic acid becomes c1ccccc1C(=O)O. The system can also detect aromaticity automatically from Kekulé structures, so C1=CC=CC=C1C(=O)O works just as well.

Simplified Subset for Organic Chemistry

Weininger recognized that most chemists work primarily with organic compounds, so he defined a simplified subset that covers the vast majority of cases. For organic molecules, you only need four rules:

1. Atoms: Use standard symbols (C, N, O, etc.)
2. Multiple bonds: Use = and # for double and triple bonds
3. Branches: Use parentheses ()
4. Rings: Use matching digits

This “basic SMILES” covers probably 90% of what most chemists encounter daily, making the system immediately accessible without having to learn all the edge cases.

Design Decisions and Edge Cases

Beyond the basic rules, the paper established several important conventions for handling ambiguous cases:

Hydrogen Handling

Hydrogens are usually implicit - the system calculates how many each atom needs based on standard valences. So C represents CH₄, N represents NH₃, and so on. This keeps strings compact and readable.

Explicit hydrogens only appear in special cases: when hydrogen connects to multiple atoms, when you need to specify an exact count, or in isotopic specifications like [2H] for deuterium.

Bond Representation

The paper made an important choice about how to represent bonds in ambiguous cases. For example, nitro groups could be written as charge-separated C[N+]([O-])[O-] or with double bonds CN(=O)=O. Weininger chose to prefer covalent bonds when possible, keeping the topology symmetric and avoiding unusual charges.

However, when covalent representation would require unusual valences, charge separation is preferred. Diazomethane becomes C=[N+]=[N-] to avoid forcing carbon into an unrealistic valence state.

Tautomers

SMILES doesn’t try to be too clever about tautomers - it represents exactly what you specify. So 2-pyridone can be written as either the enol form Oc1ncccc1 or the keto form O=c1[nH]cccc1. The system won’t automatically convert between them.

This explicit approach means you need to decide which tautomeric form to represent, but it also means the notation precisely captures what you intend.

Aromaticity Detection

One of the most sophisticated parts of the original system was automatic aromaticity detection. The algorithm uses an extended Hückel rule: a ring is aromatic if all atoms are sp² hybridized and it contains 4N+2 π-electrons.

This means you can input benzene as the Kekulé structure C1=CC=CC=C1 and the system will automatically recognize it as aromatic and convert it to c1ccccc1. The algorithm handles complex cases like tropone (O=c1ccccc1) and correctly identifies them as aromatic.

Aromatic Nitrogen

The system makes an important distinction for nitrogen in aromatic rings. Pyridine-type nitrogen (like in pyridine itself) is written as n and has no attached hydrogens. Pyrrole-type nitrogen has an attached hydrogen that must be specified explicitly: [nH]1cccc1 for pyrrole.

This distinction captures the fundamental difference in electron contribution between these two nitrogen types in aromatic systems.

Impact and Legacy

Nearly four decades later, SMILES remains one of the most important file formats in computational chemistry. The notation became the foundation for:

• Database storage - Compact, searchable molecular representations
• Substructure searching - Pattern matching in chemical databases
• Property prediction - Input format for QSAR models
• Chemical informatics - Standard exchange format between software
• Modern ML - Text-based representation for neural networks

While newer approaches like SELFIES have addressed some limitations (like the possibility of invalid strings), SMILES’ combination of simplicity and power has made it enduringly useful.

The paper established both a notation system and a design philosophy: chemical informatics tools should be powerful enough for computers while remaining accessible to working chemists. That balance remains relevant today as we develop new molecular representations for machine learning and AI applications.

This is a Method paper that introduces a new molecular string representation designed specifically for machine learning applications.

Motivation: The Invalidity Bottleneck

When neural networks generate molecules using SMILES notation, a huge fraction of output strings are invalid: either syntax errors or chemically impossible structures. This was a fundamental bottleneck: if your generative model produces 70% invalid molecules, you’re wasting computational effort and severely limiting chemical space exploration.

Novelty: A Formal Grammar Approach

The authors’ key insight was using a formal grammar approach (specifically, a Chomsky type-2, context-free grammar with self-referencing functions) where each symbol is interpreted based on chemical context. The “state of the derivation” tracks available valence bonds, preventing impossible structures like a carbon with five single bonds.

For example, generating 2-Fluoroethenimine (FC=C=N) follows a state derivation where each step restricts the available valency for the next element:

$$ \mathbf{X}_0 \xrightarrow{[F]} \text{F } \mathbf{X}_1 \xrightarrow{[=C]} \text{FC } \mathbf{X}_3 \xrightarrow{[=C]} \text{FC=C } \mathbf{X}_2 \xrightarrow{[\#N]} \text{FC=C=N} $$

This approach guarantees 100% validity: every SELFIES string corresponds to a valid molecule, and every valid molecule can be represented.

Methodology & Experiments: Validating Robustness

The authors ran a convincing set of experiments to demonstrate SELFIES’ robustness:

Random Mutation Test

They took the SELFIES and SMILES representations of MDMA and introduced random changes:

• SMILES: After just one random mutation, only 9.9% of strings remained valid (dropping to 1.1% after three mutations).
• SELFIES: 100% of mutated strings still represented valid molecules (though different from the original).

This empirical difference demonstrates why SELFIES is highly effective for evolutionary algorithms and genetic programming approaches to molecular design.

Generative Model Performance

The real test came with actual machine learning models. The authors trained Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) on both representations:

VAE Results:

• SMILES-based VAE: Large invalid regions scattered throughout the latent space
• SELFIES-based VAE: Every point in the continuous latent space mapped to a valid molecule
• The SELFIES model encoded over 100 times more diverse molecules

GAN Results:

• Best SMILES GAN: 18.5% diverse, valid molecules
• Best SELFIES GAN: 78.9% diverse, valid molecules

Evaluation Metrics:

• Validity: Percentage of generated strings representing valid molecular structures
• Diversity: Number of unique valid molecules produced
• Reconstruction Accuracy: How well the autoencoder reproduced input molecules (detailed in Table 1)

Scalability Test

The authors proved SELFIES works beyond toy molecules by successfully encoding and decoding all 72 million molecules from the PubChem database, demonstrating practical applicability to real chemical databases.

Results & Conclusions: Chemical Space Exploration

Key Findings:

• SELFIES achieves 100% validity guarantee: every string represents a valid molecule
• SELFIES-based VAEs encode over 100x more diverse molecules than SMILES-based models
• SELFIES-based GANs produce 78.9% diverse valid molecules vs. 18.5% for SMILES GANs
• Successfully validated on all 72 million PubChem molecules

Limitations Acknowledged:

• No direct canonicalization method (initially required SMILES conversion)
• Missing features: aromaticity, isotopes, complex stereochemistry
• Requires community testing and adoption

Impact:

This work demonstrated that designing ML-native molecular representations could unlock new capabilities in drug discovery and materials science. The paper sparked broader conversation about representation design and showed the effectiveness of addressing format limitations directly. SELFIES was subsequently evaluated as an alternative input representation to SMILES in ChemBERTa, a transformer pretrained on molecular strings for property prediction, where it performed comparably to SMILES on the Tox21 benchmark, though the comparison was limited to a single task.

Reproducibility Details

Data

The machine learning experiments used two distinct datasets:

• QM9 (134k molecules): Primary training dataset for VAE and GAN models
• PubChem (72M molecules): Used only to test representation coverage and scalability; not used for model training

Models

The VAE implementation included:

• Latent space: 241-dimensional with Gaussian distributions
• Input encoding: One-hot encoding of SELFIES/SMILES strings
• Full architectural details (encoder/decoder structures, layer types) provided in Supplementary Information

Algorithms

The authors found GAN performance was highly sensitive to hyperparameter selection:

• Searched 200 different hyperparameter configurations to achieve the reported 78.9% diversity
• Specific optimizers, learning rates, and training duration detailed in Supplementary Information
• Full rule generation algorithm provided in Table 2

Evaluation

All models evaluated on:

• Validity rate: Percentage of syntactically and chemically valid outputs
• Diversity: Count of unique valid molecules generated
• Reconstruction accuracy: Fidelity of autoencoder reconstruction (VAEs only)

Hardware

• Training performed on the SciNet supercomputing infrastructure.
• Specific GPU/compute details (typically NVIDIA P100/V100 nodes for SciNet during this period) are provided in the Supplementary Information.

Replication Resources

Complete technical replication is highly accessible due to the paper being published open-access in Machine Learning: Science and Technology. It primarily requires:

• The full rule generation algorithm (Table 2 in paper)
• Code and trained models: https://github.com/aspuru-guzik-group/selfies
• Supplementary Information for complete architectural and hyperparameter specifications

Note: The modern SELFIES library has evolved significantly since this foundational paper, addressing many of the implementation challenges identified by the authors.

Paper Information

Citation: Krenn, M., Häse, F., Nigam, A., Friederich, P., & Aspuru-Guzik, A. (2020). Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation. Machine Learning: Science and Technology, 1(4), 045024. https://doi.org/10.1088/2632-2153/aba947

Publication: Machine Learning: Science and Technology, 2020

@article{Krenn_2020,
	doi = {10.1088/2632-2153/aba947},
	url = {https://doi.org/10.1088%2F2632-2153%2Faba947},
	year = 2020,
	month = {aug},
	publisher = {{IOP} Publishing},
	volume = {1},
	number = {4},
	pages = {045024},
	author = {Mario Krenn and Florian H{\"{a}}se and AkshatKumar Nigam and Pascal Friederich and Alan Aspuru-Guzik},
	title = {Self-referencing embedded strings ({SELFIES}): A 100{\%} robust molecular string representation},
	journal = {Machine Learning: Science and Technology}
}

Additional Resources:

• GitHub Repository
• Modern SELFIES Documentation

This is an infrastructure/resource paper combined with a methods paper. It establishes a standard format, releases an open-source software library, and enables large-scale database operations. The methods component details the specific algorithmic rules for constructing identifiers through hashing, sorting, and layering.

The Need for Standardized Reaction Identifiers

While we have excellent standards for identifying individual molecules (like SMILES and InChI), there was no equivalent for chemical reactions. This creates real problems:

• Different researchers working on the same reaction might describe it completely differently
• Searching large reaction databases becomes nearly impossible
• No way to check if two apparently different reaction descriptions are actually the same process
• Chemical databases can’t easily link related reactions or identify duplicates

If a reaction converts “starting material A + reagent B → product C,” it is difficult to determine if that is identical to another researcher’s description of the same transformation using different names or graphical representations.

Core Innovation: Standardizing Reaction Strings

RInChI solves this by creating a standardized, machine-readable label for any chemical reaction. The key insight is to focus on the essential chemistry while ignoring experimental details that can vary between labs.

Core Principles

RInChI captures three fundamental pieces of information:

1. Starting materials: What molecules you begin with
2. Products: What molecules you end up with
3. Agents: Substances present at both the beginning and end (catalysts, solvents, etc.)

Importantly, RInChI intentionally excludes experimental conditions like temperature, pressure, yield, or reaction time. These details can vary significantly even for identical chemical transformations, so including them would make it nearly impossible for different researchers to generate the same identifier.

How RInChI Works

The RInChI String Structure

A RInChI string has six distinct layers. Crucially, Layers 2 and 3 are assigned alphabetically. This is essential for generating consistent identifiers.

Layer 1: Version

• Standard header defining the RInChI version (e.g., RInChI=1.00.1S)

Layers 2 & 3: Component Molecules

• These layers contain the InChI strings of reaction participants (reactants and products)
• Sorting Rule: The distinct groups (Reactant Group vs. Product Group) are sorted alphabetically as aggregate strings. The group that comes first alphabetically becomes Layer 2; the other becomes Layer 3
• This means if a product’s InChI is alphabetically “earlier” than the reactant’s, the product goes in Layer 2
• Formatting: Molecules within a layer are separated by !. The two layers are separated by <>

Layer 4: Agents

• Contains catalysts, solvents, and any molecule found in both the reactant and product input lists
• Algorithmic rule: Anything appearing in both the reactant list and product list must be removed from both and added to Layer 4

Layer 5: Direction (The Decoder)

• This layer determines which component layer represents the starting material:
  • /d+: Layer 2 is the Starting Material (forward direction)
  • /d-: Layer 3 is the Starting Material (reverse direction)
  • /d=: Equilibrium reaction
• Without this layer, you cannot determine reactants from products

Layer 6: No-Structure Data

• Format: /uA-B-C where the numbers indicate the count of structureless materials in Layer 2, Layer 3, and Layer 4 respectively
• Used when substances lack defined structures and cannot be represented by InChI

Separator Syntax

For parsing or generating RInChI strings, the separator characters are:

Example Structure

RInChI=1.00.1S/[Layer2 InChIs]<>[Layer3 InChIs]<>[Agent InChIs]/d+/u0-0-0

This systematic approach ensures that any researcher starting with the same reaction will generate an identical RInChI string.

RInChIKeys: Shorter Identifiers for Practical Use

Since full RInChI strings can become extremely long, the standard includes three types of shorter, hashed keys for different applications:

Long-RInChIKey

• Contains complete InChIKeys for every molecule in the reaction
• Variable length, but allows searching for reactions containing specific compounds
• Useful for substructure searches: “Show me all reactions involving compound X”

Short-RInChIKey

• Fixed length (63 characters)
• Generated by separately hashing different RInChI layers to maintain distinction between role components.
• Mathematical Formulation: The key represents an aggregate hash where the major layers ($L_{\text{major}}$, mapping to the molecular skeleton) are hashed completely separately from the minor layers ($L_{\text{minor}}$, such as stereochemistry and protonation states). $$ \text{Key}_{\text{short}} = \text{Hash}( \text{Major Components} ) \oplus \text{Hash}( \text{Minor Components} ) \oplus \dots $$
• Perfect for exact matching and database storage
• The go-to choice for linking identical reactions across different databases

Web-RInChIKey

• Shortest format (47 characters)
• Hashing algorithm: The Web-RInChIKey uses a simplified hashing structure designed to explicitly ignore molecular roles (e.g., whether something is a reactant or product). It defines a unified set of components $C = { c_1, c_2, \dots, c_n }$ consisting of all unique InChI representations across the entire reaction.
  1. Combine and sort all InChIs alphabetically
  2. Produce a two-part hash block focusing on major and minor structural layers: $$ \text{Key}_{\text{web}} = \text{Hash}(C_{\text{major}}) + \text{Hash}(C_{\text{minor}}) $$
• Ignores molecular roles as reactants vs. products
• Useful for finding related reactions where a molecule’s role might be ambiguous
• Good for discovering “reverse” reactions or alternative synthetic routes

Experimental Validation and Software Implementation

This infrastructure paper focuses on developing and validating the RInChI standard. The validation approach includes:

• Software implementation: Development of the official RInChI software library capable of parsing reaction files and generating identifiers
• Format testing: Validation that the system correctly handles standard reaction file formats (.RXN, .RD)
• Consistency verification: Ensuring identical reactions produce identical RInChI strings regardless of input variations
• Key generation: Testing all three RInChIKey variants (Long, Short, Web) for different use cases
• Database integration: Demonstrating practical application in reaction database management

Impact on Chemical Database Analytics

Practical Applications

RInChI enables systematic organization and analysis of chemical reactions:

Database Management

RInChI enables systematic organization of reaction databases. You can:

• Automatically identify and merge duplicate reaction entries
• Find all variations of a particular transformation
• Link related reactions across different data sources

Reaction Analysis

With standardized identifiers, you can perform large-scale analysis:

• Identify the most commonly used reagents or catalysts
• Find cases where identical starting materials yield different products
• Analyze reaction trends and patterns across entire databases

Multi-Step Synthesis Representation

RInChI can represent complex, multi-step syntheses as single combined identifiers, making it easier to analyze and compare different synthetic routes.

Research Integration

The standard enables better collaboration by ensuring different research groups can generate identical identifiers for the same chemical processes, facilitating data sharing and literature analysis.

Limitations and Considerations

What Gets Lost

Since RInChI builds on the Standard InChI for individual molecules, it inherits certain limitations:

• Tautomers: Different tautomeric forms are treated as identical
• Stereochemistry: Relative stereochemical relationships aren’t captured
• Experimental conditions: Temperature, pressure, yield, and reaction time are intentionally excluded

The Trade-off

This is an intentional feature. By focusing on core chemical identity, RInChI achieves its primary goal: ensuring that different researchers working on the same fundamental transformation generate the same identifier.

Implementation and Tools

Official Software

The RInChI software, available from the InChI Trust, handles the practical details:

• Accepts standard reaction file formats (.RXN, .RD)
• Generates RInChI strings, all three RInChIKey variants, and auxiliary information
• Automates the complex process of creating consistent identifiers

RAuxInfo: Preserving Visual Information

While RInChI discards graphical information (atom coordinates, drawing layout), the software can generate supplementary “RAuxInfo” strings that preserve this data. This allows reconstruction of the original visual representation when needed.

Future Directions

RInChI development continues to evolve:

• Integration: Plans for compatibility with other emerging standards like MInChI for chemical mixtures
• Extended applications: Work on representing complex, multi-component reaction systems
• Software development: Tools for generating graphical representations directly from RInChI without auxiliary information

Key Takeaways

1. Filling a critical gap: RInChI provides the first standardized way to uniquely identify chemical reactions, solving a fundamental problem in chemical informatics.

2. Focus on essential chemistry: By excluding experimental variables, RInChI achieves consistent identification of core chemical transformations.

3. Flexible searching: Multiple RInChIKey formats enable different types of database queries, from exact matching to similarity searching.

4. Practical implementation: Official software tools make RInChI generation accessible to working chemists and database managers.

5. Foundation for analysis: Standardized reaction identifiers enable large-scale analysis of chemical databases and systematic study of reaction patterns.

RInChI represents a significant step toward making reaction data as standardized and machine-readable as molecular data has become with formats like SMILES and InChI.

Paper Information

Citation: Grethe, G., Blanke, G., Kraut, H., & Goodman, J. M. (2018). International chemical identifier for reactions (RInChI). Journal of Cheminformatics, 10(1), 22. https://doi.org/10.1186/s13321-018-0277-8

Publication: Journal of Cheminformatics (2018)

@article{Grethe2018,
  title={International chemical identifier for reactions (RInChI)},
  author={Grethe, Guenter and Blanke, Gerd and Kraut, Hans and Goodman, Jonathan M},
  journal={Journal of Cheminformatics},
  volume={10},
  number={1},
  pages={22},
  year={2018},
  publisher={Springer},
  doi={10.1186/s13321-018-0277-8}
}

This software update paper documents major improvements to the SELFIES Python library, establishing it as a production-ready computational tool.

Limitations in the Original SELFIES Implementation

While the original SELFIES concept was promising, the initial 2019 implementation had critical limitations that prevented widespread adoption:

1. Performance: Too slow for production ML workflows
2. Limited chemistry: Couldn’t represent aromatic molecules, stereochemistry, or many other important chemical features
3. Poor usability: Lacked user-friendly APIs for common tasks

These barriers meant that despite SELFIES’ theoretical advantages (100% validity guarantee), researchers couldn’t practically use it for real-world applications like drug discovery or materials science.

Architectural Refactoring and New ML Integrations

The 2023 update refactors the underlying SELFIES engine, establishing it as a production-ready repository. The key updates include:

1. Graph-Based Processing: The system replaces string-based operations with directed molecular graphs internally, significantly increasing execution speed and extensibility.

2. Expanded Canonical Forms: Standardizes support for aromatic systems, stereochemistry, charged species, and isotopic data, effectively overlapping with the chemical diversity coverage of SMILES while preserving robust validity guarantees.

3. Semantic Constraint API: Introduces the set_semantic_constraints() function, allowing specification of custom valence definitions useful for theoretical studies or hypervalent states.

4. Deep Learning Hooks: Integrates tokenization, length estimation, and attribution utilities specifically engineered to plug directly into neural network encoders and decoders.

Performance Benchmarks & Validity Testing

The authors validated the library through several benchmarks:

Performance testing: Roundtrip conversion (SMILES → SELFIES → SMILES) on 300,000+ molecules from the DTP dataset completed in 252 seconds, demonstrating production-ready speed compared to previous string-based iterations (translating to a massive orders-of-magnitude faster operation footprint).

Chemical coverage: Tested on diverse molecular databases to verify the library handles most chemical diversity found in resources like PubChem.

Validity guarantee: Demonstrated that random SELFIES strings always decode to valid molecules, with controllable size distributions through symbol filtering.

Attribution system: Showed both encoder and decoder can track which input symbols produce which output symbols, useful for property alignment.

Future Trajectories for General Chemical Representations

The 2023 update successfully addresses the main adoption barriers:

1. Fast enough for large-scale ML applications (300K molecules in ~4 minutes)
2. Chemically comprehensive enough for drug discovery and materials science
3. User-friendly enough for seamless integration into existing workflows

The validity guarantee, SELFIES’ core advantage, is now practically accessible for real-world research. The roadmap includes future extensions for polymers, crystals, chemical reactions, and non-covalent interactions, which would expand SELFIES’ applicability beyond small-molecule chemistry.

Limitations acknowledged: The paper focuses on implementation improvements. Some advanced chemical systems (polymers, crystals) still need future work.

Reproducibility Details

Code

The selfies library is completely open-source and written in pure Python. It requires no extra dependencies and is available on GitHub, installable via pip install selfies. The repository includes testing suites (tox) and example benchmarking scripts to reproduce the translation speeds reported in the paper.

Hardware

Performance benchmarks (e.g., the 252-second roundtrip conversion on 300K molecules) were executed on Google Colaboratory using two 2.20GHz Intel Xeon CPUs.

Algorithms

Technical Specification: The Grammar

The core innovation of SELFIES is a Context-Free Grammar (CFG) augmented with state-machine logic to ensure that every derived string represents a valid molecule. While the software features are important, understanding the underlying derivation rules is essential for replication or extension of the system.

1. Derivation Rules: The Atom State Machine

The fundamental mechanism that guarantees validity is a state machine that tracks the remaining valence of the most recently added atom:

• State Tracking: The derivation maintains a non-terminal state $X_l$, where $l$ represents the current atom’s remaining valence (number of bonds it can still form)
• Standard Derivation: An atom symbol $[\beta \alpha]$ (bond order + atom type) transitions the state from $S$ (start) to $X_l$, where $l$ is calculated from the atom’s standard valence minus the incoming bond order
• Bond Demotion (The Key Rule): If a requested bond order $\beta$ exceeds the available valence $l$ of the previous atom, the bond is automatically demoted to $\min(l, d(\beta))$ to prevent invalid valency. This automatic adjustment is the mathematical core of the validity guarantee.

This state machine ensures that no atom ever exceeds its allowed valence, making it impossible to generate chemically invalid structures.

2. Control Symbols: Branches and Rings

Branch length calculation: SELFIES uses a hexadecimal encoding to determine branch lengths. If a branch symbol is followed by index symbols $c_1 \dots c_k$, the number of symbols $N$ to include in the branch is calculated as:

$$ \begin{aligned} N &= 1 + 16 \cdot c_k \end{aligned} $$

This formula interprets subsequent symbols as base-16 integers, allowing compact representation of arbitrarily long branches.

Ring closure queue system: Ring formation uses a lazy evaluation strategy to maintain validity. Ring symbols don’t create bonds immediately; instead, they push “closure candidates” into a queue $R$. These candidates are resolved after the main derivation completes. A ring closure candidate is rejected if either atom has no remaining valence ($m = 0$), or the closure would create a self-loop (same atom). This deferred validation prevents the creation of invalid ring structures while still allowing the grammar to remain context-free during the main derivation.

3. Symbol Structure and Standardization

SELFIES enforces a strict, standardized format for atom symbols to eliminate ambiguity:

• Canonical Format: Atom symbols follow the structure [Bond, Isotope, Element, Chirality, H-count, Charge]
• No Variation: There is only one way to write each symbol (e.g., [Fe++] and [Fe+2] are standardized to a single form)
• Order Matters: The components must appear in the specified order

4. Default Semantic Constraints

By default, the library enforces standard organic chemistry valence rules:

• Typical Valences: C=4, N=3, O=2, F=1
• Element Coverage: Supports most elements relevant to organic and medicinal chemistry
• Customizable: These constraints can be modified via set_semantic_constraints() for specialized applications (hypervalent compounds, theoretical studies, etc.)

The combination of these grammar rules with the state machine ensures that every valid SELFIES string decodes to a chemically valid molecule, regardless of how the string was generated (random, ML model output, manual construction, etc.).

Data

Benchmark dataset: DTP (Developmental Therapeutics Program) dataset with 300,000+ molecules used for performance testing.

Chemical coverage testing: Diverse molecular databases including PubChem to verify the library handles broad chemical diversity (aromatic systems, stereochemistry, charged species, isotopes).

Evaluation

Performance metric: Roundtrip conversion time (SMILES → SELFIES → SMILES) is 252 seconds for 300K molecules.

Validity testing: Random SELFIES string generation with decoding verification shows 100% of generated strings decode to valid molecules.

Attribution system: Encoder and decoder track which input symbols produce which output symbols, tested for property alignment between SMILES and SELFIES representations.

Paper Information

Citation: Lo, A., Pollice, R., Nigam, A., White, A. D., Krenn, M., & Aspuru-Guzik, A. (2023). Recent advances in the self-referencing embedded strings (SELFIES) library. Digital Discovery, 2(4), 897-908. https://doi.org/10.1039/D3DD00044C

Publication: Digital Discovery 2023

@article{lo2023recent,
  title={Recent advances in the self-referencing embedded strings (SELFIES) library},
  author={Lo, Alston and Pollice, Robert and Nigam, AkshatKumar and White, Andrew D and Krenn, Mario and Aspuru-Guzik, Al{\'a}n},
  journal={Digital Discovery},
  volume={2},
  number={4},
  pages={897--908},
  year={2023},
  publisher={Royal Society of Chemistry},
  doi={10.1039/D3DD00044C}
}

Additional Resources:

• SELFIES GitHub Repository
• Original SELFIES Paper (2020)
• SELFIES Format Overview

This is a Theory/Systematization paper that proposes a new Resource standard: the NInChI. It addresses a fundamental limitation in nanoinformatics. The result of a collaborative process (a “Request for Comments” or “RFC”), this work uses six detailed case studies to systematically develop a hierarchical, machine-readable notation for complex nanomaterials that could work across experimental research, regulatory frameworks, and computational modeling.

The Breakdown of Traditional Chemical Identifiers

Chemoinformatics has fantastic tools for representing small molecules: SMILES strings, InChI identifiers, and standardized databases that make molecular data searchable and shareable. But when you step into nanotechnology, everything breaks down.

Consider trying to describe a gold nanoparticle with a silica shell and organic surface ligands. How do you capture:

• The gold core composition and size
• The silica shell thickness and interface
• The surface chemistry and ligand density
• The overall shape and morphology

There’s simply no standardized way to represent this complexity in a machine-readable format. This creates massive problems for:

• Data sharing between research groups
• Regulatory assessment where precise identification matters
• Computational modeling that needs structured input
• Database development and search capabilities

Without a standard notation, nanomaterials research suffers from the same data fragmentation that plagued small molecule chemistry before SMILES existed.

The Five-Tier NInChI Hierarchy

The authors propose NInChI (Nanomaterials InChI), a layered extension to the existing InChI system. The clever insight is organizing nanomaterial description from the inside out, like describing an onion with a five-tier hierarchy:

1. Tier 1: Chemical Core: What’s the fundamental composition? Gold? Silver? Carbon nanotube?
2. Tier 2: Morphology: What shape and size? Spherical nanoparticle? Rod? 2D sheet?
3. Tier 3: Surface Properties: What’s the surface like? Charge, roughness, hydrophobicity?
4. Tier 4: Surface Chemistry: How are things attached? Covalent bonds? Physical adsorption?
5. Tier 5: Surface Ligands: What molecules are on the surface and how many?

This hierarchy captures the essential information needed to distinguish between different nanomaterials while building on familiar chemical concepts.

Testing the Standard: Six Case Studies

The authors tested their concept against six real-world case studies to identify what actually matters in practice.

Case Study 1: Gold Nanoparticles

Gold NPs provided a relatively simple test case: an inert metallic core with various surface functionalizations. Key insights: core composition and size are essential, surface chemistry (what molecules are attached) matters critically, shape can dramatically affect properties, and dynamic properties like protein corona formation belong outside the intrinsic NInChI representation. This established the boundary: NInChI should capture intrinsic, stable properties.

Case Study 2: Carbon Nanomaterials

Carbon nanotubes and graphene introduced additional complexity: dimensionality (1D tubes vs 2D sheets vs 0D fullerenes), chirality (the (n,m) vector that defines a nanotube’s structure), defects and impurities that can dramatically alter properties, and layer count for 2D materials. This case showed that the notation needed to handle both topological complexity and chemical composition.

Case Study 3: Complex Engineered Materials

Doped materials, alloys, and core-shell structures revealed key requirements: need to distinguish true alloys (homogeneous mixing) and core-shell structures with the same overall composition, crystal structure information becomes crucial, and component ratios must be precisely specified.

Case Study 4: Database Applications