This is primarily a Method paper with significant Resource contributions.

• Methodological Basis: The paper introduces a training pipeline (“mix-sourced distillation”) and domain-specific reinforcement learning to improve reasoning capabilities in chemical LLMs. It validates the approach through ablation studies across training stages.
• Resource Contribution: The authors constructed ChemFG, a 101 billion-token corpus annotated with “atomized” knowledge regarding functional groups and reaction centers.

Bridging the Chemical Reasoning Gap

Current chemical LLMs struggle to reason logically for two main reasons:

1. Shallow Domain Understanding: Models generally learn molecule-level properties directly, bypassing the intermediate “atomized” characteristics (e.g., functional groups) that ultimately dictate chemical behavior.
2. Specialized Reasoning Logic: Chemical logic differs fundamentally from math or code. Distilling reasoning from general teacher models like DeepSeek-R1 frequently fails because the teachers lack the domain intuition required to generate valid chemical rationales.

Atomized Knowledge and Mixed-Source Distillation

The authors introduce three structural innovations to solve the reasoning gap:

1. Atomized Knowledge Enhancement (ChemFG): A toolkit was built leveraging SMARTS notations to identify functional group changes during reactions. A critique of this approach is that it relies heavily on 2D cheminformatics abstractions, potentially missing deeper 3D stereochemical interactions.
2. Mix-Sourced Distillation: General models (DeepSeek-R1/o3-mini) are fed “pseudo-reasoning” prompts that include ground truth answers and functional group data. While this forces the teacher to generate high-quality rationales for the student to learn, it introduces a layer of hindsight bias into the generated reasoning chains. During inference, the student model lacks both the pre-calculated functional group metadata and the ground truth, forcing it to bridge an artificially steep generalization gap.
3. Chemical Reinforcement Learning: The intermediate model undergoes domain-specific reinforcement learning. The RL details are described in the paper’s Appendix D, with the authors citing the open-source DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) framework. The optimization relies on rule-based rewards (format adherence and canonicalized SMILES accuracy) across a variety of chemical tasks.

Benchmark Evaluation and Ablation Studies

The model was evaluated on comprehensive chemical benchmarks: SciKnowEval (19 tasks) and ChemEval (36 tasks).

• Baselines: Compared against similarly sized open models (Qwen2.5-14B-Instruct, Qwen3-14B), domain models (ChemLLM, MolInst), and frontier models (GPT-4o, DeepSeek-R1).
• Ablation: Evaluated across training stages (Base → ChemDFM-I → ChemDFM-R) to measure the specific impact of the instruction tuning versus the reasoning stages.
• Qualitative Analysis: The paper includes case studies demonstrating the model’s step-by-step chemical reasoning and its potential for human-AI collaboration (Sections 4.2 and 4.3).

Performance Outcomes and Numerical Limitations

• Performance vs. Baselines: ChemDFM-R outperforms similarly sized open models and domain models on molecule-centric and reaction-centric tasks, and surpasses the much larger DeepSeek-R1 on ChemEval (0.78 vs. 0.58 overall). It shows competitive results relative to o4-mini, though o4-mini leads on SciKnowEval (0.74 vs. 0.70).
• Reasoning Interactivity: The model generates readable rationales that allow users to catch structural errors or identify reaction mechanisms accurately. Section 4.3 of the paper demonstrates human-AI collaboration scenarios.
• Quantitative Limitations: The model struggles with tasks involving numerical prediction and calculation (e.g., yield extraction, molecular property calculation). The paper notes that all molecule-centric and reaction-centric tasks where ChemDFM-R falls short of Qwen2.5-14B-Instruct involve numerical reasoning.

Reproducibility Details

Data

The training data is constructed in three phases:

1. Domain Pre-training (ChemFG):

• Size: 101 billion tokens
• Composition:
  • 12M literature documents (79B tokens)
  • 30M molecules from PubChem/PubChemQC
  • 7M reactions from USPTO-FULL
• Augmentation: SMILES augmentation (10x) using R-SMILES
• Atomized Features: Annotated with a custom “Functional Group Identification Toolkit” that identifies 241 functional group types and tracks changes in reaction centers. Note: Data and toolkit are partially reproduced; while the toolkit (ChemFG-Tool) was open-sourced on GitHub, the 101 billion-token ChemFG dataset itself has not been publicly released.

2. Instruction Tuning:

• Sources: Molecule-centric (PubChem, MoleculeNet), Reaction-centric (USPTO), and Knowledge-centric (Exams, Literature QA) tasks
• Mixing: Mixed with general instruction data in a 1:2 ratio

3. Distillation Dataset:

• Sources:
  • ~70% ChemDFM-R instruction data
  • ~22% constructed pseudo-reasoning (functional group descriptions)
  • ~8% teacher rationales (from DeepSeek-R1/o3-mini)
• Mixing: Mixed with general data (including AM-Deepseek-R1-Distill-1.4M) in a 1:2 ratio

Algorithms

Functional Group Identification:

• Extends the thermo library’s SMARTS list
• For reactions, identifies “reacting functional groups” by finding reactants containing atoms involved in bond changes (reaction centers) that do not appear in the product

Mix-Sourced Distillation:

• Teacher models (DeepSeek-R1, o3-mini) are prompted with Question + Ground Truth + Functional Group Info to generate high-quality “Thoughts”
• These rationales are distilled into the student model using a supervised fine-tuning loss across target tokens $y_t$: $$ \mathcal{L}_{\text{SFT}} = - \sum_{t=1}^T \log P_\theta(y_t \mid x, y_{<t}) $$

Reinforcement Learning:

• Algorithm: The paper cites DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) as the RL framework; full details are in Appendix D of the paper. Note: While the underlying DAPO framework is open-source, the specific chemistry-oriented RL pipeline and environment used for ChemDFM-R has not been publicly released.
• Hyperparameters (from paper appendix): Learning rate 5e-7, rollout batch size 512, training batch size 128
• Rewards: The reward system applies rule-based constraints focusing on physical form and chemical validity. The total reward $R(y, y^*)$ for a generated response $y$ given target $y^*$ combines a format adherence reward ($R_{\text{format}}$) and an accuracy reward ($R_{\text{acc}}$) evaluated on canonicalized SMILES: $$ R(y, y^*) = R_{\text{format}}(y) + R_{\text{acc}}(\text{canonicalize}(y), \text{canonicalize}(y^*)) $$

Models

• Base Model: Qwen2.5-14B
• ChemDFM-I: Result of instruction tuning the domain-pretrained model for 2 epochs
• ChemDFM-R: Result of applying mix-sourced distillation (1 epoch) followed by RL on ChemDFM-I. Note: Model weights are publicly available on Hugging Face.

Hardware

Hardware and training time details are described in the paper’s appendices, which are not available in the extracted text. The details below are reported from the paper but could not be independently cross-verified against the main text:

• Compute: NVIDIA A800 Tensor Core GPUs
• Training Time: 30,840 GPU hours total (Domain Pretraining: 24,728 hours; Instruction Tuning: 3,785 hours; Distillation: 2,059 hours; Reinforcement Learning: 268 hours)

Evaluation

Benchmarks:

• SciKnowEval: 19 tasks (text-centric, molecule-centric, reaction-centric)
• ChemEval: 36 tasks, categorized similarly

Key Metrics: Accuracy, F1 Score, BLEU score (with PRS normalization for ChemEval)

Artifacts

Missing components: The 101B-token ChemFG pretraining dataset is not publicly released. The chemistry-oriented RL pipeline and training code are not open-sourced. The instruction tuning and distillation datasets are not available.

Paper Information

Citation: Zhao, Z., Chen, B., Wan, Z., Chen, L., Lin, X., Yu, S., Zhang, S., Ma, D., Zhu, Z., Zhang, D., Wang, H., Dai, Z., Wen, L., Chen, X., & Yu, K. (2025). ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledge. arXiv preprint arXiv:2507.21990. https://doi.org/10.48550/arXiv.2507.21990

Publication: arXiv 2025

@misc{zhao2025chemdfmr,
  title={ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledge},
  author={Zihan Zhao and Bo Chen and Ziping Wan and Lu Chen and Xuanze Lin and Shiyang Yu and Situo Zhang and Da Ma and Zichen Zhu and Danyang Zhang and Huayang Wang and Zhongyang Dai and Liyang Wen and Xin Chen and Kai Yu},
  year={2025},
  eprint={2507.21990},
  archivePrefix={arXiv},
  primaryClass={cs.CE},
  url={https://arxiv.org/abs/2507.21990}
}

This is primarily a Resource ($\Psi_{\text{Resource}}$) paper, with secondary Method ($\Psi_{\text{Method}}$) contributions.

• Resource Basis: The core contribution is “ChemBERTa-3,” an open-source framework integrated into DeepChem that standardizes the pretraining and benchmarking of chemical foundation models. The authors focus heavily on infrastructure (AWS/Ray integration) and correcting benchmarking inconsistencies in the field.
• Method Basis: It trains models like “c3-MoLFormer” to reproduce and validate the infrastructure.

The Pretraining Scalability Challenge

• Scalability Challenges: Building robust molecular models is difficult due to the vast size of chemical space and the computational intensity of pretraining on large datasets.
• Proprietary Barriers: Many high-performing chemical foundation models (e.g., the full MoLFormer-XL) are partially closed-source or difficult to reproduce.
• Benchmarking Inconsistencies: There is a lack of systematic comparison between architectures (e.g., Graph vs. Transformer) using unified protocols. Specifically, previous comparisons relied on reported results that used differing scaffold splitting algorithms, making them inaccurate.

Unified Infrastructure & Standardized Benchmarking

• Unified Infrastructure: Integration of DeepChem with Ray for distributed, scalable pretraining and fine-tuning of both graph and transformer models.
• Standardized Benchmarking: Identification that MoLFormer’s scaffold splitting algorithm differs from the standard DeepChem/MoleculeNet splitter, and the subsequent standardization of these benchmarks for fair comparison.
• New DeepChem Tools: Introduction of the ModularTorchModel class for flexible loss computation and HuggingFaceModel wrappers to bridge ecosystems.

Benchmarking Transformers vs. Graph Models

• Architecture Comparison: Benchmarked Transformers (ChemBERTa, MoLFormer) against Graph models (GROVER, InfoGraph, InfoMax3D, DMPNN, GCN) and baselines (Random Forest).
• Pretraining Scale Disparity:
  • Transformers were pretrained on ZINC20 subsets ranging from 10M to 1.1B molecules (combining ZINC and PubChem).
  • Graph models were limited to 250K molecule subsets due to memory and computational overhead of message passing on large graphs. While this highlights the superior scalability of Transformer architectures, comparing a 1.1B-trained Transformer to a 250K-trained Graph model provides an unbalanced evaluation of architectural capacity.
• Reproducibility Validation: Trained “c3-MoLFormer” (a reproduction of MoLFormer) on 1.1B molecules using two distinct hardware setups: AWS spot instances (Ray) and a local HPC cluster.
• Scaffold Split Analysis: Compared performance metrics using “DeepChem scaffold splits” vs. “MoLFormer scaffold splits” to quantify the impact of data leakage/overlap.

Overcoming Scaffold Splitting Inconsistencies

• Scaling Transformers vs. Graphs: Transformer-based models are significantly easier to scale to large datasets than current graph-based approaches, though performance is comparable at small scales.
• Benchmarking sensitivity: MoLFormer’s reported superiority over baselines was partly inflated by its specific scaffold splitting method, which had higher structural overlap between train and test sets (yielding a lower Tanimoto distance, generally quantified via $1 - \frac{|A \cap B|}{|A \cup B|}$) than DeepChem splits. When standardized, baselines like DMPNN perform more competitively.
• Infrastructure Viability: The framework successfully replicated large-scale training (MoLFormer-1.1B) on both cloud and on-premise HPC, confirming reproducibility.
• Open Source Release: All code, configurations, and the c3-MoLFormer-1.1B model weights are released to facilitate future research.

Reproducibility Details

Data

• Pretraining:
  • Source: ZINC20 (1.4B compounds) and PubChem.
  • Scale: Subsets of 10M, 100M, and 1.1B (100% ZINC20 + 100% PubChem) were used for Transformers. Graph models used a 250K subset.
• Fine-tuning:
  • Suite: MoleculeNet.
  • Tasks: Classification (BACE, BBBP, Tox21, HIV, SIDER, ClinTox) and Regression (ESOL, FreeSolv, Lipophilicity, QM9).
  • Splits: Critical distinction made between “DeepChem scaffold splits” (80/10/10) and “MoLFormer scaffold splits” (which can be downloaded from https://ibm.ent.box.com/v/MoLFormer-data). The paper notes these algorithms differ.

Algorithms

• Framework: DeepChem integrated with Ray for distributed training. To recreate the environment, the repository relies on a nightly version of DeepChem (pip install --pre deepchem) and specific dependencies found within the requirements.txt. Pretraining scripts are available in the chemberta3_benchmarking/pretraining directory of the repository.
• Data Preparation: Featurization workflows (e.g., CircularFingerprint, RDKitConformer) are documented under chemberta3_benchmarking/data/data_preprocessing/ in the codebase.
• Modular Training: Uses ModularTorchModel to allow loss computation from intermediate values and flexible component connection.
• Training Brittleness:
  • Optimizer: Linear learning rate scheduler with warmup.
  • Instability Handling: The authors observed significant loss spikes during warmup. Their primary mitigation strategy involved checkpointing frequently and restarting from the last stable state upon a spike, highlighting a persistent brittleness in optimizing these large chemical foundation models.
  • Numerical Issues: Addressed NaN values by pretraining on a small dataset with low LR before scaling up.

Models

• ChemBERTa: RoBERTa-based architecture trained with Masked Language Modeling (MLM) and Multitask Regression (MTR). Specific model identifiers (e.g., DeepChem/ChemBERTa-100M-MLM) are hosted on Hugging Face so researchers can pull them directly via the transformers library. The core pretraining objective minimized the standard MLM loss: $$ \mathcal{L}{\text{MLM}} = - \frac{1}{|\mathcal{M}|} \sum{i \in \mathcal{M}} \log \hat{y}{i} $$ where $\mathcal{M}$ represents the set of masked SMILES token indices, and $\hat{y}{i}$ is the model’s predicted probability for the correct token given the corrupted sequence context.
• MoLFormer (c3-MoLFormer): Re-implementation of the MoLFormer architecture (Rotary embeddings, linear attention). Specific model identifiers (e.g., DeepChem/MoLFormer-c3-1.1B) are similarly available on Hugging Face.
  • Tokenizer: ibm/MoLFormer-XL-both-10pct tokenizer.
• Graph Models:
  • GROVER: Graph Transformer with node/edge/graph level self-supervision.
  • InfoGraph: Maximizes mutual information between graph-level and substructure representations.
  • InfoMax3D: Incorporates 3D conformer data (via RDKit ETKDGv2) into contrastive pretraining.
  • DMPNN: Directed Message Passing Neural Network (Chemprop variant).

Evaluation

• Metrics: ROC-AUC for classification; RMSE for regression (MAE for QM9).
• Baselines: Random Forest, GCN, DMPNN trained on fine-tuning splits only.
• Protocol: Three independent runs per configuration to report mean and range (not a confidence interval), with the exception of the compute-heavy QM9 dataset, which only received a single run. Benchmarking execution scripts (e.g., GCN, RF, DMPNN, ChemBERTa) are stored in the repo under chemberta3_benchmarking/models_benchmarking/ and contain the specific fine-tuning hyperparameters and optimizer configurations used for each downstream task.
• Key Results:
  • c3-MoLFormer-1.1B achieved ~0.848 ROC-AUC on BACE and ~0.900 on BBBP (using MoLFormer splits). This closely matches the original IBM MoLFormer metrics, validating the reproducibility of the open-source framework.
  • When constrained to the equivalent 250K subset, Graph models (InfoGraph, GROVER) performed comparably to Transformers, indicating that Transformer superiority in chemistry is largely driven by data scalability rather than an inherent architectural advantage at small scales.

Hardware

• Cloud (AWS):
  • Compute: 40 NVIDIA T4 GPUs (g4dn.12xlarge spot instances for pretraining, g4dn.2xlarge for benchmarking).
  • Cost: ~$4000 for MoLFormer 1.1B pretraining.
  • Time: ~10 days (260 hours) for 1.1B model pretraining.
  • Setup: Setup scripts for single-node and multi-node spot EC2 clusters are provided in the GitHub repository’s infra/ and spot/ folders.
• On-Premise HPC:
  • Compute: 16 nodes (AMD EPYC), each with 4 AMD MI300A APUs.
  • Environment: Ray multi-node multi-GPU framework.

Artifacts

Paper Information

Citation: Singh, R. et al. (2026). ChemBERTa-3: an open source training framework for chemical foundation models. Digital Discovery, 5, 662-685. https://doi.org/10.1039/D5DD00348B

Publication: Digital Discovery 2026

Additional Resources:

• ChemBERTa-3 GitHub Repository
• DeepChem Project
• DeepChem Hugging Face Models

@article{singhChemBERTa3OpenSource2026,
  author = {Singh, Riya and Barsainyan, Aryan Amit and Irfan, Rida and Amorin, Connor Joseph and He, Stewart and Davis, Tony and Thiagarajan, Arun and Sankaran, Shiva and Chithrananda, Seyone and Ahmad, Walid and Jones, Derek and McLoughlin, Kevin and Kim, Hyojin and Bhutani, Anoushka and Sathyanarayana, Shreyas Vinaya and Viswanathan, Venkat and Allen, Jonathan E. and Ramsundar, Bharath},
  title = {{{ChemBERTa-3}}: an open source training framework for chemical foundation models},
  journal = {Digital Discovery},
  year = {2026},
  volume = {5},
  pages = {662-685},
  publisher = {The Royal Society of Chemistry},
  doi = {10.1039/D5DD00348B},
  url = {https://doi.org/10.1039/D5DD00348B}
}

This is primarily a Methodological paper, as it proposes a specific neural architecture (GP-MoLFormer) and a novel fine-tuning algorithm (Pair-tuning) for molecular generation. It validates these contributions against standard baselines (e.g., JT-VAE, MolGen-7b).

It also contains a secondary Theoretical contribution by establishing an empirical scaling law that relates inference compute (generation size) to the novelty of the generated molecules.

Motivation: Data Scale and Prompt-Based Optimization

While large language models (LLMs) have transformed text generation, the impact of training data scale and memorization on molecular generative models remains under-explored. Specifically, there is a need to understand how training on billion-scale datasets affects the novelty of generated molecules and whether biases in public databases (like ZINC and PubChem) perpetuate memorization. Furthermore, existing optimization methods often require computationally expensive property predictors or reinforcement learning loops; there is a practical need for more efficient “prompt-based” optimization techniques.

Core Innovations: Architecture and Pair-Tuning

1. Architecture: The application of a linear-attention transformer decoder with Rotary Positional Embeddings (RoPE) to generative chemistry, allowing for efficient training on 1.1 billion SMILES.
2. Pair-Tuning: A novel, parameter-efficient fine-tuning method that uses property-ordered molecular pairs to learn “soft prompts” for optimization without updating the base model weights.
3. Scaling Analysis: An extensive empirical investigation mapping the trade-off between inference compute (up to 10B generations) and chemical novelty, fitting an exponential decay curve that demonstrates how novelty saturates as generation volume grows.

Experimental Methodology and Downstream Tasks

The authors evaluated GP-MoLFormer on three distinct tasks, though the comparisons highlight the difficulty of evaluating foundation models against classical baselines:

1. De Novo Generation: Comparing validity, uniqueness, and novelty against baselines (CharRNN, VAE, LIMO, MolGen-7b) on a held-out test set. Notably, this is an unequal comparison; most baselines were trained on the 1.6M molecule MOSES dataset, whereas GP-MoLFormer uses up to 1.1B molecules, meaning performance gains are heavily driven by data scale.
2. Scaffold-Constrained Decoration: Generating molecules from DRD2 active binder scaffolds and measuring the hit rate of active compounds against specialized scaffold decorators.
3. Property-Guided Optimization: Using Pair-tuning to optimize for Drug-likeness (QED), Penalized logP, and DRD2 binding activity, comparing the results to graph-based and reinforcement learning benchmarks.

Additionally, they performed a Scaling Study:

• Comparing models trained on raw (1.1B) vs. de-duplicated (650M) data.
• Generating up to 10 billion molecules to fit empirical scaling laws for novelty.

Key Findings and Scaling Laws

• Scale Driven Performance: GP-MoLFormer achieves high internal diversity and validity on generation metrics. However, its baseline novelty percentage (~32%) is considerably lower than classical models. The authors attribute this to the massive training scale forcing the model to heavily prioritize matching real-world molecule frequencies over pure exploration. GP-MoLFormer’s advantage in generation metrics over LLM-baselines like MolGen-7b likely stems heavily from its 10x larger training dataset rather than fundamental architectural superiority.
• Pair-Tuning Efficacy: The proposed pair-tuning method effectively optimizes properties (e.g., improving DRD2 activity scores) without requiring full model fine-tuning or external reward loops. While successful, the text-based generation yields ~94.5% validity during optimization, which lags behind graph and SELFIES-based baselines that guarantee 100% structural validity.
• Memorization vs. Novelty: Training on de-duplicated data (GP-MoLFormer-UNIQ) yields higher novelty (approx. 5-8% higher) than training on raw data, confirming that duplication bias in public databases leads directly to memorization.
• Inference Scaling Law: Novelty decays exponentially with generation size ($y = ae^{-bx}$), yet the model maintains generative capability (~16.7% novelty) even after generating an unprecedented 10 billion molecules.

Reproducibility Details

Data

• Sources: A combination of PubChem (111M SMILES) and ZINC (1B SMILES) databases. Downloading and pre-training instructions are located in the repository’s data/README.md.
• Preprocessing:
  • All SMILES were canonicalized using RDKit (no isomeric information).
  • GP-MoLFormer (Base): Trained on the full 1.1B dataset (includes duplicates).
  • GP-MoLFormer-UNIQ: Trained on a de-duplicated subset of 650M SMILES.
• Tokenization: Uses the tokenizer from Schwaller et al. (2019) with a vocabulary size of 2,362 tokens.
• Filtering: Sequences restricted to a maximum length of 202 tokens.

Algorithms

Pair-Tuning (Algorithm 1):

• Objective: Learn task-specific soft prompts $\phi_T$ to maximize the conditional probability of target molecule $b$ given a seed molecule $a$, where pair $(a, b)$ satisfies the property condition $b > a$. The base model parameters $\theta$ remain frozen.
• Prompt Structure: Autoregressive training optimizes the continuous embeddings of $n$ enhancement tokens against the cross-entropy loss of the target sequence: $$ \mathcal{L}(\phi_T) = - \sum_{i=1}^{|b|} \log P_{\theta}(b_i | \phi_T, a, b_{<i}) $$
• Hyperparameters: Trained for 1,000 epochs with a batch size of 35 and a fixed learning rate of $3 \times 10^{-2}$.
• Inference: The learned prompt $\phi_T$ and seed molecule $a$ are prepended as context, and candidates are sampled autoregressively until a termination token is produced.

Models

• Availability: The model trained on deduplicated data (GP-MoLFormer-UNIQ) is publicly available on Hugging Face. The full 1.1B base model is not explicitly hosted. The source code repository includes a disclosure that IBM will not maintain the code going forward.
• Architecture: Transformer decoder (~47M parameters: 12 layers, 12 heads, hidden size 768).
• Attention Mechanism: Combines Linear Attention (Generalized Random Feature map, $\phi$) with Rotary Positional Embeddings (RoPE). To avoid the quadratic complexity of standard attention while maintaining relative positional awareness, RoPE is applied to queries ($Q$) and keys ($K$) prior to the random feature mapping: $$ \text{Attention}(Q, K, V) = \frac{\sum_{n=1}^N \langle \phi(R_m q_m), \phi(R_n k_n) \rangle v_n}{\sum_{n=1}^N \langle \phi(R_m q_m), \phi(R_n k_n) \rangle} $$
• Inference Speed: ~3ms per forward pass on a single A100 GPU.

Evaluation

• Generation Quality Metrics: Validity, Uniqueness, Novelty (MOSES suite), Fréchet ChemNet Distance (FCD), Scaffold similarity (Scaf), and Similarity to Nearest Neighbor (SNN).
• MoLFormer-Based Metrics: The authors introduce Fréchet MoLFormer Distance (FMD) and MoLFormer-space IntDiv2 to measure distributional similarity using their own pre-trained continuous embeddings instead of standard fingerprints.
• Optimization Metrics: Penalized logP (calculated as $\text{logP} - \text{SA} - \text{max}(\text{maxrings}(size) - 6, 0)$), Drug-likeness (QED), and DRD2 activity scores.
• Scaling Metrics: Empirical fit for novelty decay: $y = ae^{-bx}$.

Hardware

• Compute: 16 x NVIDIA A100 (80 GB) GPUs across 2 nodes connected via EDR Infiniband.
• Training Time:
  • GP-MoLFormer (1.1B data): ~115 hours total (28.75 hours/epoch for 4 epochs).
  • GP-MoLFormer-UNIQ (650M data): ~80 hours total.
• Hyperparameters: Used a batch size of 1,600 molecules per GPU with a fixed learning rate of $1.6 \times 10^{-4}$ (scaled up to $8\times$ factor as GPUs increased).
• Optimization: Used distributed data-parallel training and adaptive bucketing by sequence length to handle scale.

Artifacts

The full 1.1B base model weights are not publicly hosted. The training data (PubChem and ZINC) is publicly available, and instructions for downloading and pre-processing are in the repository’s data/README.md.

Paper Information

Citation: Ross, J., Belgodere, B., Hoffman, S. C., Chenthamarakshan, V., Navratil, J., Mroueh, Y., & Das, P. (2025). GP-MoLFormer: A Foundation Model For Molecular Generation. Digital Discovery (2025). https://doi.org/10.1039/D5DD00122F

Publication: Digital Discovery (2025)

@article{ross2025gpmolformer,
  title={GP-MoLFormer: a foundation model for molecular generation},
  author={Ross, Jerret and Belgodere, Brian and Hoffman, Samuel C and Chenthamarakshan, Vijil and Navratil, Jiri and Mroueh, Youssef and Das, Payel},
  journal={Digital Discovery},
  year={2025},
  publisher={Royal Society of Chemistry},
  doi={10.1039/D5DD00122F}
}

This is primarily a Methodological paper with a secondary Resource contribution.

It fits the Method classification because it focuses on optimizing the architecture and pretraining pipeline for molecular transformers. The authors perform extensive ablation studies (varying dataset size from 5M to 77M, comparing MLM vs. MTR objectives) to determine “how well” these strategies work compared to baselines. The secondary Resource classification applies because they open-source the trained models and establish a benchmark on a massive 77M compound dataset.

Key methodological indicators:

• Baseline comparison: The paper explicitly compares ChemBERTa-2 against standard baselines (D-MPNN, Random Forest, GCN) and its predecessor (ChemBERTa-1) with prominent benchmark tables
• Ablation studies: Extensive experiments comparing multi-task and self-supervised pretraining by varying hyperparameters and pretraining dataset size
• Scaling analysis: Systematic investigation of whether larger datasets (up to 77M compounds) yield better performance

Motivations for Scaling Molecular Transformers

The authors aim to bridge the gap between NLP success stories (like GPT-3) and molecular machine learning by developing a “chemical foundation model”.

Key motivations:

• Label scarcity: Experimental labels for molecular properties are rare and expensive, but unlabeled SMILES strings are abundant
• Scaling hypothesis: Testing if scaling pretraining data (up to 77M compounds) yields consistent downstream improvements, similar to scaling laws in NLP
• Efficiency: Optimizing the pretraining process introduced in the original ChemBERTa by comparing self-supervised (MLM) and weakly supervised (MTR, using RDKit computed properties as labels) approaches

Novelty in Multi-Task Regression Objectives

Scale: Training on 77M unique SMILES from PubChem, which is one of the largest molecular pretraining datasets used to date (compared to 10M for ChemBERTa-1 or 18.7M for SMILES-BERT).

Pipeline optimization: A direct, controlled comparison of Masked Language Modeling (MLM) vs. Multi-Task Regression (MTR) pretraining objectives on identical datasets.

Proxy selection: The finding that MLM loss correlates well with MTR loss, allowing the cheaper MLM task to be used for hyperparameter tuning before running the expensive MTR pretraining.

Experimental Pretraining Setup on 77M Compounds

Pretraining Setup

Datasets: Subsets of PubChem containing 5M, 10M, and 77M unique SMILES.

Tasks:

• MLM: Masking 15% of tokens (following RoBERTa procedure). The model is optimized by minimizing the cross-entropy loss over the predicted masked tokens: $$ \mathcal{L}_{MLM} = -\sum_{i \in \mathcal{M}} \log P(x_i \mid \mathbf{x}_{\setminus \mathcal{M}}) $$ where $\mathcal{M}$ represents the set of masked token indices.
• MTR: Predicting 200 calculated molecular properties (via RDKit) simultaneously using a mean squared error objective: $$ \mathcal{L}_{MTR} = \frac{1}{200} \sum_{j=1}^{200} \frac{1}{N} \sum_{i=1}^{N} \left( \hat{y}_{ij} - y_{ij} \right)^2 $$ Continuous target labels $y_{ij}$ are mean-normalized prior to training to equilibrate the disparate scales of different chemical properties.

Hyperparameter search: Ran 50 random configurations on the 5M dataset; selected the top 5 to scale up to 10M and 77M.

Downstream Validation

Finetuning: Evaluated on 8 tasks from MoleculeNet (BACE, BBBP, ClinTox, Delaney, etc.) using scaffold splits (80/10/10).

Analysis: Used UMAP to visualize embeddings from MLM, MTR, and ECFP to check for clustering by label without finetuning.

Key Performance Outcomes and Scaling Realities

Highly competitive performance: ChemBERTa-2 outperforms the D-MPNN baseline (chemprop) on 6 out of 8 MoleculeNet tasks, though the margins demonstrate that task-specific baselines remain notably robust.

MTR superiority: Models pretrained on Multi-Task Regression (MTR) consistently perform better on downstream tasks than those pretrained on MLM on every finetuning task evaluated. MTR is substantially slower than MLM due to the larger input size from the 200-element label vector, but MLM loss serves as a reliable proxy for MTR loss, enabling cheaper architecture search before committing to full MTR pretraining.

Scaling laws versus downstream utility: Pretraining loss improved by 25-35% when increasing the dataset from 5M to 77M compounds. However, this improvement in pretraining loss does not uniformly transfer to downstream tasks. For MTR models, SR-p53 ROC-AUC decreases monotonically from 0.834 (5M) to 0.827 (10M) to 0.817 (77M), and Lipophilicity RMSE is worse at 77M (0.798) than at 5M (0.758), despite a dip at 10M (0.744). This variability in transfer challenges the assumption that pretraining improvements always yield downstream gains.

Transfer learning: The correlation between pretraining loss and downstream performance is task-dependent; it is strong for Lipophilicity but weaker for BACE classification.

Reproducibility Details

Data

The pretraining corpus is derived from PubChem.

Algorithms

Pretraining Objectives:

1. Masked Language Modeling (MLM): Follows RoBERTa procedure. Masks 15% of tokens. Max sequence length 512.
2. Multi-Task Regression (MTR): Predicting 200 RDKit properties. Labels are mean-normalized.

Tokenizer:

• Dictionary of common SMILES characters
• Maximum vocabulary size: 591 tokens

Optimization:

• Patience: Early stopping set to one pass through the dataset to ensure full coverage
• Hyperparameter search: Random search (50 configs) varying hidden size, attention heads, dropout, intermediate size, hidden layers, and learning rate. Note: The precise configuration of the winning models that were scaled to 77M is absent from the paper.

Models

• Architecture: Based on RoBERTa (HuggingFace implementation)
• Parameter scale: Models ranged between 5M and 46M parameters
• Selection: Top 5 configurations from the 5M-dataset random search were trained on the full 77M dataset
• Checkpoints: Pre-trained weights are hosted by DeepChem on Hugging Face. Direct links include DeepChem/ChemBERTa-77M-MTR and DeepChem/ChemBERTa-77M-MLM (Note: Model cards are currently empty).
• Code Reference: While the DeepChem repository is referenced for code, isolated training scripts tailored to recreate ChemBERTa-2’s exact pipeline are not separated from the generalized deepchem library tooling.

Evaluation

Benchmarks were performed on MoleculeNet using DeepChem.

Hardware

• Compute: AWS EC2 instances with Nvidia T4 GPUs
• Strategy: AWS Spot instances were used to reduce cost; implemented frequent checkpointing to handle interruptions.
• Note: For MTR, they wrote a custom data loader wrapper around HuggingFace’s text loader to handle CSV parsing efficiency, as the default CSV loader was a major bottleneck for the 200-element target vectors.

Paper Information

Citation: Ahmad, W., Simon, E., Chithrananda, S., Grand, G., & Ramsundar, B. (2022). ChemBERTa-2: Towards Chemical Foundation Models. arXiv preprint arXiv:2209.01712. https://doi.org/10.48550/arXiv.2209.01712

Publication: arXiv 2022 (Presented at 2021 ELLIS ML for Molecule Discovery Workshop)

Additional Resources:

• ChemBERTa-1 Paper

@misc{ahmadChemBERTa2ChemicalFoundation2022,
  title = {{{ChemBERTa-2}}: {{Towards Chemical Foundation Models}}},
  shorttitle = {{{ChemBERTa-2}}},
  author = {Ahmad, Walid and Simon, Elana and Chithrananda, Seyone and Grand, Gabriel and Ramsundar, Bharath},
  year = 2022,
  month = sep,
  number = {arXiv:2209.01712},
  eprint = {2209.01712},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2209.01712},
  urldate = {2025-12-25},
  archiveprefix = {arXiv}
}

This is a Methodological ($\Psi_{\text{Method}}$) paper. It proposes an architecture adaptation (Chemformer based on BART) and a specific pre-training strategy (“Combined” masking and augmentation). The paper validates this method by benchmarking against established models on multiple tasks, including direct synthesis, retrosynthesis, and molecular optimization. It also includes a secondary Resource ($\Psi_{\text{Resource}}$) contribution by making the pre-trained models and code available.

Motivation: Computational Bottlenecks in Cheminformatics

Existing Transformer models for cheminformatics are often developed for single applications and are computationally expensive to train from scratch. For example, training a Molecular Transformer for reaction prediction can take days, limiting hyperparameter exploration. Self-supervised pre-training (like BERT or T5) has significantly advanced NLP by reducing fine-tuning time and improving performance. In chemistry, applications have traditionally focused on task-specific datasets or encoder-only architectures, which perform poorly on sequence generation tasks. The authors aim to use transfer learning on a large unlabelled dataset to create a model that converges quickly and performs well across diverse sequence-to-sequence and discriminative tasks.

Core Innovation: BART Architecture and Combined Pre-training

The primary insight lies in the adaptation of the BART architecture for chemistry and the introduction of a “Combined” self-supervised pre-training task.

• Architecture: Chemformer uses the BART encoder-decoder structure, allowing it to handle both discriminative (property prediction) and generative (reaction prediction) tasks efficiently. This provides an alternative to encoder-only (BERT) or decoder-only (GPT) models.
• Combined Pre-training: The authors introduce a task that applies both Span Masking (randomly replacing tokens with <mask>) and SMILES Augmentation (permuting atom order) simultaneously. Formally, given a canonical SMILES sequence $x$, a corrupted sequence $\tilde{x} = \text{Mask}(\text{Augment}(x))$ is generated. The model is trained using an autoregressive cross-entropy loss to reconstruct the canonical sequence from the corrupted input: $$ \mathcal{L}_{\text{pre-train}} = -\sum_{t=1}^{|x|} \log P(x_t \mid x_{<t}, \tilde{x}) $$
• Tunable Augmentation: A downstream augmentation strategy is proposed where the probability of augmenting the input/output SMILES ($p_{aug}$) is a tunable hyperparameter, performed on-the-fly.

Experimental Setup and Pre-training Tasks

The authors pre-trained Chemformer on 100 million molecules from ZINC-15 and fine-tuned it on three distinct task types:

1. Seq2Seq Reaction Prediction:
   • Direct Synthesis: USPTO-MIT dataset (Mixed and Separated).
   • Retrosynthesis: USPTO-50K dataset.
2. Molecular Optimization: Generating molecules with improved properties (LogD, solubility, clearance) starting from ChEMBL matched molecular pairs.
3. Discriminative Tasks:
   • QSAR: Predicting properties (ESOL, FreeSolv, Lipophilicity) from MoleculeNet.
   • Bioactivity: Predicting pXC50 values for 133 genes using ExCAPE data.

Ablation studies compared three pre-training strategies (Masking, Augmentation, Combined) against a randomly initialized baseline.

Results, Trade-offs, and Conclusions

• Performance: Chemformer achieved competitive top-1 accuracy on USPTO-MIT (91.3% Mixed) and USPTO-50K (53.6-54.3%), outperforming the Augmented Transformer and graph-based models (GLN, GraphRetro).
• Convergence Speed: Pre-training significantly accelerated training; fine-tuning for just 20 epochs (30 mins) outperformed the previous baselines trained for significantly longer.
• Pre-training Tasks: The “Combined” task generally performed best for reaction prediction and bioactivity, while “Masking” was superior for molecular optimization.
• Augmentation Trade-off: The augmentation strategy improved top-1 accuracy but significantly degraded top-5/10 accuracy because beam search outputs became populated with augmented versions of the same molecule. This presents a considerable limitation for practical applications like retrosynthesis mapping, where retrieving a diverse set of candidate reactions is often critical.
• Discriminative Evaluation Caveats: Chemformer underperformed specialized baselines (like D-MPNN or MolBERT) on small discriminative datasets. The authors note that direct comparison is difficult: Chemformer was trained simultaneously on multiple subtasks (multi-task learning), while the literature baselines were trained and tuned on each subtask separately. Additionally, the Chemformer encoder uses fewer than 20M parameters compared to MolBERT’s approximately 85M, and Chemformer’s pre-training does not include molecular property objectives.
• Pre-training Data Scope: The 100M pre-training dataset from ZINC-15 was selected with constraints on molecular weight ($\le 500$ Da) and LogP ($\le 5$), focusing the learned representations on small, drug-like molecules.

Reproducibility Details

Data

Note: The primary GitHub repository for Chemformer was officially archived on February 11, 2026. Pre-trained weights and datasets used in the paper are still hosted externally on Box. Active development of Chemformer models has moved to the AiZynthModels repository.

The following datasets were used for pre-training and benchmarking.

Preprocessing:

• Tokenization: Regex-based tokenization (523 tokens total) derived from ChEMBL 27 canonical SMILES.
• Augmentation: SMILES enumeration (permuting atom order) used for pre-training and on-the-fly during fine-tuning ($p_{aug}=0.5$ for Seq2Seq, $p_{aug}=1.0$ for discriminative).

Algorithms

• Pre-training Tasks:
  1. Masking: Span masking (BART style).
  2. Augmentation: Input is a randomized SMILES; target is canonical SMILES.
  3. Combined: Input is augmented then masked; target is canonical SMILES.
• Optimization:
  • Optimizer: Adam ($\beta_1=0.9, \beta_2=0.999$).
  • Schedule: Linear warm-up (8000 steps) for pre-training; One-cycle schedule for fine-tuning.
• Inference: Beam search with width 10 for Seq2Seq tasks. Used molbart/inference_score.py and molbart/retrosynthesis/round_trip_inference.py for standard and round-trip validation.

Models

Two model sizes were trained. Both use the Pre-Norm Transformer layout with GELU activation.

Evaluation

Comparisons relied on Top-N accuracy for reaction tasks and validity metrics for optimization.

Hardware

• Compute: 4 NVIDIA V100 GPUs (batch size 128 per GPU).
• Training Time:
  • Pre-training: 2.5 days (Base) / 6 days (Large) for 1M steps.
  • Fine-tuning: ~20-40 epochs for reaction prediction (<12 hours).

Paper Information

Citation: Irwin, R., Dimitriadis, S., He, J., & Bjerrum, E. J. (2022). Chemformer: a pre-trained transformer for computational chemistry. Machine Learning: Science and Technology, 3(1), 015022. https://doi.org/10.1088/2632-2153/ac3ffb

Publication: Machine Learning: Science and Technology 2022

@article{irwinChemformerPretrainedTransformer2022,
  title = {Chemformer: A Pre-Trained Transformer for Computational Chemistry},
  shorttitle = {Chemformer},
  author = {Irwin, Ross and Dimitriadis, Spyridon and He, Jiazhen and Bjerrum, Esben Jannik},
  year = 2022,
  month = jan,
  journal = {Machine Learning: Science and Technology},
  volume = {3},
  number = {1},
  pages = {015022},
  publisher = {IOP Publishing},
  issn = {2632-2153},
  doi = {10.1088/2632-2153/ac3ffb}
}

This is primarily a Method paper ($\Psi_{\text{Method}}$), with a significant Resource component ($\Psi_{\text{Resource}}$).

It is a methodological investigation because it systematically evaluates a specific architecture (Transformers/RoBERTa) against established State-of-the-Art (SOTA) baselines like directed Message Passing Neural Networks (D-MPNNs) to determine “how well does this work?” in the chemical domain. It ablates dataset size, tokenization, and input representation.

It is also a resource paper as it introduces “PubChem-77M,” a curated dataset of 77 million SMILES strings designed to facilitate large-scale self-supervised pretraining for the community.

Overcoming Data Scarcity in Property Prediction

The primary motivation is data scarcity in molecular property prediction. Graph Neural Networks (GNNs) achieve strong performance on property prediction tasks when provided with sufficient labeled data. Generating these labels requires costly and time-consuming laboratory testing, leading to severe data scarcity in specialized chemical domains.

Massive quantities of unlabeled chemical structure data exist in the form of SMILES strings. Inspired by the success of Transformers in NLP, where self-supervised pretraining on large corpora yields strong transfer learning, the authors aim to use these unlabeled datasets to learn effective molecular representations. Additionally, Transformers benefit from a mature software ecosystem (HuggingFace) that offers efficiency advantages over GNNs.

Pretraining Scaling Laws and Novelty

Previous works applied Transformers to SMILES strings. This paper advances the field by systematically evaluating scaling laws and architectural components for this domain. Specifically:

• Scaling Analysis: It explicitly tests how pretraining dataset size (100K to 10M) impacts downstream performance.
• Tokenizer Comparison: It compares standard NLP Byte-Pair Encoding (BPE) against a chemically-aware “SmilesTokenizer”.
• Representation Comparison: It evaluates if the robust SELFIES string representation offers advantages over standard SMILES in a Transformer context.

Experimental Setup: Pretraining and Finetuning

The authors trained ChemBERTa (based on RoBERTa) using Masked Language Modeling (MLM) on subsets of the PubChem dataset. The core training objective minimizes the cross-entropy loss over a corrupted input where a subset of basic tokens, denoted by $\mathcal{M}$, are masked:

$$ \mathcal{L}_{\text{MLM}} = - \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \log P(x_i \mid x_{\setminus \mathcal{M}}; \theta) $$

where $x_i$ is the exact masked token, $x_{\setminus \mathcal{M}}$ is the corrupted SMILES context string, and $\theta$ represents the network parameters.

• Pretraining: Models were pretrained on dataset sizes of 100K, 250K, 1M, and 10M compounds.
• Baselines: Performance was compared against D-MPNN (Graph Neural Network), Random Forest (RF), and SVM using 2048-bit Morgan Fingerprints.
• Downstream Tasks: Finetuning was performed individually on small MoleculeNet classification tasks: BBBP (blood-brain barrier), ClinTox (clinical toxicity), HIV, and Tox21 (p53 stress-response). This poses a transfer learning challenge, as the model must adapt from pretraining on 10 million molecules to classifying datasets ranging from ~1.5K to ~41K examples.
• Ablations:
  • Tokenization: BPE vs. SmilesTokenizer on the 1M dataset, evaluated on Tox21.
  • Input: SMILES vs. SELFIES strings on the Tox21 task.

Results vs. Graph Neural Network Baselines

The main comparison between ChemBERTa (pretrained on 10M compounds) and Chemprop baselines on MoleculeNet tasks is summarized below (Table 1 from the paper):

• Scaling Improvements & Training Dynamics: Performance scales predictably with pretraining data size. Increasing data from 100K to 10M improved ROC-AUC by +0.110 and PRC-AUC by +0.059 on average across BBBP, ClinTox, and Tox21 (HIV was omitted due to resource constraints). Notably, researchers had to halt pretraining on the 10M subset after just 3 epochs due to overfitting, suggesting that simple 15% token masking might not provide a sufficiently difficult learning curvature for large-scale chemical representation.
• Performance Limits vs. GNNs: ChemBERTa generally performs below the D-MPNN baseline. On the Tox21 dataset, ChemBERTa-10M achieved a higher ROC-AUC (0.728) than D-MPNN (0.688); nonetheless, it recorded a substantially lower PRC-AUC (0.207 vs 0.429). This gap indicates that current Transformer iterations lack the explicit inductive biases of graph algorithms and struggle with the severe class imbalances typical of chemical datasets.
• Ablation Limitations (Tokenization & SELFIES): The authors’ ablation studies for tokenization (SmilesTokenizer narrowly beating BPE) and input representation (SELFIES performing comparably to SMILES) were evaluated exclusively on the single Tox21 task. Deriving broad architectural conclusions regarding “semantically-aware tokenization” or string robustness from an $N=1$ empirical evaluation is a significant limitation of the study. Broader benchmarking is required to validate these findings.
• Interpretability: Attention heads organically learn to track chemically relevant substructures (like specific functional groups and aromatic rings), mimicking the inductive biases of graph convolutions.

Reproducibility Details

Data

The authors curated a massive dataset for pretraining and utilized standard benchmarks for evaluation.

• Pretraining Data: PubChem-77M.
  • Source: 77 million unique SMILES from PubChem.
  • Preprocessing: Canonicalized and globally shuffled.
  • Subsets used: 100K, 250K, 1M, and 10M subsets.
  • Availability Note: The authors provided a direct link to the canonicalized 10M compound subset used for their largest experiments. Full reproducibility of the smaller (100K, 250K, 1M) or full 77M sets may require re-extracting from PubChem.
• Evaluation Data: MoleculeNet.
  • Tasks: BBBP (2,039), ClinTox (1,478), HIV (41,127), Tox21 (7,831).
  • Splitting: 80/10/10 train/valid/test split using a scaffold splitter to ensure chemical diversity between splits.

Algorithms

The core training methodology mirrors standard BERT/RoBERTa procedures adapted for chemical strings.

• Objective: Masked Language Modeling (MLM) with 15% token masking.
• Tokenization:
  • BPE: Byte-Pair Encoder (vocab size 52K).
  • SmilesTokenizer: Regex-based custom tokenizer available in DeepChem (documented here).
• Sequence Length: Maximum sequence length of 512 tokens.
• Finetuning: Appended a linear classification layer; backpropagated through the base model for up to 25 epochs with early stopping on ROC-AUC.

Models

• Architecture: RoBERTa (via HuggingFace).
  • Layers: 6
  • Attention Heads: 12 (72 distinct mechanisms total).
  • Implementation Note: The original training notebooks and scripts are maintained in the authors’ bert-loves-chemistry repository, alongside the primary downstream tasks integrated into DeepChem. A full Tox21 transfer learning tutorial has been incorporated into the DeepChem repository.
• Baselines (via Chemprop library):
  • D-MPNN: Directed Message Passing Neural Network with default hyperparameters.
  • RF/SVM: Scikit-learn Random Forest and SVM using 2048-bit Morgan fingerprints (RDKit).

Evaluation

Performance is measured using dual metrics to account for class imbalance common in toxicity datasets.

Hardware

• Compute: Single NVIDIA V100 GPU.
• Training Time: Approximately 48 hours for the 10M compound subset.
• Carbon Footprint: Estimated 17.1 kg $\text{CO}_2\text{eq}$ (offset by Google Cloud).

Artifacts

Reproducibility status: Partially Reproducible. Code and pre-trained models are available, and the 10M pretraining subset is downloadable. However, smaller subsets (100K, 250K, 1M) may need re-extraction from PubChem, and exact hyperparameter details for finetuning (learning rate, batch size) are not fully specified in the paper.

Paper Information

Citation: Chithrananda, S., Grand, G., & Ramsundar, B. (2020). ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction. arXiv preprint arXiv:2010.09885. https://doi.org/10.48550/arXiv.2010.09885

Publication: arXiv 2020 (Preprint)

Additional Resources:

• HuggingFace Model Hub (ChemBERTa-zinc-base-v1) - Additional pre-trained variations on PubChem & ZINC datasets are available on the author’s seyonec HF profile.
• bert-loves-chemistry GitHub Repository - Notebooks and scripts used for MLM pretraining and finetuning evaluations.

BibTeX

@misc{chithranandaChemBERTaLargeScaleSelfSupervised2020,
  title = {{{ChemBERTa}}: {{Large-Scale Self-Supervised Pretraining}} for {{Molecular Property Prediction}}},
  shorttitle = {{{ChemBERTa}}},
  author = {Chithrananda, Seyone and Grand, Gabriel and Ramsundar, Bharath},
  year = 2020,
  month = oct,
  number = {arXiv:2010.09885},
  eprint = {2010.09885},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2010.09885},
  urldate = {2025-12-24},
  archiveprefix = {arXiv}
}

This is primarily a Method paper. It proposes a unified framework that generalizes previous discrete score-based models (SMLD and DDPM) into continuous-time Stochastic Differential Equations (SDEs). The paper introduces novel algorithms for sampling (Predictor-Corrector) and likelihood computation (Probability Flow ODE), validated by achieving state-of-the-art results. It also contains elements of Systematization by showing how existing methods are special cases of this broader framework.

What is the motivation?

Prior successful generative models, specifically Score Matching with Langevin Dynamics (SMLD) and Denoising Diffusion Probabilistic Models (DDPM), operate by sequentially corrupting data with slowly increasing noise and learning to reverse the process. Both methods treat the noise scales as a finite set of discrete steps. The authors aim to generalize this to a continuum of noise scales by modeling the diffusion process as a Stochastic Differential Equation (SDE). This continuous formulation enables:

• Flexible sampling: Use of general-purpose SDE solvers.
• Exact likelihood computation: Via connection to Neural ODEs.
• Controllable generation: Solving inverse problems (inpainting, colorization) without retraining.

What is the novelty here?

The core novelty is the SDE framework for score-based generative modeling:

• Continuous Generalization: Proving that SMLD and DDPM noise perturbations correspond to discretizations of Variance Exploding (VE) SDEs and Variance Preserving (VP) SDEs, respectively.
• Reverse-Time SDE: Leveraging Anderson’s result (Anderson, 1982: a result on time-reversal of diffusion processes showing that the reverse is also a diffusion, with the forward drift reversed and a correction term involving the score of the marginal density) that the reverse of a diffusion process is also a diffusion process, governed by the score (gradient of log density).
• Predictor-Corrector (PC) Samplers: A hybrid sampling strategy where a numerical SDE solver (Predictor) estimates the next step, and a score-based MCMC approach (Corrector) corrects the marginal distribution.
• Probability Flow ODE: Deriving a deterministic ODE that shares the same marginal densities as the SDE, enabling near-exact likelihood computation (accuracy is limited by both numerical ODE solver discretization and variance of the unbiased Hutchinson trace estimator) and latent space manipulation.
• Sub-VP SDE: A new SDE class proposed to improve likelihoods by bounding variance tighter than the VP SDE.

What experiments were performed?

The authors validated the framework on standard image benchmarks:

• Datasets: CIFAR-10 (32x32), CelebA (64x64), LSUN (Bedroom, Church), and CelebA-HQ (256x256 and 1024x1024).
• Ablation Studies: Comparing samplers (Ancestral vs. Reverse Diffusion vs. Probability Flow vs. PC) and SDE types (VE, VP, sub-VP).
• Architecture Search: Exploring improvements like FIR up/downsampling, rescaling skip connections, and increasing depth (leading to NCSN++ and DDPM++ architectures).
• Likelihood Evaluation: Computing Negative Log-Likelihood (NLL) in bits/dim using the Probability Flow ODE.
• Inverse Problems: Testing class-conditional generation, inpainting, and colorization using the conditional reverse-time SDE.

What outcomes/conclusions?

• Record Performance: The NCSN++ cont. (deep, VE) model achieved an Inception Score of 9.89 and FID of 2.20 on CIFAR-10 (as of ICLR 2021).
• High-Fidelity Generation: First score-based model to generate 1024x1024 images (CelebA-HQ).
• Competitive Likelihoods: The DDPM++ cont. (deep, sub-VP) model achieved 2.99 bits/dim on uniformly dequantized CIFAR-10, a record at the time.
• Sampling Efficiency: PC samplers consistently outperformed predictor-only methods (like standard ancestral sampling) for the same computational cost.
• Controllable Generation: Successful application to inpainting and colorization using a single unconditional model.

Reproducibility Details

Data

• CIFAR-10: Used for main benchmarking (FID, Inception Score, NLL).
• CelebA-HQ: Used for high-resolution experiments at 256x256 and 1024x1024.
• LSUN: Bedroom and Church Outdoor categories used for inpainting/colorization tests.
• Preprocessing: CIFAR-10 images are 32x32; CelebA pre-processed to 64x64 following Song & Ermon (2020). Data is typically scaled to $[0, 1]$ or standardized depending on the specific SDE config.

Algorithms

Forward SDEs:

Here $dw$ denotes a Wiener process increment (a small, independent Gaussian noise burst at each timestep).

• VE SDE (Variance Exploding): $dx = \sqrt{\frac{d[\sigma^2(t)]}{dt}} dw$. Corresponds to SMLD. Used with $\sigma_{\min}=0.01$ and $\sigma_{\max}$ chosen via heuristics.
• VP SDE (Variance Preserving): $dx = -\frac{1}{2}\beta(t)x dt + \sqrt{\beta(t)} dw$. Corresponds to DDPM.
• Sub-VP SDE: $dx = -\frac{1}{2}\beta(t)x dt + \sqrt{\beta(t)(1 - e^{-2\int_0^t \beta(s)ds})} dw$. Bounded variance, good for likelihoods.

Reverse-Time SDE Solver (Predictor):

• Discretized via Reverse Diffusion Sampling, which matches the forward discretization.
• Euler-Maruyama solver used for continuously-trained models.

Corrector Algorithm:

• Langevin MCMC: Applies annealed Langevin dynamics: adds noise and takes a score-guided gradient step to correct the marginal distribution at each timestep.
• PC Sampling: Alternates between one step of the Predictor and one step of the Corrector.
• Signal-to-Noise Ratio ($r$): A hyperparameter for the corrector step size. Tuned values: $r \approx 0.16$ for VE SDEs on CIFAR-10.

Models

• NCSN++: Optimized architecture for VE SDEs. Key features:
  • 4 residual blocks per resolution.
  • BigGAN-type residual blocks.
  • Rescaling skip connections by $1/\sqrt{2}$.
  • FIR (Finite Impulse Response) up/downsampling.
  • No progressive growing for output.
• DDPM++: Optimized architecture for VP/sub-VP SDEs. Similar to NCSN++ but without FIR upsampling and no progressive growing.
• Deep Variants: “cont. (deep)” models double the depth (from 4 to 8 blocks per resolution) for SOTA results.
• Conditioning: Time $t$ is conditioned via random Fourier feature embeddings (scale 16) for continuous models.

Evaluation

Metrics:

• FID (Fréchet Inception Distance): Computed on 50k samples.
• Inception Score: Reported for CIFAR-10.
• NLL (Negative Log-Likelihood): Reported in bits/dim on uniformly dequantized data using the Probability Flow ODE.

Denoising: A single denoising step using Tweedie’s formula is applied at the end of sampling to remove residual noise, which significantly improves FID.

Hardware

Training:

• Batch size: 128 for CIFAR-10, 64 for LSUN, 8 for high-res CelebA-HQ.
• Iterations: Models trained for ~1.3M iterations. High-res CelebA-HQ trained for 2.4M iterations.
• EMA: Exponential Moving Average rate of 0.999 used for VE models, 0.9999 for VP models.

Paper Information

Citation: Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021. https://arxiv.org/abs/2011.13456

Publication: ICLR 2021

@inproceedings{song2021scorebased,
  title     = {Score-Based Generative Modeling through Stochastic Differential Equations},
  author    = {Song, Yang and Sohl-Dickstein, Jascha and Kingma, Diederik P and Kumar, Abhishek and Ermon, Stefano and Poole, Ben},
  booktitle = {International Conference on Learning Representations},
  year      = {2021},
  url       = {https://openreview.net/forum?id=PxTIG12RRHS}
}

Additional Resources:

• GitHub Repository
• Score Matching and Denoising Autoencoders

This is a Theory Paper.

Its primary contribution is a formal mathematical derivation connecting two previously distinct techniques: Score Matching (SM) and Denoising Autoencoders (DAE). It provides the “why” behind the empirical success of DAEs by grounding them in the probabilistic framework of energy-based models. It relies on proofs and equivalence relations (e.g., $J_{ESMq_{\sigma}} \sim J_{DSMq_{\sigma}}$).

What is the motivation?

The paper bridges a gap between two successful but disconnected approaches in unsupervised learning:

1. Denoising Autoencoders (DAE): Empirically successful for pre-training deep networks. They previously lacked a clear probabilistic interpretation.
2. Score Matching (SM): A theoretically sound method for estimating unnormalized density models that avoids the partition function problem but requires computing expensive second derivatives.

By connecting them, the authors aim to define a proper probabilistic model for DAEs (allowing sampling/ranking) and find a simpler way to apply score matching that avoids second derivatives.

What is the novelty here?

The core novelty is the Denoising Score Matching (DSM) framework and the proof of its equivalence to DAEs. Key contributions include:

• Equivalence Proof: Showing that training a DAE with Gaussian noise is equivalent to matching the score of a model against a non-parametric Parzen density estimator of the data.
• Denoising Score Matching ($J_{DSM}$): A new objective that learns a score function by trying to denoise corrupted samples. This avoids the explicit second derivatives required by standard Implicit Score Matching ($J_{ISM}$).
• Explicit Energy Function: Deriving the specific energy function $E(x;\theta)$ that corresponds to the standard sigmoid DAE architecture.
• Justification for Tied Weights: Providing a theoretical justification for tying encoder and decoder weights, which arises naturally from differentiating the energy function.

What experiments were performed?

The validation in this theoretical paper is purely mathematical and focuses on formal proofs:

• Derivation of Equivalence: The paper formally proves the chain of equivalences: $$J_{ISMq_{\sigma}} \sim J_{ESMq_{\sigma}} \sim J_{DSMq_{\sigma}} \sim J_{DAE\sigma}$$ where $q_{\sigma}$ is the Parzen density estimate.
• Appendix Proof: A detailed proof is provided to show that Explicit Score Matching ($J_{ESM}$) on the Parzen density is equivalent to the proposed Denoising Score Matching ($J_{DSM}$) objective.

What outcomes/conclusions?

• Theoretical Unification: DAE training is formally equivalent to Score Matching on a smoothed data distribution ($q_{\sigma}$).
• New Training Objective: The $J_{DSM}$ objective offers a computationally efficient way to perform score matching (no Hessian required) by using a denoising objective.
• Probabilistic Interpretation: DAEs can now be understood as Energy-Based Models (EBMs), allowing for operations like sampling (via Hybrid Monte Carlo) and likelihood ranking, which were previously ill-defined for standard autoencoders.
• Regularization Insight: The smoothing kernel width $\sigma$ in the Parzen estimator corresponds to the noise level in the DAE. This suggests that DAEs are learning a regularized version of the score, which may explain their robustness.

Key Concepts Explained

1. “Score” and “Score Matching”

What does “score” actually mean?

In this paper (and probabilistic modeling generally), the score is the gradient of the log-density with respect to the data vector $x$.

• Definition: $\psi(x) = \nabla_x \log p(x)$.
• Intuition: It is a vector field pointing in the direction of highest probability increase. Crucially, calculating the score avoids the intractable partition function $Z$, because $\nabla_x \log p(x) = \nabla_x \log \tilde{p}(x) - \nabla_x \log Z = \nabla_x \log \tilde{p}(x)$. The constant $Z$ vanishes upon differentiation.

What is Score Matching?

Score Matching is a training objective for unnormalized models. It minimizes the squared Euclidean distance between the model’s score $\psi(x;\theta)$ and the data’s true score $\nabla_x \log q(x)$.

2. The Parzen Density Estimator

What is it?

It is a non-parametric method for estimating a probability density function from finite data. It places a smooth kernel (here, a Gaussian) centered at every data point in the training set $D_n$.

• Formula: $q_{\sigma}(\tilde{x}) = \frac{1}{n} \sum_{t=1}^n \mathcal{N}(\tilde{x}; x^{(t)}, \sigma^2 I)$.

Why smooth the data?

1. To define the score: The empirical data distribution is a set of Dirac deltas (spikes). The gradient (score) of a Dirac delta is undefined. Smoothing creates a differentiable surface, allowing a valid target score $\nabla_{\tilde{x}} \log q_{\sigma}(\tilde{x})$ to be computed.

2. To model corruption: The Parzen estimator with Gaussian kernels mathematically models the process of taking a clean data point $x$ and adding Gaussian noise - the exact procedure used in Denoising Autoencoders.

3. Why avoiding second derivatives matters

Standard Implicit Score Matching (ISM) eliminates the need for the unknown data score, but introduces a new cost: it requires computing the trace of the Hessian (the sum of second partial derivatives) of the log-density.

• The Cost: For high-dimensional data (like images) and deep networks, computing second derivatives is computationally prohibitive ($O(d^2)$ operations per sample).
• This paper shows that Denoising Score Matching (DSM) allows you to bypass Hessian computation entirely. By using the Parzen target, the objective simplifies to matching a first-order vector, making it scalable to deep neural networks.

4. The equivalence chain - why each step?

The chain $J_{ISMq_{\sigma}} \sim J_{ESMq_{\sigma}} \sim J_{DSMq_{\sigma}} \sim J_{DAE\sigma}$ connects the concepts.

• $J_{ISMq_{\sigma}} \sim J_{ESMq_{\sigma}}$ (Implicit $\to$ Explicit): Why: Integration by parts. This is Hyvärinen’s original proof (2005): integration by parts moves the derivative from $\psi$ onto the data density $q$, producing a term involving $q$’s gradient (the score). The boundary term vanishes because $q_{\sigma}$ decays to zero at infinity (Hyvärinen’s 2005 regularity condition for Implicit Score Matching). The result allows replacing the unknown data score with a computable term involving only the model’s score and its Jacobian.

• $J_{ESMq_{\sigma}} \sim J_{DSMq_{\sigma}}$ (Explicit $\to$ Denoising): Why: The explicit score of the Parzen density is known. When $x$ is perturbed to $\tilde{x}$ by Gaussian noise $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$, the gradient of the log-density pointing back to the mean is exactly $\frac{1}{\sigma^2}(x - \tilde{x})$. Minimizing the error against the true score becomes minimizing the error against this restoration vector.

• $J_{DSMq_{\sigma}} \sim J_{DAE\sigma}$ (Denoising $\to$ Autoencoder): Why: Algebraic substitution. If you define the model’s score $\psi(\tilde{x};\theta)$ to be proportional to the reconstruction error ($\propto x^r - \tilde{x}$), the score matching loss $J_{DSM}$ becomes proportional to the standard autoencoder squared loss $|x^r - x|^2$.

5. Energy-Based Models (EBMs) connection

What is an EBM?

An EBM defines a probability distribution via an energy function $E(x;\theta)$, where $p(x;\theta) \propto e^{-E(x;\theta)}$.

Why standard autoencoders lack probabilistic interpretation:

A standard autoencoder acts as a deterministic map $x \to x^r$, providing a reconstruction error. It lacks a normalization constant or a defined density function to support sampling or probability queries.

What does this enable?

By proving the equivalence, the DAE is formally defined as an EBM. This enables:

1. Sampling: Using MCMC methods (like Hybrid Monte Carlo) to generate new data from the DAE.
2. Ranking: Calculating the energy of inputs to determine which are more “likely” or “normal” (useful for anomaly detection).

6. The specific energy function form

The function is:

$$E(x; W, b, c) = - \frac{1}{\sigma^2} \left( \langle c, x \rangle - \frac{1}{2}|x|^2 + \sum_{j=1}^{d_h} \text{softplus}(\langle W_j, x \rangle + b_j) \right)$$

Why does it have that specific form?

It was derived via integration to ensure its derivative matches the DAE architecture. The authors worked backward from the DAE’s reconstruction function (sigmoid + linear) to find the scalar field that generates it.

Where does the quadratic term come from?

The score (derivative) needs to look like $\psi(x) \propto c + x + f(Wx + b)$.

• The term $+x$ (i.e., $\nabla_x (\frac{1}{2}|x|^2) = x$) in the derivative must come from a $-\frac{1}{2}|x|^2$ term in the energy potential.

How does differentiating it recover the DAE reconstruction?

• $\nabla_x \sum_j \text{softplus}(\langle W_j, x \rangle + b_j) = W^T \sigma(Wx + b)$ (The encoder part).
• $\nabla_x \langle c, x \rangle = c$ (The bias).
• $\nabla_x (-\frac{1}{2}|x|^2) = -x$ (The input subtraction).
• Result: $-\nabla_x E \propto c + W^T h - x = x^r - x$.

7. “Tied weights” justification

What does it mean for weights to be “tied”?

The decoder matrix is the transpose of the encoder matrix ($W^T$).

Why is this theoretically justified?

Because the reconstruction function is interpreted as the gradient of an energy function. A vector field can only be the gradient of a scalar field if its Jacobian is symmetric.

• In the DAE energy derivative, the encoder contributes $W^T \sigma(Wx + b)$. If the decoder used a separate matrix $U$, the resulting vector field would not be a valid gradient of any scalar energy function (unless $U = W^T$).
• Therefore, for a DAE to correspond to a valid probabilistic Energy-Based Model, the weights must be tied.

The necessity of tied weights:

Within this parametrization, tied weights are a mathematical necessity: a separate decoder matrix $U \neq W^T$ would make the reconstruction function an invalid gradient of any scalar energy, breaking the EBM correspondence.

Reproducibility Details

Since this is a theoretical paper, the “reproducibility” lies in the mathematical formulations derived.

Data

• Input Data ($D_n$): The theory assumes a set of training examples $D_n = {x^{(1)}, …, x^{(n)}}$ drawn from an unknown true pdf $q(x)$.
• Parzen Density Estimate ($q_{\sigma}$): The theoretical targets are derived from a kernel-smoothed empirical distribution: $$q_{\sigma}(\tilde{x}) = \frac{1}{n} \sum_{t=1}^n q_{\sigma}(\tilde{x}|x^{(t)})$$ where the kernel is an isotropic Gaussian of variance $\sigma^2$.

Algorithms

1. Denoising Score Matching (DSM) Objective

The paper proposes this objective as a tractable alternative to standard score matching. It minimizes the distance between the model score and the gradient of the log-noise density:

$$J_{DSMq_{\sigma}}(\theta) = \mathbb{E}_{q_{\sigma}(x,\tilde{x})} \left[ \frac{1}{2} \left| \psi(\tilde{x};\theta) - \frac{\partial \log q_{\sigma}(\tilde{x}|x)}{\partial \tilde{x}} \right|^2 \right]$$

For Gaussian noise, the target score is simply $\frac{1}{\sigma^2}(x - \tilde{x})$.

2. Equivalence Chain

The central result connects four objectives:

$$J_{ISMq_{\sigma}} \sim J_{ESMq_{\sigma}} \sim J_{DSMq_{\sigma}} \sim J_{DAE\sigma}$$

This implies optimizing the DAE reconstruction error is minimizing a score matching objective.

Models

1. The Denoising Autoencoder (DAE)

• Corruption: Additive isotropic Gaussian noise $\tilde{x} = x + \epsilon, \epsilon \sim \mathcal{N}(0, \sigma^2 I)$.
• Encoder: $h = \text{sigmoid}(W\tilde{x} + b)$.
• Decoder: $x^r = W^T h + c$ (Tied weights $W$).
• Loss: Squared reconstruction error $\frac{1}{2\sigma^4} |x^r - x|^2$.

2. The Corresponding Energy Function

To make the DAE equivalent to Score Matching, the underlying Energy-Based Model $p(x;\theta) \propto e^{-E(x;\theta)}$ must have the following energy function:

$$E(x; W, b, c) = - \frac{1}{\sigma^2} \left( \langle c, x \rangle - \frac{1}{2}|x|^2 + \sum_{j=1}^{d_h} \text{softplus}(\langle W_j, x \rangle + b_j) \right)$$

Note the scaling by $1/\sigma^2$ and the quadratic term $|x|^2$.

Evaluation

• Metric: Theoretical Equivalence ($\sim$).
• Condition: The equivalence holds provided $\sigma > 0$ and the density $q_{\sigma}$ is differentiable and vanishes at infinity (Hyvärinen’s 2005 regularity condition for Implicit Score Matching).

Paper Information

Citation: Vincent, P. (2011). A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7), 1661-1674. https://doi.org/10.1162/NECO_a_00142

Publication: Neural Computation 2011

@article{vincentConnectionScoreMatching2011,
  title = {A {{Connection Between Score Matching}} and {{Denoising Autoencoders}}},
  author = {Vincent, Pascal},
  year = 2011,
  month = jul,
  journal = {Neural Computation},
  volume = {23},
  number = {7},
  pages = {1661--1674},
  doi = {10.1162/NECO_a_00142}
}

Additional Resources:

• Official PDF

Key Prerequisites: Before diving in, note that for the ODE solver to guarantee a unique solution, the neural network $f(h(t), t, \theta)$ parameterizing the dynamics must be Lipschitz continuous. This ensures the Picard-Lindelöf theorem holds, preventing trajectories from crossing and guaranteeing a well-defined backward pass.

What kind of paper is this?

This is primarily a Method paper, with a strong secondary Theory component.

• Method: It proposes a novel family of deep neural network models where the derivative of the hidden state is parameterized by a neural network. It provides specific algorithms (Algorithm 1) for training these models scalably.
• Theory: It derives the adjoint sensitivity method for backpropagating through black-box ODE solvers and proves the “Instantaneous Change of Variables” theorem (Theorem 1) for continuous normalizing flows.

What is the motivation?

The authors aim to address limitations in discrete deep learning architectures:

• Discrete vs. Continuous: Existing models like Residual Networks build transformations by composing discrete steps, which can be seen as an Euler discretization of a continuous transformation. The authors investigate the limit as step sizes go to zero.
• Memory Efficiency: Backpropagating through deep discrete networks requires storing intermediate activations, leading to linear memory cost in terms of depth, which is a major bottleneck.
• Irregular Data: Recurrent Neural Networks (RNNs) struggle with data arriving at arbitrary times, typically requiring discretization into fixed bins.
• Normalizing Flow Costs: Standard normalizing flows have a bottleneck in computing the determinant of the Jacobian, which is computationally expensive ($O(D^3)$).

What is the novelty here?

The core contribution is the Neural ODE formulation: $$\frac{dh(t)}{dt} = f(h(t), t, \theta)$$ where the output is computed using a black-box differential equation solver.

Key technical innovations include:

1. Adjoint Sensitivity Method for Backprop: The authors treat the solver as a black box and compute gradients by solving a second, augmented ODE backwards in time. This allows for constant memory cost regardless of depth.
2. Adaptive Computation: The model uses modern ODE solvers that adapt evaluation steps based on error tolerance, allowing the model to trade precision for speed explicitly.
3. Continuous Normalizing Flows (CNF): By moving to continuous time, the change of variables formula simplifies from a log-determinant (cubic cost) to a trace operation (linear cost), enabling scalable generative modeling.
4. Latent ODEs: A generative time-series model that represents time-series as latent trajectories determined by a local initial state and global shared dynamics, handling irregular sampling naturally.

What experiments were performed?

The authors validated the method across three distinct domains:

1. Supervised Learning (MNIST):
   • Compared ODE-Net against a standard ResNet and a Runge-Kutta network (RK-Net).
   • Measured test error, parameter count, and memory usage.
   • Analyzed the trade-off between numerical precision (tolerance) and speed (NFE).
2. Continuous Normalizing Flows (Generative):
   • Compared CNF against standard Normalizing Flows (NF) on density estimation tasks using toy 2D datasets (Target, Two Circles, Two Moons).
   • Evaluated training loss (KL divergence) and maximum likelihood estimation.
3. Time-Series Modeling (Latent ODE):
   • Tested on a dataset of bi-directional spirals with irregular timestamps and Gaussian noise.
   • Compared Latent ODEs against RNNs and RNNs with time-concatenation on predictive RMSE.

What outcomes/conclusions?

• Efficiency: ODE-Nets achieved roughly equivalent accuracy to ResNets on MNIST (0.42% vs 0.41% error) but with constant memory cost ($O(1)$) compared to ResNet’s linear cost ($O(L)$).
• Adaptive Depth: The number of function evaluations (NFE) in ODE-Nets increases with training epoch, suggesting the model adapts its complexity as it learns.
• Generative Performance: Continuous Normalizing Flows (CNF) achieved lower KL divergence loss than standard Normalizing Flows (NF), trained with only 10,000 iterations (Adam) compared to 500,000 iterations (RMSprop) for NF. The paper acknowledges the optimizer difference (Adam vs RMSprop) as a confound, but notes the iteration gap is large enough that CNF’s efficiency advantage holds. CNF can also expand capacity by increasing width ($M$) without architectural constraints.
• Irregular Time-Series: Latent ODEs significantly outperformed RNNs across all observation counts on irregular spiral data. The advantage is most pronounced with sparse observations (0.1642 vs 0.3937 RMSE at 30 obs), and the model learns interpretable latent trajectories that switch direction smoothly.

Reproducibility Details

Data

• MNIST: Standard handwritten digit dataset used for supervised learning benchmarks.
• Toy 2D Densities: “Two Circles” and “Two Moons” distributions used for visualizing normalizing flows.
• Bi-directional Spirals: A generated dataset of 1,000 2D spirals (half clockwise, half counter-clockwise). Each spiral is sampled at 100 equally-spaced timesteps with added Gaussian noise. For training, each spiral is then subsampled without replacement to $n \in {30, 50, 100}$ irregularly-spaced observations, simulating realistic missing data.

Algorithms

1. Adjoint Sensitivity Method (Backpropagation)

To optimize the parameters of the ODE-Net, the authors use the adjoint sensitivity method to compute gradients. Standard backpropagation would require storing the activations at every step of the ODE solver, incurring a high memory cost that scales linearly with the number of steps.

Instead, this method treats the ODE solver as a “black box” and computes gradients by solving a second, augmented ODE backwards in time from the final state $t_1$ to the initial state $t_0$.

The augmented state contains three components that are solved simultaneously:

1. The State: The original hidden state $z(t)$, which is reconstructed backwards.
2. The Adjoint: The sensitivity of the loss with respect to the state, $a(t) = \partial L / \partial z(t)$.
3. The Gradient: The accumulating gradients with respect to parameters, $\partial L / \partial \theta$.

The dynamics of this augmented system are defined as: $$\frac{d}{dt}\begin{bmatrix} z(t) \ a(t) \ \partial L/\partial \theta \end{bmatrix} = \begin{bmatrix} f(z(t), t, \theta) \ -a(t)^T \frac{\partial f}{\partial z} \ -a(t)^T \frac{\partial f}{\partial \theta} \end{bmatrix}$$

Using this approach, the vector-Jacobian products (e.g., $a(t)^T \frac{\partial f}{\partial z}$) are evaluated efficiently using automatic differentiation.

Why: Reconstructing $z(t)$ backwards avoids storing the forward pass, enabling constant memory cost ($O(1)$) regardless of depth.

Origin: Adapted from Pontryagin’s maximum principle (1962) for optimal control.

import torch
import torch.nn as nn
from torchdiffeq import odeint_adjoint

class ODEFunc(nn.Module):
    def __init__(self, dim):
        super(ODEFunc, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(dim, 50),
            nn.Tanh(),
            nn.Linear(50, dim),
        )

    def forward(self, t, y):
        # Defines dy/dt = f(y, t)
        return self.net(y)

# Usage with adjoint method for O(1) memory backprop
func = ODEFunc(dim=2)
y0 = torch.tensor([[1., 0.]]) # Initial state
t = torch.linspace(0., 1., 10) # Time points to solve for

# 'odeint_adjoint' automatically handles the augmented state backward pass
out = odeint_adjoint(func, y0, t, method='dopri5')

2. Instantaneous Change of Variables (CNF)

For generative modeling, the authors introduce Continuous Normalizing Flows (CNF). In discrete normalizing flows, the probability density of a transformed variable is calculated using the change of variables theorem, which requires computing the log-determinant of the Jacobian: $\log p(z_1) = \log p(z_0) - \log |\det \frac{\partial z_1}{\partial z_0}|$. This operation is computationally expensive ($O(D^3)$) and often restricts model architectures to ensure the Jacobian is easy to compute (e.g., triangular).

Moving to continuous time simplifies this requirement. The paper proves that if the transformation is defined by an ODE, the change in log-probability follows a differential equation determined by the trace of the Jacobian: $$\frac{\partial \log p(z(t))}{\partial t} = -\text{tr}\left( \frac{\partial f}{\partial z(t)} \right)$$

The total change in log-density is obtained by integrating this value over time.

def get_trace(y, f):
    """
    Computes trace of Jacobian df/dy.
    For high dimensions, use Hutchinson's trace estimator (approximate).
    """
    tr = 0.
    for i in range(y.size(1)):
        # Gradients of f's i-th component w.r.t y's i-th component
        tr += torch.autograd.grad(f[:, i].sum(), y, create_graph=True)[0][:, i]
    return tr

# In the ODE function:
# d(log_p)/dt = -trace(df/dy)

Why: The trace operator has linear cost ($O(D)$), whereas the determinant has cubic cost ($O(D^3)$). This allows for unrestricted, “wide” architectures that are automatically bijective.

Origin: This is the “Instantaneous Change of Variables” theorem (Theorem 1), derived in Appendix A of the paper.

Models

ODE-Net (MNIST Classification):

• Input: Downsamples input twice.
• Core: 6 standard residual blocks replaced by a single ODESolve module.
• Output: Global average pooling + Fully connected layer.
• Solver: Implicit Adams method.

class ODEBlock(nn.Module):
    def __init__(self, odefunc):
        super(ODEBlock, self).__init__()
        self.odefunc = odefunc
        self.integration_time = torch.tensor([0, 1]).float()

    def forward(self, x):
        self.integration_time = self.integration_time.type_as(x)
        # Returns [x(t0), x(t1)]; we only want final state x(t1)
        out = odeint_adjoint(self.odefunc, x, self.integration_time)
        return out[1]

# ResNet-like architecture with ODE block
model = nn.Sequential(
    nn.Conv2d(1, 64, 3, 1),
    nn.ReLU(inplace=True),
    ODEBlock(ODEFunc(64)), # Continuous-depth layer replacement
    nn.BatchNorm2d(64),
    nn.AdaptiveAvgPool2d((1, 1)),
    nn.Flatten(),
    nn.Linear(64, 10)
)

Latent ODE (Time-Series):

• Encoder: RNN with 25 hidden units processing data backwards to produce $q(z_0|x)$. It runs backwards so the final RNN state summarizes the entire sequence at $t_0$, parameterizing the initial latent state $z_0$ for the forward-running ODE.
• Latent Space: 4-dimensional latent state $z_0$.
• Dynamics ($f$): Neural network with one hidden layer of 20 units.
• Decoder: Neural network with one hidden layer of 20 units computing $p(x_{t_i}|z_{t_i})$.
• Likelihood: Gaussian log-likelihood for the spiral reconstruction task. The paper also describes an optional Poisson process likelihood $\lambda(z(t))$ for event-time data (e.g., medical records), but this is not used in the spiral experiment.

Evaluation

Hardware

• Implementation: Hidden state dynamics evaluated on GPU using TensorFlow.
• Solvers: Fortran ODE solvers (LSODE, VODE) from scipy.integrate were used for the actual integration.
• Note: While the original paper used TensorFlow/Scipy, the authors later released torchdiffeq (PyTorch), which has become the standard implementation for this architecture. The code samples above reflect this modern standard.
• Interface: Python’s autograd framework bridged the TensorFlow dynamics and Scipy solvers.

Paper Information

Citation: Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018). Neural ordinary differential equations. Proceedings of the 32nd International Conference on Neural Information Processing Systems, 6572-6583.

Publication: NeurIPS 2018

@inproceedings{chen2018neural,
  title={Neural ordinary differential equations},
  author={Chen, Ricky T. Q. and Rubanova, Yulia and Bettencourt, Jesse and Duvenaud, David},
  booktitle={Proceedings of the 32nd International Conference on Neural Information Processing Systems},
  pages={6572--6583},
  year={2018}
}

Additional Resources:

• Official PyTorch Implementation

This is primarily a Method paper, as it introduces “Flow Matching” (FM), a novel simulation-free paradigm for training Continuous Normalizing Flows (CNFs) at scale. It is supported by a strong Theory basis, providing formal theorems that allow the intractable marginal vector field regression to be solved via a tractable conditional objective. It also touches on Systematization by showing that existing diffusion paths are specific instances of the proposed Gaussian probability path framework.

What is the motivation?

The paper aims to overcome the scaling limitations of Continuous Normalizing Flows (CNFs).

• Problem: Standard Maximum Likelihood training for CNFs requires expensive numerical ODE simulations during training, which scales poorly. Existing simulation-free methods often involve intractable integrals or result in biased gradients.
• Gap: Diffusion models scale well, yet they are restricted to specific, curved probability paths (e.g., VP, VE) that can result in slow sampling and long training times.
• Goal: To develop an efficient, simulation-free training method for CNFs that supports arbitrary probability paths, specifically allowing for straighter, more efficient trajectories like those from Optimal Transport.

What is the novelty here?

The core novelty is Flow Matching (FM) and specifically the Conditional Flow Matching (CFM) objective.

• Direct Vector Field Regression: The model regresses a target vector field $u_t$ that generates a desired probability path $p_t$.
• Conditional Flow Matching (CFM): The authors prove that regressing the vector field of conditional paths (e.g., $p_t(x|x_1)$ given a single data point) yields the same gradients as regressing the intractable marginal vector field. This bypasses the need to know the marginal score or vector field.
• Optimal Transport Paths: The framework enables the use of Optimal Transport (OT) displacement interpolation for probability paths. OT paths are straight lines with constant speed, leading to faster training and easier sampling.

Concurrent work note: Rectified Flow (Liu et al., 2023) and Stochastic Interpolants (Albergo & Vanden-Eijnden, 2023) were published concurrently at ICLR 2023 with structurally similar contributions under different names. All three independently propose simulation-free training of continuous flows via direct vector field regression; the differences lie in the specific interpolation schemes, theoretical framing, and experimental focus. The related notes linked in the frontmatter cover both concurrent papers.

What experiments were performed?

• Domains: 2D Checkerboard data, CIFAR-10, and ImageNet at resolutions $32 \times 32$, $64 \times 64$, and $128 \times 128$.
• Task: Unconditional generative modeling (density estimation and sample quality) and conditional super-resolution ($64 \to 256$).
• Baselines: Compared against Diffusion-based methods on the same architecture (U-Net): DDPM (Noise Matching), Score Matching (SM), and ScoreFlow.
• Ablations: Specifically compared FM with Diffusion paths vs. FM with Optimal Transport (OT) paths to isolate the benefit of the training objective vs. the path choice.

What outcomes/conclusions?

• Outperforms diffusion baselines: FM-OT consistently outperforms all diffusion-based methods (DDPM, Score Matching, ScoreFlow) in both Likelihood (NLL) and Sample Quality (FID) across CIFAR-10 and ImageNet, using the same U-Net architecture and training budget. Selected rows from Table 1 (NLL in bits per dimension, BPD; lower is better for all three metrics; “FM [w] / OT” and “FM [w] / Diffusion” refer to FM trained with OT paths and Diffusion paths respectively):

• Training stability: FM with diffusion paths (FM [w] / Diffusion) is itself a more stable alternative to diffusion training than DDPM and Score Matching, as shown by training curves in the paper (Figure 2), even before switching to OT paths. The OT path then provides further gains.
• Sampling speed: The straight trajectories of OT paths allow accurate sampling with significantly fewer function evaluations (NFE) compared to diffusion paths.
• Generality: Diffusion is a specific instance of Gaussian probability paths within FM. OT paths are a better-optimized alternative available within the same framework.
• Downstream adoption: Flow matching has been adopted beyond image generation. DynamicFlow uses it as the generative backbone for simultaneously generating ligand molecules and transforming protein pockets, extending flow matching to structure-based drug design.

Reproducibility Details

Data

• Datasets: CIFAR-10, ImageNet ($32 \times 32$, $64 \times 64$, $128 \times 128$).
• Preprocessing:
  • Images are center-cropped and resized.
  • For $32 \times 32$ and $64 \times 64$, the preprocessing follows Chrabaszcz et al. (2017).
  • Data is transformed via $\varphi(y) = 2^7(y+1)$ mapping $[-1, 1]$ pixel values to $[0, 256]$ for BPD computation.

Algorithms

1. Conditional Flow Matching (CFM) Objective

The practical training objective used is the CFM loss, which bypasses intractable marginalization:

$$\mathcal{L}_{CFM}(\theta) = \mathbb{E}_{t, q(x_1), p(x_0)} | v_t(\psi_t(x_0)) - u_t(\psi_t(x_0) | x_1) |^2$$

Where $t \sim \mathcal{U}[0,1]$, $x_1 \sim q(x_1)$ (data), and $x_0 \sim p(x_0)$ (noise).

2. Optimal Transport (OT) Probability Path

The authors recommend the OT path for efficiency.

• Mean/Std Schedule: $\mu_t(x) = t x_1$ and $\sigma_t(x) = 1 - (1 - \sigma_{min})t$.
• Conditional Flow Map: $\psi_t(x) = (1 - (1 - \sigma_{min})t)x + t x_1$.
• Target Vector Field: The closed-form regression target for OT is: $$u_t(x|x_1) = \frac{x_1 - (1 - \sigma_{min})x}{1 - (1 - \sigma_{min})t}$$

3. Sampling

Sampling is performed by solving the ODE $\frac{d}{dt}\phi_t(x) = v_t(\phi_t(x))$ from $t=0$ to $t=1$ using the learned vector field $v_t$.

• Solver: dopri5 (adaptive) is used for robust evaluation. Fixed-step solvers (Euler, Midpoint) are used for low-NFE efficiency tests.

Models

• Architecture: U-Net architecture from Dhariwal & Nichol (2021) is used for all image experiments.
• Toy Data: 5-layer MLP with 512 neurons.
• Hyperparameters:
  • Optimizer: Adam ($\beta_1=0.9, \beta_2=0.999$, weight decay=0.0).
  • Learning Rate: Polynomial decay or constant (see Table 3 in paper).
  • $\sigma_{min}$: Set to a small value (e.g., $1e-5$).

Evaluation

• Metrics:
  • NLL (BPD): Computed using the continuous change of variables formula, estimated via the Hutchinson trace estimator to bypass $O(d^3)$ divergence computation.
  • FID: Frechet Inception Distance for sample quality.
  • NFE: Number of Function Evaluations required by the solver.
• Likelihood Computation: Requires solving an augmented ODE to track the log-density change: $$\frac{d}{dt} \begin{bmatrix} \phi_t(x) \ f(t) \end{bmatrix} = \begin{bmatrix} v_t(\phi_t(x)) \ -\text{div}(v_t(\phi_t(x))) \end{bmatrix}$$

Hardware

• CIFAR-10: 2 GPUs.
• ImageNet-32: 4 GPUs.
• ImageNet-64: 16 GPUs.
• ImageNet-128: 32 GPUs.
• Precision: Full 32-bit for CIFAR/IM-32; 16-bit mixed precision for IM-64/128.

Theoretical Notes: Why CFM Works

The paper relies on three key theorems to make training tractable.

Theorem 1 (Marginal Generation):

Marginalizing conditional vector fields $u_t(x|x_1)$ yields the correct marginal vector field $u_t(x)$ that generates the marginal probability path $p_t(x)$.

$$u_t(x) = \int u_t(x|x_1) \frac{p_t(x|x_1)q(x_1)}{p_t(x)} dx_1$$

Understanding the Proof:

To understand why this theorem holds, we have to look at the Continuity Equation, which is the fundamental partial differential equation (PDE) that links a probability density path $p_t$ to a vector field $u_t$.

A vector field $u_t$ is said to “generate” a probability path $p_t$ if and only if they satisfy the continuity equation:

$$\frac{\partial p_t(x)}{\partial t} + \nabla \cdot (p_t(x) u_t(x)) = 0$$

The proof of Theorem 1 relies on substituting the definitions of the marginal path and vector field into this equation to see if they balance out.

Step-by-Step Proof:

1. Start with the time derivative of the marginal path: We begin by differentiating the marginal probability path $p_t(x)$ with respect to time. By definition, the marginal path is the integral of the conditional paths over the data distribution: $$\frac{\partial p_t(x)}{\partial t} = \frac{\partial}{\partial t} \int p_t(x|x_1) q(x_1) dx_1$$

2. Swap derivative and integral: Assuming standard regularity conditions (Leibniz Rule), we can move the time derivative inside the integral: $$\frac{\partial p_t(x)}{\partial t} = \int \frac{\partial p_t(x|x_1)}{\partial t} q(x_1) dx_1$$

3. Apply the Conditional Continuity Equation: This is the critical step. We know that the conditional vector field $u_t(x|x_1)$ generates the conditional path $p_t(x|x_1)$. Therefore, for every single sample $x_1$, the pair satisfies the continuity equation: $$\frac{\partial p_t(x|x_1)}{\partial t} = -\nabla \cdot (p_t(x|x_1) u_t(x|x_1))$$

   Substituting this into our integral gives: $$\frac{\partial p_t(x)}{\partial t} = -\int \nabla \cdot (p_t(x|x_1) u_t(x|x_1)) q(x_1) dx_1$$

4. Pull the Divergence out: Since the divergence operator ($\nabla \cdot$) acts on $x$ and the integral is over $x_1$, we can pull the divergence operator outside the integral (by linearity): $$\frac{\partial p_t(x)}{\partial t} = -\nabla \cdot \left( \int p_t(x|x_1) u_t(x|x_1) q(x_1) dx_1 \right)$$

5. Match with the Marginal Vector Field Definition: Now, look at the term inside the parentheses. The paper defines the marginal vector field $u_t(x)$ specifically to make this term simpler. Rearranging the definition of $u_t(x)$ provided in the theorem: $$p_t(x) u_t(x) = \int p_t(x|x_1) u_t(x|x_1) q(x_1) dx_1$$

   Substitute $p_t(x) u_t(x)$ back into our equation from Step 4: $$\frac{\partial p_t(x)}{\partial t} = -\nabla \cdot (p_t(x) u_t(x))$$

Conclusion: We have just shown that $\frac{\partial p_t(x)}{\partial t} + \nabla \cdot (p_t(x) u_t(x)) = 0$. This is exactly the continuity equation. Because the marginal path and the aggregated marginal vector field satisfy this equation, the vector field is proven to generate the path.

Theorem 2 (Gradient Equivalence):

The intractable Flow Matching objective $\mathcal{L}_{FM}$ (which requires $u_t(x)$) has the same gradients as the tractable Conditional Flow Matching objective $\mathcal{L}_{CFM}$.

$$\nabla_\theta \mathcal{L}_{FM}(\theta) = \nabla_\theta \mathcal{L}_{CFM}(\theta)$$

This allows the model to learn the marginal vector field by only seeing conditional sample paths.

Understanding the Proof:

The reason Theorem 2 holds is that the “Conditional Flow Matching” (CFM) objective is essentially an unbiased estimator of the “Flow Matching” (FM) objective (up to a constant). When we average over all the conditional data points $x_1$, the “cross-term” in the loss function aligns perfectly with the marginal vector field.

1. Expand the Loss Functions

First, let’s look at the squared error in both objectives. Recall that $v_t$ is our neural network (parameterized by $\theta$), $u_t$ is the intractable marginal target, and $u_t(x|x_1)$ is the tractable conditional target.

Expanding the squared norms:

• FM Objective: $$\mathcal{L}_{FM}(\theta) = \mathbb{E}_{t, p_t(x)} \left[ |v_t(x)|^2 - 2v_t(x) \cdot u_t(x) + |u_t(x)|^2 \right]$$

• CFM Objective: $$\mathcal{L}_{CFM}(\theta) = \mathbb{E}_{t, q(x_1), p_t(x|x_1)} \left[ |v_t(x)|^2 - 2v_t(x) \cdot u_t(x|x_1) + |u_t(x|x_1)|^2 \right]$$

Key Insight: When we take the gradient $\nabla_\theta$, the last term in both equations disappears because the targets ($u_t$) are independent of the network weights $\theta$. We only need to show that the expectations of the first two terms match.

2. Matching the First Term ($|v_t(x)|^2$)

This part is straightforward. The expectation of $|v_t(x)|^2$ is the same in both cases because of how the marginal density $p_t(x)$ is defined.

• FM: averages over $p_t(x)$.
• CFM: averages over $p_t(x|x_1)q(x_1)$.

Since $p_t(x) = \int p_t(x|x_1) q(x_1) dx_1$ (by definition), averaging over the joint distribution is mathematically identical to averaging over the marginal $p_t(x)$.

3. Matching the Cross Term (The “Trick”)

This is the critical part of the proof. We need to show that the interaction between the network and the marginal field equals the interaction between the network and the conditional field.

The Goal: Show $\mathbb{E}_{t, p_t(x)} [v_t(x) \cdot u_t(x)] = \mathbb{E}_{t, q(x_1), p_t(x|x_1)} [v_t(x) \cdot u_t(x|x_1)]$.

The Proof:

1. Start with the FM cross-term (marginal): $$\mathbb{E}_{t, p_t(x)} [v_t(x) \cdot u_t(x)]$$

2. Substitute the definition of the marginal vector field $u_t(x)$ derived in Theorem 1: $$u_t(x) = \int u_t(x|x_1) \frac{p_t(x|x_1) q(x_1)}{p_t(x)} dx_1$$

3. Plug this into the integral. The $p_t(x)$ terms cancel: $$\mathbb{E}_{t, p_t(x)} [v_t(x) \cdot u_t(x)] = \int_t \int_x p_t(x) v_t(x) \cdot \left[ \int_{x_1} u_t(x|x_1) \frac{p_t(x|x_1) q(x_1)}{p_t(x)} dx_1 \right] dx$$

4. This simplifies to: $$= \int_t \int_x \int_{x_1} v_t(x) \cdot u_t(x|x_1) p_t(x|x_1) q(x_1) dx_1 dx dt$$

5. This is exactly the definition of the expectation in the CFM objective: $$= \mathbb{E}_{t, q(x_1), p_t(x|x_1)} [v_t(x) \cdot u_t(x|x_1)]$$

Conclusion: Because the expectations of all terms involving $\theta$ are identical, the gradients must be identical.

Intuitively, this works like Denoising Score Matching or Stochastic Gradient Descent: even though each individual conditional vector field $u_t(x|x_1)$ points to a specific data point $x_1$ (which may differ from the true marginal direction), the average of all these pulls equals the true marginal vector field $u_t(x)$.

Theorem 3 (Gaussian Conditional VFs):

For any Gaussian probability path $p_t(x|x_1) = \mathcal{N}(x | \mu_t(x_1), \sigma_t(x_1)^2 I)$, the unique vector field generating it is available in closed form:

$$u_t(x|x_1) = \frac{\sigma’_t(x_1)}{\sigma_t(x_1)}(x - \mu_t(x_1)) + \mu’_t(x_1)$$

This theorem allows explicitly defining targets for both Diffusion (curved) and Optimal Transport (straight) paths.

Understanding the Proof:

The derivation of Theorem 3 comes from the direct relationship between a flow map $\psi_t$ and its generating vector field. Because we chose a specific, simple path (Gaussian), we can invert the flow map to find the vector field in closed form.

1. Define the Flow Map $\psi_t$

We start by defining the conditional probability path as a Gaussian:

$$p_t(x|x_1) = \mathcal{N}(x | \mu_t(x_1), \sigma_t(x_1)^2 I)$$

The simplest way to “push” a standard normal distribution (noise) $p_0 = \mathcal{N}(0, I)$ to this Gaussian is using an affine transformation (scaling and shifting). We define the flow map $\psi_t$ as:

$$\psi_t(x_0) = \sigma_t(x_1) x_0 + \mu_t(x_1)$$

This map takes a noise sample $x_0$ and transforms it into a sample $x$ at time $t$.

2. The Definition of a Generating Vector Field

By definition, a vector field $u_t$ generates a flow $\psi_t$ if the vector field describes the instantaneous velocity of the flow at any point. Mathematically:

$$u_t(\psi_t(x_0)) = \frac{d}{dt}\psi_t(x_0)$$

Let $x = \psi_t(x_0)$ be the position of the particle at time $t$. We want to find $u_t(x)$.

3. Invert the Flow Map

To find $u_t(x)$, we must express the equation in terms of $x$ rather than $x_0$. Since our flow map is a simple affine transformation (multiply and add), it is easily invertible (assuming $\sigma_t(x_1) \neq 0$):

$$x_0 = \frac{x - \mu_t(x_1)}{\sigma_t(x_1)}$$

We will call this inverse map $\psi_t^{-1}(x)$.

4. Differentiate the Flow Map

Now we calculate the left side of our definition equation (velocity): $\frac{d}{dt}\psi_t(x_0)$.

Taking the time derivative of $\psi_t(x_0) = \sigma_t(x_1) x_0 + \mu_t(x_1)$:

$$\frac{d}{dt}\psi_t(x_0) = \sigma’_t(x_1) x_0 + \mu’_t(x_1)$$

(Note: $\sigma’_t$ and $\mu’_t$ denote time derivatives).

5. Substitute and Solve

Now we combine everything. We know $u_t(\psi_t(x_0)) = \frac{d}{dt}\psi_t(x_0)$.

Substitute the result from Step 4 into this equation:

$$u_t(\psi_t(x_0)) = \sigma’_t(x_1) x_0 + \mu’_t(x_1)$$

This expresses the vector field in terms of the initial point $x_0$. We must express it in terms of the current point $x$. So, we plug in the inverse formula for $x_0$ derived in Step 3:

$$u_t(x|x_1) = \sigma’_t(x_1) \frac{x - \mu_t(x_1)}{\sigma_t(x_1)} + \mu’_t(x_1)$$

Rearranging terms gives the final closed form:

$$u_t(x|x_1) = \frac{\sigma’_t(x_1)}{\sigma_t(x_1)}(x - \mu_t(x_1)) + \mu’_t(x_1)$$

Why is this useful?

This formula means that as long as you can define a mean schedule $\mu_t(x_1)$ and a standard deviation schedule $\sigma_t(x_1)$ (which is easy to do for both Diffusion and Optimal Transport), you immediately get the exact vector field target $u_t(x|x_1)$ needed to train your neural network, bypassing complex ODE solving or score matching approximations.

Paper Information

Citation: Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., & Le, M. (2023). Flow Matching for Generative Modeling. International Conference on Learning Representations (ICLR).

Publication: ICLR 2023

@inproceedings{lipmanFlowMatchingGenerative2023,
  title = {Flow Matching for Generative Modeling},
  author = {Lipman, Yaron and Chen, Ricky T. Q. and Ben-Hamu, Heli and Nickel, Maximilian and Le, Matt},
  booktitle = {International Conference on Learning Representations},
  year = {2023}
}

Additional Resources:

• ArXiv

This is primarily a Method paper, with significant Theory contributions.

The authors propose a specific algorithm (“InterFlow”) for constructing generative models based on continuous-time normalizing flows. The work is characterized by the derivation of a new training objective (a simple quadratic loss) that bypasses the computational bottlenecks of previous methods. It includes prominent baseline comparisons against continuous flow methods (FFJORD, OT-Flow) and diffusion models. The theoretical component establishes the validity of the interpolant density satisfying the continuity equation (a conservation law governing how probability mass flows) and bounds the Wasserstein-2 distance (a measure of transport cost between distributions, penalizing squared displacement) of the transport.

What is the motivation?

The primary motivation is to overcome the computational inefficiency of training Continuous Normalizing Flows (CNFs) using Maximum Likelihood Estimation (MLE). Standard CNF training requires backpropagating through numerical ODE solvers, which is costly and limits scalability.

Additionally, while score-based diffusion models (SDEs) have achieved high sample quality, they theoretically require infinite time integration and rely on specific noise schedules. The authors aim to establish a method that works strictly with Probability Flow ODEs on finite time intervals, retaining the flexibility to connect arbitrary densities without the complexity of SDEs or the cost of standard ODE adjoint methods.

What is the novelty here?

The core novelty is the Stochastic Interpolant framework:

• Explicit Interpolant Construction: The method defines a time-dependent interpolant $x_t = I_t(x_0, x_1)$ (e.g., trigonometric interpolation) that connects samples from the base density $\rho_0$ and target $\rho_1$.
• Simulation-Free Training: The velocity field $v_t(x)$ of the probability flow is learned by minimizing a simple quadratic objective: $G(\hat{v}) = \mathbb{E}[|\hat{v}_t(x_t)|^2 - 2\partial_t x_t \cdot \hat{v}_t(x_t)]$. Because $\partial_t I_t$ is known analytically from the interpolant definition, the expectation can be estimated by sampling $(x_0, x_1, t)$ directly. This avoids ODE integration during training (ODE integration is still required at inference).
• Decoupling Path and Optimization: The choice of path (interpolant) is separated from the optimization of the velocity field. MLE methods couple the path and objective.
• Connection to Score-Based Models: The authors show that for Gaussian base densities and trigonometric interpolants, the learned velocity field is explicitly related to the score function $\nabla \log \rho_t$, providing a theoretical bridge between CNFs and diffusion models.

What experiments were performed?

The authors performed validation across synthetic, tabular, and image domains:

• 2D Density Estimation: Benchmarked on “Checkerboard”, “8 Gaussians”, and “Spirals” to visualize mode coverage and transport smoothness.
• High-Dimensional Tabular Data: Evaluated on standard benchmarks (POWER, GAS, HEPMASS, MINIBOONE, BSDS300) comparing Negative Log Likelihood (NLL) against FFJORD, OT-Flow, and others.
• Image Generation: Trained models on CIFAR-10 ($32 \times 32$), ImageNet ($32 \times 32$), and Oxford Flowers ($128 \times 128$) to test scalability.
• Ablations: Investigated optimizing the interpolant path itself (e.g., learning Fourier coefficients for the path) to approach optimal transport and minimize path length.

What outcomes/conclusions?

• Performance: The method matches or supersedes conventional ODE flows (like FFJORD) in terms of NLL while being significantly cheaper to train.
• Efficiency: The training cost per epoch is constant (simulation-free), whereas MLE-based ODE methods see growing costs as the dynamics become more complex.
• Scalability: The method successfully scales to $128 \times 128$ resolution on a single GPU, a resolution that prior ab-initio ODE flows had not demonstrated.
• Flexibility: The framework can connect any two arbitrary densities (e.g., connecting two different complex 2D distributions) without needing one to be Gaussian.
• Optimal Transport: For a fixed interpolant, minimizing $G(\hat{v})$ over the velocity field recovers the velocity for that specific path. Additionally optimizing over the interpolant family yields a solution to the Benamou-Brenier optimal transport problem.

Reproducibility Details

Data

• Tabular Datasets: POWER (6D), GAS (8D), HEPMASS (21D), MINIBOONE (43D), BSDS300 (63D).
  • Training points range from ~30k (MINIBOONE) to ~1.6M (POWER).
• Image Datasets:
  • CIFAR-10 ($32 \times 32$, 50k training points).
  • ImageNet ($32 \times 32$, ~1.28M training points).
  • Oxford Flowers ($128 \times 128$, ~315k training points).
• Time Sampling: Time $t$ is sampled from a Beta distribution during training (reweighting) to focus learning near the target.

Algorithms

• Interpolant: The primary interpolant used is trigonometric: $I_t(x_0, x_1) = \cos(\frac{\pi t}{2})x_0 + \sin(\frac{\pi t}{2})x_1$.
  • Alternative linear interpolant: $I_t = a_t x_0 + b_t x_1$.
• Loss Function: $$G(\hat{v}) = \mathbb{E}_{t, x_0, x_1}[|\hat{v}_t(x_t)|^2 - 2\partial_t I_t(x_0, x_1) \cdot \hat{v}_t(x_t)]$$
  • The expectation is amenable to empirical estimation using batches of $x_0, x_1, t$.
• Sampling: Numerical integration using Dormand-Prince (Runge-Kutta 4/5).
• Optimization: SGD/Adam variants used for optimization.

Models

• Tabular Architectures:
  • Feed-forward networks with 3-5 hidden layers.
  • Hidden widths: 512-1024 units.
  • Activation: ReLU (general) or ELU (BSDS300).
• Image Architectures:
  • U-Net based on the DDPM implementation.
  • Dimensions: 256 hidden dimension.
  • Sinusoidal time embeddings used.

Evaluation

• Metrics: Negative Log Likelihood (NLL) in nats, Frechet Inception Distance (FID) for images.
• Baselines: FFJORD, Glow, Real NVP, OT-Flow, ScoreFlow, DDPM.

Tabular NLL (nats, lower is better; Table 2 Left):

Image Generation NLL and FID (Table 2 Right; all numbers as reported in Albergo & Vanden-Eijnden (2023)):

Note: DDPM++ and ScoreSDE are both architectural variants from Song et al. (2021); NLL and FID figures in that row correspond to different model configurations within the same paper (NLL 2.99 from the VP-SDE Probability Flow ODE; FID 2.92 from the VE-SDE configuration). InterFlow matches ScoreSDE on CIFAR-10 NLL (2.99) while being simulation-free. FID is weaker than dedicated image models (10.27 vs 2.92 for ScoreSDE), reflecting the paper’s primary focus on tractable likelihood rather than sample quality.

Hardware

• Compute: All models were trained on a single NVIDIA A100 GPU.
• Training Time:
  • Tabular: $10^5$ steps.
  • Images: $1.5 \times 10^5$ to $6 \times 10^5$ steps.
  • Speedup: Demonstrated ~400x speedup compared to FFJORD on MiniBooNE dataset.

Paper Information

Citation: Albergo, M. S., & Vanden-Eijnden, E. (2023). Building Normalizing Flows with Stochastic Interpolants. The Eleventh International Conference on Learning Representations.

Publication: ICLR 2023

@inproceedings{albergoBuildingNormalizingFlows2022,
  title = {Building {{Normalizing Flows}} with {{Stochastic Interpolants}}},
  booktitle = {The {{Eleventh International Conference}} on {{Learning Representations}}},
  author = {Albergo, Michael Samuel and {Vanden-Eijnden}, Eric},
  year = 2023,
  url = {https://openreview.net/forum?id=li7qeBbCR1t}
}

Additional Resources:

• OpenReview
• arXiv

This is primarily a Method paper. It adapts a specific architecture (Transformer) to a specific task (InChI-to-IUPAC translation) and evaluates its performance against both machine learning and commercial baselines. It also has a secondary Resource contribution, as the trained model and scripts are released as open-source software.

Motivation: The Bottleneck in Algorithmic IUPAC Nomenclature

Generating correct IUPAC names is difficult due to the comprehensive but complex rules defined by the International Union of Pure and Applied Chemistry. Commercial software generates names from structures but remains closed-source with opaque methodologies and frequent inter-package disagreements. Open identifiers like InChI and SMILES lack direct human readability. This creates a need for an open, automated method to generate informative IUPAC names from standard identifiers like InChI, which are ubiquitous in online chemical databases.

Novelty: Treating Chemical Translation as a Character-Level Sequence

The key novelty is treating chemical nomenclature translation as a character-level sequence-to-sequence problem using a Transformer architecture, specifically using InChI as the source language.

• Standard Neural Machine Translation (NMT) uses sub-word tokenization. This model processes InChI and predicts IUPAC names character-by-character.
• It demonstrates that character-level tokenization outperforms byte-pair encoding or unigram models for this specific chemical task.
• It uses InChI’s standardization to avoid the canonicalization issues inherent in SMILES-based approaches.
• The attention mechanism allows the decoder to align specific parts of the generated IUPAC name with corresponding structural features in the source InChI string, operating via the standard scaled dot-product attention: $$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$

Methodology & Experimental Validation

• Training: The model was trained on 10 million InChI/IUPAC pairs sampled from PubChem using a character-level objective. The model is supervised using categorical cross-entropy loss across the vocabulary of characters: $$ \mathcal{L} = -\sum_{i=1}^{N} y_i \log(\hat{y}_i) $$
• Ablation Studies: The authors experimentally validated architecture choices, finding that LSTM models and sub-word tokenization (BPE) performed worse than the Transformer with character tokenization. They also optimized dropout rates.
• Performance Benchmarking: The model was evaluated on a held-out test set of 200,000 samples. Performance was quantified primarily by Whole-Name Accuracy and Normalized Edit Distance (based on the Damerau-Levenshtein distance, scaled by the maximum string length).
• Commercial Comparison: The authors compared their model against four major commercial packages (ACD/I-Labs, ChemAxon, Mestrelab, and PubChem’s Lexichem). However, this evaluation used a highly limited test set of only 100 molecules, restricting the statistical confidence of the external baseline.
• Error Analysis: They analyzed performance across different chemical classes (organics, charged species, macrocycles, inorganics) and visualized attention coefficients to interpret model focus.

Key Results and the Inorganic Challenge

• High Accuracy on Organics: The model achieved 91% whole-name accuracy on the test set, performing particularly well on organic compounds.
• Comparable to Commercial Tools: On the limited 100-molecule benchmark, the edit distance between the model’s predictions and commercial packages (15-23%) was similar to the variation found between the commercial packages themselves (16-21%).
• Limitations on Inorganics: The model performed poorly on inorganic (14% accuracy) and organometallic compounds (20% accuracy). This is attributed to inherent data limitations in the standard InChI format (which deliberately disconnects metal atoms from their ligands) and low training data coverage for those classes.
• Character-Level Superiority: Character-level tokenization was found to be essential; byte-pair encoding reduced accuracy significantly.

Reproducibility Details

Data

The dataset was derived from PubChem’s public FTP server (CID-SMILES.gz and CID-IUPAC.gz).

Algorithms

• Framework: OpenNMT-py 2.0.0 (using PyTorch). Training scripts and vocabularies are available as supplementary files to the original publication. Pre-trained model weights are hosted on Zenodo.
• Architecture Type: Transformer Encoder-Decoder.
• Optimization: ADAM optimizer ($\beta_1=0.9, \beta_2=0.998$).
• Learning Rate: Linear warmup over 8000 steps to 0.0005, then decayed by inverse square root of iteration.
• Regularization:
  • Dropout: 0.1 (applied to dense and attentional layers).
  • Label Smoothing: Magnitude 0.1.
• Training Strategy: Teacher forcing used for both training and validation.
• Gradient Accumulation: Gradients accumulated over 4 batches before updating parameters.
• Inference: Beam search with width 10 and length penalty 1.0.

Models

• Structure: 6 layers in encoder, 6 layers in decoder.
• Attention: 8 heads per attention sub-layer.
• Dimensions:
  • Feed-forward hidden state size: 2048.
  • Embedding vector length: 512.
• Initialization: Glorot’s method.
• Position: Positional encoding added to word vectors.

Evaluation

Metrics reported include Whole-Name Accuracy (percentage of exact matches) and Normalized Edit Distance (Damerau-Levenshtein, scale 0-1).

Hardware

• GPU: Tesla K80.
• Training Time: 7 days.
• Throughput: ~6000 tokens/sec (InChI) and ~3800 tokens/sec (IUPAC).
• Batch Size: 4096 tokens (approx. 30 compounds).

Artifacts

Paper Information

Citation: Handsel, J., Matthews, B., Knight, N. J., & Coles, S. J. (2021). Translating the InChI: Adapting Neural Machine Translation to Predict IUPAC Names from a Chemical Identifier. Journal of Cheminformatics, 13(1), 79. https://doi.org/10.1186/s13321-021-00535-x

Publication: Journal of Cheminformatics 2021

@article{handselTranslatingInChIAdapting2021a,
  title = {Translating the {{InChI}}: Adapting Neural Machine Translation to Predict {{IUPAC}} Names from a Chemical Identifier},
  shorttitle = {Translating the {{InChI}}},
  author = {Handsel, Jennifer and Matthews, Brian and Knight, Nicola J. and Coles, Simon J.},
  year = 2021,
  month = oct,
  journal = {Journal of Cheminformatics},
  volume = {13},
  number = {1},
  pages = {79},
  issn = {1758-2946},
  doi = {10.1186/s13321-021-00535-x},
  urldate = {2025-12-20},
  abstract = {We present a sequence-to-sequence machine learning model for predicting the IUPAC name of a chemical from its standard International Chemical Identifier (InChI). The model uses two stacks of transformers in an encoder-decoder architecture, a setup similar to the neural networks used in state-of-the-art machine translation. Unlike neural machine translation, which usually tokenizes input and output into words or sub-words, our model processes the InChI and predicts the IUPAC name character by character. The model was trained on a dataset of 10 million InChI/IUPAC name pairs freely downloaded from the National Library of Medicine's online PubChem service. Training took seven days on a Tesla K80 GPU, and the model achieved a test set accuracy of 91\%. The model performed particularly well on organics, with the exception of macrocycles, and was comparable to commercial IUPAC name generation software. The predictions were less accurate for inorganic and organometallic compounds. This can be explained by inherent limitations of standard InChI for representing inorganics, as well as low coverage in the training data.},
  langid = {english},
  keywords = {Attention,GPU,InChI,IUPAC,seq2seq,Transformer}
}

This is primarily a Method paper with significant elements of Position.

• Method: The authors propose a specific neural architecture (Transformer with custom tokenization) and a verification pipeline (round-trip check) to solve the SMILES $\leftrightarrow$ IUPAC translation task. They rigorously benchmark this against rule-based baselines (OPSIN).
• Position: The authors explicitly argue for a paradigm shift, suggesting that “heavy” neural architectures should replace complex, costly rule-based legacy systems even for “exact” algorithmic tasks.

The Cost of Rule-Based Chemical Naming

• Complexity of Naming: Generating IUPAC names manually is error-prone and requires deep algorithmic knowledge.
• Lack of Open Source Tools: While open-source tools exist for Name-to-Structure (e.g., OPSIN), there were no open-source tools for the inverse “Structure-to-Name” conversion at the time of writing.
• Cost of Development: Developing rule-based converters “from scratch” is prohibitively expensive and time-consuming compared to training a neural model on existing data.

Struct2IUPAC Core Innovation

• Struct2IUPAC: The first effective open-source neural model for converting SMILES to IUPAC names, treating chemical translation as a Neural Machine Translation (NMT) problem.
• Verification Loop: A novel inference pipeline that generates multiple candidates via beam search and validates them using a reverse converter (OPSIN) to ensure the generated name maps back to the original structure.
• Custom Tokenization: A manually curated rule-based tokenizer for IUPAC names that handles specific chemical suffixes, prefixes, and stereochemical markers.

Experimental Setup and Stress Testing

• Accuracy Benchmarking: The models were tested on a held-out subset of 100,000 molecules from PubChem. The authors measured accuracy across different beam sizes (1, 3, 5).
• Comparison to Rules: The neural IUPAC2Struct model was compared directly against the rule-based OPSIN tool.
• Stress Testing:
  • Sequence Length: Evaluated performance across varying token lengths, identifying a “sweet spot” (10-60 tokens) and failure modes for very short (e.g., methane) or long molecules.
  • Stereochemistry: Tested on “stereo-dense” compounds. The authors define a “stereo-density” index ($I$) as the ratio of stereocenters ($S$) to total tokens ($N$): $$I = \frac{S}{N}$$ They observed a performance drop for these dense molecules, though the model still handled many stereocenters robustly.
  • Tautomers: Verified the model’s ability to handle different tautomeric forms (e.g., Guanine and Uracil variants).
• Latency Analysis: Benchmarked inference speeds on CPU vs. GPU relative to output sequence length.

Benchmarks and Outcomes

• High Accuracy: The Struct2IUPAC model achieved 98.9% accuracy (Beam 5 with verification). The reverse model (IUPAC2Struct) achieved 99.1%, comparable to OPSIN’s 99.4%.
• Distribution Modeling vs. Intuition: The authors claim the model infers “chemical logic,” because it correctly generates multiple valid IUPAC names for single molecules where naming ambiguity exists (e.g., parent group selection). However, this more likely reflects the Transformer successfully modeling the high-frequency conditional probability distribution of synonymous names present in the PubChem training data, rather than learning intrinsic chemical rules.
• Production Readiness: Inference on GPU takes less than 0.5 seconds even for long names, making it viable for production use.
• Paradigm Shift: The authors conclude that neural networks are a viable, cost-effective alternative to developing rule-based algorithms for legacy notation conversion.

Reproducibility Details

Data

The study utilized the PubChem database.

• Cleaning: Molecules that could not be processed by RDKit were removed. Molecules containing tokens not in the tokenizer (e.g., aromatic selenium) were excluded.
• Availability: A subset of 100,000 test molecules is available on GitHub (data/test_100000.csv) and Zenodo. The full train/test splits are not explicitly provided.

Algorithms

• Tokenization:
  • SMILES: Character-based tokenization.
  • IUPAC: Custom rule-based tokenizer splitting suffixes (-one, -al), prefixes (-oxy, -di), and special symbols ((, ), R(S)).
• Verification Step:
  1. Generate $N$ names using Beam Search ($N=5$).
  2. Reverse translate the candidate name using OPSIN.
  3. Check if the OPSIN structure matches the original input SMILES.
  4. Display the first verified match; otherwise, report failure.

Models

• Architecture: Standard Transformer with 6 encoder layers and 6 decoder layers.
• Hyperparameters:
  • Attention Heads: 8
  • Attention Dimension ($d_{\text{model}}$): 512
  • Feed-Forward Dimension ($d_{\text{ff}}$): 2048
• Training Objective: The models were trained using standard autoregressive cross-entropy loss over the target token sequence $y$ given the input string $x$: $$\mathcal{L} = - \sum_{t=1}^{T} \log P(y_t \mid y_{<t}, x)$$
• Training: Two separate models were trained: Struct2IUPAC (SMILES $\to$ IUPAC) and IUPAC2Struct (IUPAC $\to$ SMILES).
• Availability: Code for model architecture is provided in the GitHub repository. Pre-trained weights for the IUPAC2Struct model are available, but the Struct2IUPAC model weights are not publicly released, meaning researchers would need to retrain that model on their own PubChem data to reproduce those results.

Evaluation

Evaluation was performed on a random subset of 100,000 molecules from the test set.

• Robustness: Accuracy drops significantly for augmented (non-canonical) SMILES (37.16%) and stereo-enriched compounds (66.52%).

Hardware

• Training Infrastructure: 4 $\times$ Tesla V100 GPUs and 36 CPUs.
• Training Time: Approximately 10 days under full load.
• Inference Speed: <0.5s per molecule on GPU; scale is linear with output token length.

Artifacts

Paper Information

Citation: Krasnov, L., Khokhlov, I., Fedorov, M. V., & Sosnin, S. (2021). Transformer-based artificial neural networks for the conversion between chemical notations. Scientific Reports, 11(1), 14798. https://doi.org/10.1038/s41598-021-94082-y

Publication: Scientific Reports 2021

@article{krasnovTransformerbasedArtificialNeural2021a,
  title = {Transformer-Based Artificial Neural Networks for the Conversion between Chemical Notations},
  author = {Krasnov, Lev and Khokhlov, Ivan and Fedorov, Maxim V. and Sosnin, Sergey},
  year = 2021,
  month = jul,
  journal = {Scientific Reports},
  volume = {11},
  number = {1},
  pages = {14798},
  publisher = {Nature Publishing Group},
  doi = {10.1038/s41598-021-94082-y}
}

Additional Resources:

• GitHub Repository
• Web Demo

This is primarily a Method paper, with a strong secondary contribution as a Resource paper.

• Method: It proposes a neural machine translation (NMT) architecture to approximate the complex, rule-based algorithm of IUPAC naming, treating it as a language translation task.
• Resource: It provides an open-source tool and trained models to the community, addressing a gap where such functionality was previously limited to proprietary software.

Motivation: Democratizing IUPAC Nomenclature

The International Union of Pure and Applied Chemistry (IUPAC) naming scheme is universally accepted but algorithmically complex. Generating these names correctly is challenging for humans, and automated generation is largely missing from major open-source toolkits like CDK, RDKit, or Open Babel. While reliable commercial tools exist (e.g., ChemAxon’s molconvert), there was a lack of open-source alternatives for the scientific community. STOUT aims to fill this gap using a data-driven approach.

Core Innovation: Sequence-to-Sequence Naming

• Language Translation Approach: The authors treat chemical representations (SMILES/SELFIES) and IUPAC names as two different languages, applying Neural Machine Translation (NMT) to translate between them.
• Use of SELFIES: The work establishes SELFIES (Self-Referencing Embedded Strings) as a robust choice over SMILES for deep learning tokenization in this specific task, capitalizing on its syntactic robustness.
• Hardware Acceleration: The paper benchmarks GPU versus TPU training and highlights the practical necessity of Tensor Processing Units (TPUs) for training large-scale chemical language models, reducing training time by an order of magnitude.

Methodology & Translation Validation

• Data Scale: The model was trained on datasets of 30 million and 60 million molecules derived from PubChem.
• Hardware Benchmarking: Training efficiency was compared between an nVidia Tesla V100 GPU and Google TPU v3-8/v3-32 units.
• Bidirectional Translation: The system was tested on two distinct tasks:
  1. Forward: SELFIES → IUPAC names
  2. Reverse: IUPAC names → SELFIES
• Validation: Performance was evaluated on a held-out test set of 2.2 million molecules.

Translation Accuracy & Hardware Scaling

• High Accuracy: The model achieved an average BLEU score of ~90% and a Tanimoto similarity index > 0.9 for both translation directions.
• Generalization: Even when predictions were textually mismatched (low BLEU score), the underlying chemical structures often remained highly similar (high Tanimoto similarity), suggesting the system captures fundamental chemical semantics rather than merely memorizing strings.
• Impact of Data Size: Expanding training from 30 million to 60 million molecules yielded consistent performance gains without saturating.
• Hardware Necessity: Training on TPUs proved up to 54 times faster than a standard GPU baseline (Tesla V100), making scaling highly computationally tractable.

Reproducibility

Data

The dataset was curated from PubChem (111 million molecules). Note that the specific 30M and 60M subsets are not directly linked in the publication repository, which means a user would have to reconstruct the filtering process.

Preprocessing & Filtering:

• Explicit hydrogens removed; converted to canonical SMILES.
• Filtering Rules: MW < 1500 Da, no counter ions, limited element set (C, H, O, N, P, S, F, Cl, Br, I, Se, B), no hydrogen isotopes, 3-40 bonds, no charged groups.
• Ground Truth Generation: ChemAxon’s molconvert (Marvin Suite 20.15) was used to generate target IUPAC names for training.
• Representation: All SMILES were converted to SELFIES for training.

Algorithms

• Tokenization:
  • SELFIES: Split iteratively by brackets [ and ].
  • IUPAC: Split via punctuation ((, ), {, }, [, ], -, ., ,) and a discrete set of sub-word chemical morphemes (e.g., methyl, benzene, fluoro).
  • Padding: SELFIES padded to 48 tokens; IUPAC padded to 78 tokens. “Start” and “End” sequence markers append each chain.
• Optimization: Adam optimizer instantiated with a learning rate of $0.0005$.
• Objective Function: Sparse categorical cross-entropy, assessing prediction probabilities for token $i$ over vocabulary $V$: $$ \mathcal{L} = -\sum_{i=1}^{V} y_i \log(\hat{y}_i) $$

Models

• Architecture: Encoder-Decoder sequence-to-sequence network with Bahdanau attention mechanism context weighting.
• Components:
  • Encoder/Decoder: Recurrent Neural Networks (RNN) constructed using Gated Recurrent Units (GRU).
  • Attention: Bahdanau (additive) soft attention, which calculates alignment scores to softly weight encoder hidden states natively: $$ e_{tj} = v_a^\top \tanh(W_a s_{t-1} + U_a h_j) $$
  • Embedding: Decoder output passes through a continuous embedding layer before concatenating with the attention context vector.
• Implementation: Python 3 backend using TensorFlow 2.3.0. Note: The linked GitHub repository currently defaults to the STOUT V2.0 transformer models, so researchers aiming to reproduce this specific V1 RNN paper should reference the older tag/commit history.

Evaluation

Metrics heavily emphasize both linguistic accuracy and cheminformatic structural correctness:

Hardware

Comparison of hardware efficiency for training large chemical language models:

Paper Information

Citation: Rajan, K., Zielesny, A., & Steinbeck, C. (2021). STOUT: SMILES to IUPAC names using neural machine translation. Journal of Cheminformatics, 13(1), 34. https://doi.org/10.1186/s13321-021-00512-4

Publication: Journal of Cheminformatics 2021

@article{rajanSTOUTSMILESIUPAC2021,
  title = {STOUT: SMILES to IUPAC Names Using Neural Machine Translation},
  shorttitle = {STOUT},
  author = {Rajan, Kohulan and Zielesny, Achim and Steinbeck, Christoph},
  year = {2021},
  month = apr,
  journal = {Journal of Cheminformatics},
  volume = {13},
  number = {1},
  pages = {34},
  issn = {1758-2946},
  doi = {10.1186/s13321-021-00512-4},
  urldate = {2025-09-22},
  abstract = {Chemical compounds can be identified through a graphical depiction, a suitable string representation, or a chemical name. A universally accepted naming scheme for chemistry was established by the International Union of Pure and Applied Chemistry (IUPAC) based on a set of rules. Due to the complexity of this ruleset a correct chemical name assignment remains challenging for human beings and there are only a few rule-based cheminformatics toolkits available that support this task in an automated manner. Here we present STOUT (SMILES-TO-IUPAC-name translator), a deep-learning neural machine translation approach to generate the IUPAC name for a given molecule from its SMILES string as well as the reverse translation, i.e. predicting the SMILES string from the IUPAC name. In both cases, the system is able to predict with an average BLEU score of about 90% and a Tanimoto similarity index of more than 0.9. Also incorrect predictions show a remarkable similarity between true and predicted compounds.},
  langid = {english},
  keywords = {Attention mechanism,Chemical language,Deep neural network,DeepSMILES,IUPAC names,Neural machine translation,Recurrent neural network,SELFIES,SMILES}
}

Additional Resources:

• GitHub Repository
• STOUT V2.0 Note
• Struct2IUPAC Note
• HandSEL Note (InChI to IUPAC)

Method (Primary) / Resource (Secondary)

This paper presents a Methodological contribution by developing and validating a Transformer-based neural machine translation model (STOUT V2) for bidirectional chemical nomenclature (SMILES $\leftrightarrow$ IUPAC). It systematically compares this new architecture against previous RNN-based baselines (STOUT V1) and performs ablation studies on tokenization strategies.

It also serves as a significant Resource contribution by generating a massive training dataset of nearly 1 billion SMILES-IUPAC pairs (curated via commercial Lexichem software) and releasing the resulting models and code as open-source tools for chemical naming.

The Need for Robust Open-Source IUPAC Nomenclature Rules

Assigning systematic IUPAC names to chemical structures requires adherence to complex rules, challenging human consistency. Deterministic, rule-based software options like OpenEye Lexichem and ChemAxon are reliable commercial solutions. Existing open-source tools like OPSIN focus on parsing names to structures.

The previous version of STOUT (V1), based on RNNs/GRUs, achieved ~90% BLEU accuracy, with known limitations in capturing long-distance dependencies required for stereochemistry handling. This work uses the sequence-learning capabilities of Transformers combined with large-scale datasets to create a competitive open-source IUPAC naming tool.

Architectural Shift and Billion-Scale Training

The primary advancements over previous iterations address both architecture and dataset scale:

1. Architecture Shift: Moving from an RNN-based Seq2Seq model to a Transformer-based architecture (4 layers, 8 heads), which captures intricate chemical patterns better than GRUs.
2. Billion-Scale Training: Training on a dataset of nearly 1 billion molecules (combining PubChem and ZINC15), significantly larger than the 60 million used for STOUT V1.
3. Tokenization Strategy: Determining that character-wise tokenization for IUPAC names is superior to word-wise tokenization in terms of both accuracy and training efficiency (15% faster).

Experimental Validation and Scaling Limits

The authors conducted three primary experiments to validate bidirectional translation (SMILES $\rightarrow$ IUPAC and IUPAC $\rightarrow$ SMILES):

• Experiment 1 (Optimization): Assessed the impact of dataset size (1M vs 10M vs 50M) and tokenization strategy on SMILES-to-IUPAC performance.
• Experiment 2 (Scaling): Trained models on 110 million PubChem molecules for both forward and reverse translation tasks to test performance on longer sequences.
• Experiment 3 (Generalization): Trained on the full ~1 billion dataset (PubChem + ZINC15) for both translation directions.
• External Validation: Benchmarked against an external dataset from ChEBI (1,485 molecules) and ChEMBL34 to test generalization to unseen data.

Evaluation Metrics:

• Textual Accuracy: BLEU scores (1-4) and Exact String Match.
• Chemical Validity: Retranslation of generated names back to SMILES using OPSIN, followed by Tanimoto similarity checks (PubChem fingerprints) against the original input.

Translation Accuracy and Structural Validity

• Superior Performance: STOUT V2 achieved an average BLEU score of 0.99 (vs 0.94 for V1). While exact string matches varied by experiment (83-89%), the model notably achieved a perfect BLEU score (1.0) on 97.49% of a specific test set where STOUT V1 only reached 66.65%.
• Structural Validity (“Near Misses”): When the generated name differed from the ground truth string, the re-generated structure often remained chemically valid. The model maintained an average Tanimoto similarity $T(A,B)$ of 0.68 for these divergent names between bit-vector fingerprints $A$ and $B$, roughly defined as: $$ T(A,B) = \frac{\sum (A \cap B)}{\sum (A \cup B)} $$ Critique: Note that an average Tanimoto coefficient of 0.68 typically suggests moderate structural similarity/drift, not an almost-identical “near miss” (which would be $>0.85$). This implies the model constructs chemically related but structurally distinct outputs when it fails exact string matching.
• Tokenization: Character-level splitting for IUPAC names outperformed word-level splitting and was more computationally efficient.
• Data Imbalance & Generalization: The model’s drop in performance for sequences >600 characters highlights a systemic issue in open chemical databases: long, highly complex SMILES strings are significantly underrepresented. Even billion-scale training datasets are still bound by the chemical diversity of their source material.
• Limitations:
  • Preferred Names (PINs): The model mimics Lexichem’s naming conventions, generating valid IUPAC names distinct from strict Preferred IUPAC Names (PINs).
  • Sequence Length: Performance degrades for very long SMILES (>600 characters) due to scarcity in the training data.
  • Algorithmic Distillation Bottleneck: Because the 1 billion training pairs were generated entirely by OpenEye’s Lexichem, STOUT V2 acts as a knowledge distillation of that specific commercial algorithm. The model learns Lexichem’s heuristic mapping, specific dialects, and potential systematic errors, rather than deriving true nomenclature rules from first principles.

Reproducibility Details

Data

The training data was derived from PubChem and ZINC15. Ground truth IUPAC names were generated using OpenEye Lexichem TK 2.8.1 to ensure consistency.

Preprocessing:

• SMILES: Canonicalized, isomeric, and kekulized using RDKit (v2023.03.1).
• Formatting: Converted to TFRecord format in 100 MB chunks for TPU efficiency.

Algorithms

• SMILES Tokenization: Regex-based splitting. Atoms (e.g., “Cl”, “Au”), bonds, brackets, and digits are separate tokens.
• IUPAC Tokenization: Character-wise split was selected as the optimal strategy (treating every character as a token).
• Optimization: Adam optimizer with a custom learning rate scheduler based on model dimensions.
• Loss Function: Trained to minimize the Sparse Categorical Cross-Entropy $L$, masking padding tokens. For a correctly predicted target class $t$ alongside probabilities $p_i$, the masked loss is represented mathematically as: $$ L = - \sum_{i=1}^{m} m_i y_{i} \log(p_{i}) $$ where $m_i$ masks padded positions.
• Code Availability: The main STOUT V2 repository contains the inference package. The training pipeline/instructions (originally linked to a separate repo that is currently a 404) can still be found within the Zenodo archive release.

Models

The model follows the standard Transformer architecture from “Attention is All You Need” (Vaswani et al.).

• Architecture: 4 Transformer layers (encoder/decoder stack).
• Attention: Multi-head attention with 8 heads.
• Dimensions: Embedding size ($d_{model}$) = 512; Feed-forward dimension ($d_{ff}$) = 2048.
• Regularization: Dropout rate of 0.1.
• Context Window: Max input length (SMILES) = 600; Max output length (IUPAC) = 700-1000.
• Weights: Model weights for forward and reverse architectures are available via Zenodo (v3).

Evaluation

Evaluation focused on both string similarity and chemical structural integrity.

Artifacts

Hardware

Training was conducted entirely on Google Cloud Platform (GCP) TPUs.

• STOUT V1: Trained on TPU v3-8.
• STOUT V2: Trained on TPU v4-128 pod slices (128 nodes).
• Large Scale (Exp 3): Trained on TPU v4-256 pod slice (256 nodes).
• Training Time: Average of 15 hours and 2 minutes per epoch for the 1 billion dataset.
• Framework: TensorFlow 2.15.0-pjrt with Keras.

Paper Information

Citation: Rajan, K., Zielesny, A., & Steinbeck, C. (2024). STOUT V2.0: SMILES to IUPAC name conversion using transformer models. Journal of Cheminformatics, 16(146). https://doi.org/10.1186/s13321-024-00941-x

Publication: Journal of Cheminformatics 2024

@article{rajanSTOUTV20SMILES2024,
  title = {{{STOUT V2}}.0: {{SMILES}} to {{IUPAC}} Name Conversion Using Transformer Models},
  shorttitle = {{{STOUT V2}}.0},
  author = {Rajan, Kohulan and Zielesny, Achim and Steinbeck, Christoph},
  year = 2024,
  month = dec,
  journal = {Journal of Cheminformatics},
  volume = {16},
  number = {1},
  pages = {146},
  issn = {1758-2946},
  doi = {10.1186/s13321-024-00941-x}
}

Additional Resources:

• Web Application (Includes Ketcher drawing, bulk submission, and DECIMER integration)
• DECIMER Project
• STOUT V1 Note
• Zenodo Archive (Code Snapshot)

This paper represents a $\Psi_{\text{Method}}$ projection that proposes a novel architectural pipeline for indexing and searching chemical literature. The framework unifies text, molecular diagrams, and structured reaction records. It also contains a secondary $\Psi_{\text{Resource}}$ projection, providing a functional demonstration tool and curating a specific benchmark dataset for Suzuki coupling reactions.

The Gap in Passage-Level Chemical Retrieval

Scientific literature documents chemical reactions through a combination of text and visual diagrams. Textual descriptions detail parameters like yield and operational temperature, whereas diagrams graphically model these structural transformations. Existing tools such as SciFinder or Reaxys perform document-level or individual compound retrieval. They fail to explicitly link molecular figures to localized textual descriptions. This structure prevents researchers from directly extracting a corresponding reaction diagram alongside the exact textual protocol. Researchers require passage-level retrieval of synthesis protocols to efficiently access complete reaction conditions.

Core Innovation: Unified Multimodal Indexing

The core methodological innovation is a multimodal passage-level indexing and linking pipeline.

• Unified Indexing: The framework processes text and diagrams in parallel and directly links them into a single index structure. This architecture supports search queries utilizing raw text, discrete SMILES strings, or multimodal combinations.
• Compound-Passage Linking: The mechanism applies conflict-resolution logic linking chemical diagrams to specific text citations using two parallel heuristics:
  1. Token-based Alignment: Matching parsed diagram labels against documented text strings (e.g., “compound 5”) using normalized Levenshtein distance.
  2. Fingerprint-based Alignment: Matching chemical structures against generated SMILES strings via structural Tanimoto Similarity.
• ReactionMiner Integration: The pipeline parses and incorporates formatted reaction records (reactants, products, catalysts, quantitative yields) directly derived from segmented text passages.

Methodology & Expert Evaluation

The authors evaluated the system utilizing a chemical case study targeting specific synthesis domains alongside qualitative expert assessment.

• Dataset: Evaluators processed a corpus of 7 research manuscripts and 6 supplementary data documents detailing Suzuki coupling reactions.
• Volume: The resulting index processed 1,282 extracted passages (indexing 538), extracted 383 unique SMILES, and logged 219 parsed reactions.
• Qualitative Evaluation: Practicing structural chemists developed real-world queries (such as cross-referencing the conceptual “Burke group” alongside an explicit structural SMARTS pattern) to gauge retrieval capability.

Key Findings & System Limitations

• Diagram-to-Text Linking: The pipeline accurately paired visual molecular diagrams with structurally derived text details, permitting testers to navigate directly from a molecule query card to the exact origin passage within the source PDF.
• Contextual Insight Extraction: Specialized chemists found the parsed reaction representations (yield metrics, isolated catalysts) functionally pragmatic as high-level extractive summaries.
• Extrapolative Retrieval: The architecture permitted the effective retrieval of targeted chemical derivatives (such as benzo[b]thiophen-2-ylboronic acid) via structurally related input queries (dibenzothiophene).

The system evaluation highlights several architectural restrictions:

• Domain-Restricted Validation: The initial validation is entirely qualitative and bounded to the specific subclass of Suzuki coupling reactions. The evaluation omits standardized quantitative retrieval baselines (e.g., MAP, NDCG) and lacks systematic ablation data for the fusion scoring mechanism.
• Algorithmic Transparency: The multimodal query routing mechanism does not clearly indicate the dominant retrieval feature. This hides whether keyword text or structural similarity actually drove the final result placement. This ambiguity limits operator control.
• Optical Processing Brittleness: The embedded vision inference and primitive parsing pipelines display inherent fragility, producing intermittent failures when associating text passages with correctly parsed molecular diagrams.
• Metadata Logging Incompleteness: Practicing chemists requested additional structured metadata targets (such as specific molar equivalents and parameterized mol% values) to successfully bridge the extracted data stream directly into digital electronic lab notebooks.

Reproducibility

Artifacts

Data

• Source: The corpus features 7 primary research papers and 6 auxiliary supplementary information documents focusing on Suzuki coupling reactions, sourced from practicing chemists at UIUC. This evaluation dataset is strictly internal and not publicly available.
• Preprocessing:
  • Engineers convert source PDFs to full-page raster images.
  • The system extracts localized graphical layout and raw text via PyTesseract.
  • The pipeline segments valid passage chunks emphasizing reaction-related sentences utilizing product-indicative lexicons and topic modeling.

Algorithms

• Diagram Extraction: A YOLOv8 model identifies and segments molecular regions within structured PDF pages.
• Diagram Parsing: The architecture relies on ChemScraper to infer structural semantics from raw diagrams:
  • Born-digital PDFs: SymbolScraper extracts vector lines and polygons directly from bounding box definitions.
  • Raster images: The system employs the Line Segment Detector (LSD) and watershed bounding algorithms to isolate native geometric primitives.
• Text Entity Extraction: The framework deploys ChemDataExtractor 2.0 to extract explicit molecular aliases. A translation layer maps these entities to string representations via OPSIN.
• Linking Logic (Fusion Score):
  • Text Link: The algorithm calculates a normalized Levenshtein ratio connecting visual diagram labels against proximal text mentions based on calculated edit distance.
  • Structure Link: The algorithm computes the discrete Tanimoto Similarity between generated 2048-bit Morgan fingerprints extracted from localized visual diagram features and baseline text SMILES queries: $$ T(A, B) = \frac{A \cdot B}{|A|^{2} + |B|^{2} - A \cdot B} $$ where $A$ and $B$ represent the boolean bit vectors of the respective fingerprint pairs.
  • Conflict Resolution Protocol: The system fuses structural geometry bounds and discrete textual tokenization metrics, prioritizing the ranking sequence that yields a higher terminal similarity score. During final retrieval, the candidate subset is systematically re-ranked leveraging the hybrid calculation of the BM25 explicit metric and the localized count of exact SMILES pattern hits.

Models

• Reaction Extraction Parameters: The engineers configure a LLaMA-3.1-8b model fine-tuned entirely via LoRA targeting custom tokens representing reaction entities (compounds, reagents, thermal inputs) directly pulled from text sub-chunks. Exact prompt constraints, the fine-tuning dataset, and specific LoRA hyperparameters are omitted from the source text.
• Diagram Processing Bounds: The codebase incorporates a segmentation-aware multi-task neural network topology built into ChemScraper to execute low-level raster image parsing tasks.

Evaluation

• Search Engine Base: The authors implemented their indexing framework scaling atop PyTerrier.
• Text Feature Ranking: The metric utilizes standalone BM25 bounds mapping keyword-similarity.
• Structure Feature Operations: The topology operates RDKit bindings powering substructure coordinate mapping logic and exact molecular similarity searches.
• Multimodal Fusion Processing:
  • The algorithm filters out terminal candidates mapping initial structural properties (SMILES queries) against the document-wide lexical properties (BM25 scores).
  • The final fusion routing assigns the strongest positive weight to retrieved passages that accumulate dense local clusters of structurally exact verified SMILES patterns.

Hardware

• Compute Infrastructure: The hardware and parameter requirements to host the multi-stage vision extractors (YOLOv8, ChemScraper) alongside a local 8B LLM are entirely unspecified in the paper.

Paper Information

Citation: Shah, A. K., et al. (2025). Multimodal Search in Chemical Documents and Reactions. arXiv preprint arXiv:2502.16865. https://doi.org/10.48550/arXiv.2502.16865

Publication: SIGIR ‘25 (Demo Track), 2025

@misc{shahMultimodalSearchChemical2025,
  title = {Multimodal {{Search}} in {{Chemical Documents}} and {{Reactions}}},
  author = {Shah, Ayush Kumar and Dey, Abhisek and Luo, Leo and Amador, Bryan and Philippy, Patrick and Zhong, Ming and Ouyang, Siru and Friday, David Mark and Bianchi, David and Jackson, Nick and Zanibbi, Richard and Han, Jiawei},
  year = 2025,
  month = feb,
  number = {arXiv:2502.16865},
  eprint = {2502.16865},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2502.16865},
  archiveprefix = {arXiv}
}

Additional Resources:

• Online Demo (Note: While the landing page advertises the system as open-source, the exact repository URL and installation prerequisites are omitted from the official manuscript.)

This is a Methodological Paper ($\Psi_{\text{Method}}$).

It introduces MOFFlow, a generative architecture and training framework designed specifically for the structure prediction of Metal-Organic Frameworks (MOFs). The paper focuses on the algorithmic innovation of decomposing the problem into rigid-body assembly on a Riemannian manifold, validates this through comparison against existing baselines, and performs ablation studies to justify architectural choices. While it leverages the theory of flow matching, its primary contribution is the application-specific architecture and the handling of modular constraints.

Motivation: Scaling Limits of Atom-Level Generation

The primary motivation is to overcome the scalability and accuracy limitations of existing methods for MOF structure prediction.

• Computational Cost of DFT: Conventional approaches rely on ab initio calculations (DFT) combined with random search, which are computationally prohibitive for large, complex systems like MOFs.
• Failure of General CSP: Existing deep generative models for general Crystal Structure Prediction (CSP) operate on an atom-by-atom basis. They fail to scale to MOFs, which often contain hundreds or thousands of atoms per unit cell, and do not exploit the inherent modular nature (building blocks) of MOFs.
• Tunability: MOFs have applications in carbon capture and drug delivery due to their tunable porosity, making automated design tools valuable.

Core Innovation: Rigid-Body Flow Matching on SE(3)

MOFFlow introduces a hierarchical, rigid-body flow matching framework tailored for MOFs.

• Rigid Body Decomposition: MOFFlow treats metal nodes and organic linkers as rigid bodies, reducing the search space from $3N$ (atoms) to $6M$ (roto-translation of $M$ blocks) compared to atom-based methods.
• Riemannian Flow Matching on $SE(3)$: It is the first end-to-end model to jointly generate block-level rotations ($SO(3)$), translations ($\mathbb{R}^3$), and lattice parameters using Riemannian flow matching.
• MOFAttention: A custom attention module designed to encode the geometric relationships between building blocks, lattice parameters, and rotational constraints.
• Constraint Handling: It incorporates domain knowledge by operating on a mean-free system for translation invariance and using canonicalized coordinates for rotation invariance.

Experimental Setup and Baselines

The authors evaluated MOFFlow on structure prediction accuracy, physical property preservation, and scalability.

• Dataset: The Boyd et al. (2019) dataset consisting of 324,426 hypothetical MOF structures, decomposed into building blocks using the MOFid algorithm. Filtered to structures with <200 blocks, yielding 308,829 structures (247,066 train / 30,883 val / 30,880 test). Structures contain up to approximately 2,400 atoms per unit cell.
• Baselines:
  • Optimization-based: Random Search (RS) and Evolutionary Algorithm (EA) using CrySPY and CHGNet.
  • Deep Learning: DiffCSP (deep generative model for general crystals).
  • Self-Assembly: A heuristic algorithm used in MOFDiff (adapted for comparison).
• Metrics:
  • Match Rate (MR): Percentage of generated structures matching ground truth within tolerance.
  • RMSE: Root mean squared displacement normalized by average free length per atom.
  • Structural Properties: Volumetric/Gravimetric Surface Area (VSA/GSA), Pore Limiting Diameter (PLD), Void Fraction, etc., calculated via Zeo++.
  • Scalability: Performance vs. number of atoms and building blocks.

Results and Generative Performance

MOFFlow outperformed all baselines in accuracy and efficiency, particularly for large structures.

• Accuracy: With a single sample, MOFFlow achieved a 31.69% match rate (stol=0.5) and 87.46% (stol=1.0) on the full test set (30,880 structures). With 5 samples, these rose to 44.75% (stol=0.5) and 100.0% (stol=1.0). RS and EA (tested on 100 and 15 samples respectively due to computational cost, generating 20 candidates each) achieved 0.00% MR at both tolerance levels. DiffCSP reached 0.09% (stol=0.5) and 23.12% (stol=1.0) with 1 sample.
• Speed: Inference took 1.94 seconds per structure, compared to 5.37s for DiffCSP, 332s for RS, and 1,959s for EA.
• Scalability: MOFFlow preserved high match rates across all system sizes, while DiffCSP’s match rate dropped sharply beyond 200 atoms.
• Property Preservation: The distributions of physical properties (e.g., surface area, void fraction) for MOFFlow-generated structures closely matched the ground truth. DiffCSP frequently reduced volumetric surface area and void fraction to zero.
• Self-Assembly Comparison: In a controlled comparison where the self-assembly (SA) algorithm received MOFFlow’s predicted translations and lattice, MOFFlow (MR=31.69%, RMSE=0.2820) outperformed SA (MR=30.04%, RMSE=0.3084), confirming the value of the learned rotational vector fields. In an extended scalability comparison, SA scaled better for structures with many building blocks, but MOFFlow achieved higher overall match rate (31.69% vs. 27.14%).
• Batch Implementation: A refactored Batch version achieves improved results: 32.73% MR (stol=0.5), RMSE of 0.2743, inference in 0.19s per structure (10x faster), and training in roughly 1/3 the GPU hours.

Limitations

The paper identifies three key limitations:

1. Hypothetical-only evaluation: All experiments use the Boyd et al. hypothetical database. Evaluation on more challenging real-world datasets remains needed.
2. Rigid-body assumption: The model assumes that local building block structures are known, which may be impractical for rare building blocks whose structural information is missing from existing libraries or is inaccurate.
3. Periodic invariance: The model is not invariant to periodic transformations of the input. Explicitly modeling periodic invariance could further improve performance.

Reproducibility Details

Data

• Source: MOF dataset by Boyd et al. (2019).
• Preprocessing: Structures were decomposed using the metal-oxo decomposition algorithm from MOFid.
• Filtering: Structures with fewer than 200 building blocks were used, yielding 308,829 structures.
• Splits: Train/Validation/Test ratio of 8:1:1 (247,066 / 30,883 / 30,880).
• Availability: Pre-processed dataset is available on Zenodo.
• Representations:
  • Atom-level: Tuple $(X, a, l)$ (coordinates, types, lattice).
  • Block-level: Tuple $(\mathcal{B}, q, \tau, l)$ (blocks, rotations, translations, lattice).

Algorithms

• Framework: Riemannian Flow Matching.
• Objective: Conditional Flow Matching (CFM) loss regressing to clean data $q_1, \tau_1, l_1$. $$ \begin{aligned} \mathcal{L}(\theta) = \mathbb{E}_{t, \mathcal{S}^{(1)}} \left[ \frac{1}{(1-t)^2} \left( \lambda_1 |\log_{q_t}(\hat{q}_1) - \log_{q_t}(q_1)|^2 + \dots \right) \right] \end{aligned} $$
• Priors:
  • Rotations ($q$): Uniform on $SO(3)$.
  • Translations ($\tau$): Standard normal on $\mathbb{R}^3$.
  • Lattice ($l$): Log-normal for lengths, Uniform(60, 120) for angles (Niggli reduced).
• Inference: ODE solver with 50 integration steps.
• Local Coordinates: Defined using PCA axes, corrected for symmetry to ensure consistency.

Models

• Architecture: Hierarchical structure with two key modules.
  • Atom-level Update Layers: 4-layer EGNN-like structure to encode building block features $h_m$ from atomic graphs (cutoff 5Å).
  • Block-level Update Layers: 6 layers that iteratively update $q, \tau, l$ using the MOFAttention module.
• MOFAttention: Modified Invariant Point Attention (IPA) that incorporates lattice parameters as offsets to the attention matrix.
• Hyperparameters:
  • Node dimension: 256 (block-level), 64 (atom-level).
  • Attention heads: 24.
  • Loss coefficients: $\lambda_1=1.0$ (rot), $\lambda_2=2.0$ (trans), $\lambda_3=0.1$ (lattice).
• Checkpoints: Pre-trained weights and models are openly provided on Zenodo.

Evaluation

• Metrics:
  • Match Rate: Using StructureMatcher from pymatgen. Tolerances: stol=0.5/1.0, ltol=0.3, angle_tol=10.0.
  • RMSE: Normalized by average free length per atom.
• Tools: Zeo++ for structural property calculations (Surface Area, Pore Diameter, etc.).

Hardware

• Training Hardware: 8 $\times$ NVIDIA RTX 3090 (24GB VRAM).
• Training Time:
  • TimestepBatch version (main paper): ~5 days 15 hours.
  • Batch version: ~1 day 17 hours (332.74 GPU hours). The authors also release this refactored implementation, which achieves comparable performance with faster convergence.
• Batch Size: 160 (capped at $1.6 \times 10^6$ atoms squared for memory).

Paper Information

Citation: Kim, N., Kim, S., Kim, M., Park, J., & Ahn, S. (2025). MOFFlow: Flow Matching for Structure Prediction of Metal-Organic Frameworks. International Conference on Learning Representations (ICLR).

Publication: ICLR 2025

@inproceedings{kimMOFFlowFlowMatching2025,
  title={MOFFlow: Flow Matching for Structure Prediction of Metal-Organic Frameworks},
  author={Kim, Nayoung and Kim, Seongsu and Kim, Minsu and Park, Jinkyoo and Ahn, Sungsoo},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025},
  url={https://openreview.net/forum?id=dNT3abOsLo}
}

Additional Resources:

• OpenReview Discussion
• Official Code Repository

This is a Methodological ($\Psi_{\text{Method}}$) paper. It introduces a novel architecture, InvMSAFold, which hybridizes deep learning encoders with statistical physics-based decoders (Potts models). The rhetorical structure focuses on architectural innovation (low-rank parameter generation), ablation of speed/diversity against baselines (ESM-IF1), and algorithmic efficiency.

What is the motivation?

Standard inverse folding models (like ESM-IF1 or ProteinMPNN) solve a “one-to-one” mapping: given a structure, predict the single native sequence. However, in nature, folding is “many-to-one”: many homologous sequences fold into the same structure.

The authors identify two key gaps:

1. Lack of Diversity: Standard autoregressive models maximize probability for the ground truth sequence, often failing to capture the broad evolutionary landscape of viable homologs.
2. Slow Inference: Autoregressive sampling requires a full neural network pass for every amino acid, making high-throughput screening (e.g., millions of candidates) computationally prohibitive.

What is the novelty here?

The core novelty is shifting the learning objective from predicting sequences to predicting probability distributions.

InvMSAFold outputs the parameters (couplings $\mathbf{J}$ and fields $\mathbf{h}$) of a Potts Model (a pairwise Markov Random Field).

• Low-Rank Decomposition: To handle the massive parameter space of pairwise couplings ($L xL xq xq$), the model predicts a low-rank approximation $\mathbf{V}$ ($L xK xq$), reducing complexity from $\mathcal{O}(L^2)$ to $\mathcal{O}(L)$.
• One-Shot Generation: The deep network runs only once to generate the Potts parameters. Sampling sequences from this Potts model is then performed on CPU via MCMC or autoregressive approximation, which is orders of magnitude faster than running a Transformer decoder for every step.

What experiments were performed?

The authors validated the model on three CATH-based test sets (Inter-cluster, Intra-cluster, MSA) to test generalization to unseen folds.

• Speed Benchmarking: Compared wall-clock sampling time vs. ESM-IF1 on CPU/GPU.
• Covariance Reconstruction: Checked if generated sequences recover the evolutionary correlations found in natural MSAs (Pearson correlation of covariance matrices).
• Structural Fidelity: Generated sequences with high Hamming distance from native, folded them with AlphaFold 2 (no templates), and measured RMSD to the target structure.
• Property Profiling: Analyzed the distribution of predicted solubility (Protein-Sol) and thermostability (Thermoprot) to demonstrate diversity.

What outcomes/conclusions?

• Massive Speedup: InvMSAFold is orders of magnitude faster than ESM-IF1 (CPU vs. GPU; the comparison is not hardware-matched). Because the “heavy lifting” (generating Potts parameters) happens once, sampling millions of sequences becomes trivial on CPUs.
• Better Diversity: The model captures evolutionary covariances significantly better than ESM-IF1 and ProteinMPNN (which shares similar covariance recovery to ESM-IF1). A PCA-based KL-divergence analysis (lower is better; 0 means a perfect match to the natural MSA distribution) shows InvMSAFold-AR scores of $0.49$ (Inter-cluster) and $0.67$ (Intra-cluster), compared to $15.8$ and $11.9$ for ESM-IF1, demonstrating that the generated sequences occupy a distribution much closer to natural MSAs.
• Robust Folding: Sequences generated far from the native sequence (high Hamming distance) still fold into the correct structure (low RMSD), whereas ESM-IF1 struggles to produce diverse valid sequences.
• Property Expansion: The method generates a wider spread of biochemical properties (solubility/thermostability), making it more suitable for screening libraries in protein design.

Reproducibility Details

Data

Source: CATH v4.2 database (40% non-redundant dataset).

Splits:

• Training: ~22k domains.
• Inter-cluster Test: 10% of clusters held out (unseen folds).
• Intra-cluster Test: Unseen domains from seen clusters.
• Augmentation: MSAs generated using MMseqs2 against UniRef50. Training uses random subsamples of these MSAs ($|M_X| = 64$) to teach the model evolutionary variance.

Algorithms

Architecture:

• Encoder: Pre-trained ESM-IF1 encoder. The paper does not explicitly state whether the encoder is frozen or jointly fine-tuned during training; this is an unresolved reproducibility detail.
• Decoder: 6-layer Transformer (8 heads) that outputs a latent tensor.
• Projection: Linear layers project latent tensor to fields $\mathbf{h}$ ($L xq$) and low-rank tensor $\mathbf{V}$ ($L xK xq$).

Coupling Construction: The full coupling tensor $\mathcal{J}$ is approximated via: $$\mathcal{J}_{i,a,j,b} = \frac{1}{\sqrt{K}} \sum_{k=1}^{K} \mathcal{V}_{i,k,a} \mathcal{V}_{j,k,b}$$ Rank $K=48$ was used.

Loss Functions: Two variants were trained:

1. InvMSAFold-PW: Trained via Pseudo-Likelihood (PL). Computation is optimized to $\mathcal{O}(L)$ time using the low-rank property.
2. InvMSAFold-AR: Trained via Autoregressive Likelihood. Couplings are masked ($J_{ij} = 0$ if $i<j$) to allow exact likelihood computation and direct sampling without MCMC.

Models

• InvMSAFold-PW: Requires MCMC sampling (Metropolis-Hastings) at inference.
• InvMSAFold-AR: Allows direct, fast autoregressive sampling.
• Hyperparameters: AdamW optimizer, lr=$10^{-4}$ (PW) / $3.4 x10^{-4}$ (AR), 94 epochs. L2 regularization on fields/couplings.

Evaluation

Metrics:

• scRMSD: Structure fidelity (AlphaFold2 prediction vs. Native).
• Covariance Pearson Correlation: Measures recovery of evolutionary pairwise statistics.
• Sampling Speed: Wall-clock time vs. sequence length/batch size.

Hardware

• Training: Unspecified; described as “fast” due to low rank.
• Inference:
  • ESM-IF1: NVIDIA GeForce RTX 4060 Laptop (8GB).
  • InvMSAFold: Single core of Intel i9-13905H CPU.

Paper Information

Citation: Silva, L. A., Meynard-Piganeau, B., Lucibello, C., & Feinauer, C. (2025). Fast Uncovering of Protein Sequence Diversity from Structure. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2406.11975

Publication: ICLR 2025

@inproceedings{silvaFastUncoveringProtein2025,
  title = {Fast {{Uncovering}} of {{Protein Sequence Diversity}} from {{Structure}}},
  booktitle = {The {{Thirteenth International Conference}} on {{Learning Representations}}},
  author = {Silva, Luca Alessandro and {Meynard-Piganeau}, Barthelemy and Lucibello, Carlo and Feinauer, Christoph},
  year = {2025},
  url = {https://openreview.net/forum?id=1iuaxjssVp}
}

Additional Resources:

• OpenReview Page
• GitHub Repository

Methodological Paper ($\Psi_{\text{Method}}$)

This work proposes InstructMol, a novel multi-modal architecture and training paradigm. It focuses on engineering a system that aligns a pre-trained molecular graph encoder with a general-purpose Large Language Model (LLM). The paper’s primary contribution is the Two-Stage Instruction Tuning strategy (Alignment Pre-training + Task-Specific Tuning) designed to bridge the modality gap between 2D molecular graphs and natural language.

Bridging Specialist and Generalist Models

Current AI approaches in drug discovery typically fall into two categories. Specialist models deliver high accuracy on specific tasks (such as property prediction) but require extensive labeled datasets and lack conversational adaptability. Conversely, generalist LLMs offer strong reasoning and dialogue capabilities but struggle to natively interpret complex structural data, often relying on brittle 1D text representations of molecules like SMILES.

There is a practical need for a unified “Molecular Assistant” capable of visually interpreting molecular graphs, reasoning about structure in natural language, and adapting across tasks like synthesis planning and property analysis without training from scratch.

Two-Stage Modality Alignment

The core novelty lies in the architecture and the two-stage training pipeline designed to align differing modalities efficiently:

1. MoleculeSTM Integration: InstructMol initializes its graph encoder with MoleculeSTM, which is already pre-aligned with text via contrastive learning, facilitating easier downstream alignment.
2. Two-Stage Alignment Strategy:
   • Stage 1 (Alignment Pre-training): Freezes both the LLM and Graph Encoder; trains only a linear projector using a massive dataset of molecule-description pairs to map graph features into the LLM’s token space.
   • Stage 2 (Task-Specific Instruction Tuning): Freezes the Graph Encoder; fine-tunes the Projector and the LLM (using LoRA) on specific downstream tasks. This allows the model to adapt its reasoning capabilities while preserving the structural understanding gained in Stage 1.

Task Evaluation in Drug Discovery

The authors evaluated InstructMol across three distinct categories of drug discovery tasks, comparing it against generalist LLMs (Vicuna, LLaMA, Galactica) and specialist models (ChemBERTa, MolT5):

1. Property Prediction:
   • Regression: Predicting quantum mechanical properties (HOMO, LUMO, Gap) using the QM9 dataset.
   • Classification: Predicting biological activity (BACE, BBBP, HIV) using MoleculeNet.
2. Molecule Description Generation: Generating natural language descriptions of molecules using the ChEBI-20 dataset.
3. Chemical Reaction Analysis:
   • Forward Reaction Prediction: Predicting products from reactants.
   • Reagent Prediction: Identifying necessary reagents.
   • Retrosynthesis: Suggesting reactants for a given product.

Ablation Studies tested the impact of the projector type (Linear vs. MLP), LLM scale (7B vs 13B), and the necessity of the two-stage training approach.

Core Findings and Limitations

• Improvement Over Baseline Generalists: InstructMol significantly outperformed generalist LLMs (like LLaMA and Galactica) on all tasks, demonstrating the value of incorporating explicit graph modalities.
• Reducing the Gap with Specialists: While InstructMol brings versatile reasoning capabilities, it still trails highly optimized specialist models (such as Uni-Mol and MolT5) on tasks like molecule description generation. This remaining gap likely stems from its reliance on a relatively small alignment pre-training dataset (~264K PubChem pairs) and the information bottleneck of using a simple linear projector, compared to the millions of structures used to train expert foundational models.
• Importance of Alignment: Ablation studies confirmed that skipping Stage 1 (Alignment Pre-training) degraded performance, proving that a dedicated phase for projecting graph features into text space is crucial.
• Limitation: The model struggles with highly imbalanced datasets (e.g., HIV) and complex reaction mixtures where mapping multiple graph tokens to text becomes ambiguous.

Reproducibility Details

Data

The training pipeline utilizes distinct datasets for the two stages. Note: As of the latest repository update, the finely-processed instruction-tuning datasets (e.g., the filtered ~264K PubChem pairs and instruction-formatted subset pairs) are listed as “coming soon”, requiring manual recreation for full reproduction.

Algorithms

• Two-Stage Training:
  1. Alignment Pre-training: Updates only the Projector. The objective maximizes the probability of generating the target description token sequence $\mathbf{X}_A$ given the molecule input $\mathbf{X}_M$ and instruction $\mathbf{X}_I$: $$p(\mathbf{X}_A | \mathbf{X}_M, \mathbf{X}_I) = \prod_{i=1}^L p_\theta(x_i | \mathbf{X}_G \parallel \mathbf{X}_S, \mathbf{X}_I, \mathbf{X}_{A,<i})$$
  2. Instruction Tuning: Updates Projector + LLM (via LoRA) using standard autoregressive language modeling on task-specific instructions. The objective minimizes the negative log-likelihood of generating the target response $R$ of length $L$: $$\mathcal{L}(\theta) = -\sum_{i=1}^L \log p(R_i | I, M, R_{<i}; \theta)$$ where $I$ represents the instruction and $M$ is the multi-modal molecular input.
• LoRA (Low-Rank Adaptation): Applied to the LLM in Stage 2. Rank $r=64$, Scaling $\alpha=16$.
• Optimization: AdamW optimizer. Learning rate starts at 2e-3 (Stage 1) and 8e-5 (Stage 2) with cosine decay. Warm-up ratio 0.03.

Models

Note: The official repository currently lists the final fine-tuned InstructMol weights as “coming soon.” Consequently, one must fine-tune the components using the provided scripts. Base model weights (Vicuna-7B and MoleculeSTM) are publicly available via Hugging Face.

• Graph Encoder ($f_g$):
  • Architecture: Graph Isomorphism Network (GIN) with 5 layers.
  • Hidden Dimension: 300.
  • Initialization: MoleculeSTM checkpoint (pre-trained via contrastive learning).
  • Status: Frozen during Stage 2.
• LLM:
  • Base: Vicuna-v1.3-7B.
  • Status: Frozen in Stage 1; LoRA fine-tuned in Stage 2.
• Projector:
  • Architecture: Linear Layer.
  • Function: Maps node-level graph representation $Z_G \in \mathbb{R}^{N \times d}$ to the LLM’s word embedding space dimensions.

Evaluation

• Metric Libraries: RDKit for validity/fingerprints, standard NLP libraries for BLEU/ROUGE.
• Reaction Metrics: Fingerprint Tanimoto Similarity (FTS), Exact Match, Levenshtein distance, and validity (via RDKit).
• Description Metrics: BLEU-2, BLEU-4, ROUGE-1, ROUGE-2, ROUGE-L, METEOR.

Hardware

• Compute: 4 x NVIDIA RTX A6000 (48GB VRAM).
• Training Time:
  • Stage 1: 5 epochs.
  • Stage 2: 20-50 epochs (Description Generation), 10 epochs (Properties/Reactions).
• Batch Size: 128 for both stages.

Artifacts

Paper Information

Citation: Cao, H., Liu, Z., Lu, X., Yao, Y., & Li, Y. (2025). InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery. Proceedings of the 31st International Conference on Computational Linguistics, 354-379.

Publication: COLING 2025

@inproceedings{caoInstructMolMultiModalIntegration2025,
  title = {{{InstructMol}}: {{Multi-Modal Integration}} for {{Building}} a {{Versatile}} and {{Reliable Molecular Assistant}} in {{Drug Discovery}}},
  shorttitle = {{{InstructMol}}},
  booktitle = {Proceedings of the 31st {{International Conference}} on {{Computational Linguistics}}},
  author = {Cao, He and Liu, Zijing and Lu, Xingyu and Yao, Yuan and Li, Yu},
  editor = {Rambow, Owen and Wanner, Leo and Apidianaki, Marianna and {Al-Khalifa}, Hend and Eugenio, Barbara Di and Schockaert, Steven},
  year = 2025,
  month = jan,
  pages = {354--379},
  publisher = {Association for Computational Linguistics},
  address = {Abu Dhabi, UAE},
  abstract = {The rapid evolution of artificial intelligence in drug discovery encounters challenges with generalization and extensive training, yet Large Language Models (LLMs) offer promise in reshaping interactions with complex molecular data. Our novel contribution, InstructMol, a multi-modal LLM, effectively aligns molecular structures with natural language via an instruction-tuning approach, utilizing a two-stage training strategy that adeptly combines limited domain-specific data with molecular and textual information. InstructMol showcases substantial performance improvements in drug discovery-related molecular tasks, surpassing leading LLMs and significantly reducing the gap with specialists, thereby establishing a robust foundation for a versatile and dependable drug discovery assistant.}
}

Additional Resources:

• Official Repository

This note provides a comparative analysis of image-to-sequence methods for Optical Chemical Structure Recognition (OCSR). These methods treat molecular structure recognition as an image captioning task, using encoder-decoder architectures to generate sequential molecular representations (SMILES, SELFIES, InChI) directly from pixels.

For the full taxonomy of OCSR approaches including image-to-graph and rule-based methods, see the OCSR Methods taxonomy.

Architectural Evolution (2019-2025)

The field has undergone rapid architectural evolution, with clear generational shifts in both encoder and decoder design.

Timeline

Encoder Architectures

Decoder Architectures

Output Representation Comparison

The choice of molecular string representation significantly impacts model performance. Representations fall into three categories: core molecular formats for single structures, extended formats for molecular families and variable structures (primarily Markush structures in patents), and specialized representations optimizing for specific recognition challenges.

The Rajan et al. 2022 ablation study provides a comparison of core formats.

Core Molecular Formats

These represent specific, concrete molecular structures.

SMILES and Variants

SMILES remains the dominant format due to its compactness and highest accuracy on clean data. Standard SMILES uses single characters for ring closures and branches that may appear far apart in the sequence, creating learning challenges for sequence models.

DeepSMILES addresses these non-local syntax dependencies by modifying how branches and ring closures are encoded, making sequences more learnable for neural models. Despite this modification, DeepSMILES sequences are ~1.1x longer than standard SMILES (not shorter). The format offers partial validity improvements through regex-based tokenization with a compact 76-token vocabulary, providing a middle ground between SMILES accuracy and guaranteed validity.

SELFIES guarantees 100% valid molecules by design through a context-free grammar, eliminating invalid outputs entirely. This comes at the cost of ~1.5x longer sequences and a typical 2-5% accuracy drop compared to SMILES on exact-match metrics. The validity guarantee makes SELFIES particularly attractive for generative modeling applications.

InChI uses a layered canonical syntax fundamentally different from SMILES-based formats. While valuable for unique molecular identification, its complex multi-layer structure (formula, connectivity, stereochemistry, isotopes, etc.) and longer sequences make it less suitable for image-to-sequence learning, resulting in lower recognition accuracy.

Key Findings from Rajan et al. 2022

1. SMILES achieves highest exact-match accuracy on clean synthetic data
2. SELFIES guarantees 100% valid molecules but at cost of ~2-5% accuracy drop
3. InChI is problematic due to complex layered syntax and longer sequences
4. DeepSMILES offers middle ground with partial validity improvements through modified syntax

Extended Formats for Variable Structures

Markush structures represent families of molecules, using variable groups (R1, R2, etc.) with textual definitions. They are ubiquitous in patent documents for intellectual property protection. Standard SMILES cannot represent these variable structures.

E-SMILES (Extended SMILES) maintains backward compatibility by using a <sep> token to separate core SMILES from XML-like annotations. Annotations encode Markush substituents (<a>index:group</a>), polymer structures (<p>polymer_info</p>), and abstract ring patterns (<r>abstract_ring</r>). The core structure remains parseable by standard RDKit.

CXSMILES optimizes representation by moving variable groups directly into the main SMILES string as special atoms with explicit atom indexing (e.g., C:1) to link to an extension block containing substituent tables. This handles both frequency variation and position variation in Markush structures.

Specialized Representations

These formats optimize for specific recognition challenges beyond standard single-molecule tasks.

RFL: Ring-Free Language

RFL fundamentally restructures molecular serialization through hierarchical ring decomposition, addressing a core challenge: standard 1D formats (SMILES, SSML) flatten complex 2D molecular graphs, losing explicit spatial relationships.

Mechanism: RFL decomposes molecules into three explicit components:

• Molecular Skeleton (𝒮): Main graph with rings “collapsed”
• Ring Structures (ℛ): Individual ring components stored separately
• Branch Information (ℱ): Connectivity between skeleton and rings

Technical approach:

1. Detect all non-nested rings using DFS
2. Calculate adjacency ($\gamma$) between rings based on shared edges
3. Merge isolated rings ($\gamma=0$) into SuperAtoms (single node placeholders)
4. Merge adjacent rings ($\gamma>0$) into SuperBonds (edge placeholders)
5. Progressive decoding: predict skeleton first, then conditionally decode rings using stored hidden states

Performance: RFL achieves SOTA results on both handwritten (95.38% EM) and printed (95.58% EM) structures, with particular strength on high-complexity molecules where standard baselines fail completely (0% → ~30% on hardest tier).

Note: RFL does not preserve original drawing orientation; it’s focused on computational efficiency through hierarchical decomposition.

SSML: Structure-Specific Markup Language

SSML is the primary orientation-preserving format in OCSR. Based on Chemfig (LaTeX chemical drawing package), it provides step-by-step drawing instructions.

Key characteristics:

• Describes how to draw the molecule alongside its graph structure
• Uses “reconnection marks” for cyclic structures
• Preserves branch angles and spatial relationships
• Significantly outperformed SMILES for handwritten recognition: 92.09% vs 81.89% EM (Hu et al. RCGD 2023)

Use case: Particularly valuable for hand-drawn structure recognition where visual alignment between image and reconstruction sequence aids model learning.

Training Data Comparison

Training data scale has grown dramatically, with a shift toward combining synthetic and real-world images.

Data Scale Evolution

Synthetic vs Real Data

Data Augmentation Strategies

Common augmentation techniques across methods:

Hardware and Compute Requirements

Hardware requirements span several orders of magnitude, from consumer GPUs to TPU pods.

Training Hardware Comparison

Inference Considerations

Few papers report inference speed consistently. Available data:

Accessibility Tiers

Benchmark Performance

Common Evaluation Datasets

Accuracy Comparison (Exact Match %)

Methods are roughly grouped by evaluation era; direct comparison is complicated by different test sets.

Note: Numbers are approximate and may vary by specific test split. See individual paper notes for precise figures.

Stereochemistry Recognition

Stereochemistry remains a persistent challenge across all methods:

Hand-Drawn Recognition

A distinct sub-lineage focuses on hand-drawn/sketched chemical structures.

Hand-Drawn vs Printed Trade-offs

• Hand-drawn methods sacrifice some accuracy on clean printed images
• Require specialized training data (synthetic hand-drawn simulation)
• Generally smaller training sets due to data collection difficulty
• Better suited for educational and lab notebook applications

Key Innovations by Method

Open Challenges

1. Stereochemistry: Consistent challenge across all methods; RL approaches (MolSight) show promise
2. Abbreviations/R-groups: E-SMILES and Markush-specific methods emerging
3. Real-world robustness: Gap between synthetic training and patent/paper images
4. Inference speed: Rarely reported; important for production deployment
5. Memory efficiency: Almost never documented; limits accessibility
6. Multi-molecule images: Most methods assume single isolated structure

References

Individual paper notes linked throughout. For the complete method listing, see the OCSR Methods taxonomy.

This is primarily a Methodological Paper ($\Psi_{\text{Method}}$) with a strong Resource ($\Psi_{\text{Resource}}$) component.

• Method: It proposes DynamicFlow, a novel multiscale architecture combining atom-level SE(3)-equivariant GNNs (SE(3) is the special Euclidean group in 3D: the set of all 3D rotations and translations, and equivariance means predictions transform consistently under those symmetries) and residue-level Transformers within a flow matching framework to model the joint distribution of ligand generation and protein conformational change.
• Resource: It curates a significant dataset derived from MISATO, pairing AlphaFold2-predicted apo structures with multiple MD-simulated holo states, specifically filtered for flow matching tasks.

What is the motivation?

Traditional Structure-Based Drug Design (SBDD) methods typically assume the protein target is rigid, which limits their applicability because proteins are dynamic and undergo conformational changes (induced fit) upon ligand binding.

• Biological Reality: Proteins exist as ensembles of states; binding often involves transitions from “apo” (unbound) to “holo” (bound) conformational changes, sometimes revealing cryptic pockets.
• Computational Bottleneck: Molecular Dynamics (MD) simulates these changes but incurs high computational costs due to energy barriers.
• Gap: Existing generative models for SBDD mostly condition on a fixed pocket structure, ignoring the co-adaptation of the protein and ligand.

What is the novelty here?

The core novelty is the simultaneous modeling of ligand generation and protein conformational dynamics using a unified flow matching framework.

• DynamicFlow Architecture: A multiscale model that treats the protein as both full-atom (for interaction) and residue-level frames (for large-scale dynamics), utilizing separate flow matching objectives for backbone frames, side-chain torsions, and ligand atoms.
• Stochastic Flow (SDE): Introduction of a stochastic variant (DynamicFlow-SDE) that improves robustness and diversity compared to the deterministic ODE flow.
• Coupled Generation: The model learns to transport the apo pocket distribution to the holo pocket distribution while simultaneously denoising the ligand, advancing beyond rigid pocket docking methods.

What experiments were performed?

The authors validated the method on a curated dataset of 5,692 protein-ligand complexes.

• Baselines: Compared against rigid-pocket SBDD methods: Pocket2Mol, TargetDiff, and IPDiff (adapted as TargetDiff* and IPDiff* for fair comparison of atom numbers). Also compared against conformation sampling baselines (Str2Str).
• Metrics:
  • Ligand Quality: Vina Score (binding affinity), QED (drug-likeness), SA (synthesizability), Lipinski’s rule of 5.
  • Pocket Quality: RMSD between generated and ground-truth holo pockets, Cover Ratio (percentage of holo states successfully retrieved), and Pocket Volume distributions.
  • Interaction: Protein-Ligand Interaction Profiler (PLIP) to measure specific non-covalent interactions.
• Ablations: Tested the impact of the interaction loss, residue-level Transformer, and SDE vs. ODE formulations.

What outcomes/conclusions?

• Improved Affinity: DynamicFlow-SDE achieved the best (lowest) Vina scores ($-7.65$) compared to baselines like TargetDiff ($-5.09$) and Pocket2Mol ($-5.50$). Note that Vina scores are a computational proxy and do not directly predict experimental binding affinity. Moreover, Vina score optimization is gameable: molecules can achieve strong computed binding energies while remaining synthetically inaccessible. QED and SA scores, which assess drug-likeness and synthesizability respectively, were reported but were not primary optimization targets in the paper, which limits the strength of this affinity claim.
• Realistic Dynamics: The model successfully generated holo-like pocket conformations with volume distributions and interaction profiles closer to ground-truth MD simulations than the initial apo structures.
• Enhancing Rigid Methods: Holo pockets generated by DynamicFlow served as better inputs for rigid-SBDD baselines (e.g., TargetDiff improved from $-5.09$ to $-9.00$ and IPDiff improved from $-5.69$ to $-11.04$ when using “Our Pocket”), suggesting the method can act as a “pocket refiner”.
• ODE vs. SDE Trade-off: The deterministic ODE variant achieves better pocket RMSD, while the stochastic SDE variant achieves better Cover Ratio (diversity of holo states captured) and binding affinity. Neither dominates uniformly.
• Conformation Sampling Baseline: Str2Str, a dedicated conformation sampling baseline, performed worse than simply perturbing the apo structure with noise. One interpretation is that this highlights the difficulty of the apo-to-holo prediction task; another is that Str2Str was not designed specifically for apo-to-holo prediction, making it a limited test of its capabilities.

Reproducibility Details

Data

The dataset is derived from MISATO, which contains MD trajectories for PDBbind complexes.

Preprocessing:

• Clustering: Holo-ligand conformations clustered with RMSD threshold $1.0\text{\AA}$; top 10 clusters kept per complex.
• Pocket Definition: Residues within $7\text{\AA}$ of the ligand.
• Alignment: AlphaFold predicted structures (apo) aligned to MISATO holo structures using sequence alignment (Smith-Waterman) to identify pocket residues.

Algorithms

Flow Matching Framework:

• Continuous Variables (Pocket translation/rotation/torsions, Ligand positions): Modeled using Conditional Flow Matching (CFM).
  • Prior: Apo state for pocket; Normal distribution for ligand positions.
  • Target: Holo state from MD; Ground truth ligand.
  • Interpolant: Linear interpolation for Euclidean variables; Geodesic for rotations ($SO(3)$, the rotation-only subgroup of SE(3) containing all 3D rotations but not translations); Wrapped linear interpolation for torsions (Torus).
• Discrete Variables (Ligand atom/bond types): Modeled using Discrete Flow Matching based on Continuous-Time Markov Chains (CTMC).
  • Rate Matrix: Interpolates between mask token and data distribution.
• Loss Function: Weighted sum of 7 losses:
  1. Translation CFM (Eq 5)
  2. Rotation CFM (Eq 7)
  3. Torsion CFM (Eq 11)
  4. Ligand Position CFM
  5. Ligand Atom Type CTMC (Eq 14)
  6. Ligand Bond Type CTMC
  7. Interaction Loss (Eq 18): Explicitly penalizes deviations in pairwise distances between protein and ligand atoms for pairs $< 3.5\text{\AA}$.

Models

Architecture: DynamicFlow is a multiscale model with 15.9M parameters.

1. Atom-Level SE(3)-Equivariant GNN:
   • Input: Complex graph (k-NN, $k=32$) and Ligand graph (fully connected).
   • Layers: 6 EGNN blocks modified to maintain node and edge hidden states.
   • Function: Updates ligand positions and predicts ligand atom/bond types.
2. Residue-Level Transformer:
   • Input: Aggregated atom features from the GNN + Residue frames/torsions.
   • Layers: 4 Transformer blocks with Invariant Point Attention (IPA).
   • Function: Updates protein residue frames (translation/rotation) and predicts side-chain torsions.

Evaluation

Metrics:

• Vina Score: vina_minimize mode used for binding affinity.
• RMSD: Minimum RMSD between generated pocket and ground-truth holo conformations.
• Cover Ratio: % of ground-truth holo conformations covered by at least one generated sample (threshold $1.42\text{\AA}$).
• POVME 3: For pocket volume calculation.

Hardware

• Inference Benchmark: 1x Tesla V100-SXM2-32GB.
• Speed: Generates 10 ligands in ~35-36 seconds (100 NFE), significantly faster than diffusion baselines like Pocket2Mol (980s) or TargetDiff (156s).

Paper Information

Citation: Zhou, X., Xiao, Y., Lin, H., He, X., Guan, J., Wang, Y., Liu, Q., Zhou, F., Wang, L., & Ma, J. (2025). Integrating Protein Dynamics into Structure-Based Drug Design via Full-Atom Stochastic Flows. International Conference on Learning Representations (ICLR). https://doi.org/10.48550/arXiv.2503.03989

Publication: ICLR 2025

@inproceedings{zhouIntegratingProteinDynamics2025,
  title = {Integrating Protein Dynamics into Structure-Based Drug Design via Full-Atom Stochastic Flows},
  author = {Zhou, Xiangxin and Xiao, Yi and Lin, Haowei and He, Xinheng and Guan, Jiaqi and Wang, Yang and Liu, Qiang and Zhou, Feng and Wang, Liang and Ma, Jianzhu},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=uMAujpVi9m}
}

Additional Resources:

• OpenReview Page
• Code: no public repository available at time of writing

This is primarily a Method paper with a significant Resource contribution.

Method: The paper proposes a novel “Cross-modal Dialogue Foundation Model” architecture that aligns five distinct chemical modalities (2D graphs, 3D conformations, images, MS2 spectra, IR spectra) to a single LLM decoder using separate encoders and projection modules. It establishes strong baseline performance across multiple modalities compared against current generalist models.

Resource: The paper addresses the scarcity of multimodal chemical data by constructing a 7.6M instruction-tuning dataset. This dataset is largely synthesized from seed SMILES strings using approximate calculations (MMFF94, CFM-ID, Chemprop-IR) and specialist model predictions.

Bridging Experimental Data and LLMs

Existing chemical AI models generally fall into two distinct categories. Task-specific specialist models achieve high accuracy on singular objectives, such as property prediction or molecular generation, but require strict formatting and lack conversational flexibility. Conversely, early chemical large language models provide natural language interaction but are restricted to text and SMILES strings. ChemDFM-X addresses this gap by enabling large multimodal models to process the experimental characterization data (MS2 and IR spectra) and visual data routinely used in practical chemistry workflows.

Synthetic Data Scaling for Modality Alignment

The core novelty lies in the “Any-to-Text” alignment strategy via synthetic data scaling:

1. Comprehensive Modality Support: ChemDFM-X incorporates experimental characterization data (MS2 and IR spectra) alongside 2D graphs, 3D conformations, and images. The data representations are formally defined mathematically rather than as raw pixels:

   • Molecular Graph: An undirected graph $G = (\textbf{V}, \textbf{E})$ with atom set $\textbf{V}$ and bond set $\textbf{E}$.
   • Molecular Conformation: An undirected graph $G = (\textbf{V}’, \textbf{E})$ storing spatial coordinates: $\textbf{v}_i = (x_i, y_i, z_i, a_i)$.
   • MS2 Spectrum: Treated as a point sequence of discrete mass-to-charge ratios and intensities, tokenized via a discrete codebook: $\textbf{M} = ((r_1, I_1), (r_2, I_2), \dots, (r_n, I_n))$.
   • IR Spectrum: Treated as a dense sequence of continuous wave lengths and absorption intensities, directly reshaped for feature extraction: $\textbf{R} = ((w_1, t_1), (w_2, t_2), \dots, (w_l, t_l))$.

   The authors trained new Sequence Transformer encoders from scratch for the MS2 and IR modalities since suitable pre-trained models did not exist.

2. Synthetic Data Generation Pipeline: The authors generated a 7.6M sample dataset by starting with 1.3M seed SMILES and using “approximate calculations” to generate missing modalities:

   • 3D conformations via MMFF94 force field optimization
   • MS2 spectra via CFM-ID 4.0 (Competitive Fragmentation Modeling)
   • IR spectra via Chemprop-IR (Message Passing Neural Network)

3. Cross-Modal Synergy: The model demonstrates that training on reaction images improves recognition performance by leveraging semantic chemical knowledge (reaction rules) to correct visual recognition errors, an emergent capability from multimodal training.

Multimodal Benchmarking with ChemLLMBench

The model was evaluated using a customized version of ChemLLMBench and MoleculeNet across three modality categories:

1. Structural Modalities (2D Graphs & 3D Conformations):

   • Molecule recognition and captioning
   • Property prediction (MoleculeNet: BACE, BBBP, ClinTox, HIV, Tox21)
   • Compared against specialist models (Mole-BERT, Uni-Mol, MolXPT, MolCA) and generalist models (3D-MoLM, ChemDFM, ChemLLM)

2. Visual Modalities (Images):

   • Single molecule image recognition
   • Reaction image recognition
   • Compared against GPT-4O, Gemini 1.5 Pro, Qwen-VL, LLaVA, and specialist models MolNextr and MolScribe

3. Characterization Modalities (MS2 & IR Spectra):

   • Spectral analysis tasks (identifying molecules from spectra)
   • Contextualized spectral interpretation (combining spectra with reaction context)
   • Novel evaluation requiring integration of spectroscopic data with reaction knowledge

Cross-Modal Synergy and Generalist Performance

Key Findings:

1. Leading Generalist Performance: ChemDFM-X establishes a new benchmark among existing generalist models (such as 3D-MOLM and ChemLLM), achieving performance metrics that match dedicated specialist models across several multimodal tasks.

2. Failure of General LMMs: General vision models (GPT-4O, Gemini 1.5 Pro, Qwen-VL, LLaVA, InternLM-XComposer2, DocOwl) failed significantly on chemical image recognition tasks (0% accuracy for most models on molecule and reaction recognition, Table 9), demonstrating that chemical domain knowledge cannot be assumed from general pre-training.

3. Cross-Modal Error Correction: In reaction image recognition, ChemDFM-X achieved higher accuracy (53.0%) than on single molecules (46.0%) (Table 9). The authors conclude the model uses its internal knowledge of chemical reaction rules to correct recognition errors in the visual modality, an emergent capability from multimodal training.

4. Reliance on Reaction Context for Spectra: In zero-shot scenarios, ChemDFM-X essentially fails at pure spectral recognition (achieving 0% and 1% top-1 accuracy on MS2 and IR spectra alone, Table 11). However, when SMILES-based reaction context is included, performance rises to 45% (MS2) and 64% (IR) on the reaction prediction task, and 29% (MS2) and 60% (IR) on retrosynthesis (Table 11). This indicates the model uses spectral data as a soft prior to constrain textual deductions. Furthermore, the paper compares ChemDFM-X’s spectral identification performance exclusively against text-only LLMs that cannot process spectra, omitting comparisons against established specialist tools.

5. Surrogate Distillation Trade-offs: Because the spectral training data relies entirely on outputs from CFM-ID 4.0 and Chemprop-IR, ChemDFM-X effectively distills these surrogate models. Any inherent predictive biases or inaccuracies from these underlying tools are permanently embedded in the new ChemDFM-X encoders.

Main Conclusion: The “separate encoders + unified decoder” architecture with synthetic data generation enables effective multimodal chemical understanding, bridging the gap between specialist and generalist AI systems for chemistry.

Reproducibility Details

Data

The authors constructed a 7.6M sample instruction-tuning dataset derived from 1.3M seed SMILES (sourced from PubChem and USPTO). Note: The final 7.6M multimodal tuning dataset itself isn’t publicly available.

Generation Pipeline:

Data Augmentation:

• Molecule images augmented with “handwritten” style using the ChemPix pipeline
• Multiple rendering styles (RDKit default, Indigo clean)
• Spectra generated at multiple energy levels (10eV, 20eV, 40eV for MS2)

Algorithms

Architecture: “Separate Encoders + Unified Decoder”

Code Availability: The authors have only released inference logic. The cross-modal projection training and synthetic data-generation scripts are closed.

Modality Alignment:

• Each modality has a dedicated encoder (frozen pre-trained models where available)
• For graph, conformation, MS2, and IR modalities: 2-layer MLP projector (Linear, GELU, Linear) maps encoder features to LLM input space
• For images: H-Reducer module compresses image tokens by factor of $n=8$ to handle high-resolution chemical images, then projects to LLM input space
• All projected features are concatenated and fed to the unified LLM decoder

Models

Base LLM:

• ChemDFM (13B): LLaMA-based model pre-trained on chemical text and SMILES

Modality Encoders:

Design Rationale: MS2 and IR encoders trained from scratch as Sequence Transformers treating spectral peaks as token sequences, since no suitable pre-trained models exist for chemical spectra.

Evaluation

Metrics:

• Accuracy (Acc) for recognition tasks
• BLEU-2/4 and METEOR for captioning tasks
• AUC-ROC for property prediction (classification)

Code Availability: The adapted code for evaluating on ChemLLMBench and their custom spectral recognition tasks is closed-source.

Benchmarks:

• ChemLLMBench: Adapted for multimodal inputs across molecule captioning, property prediction, and reaction understanding
• MoleculeNet: Standard molecular property prediction tasks (BACE, BBBP, ClinTox, HIV, Tox21)
• USPTO: Reaction prediction and retrosynthesis tasks
• Custom Spectral Tasks: Novel evaluations requiring spectral interpretation

Hardware

Note: The type and quantity of GPUs used, along with the total training wall-time, were not published.

Training Configuration:

• Total Batch Size: 256
• Epochs: 3
• Optimizer: AdamW

Modality-Specific Learning Rates (Peak):

Note: Different learning rates reflect the varying degrees of domain adaptation required. Images (general CLIP) need more adaptation than graphs (chemical Mole-BERT).

Artifacts

Paper Information

Citation: Zhao, Z., Chen, B., Li, J., Chen, L., Wen, L., Wang, P., Zhu, Z., Zhang, D., Wan, Z., Li, Y., Dai, Z., Chen, X., & Yu, K. (2024). ChemDFM-X: Towards Large Multimodal Model for Chemistry. Science China Information Sciences, 67(12), 220109. https://doi.org/10.1007/s11432-024-4243-0

Publication: Science China Information Sciences, December 2024

Additional Resources:

• arXiv Version
• Code Repository

@article{zhaoChemDFMXLargeMultimodal2024,
  title = {{{ChemDFM-X}}: {{Towards Large Multimodal Model}} for {{Chemistry}}},
  author = {Zhao, Zihan and Chen, Bo and Li, Jingpiao and Chen, Lu and Wen, Liyang and Wang, Pengyu and Zhu, Zichen and Zhang, Danyang and Wan, Ziping and Li, Yansi and Dai, Zhongyang and Chen, Xin and Yu, Kai},
  year = {2024},
  month = dec,
  journal = {Science China Information Sciences},
  volume = {67},
  number = {12},
  pages = {220109},
  doi = {10.1007/s11432-024-4243-0},
  archiveprefix = {arXiv},
  eprint = {2409.13194},
  primaryclass = {cs.LG}
}

Citation: Chang, Q., Chen, M., Pi, C., et al. (2024). RFL: Simplifying Chemical Structure Recognition with Ring-Free Language. arXiv preprint arXiv:2412.07594. https://doi.org/10.48550/arXiv.2412.07594

Publication: arXiv preprint (December 2024)

Additional Resources:

• Official Code Repository

Methodological Contribution

This is a Methodological paper ($\Psi_{\text{Method}}$). It introduces a novel representation system (Ring-Free Language) and a specialized neural architecture (Molecular Skeleton Decoder) designed to solve specific limitations in converting 2D images to 1D chemical strings. The paper validates this method through direct comparison with state-of-the-art baselines and ablation studies.

Motivation: Limitations of 1D Serialization

Current Optical Chemical Structure Recognition (OCSR) methods typically rely on “unstructured modeling,” where 2D molecular graphs are flattened into 1D strings like SMILES or SSML. While simple, these linear formats struggle to explicitly capture complex spatial relationships, particularly in molecules with multiple rings and branches. End-to-end models often fail to “understand” the graph structure when forced to predict these implicit 1D sequences, leading to error accumulation in complex scenarios.

Innovation: Ring-Free Language (RFL) and Molecular Skeleton Decoder (MSD)

The authors propose two primary contributions to decouple spatial complexity:

1. Ring-Free Language (RFL): A divide-and-conquer representation that splits a molecular graph $G$ into three explicit components: a molecular skeleton $\mathcal{S}$, individual ring structures $\mathcal{R}$, and branch information $\mathcal{F}$. This allows rings to be collapsed into “SuperAtoms” or “SuperBonds” during initial parsing.
2. Molecular Skeleton Decoder (MSD): A hierarchical architecture that progressively predicts the skeleton first, then the individual rings (using SuperAtom features as conditions), and finally classifies the branch connections.

Methodology and Experiments

The method was evaluated on both handwritten and printed chemical structures against two baselines: DenseWAP (Zhang et al. 2018) and RCGD (Hu et al. 2023).

• Datasets:
  • EDU-CHEMC: ~49k handwritten samples (challenging, diverse styles)
  • Mini-CASIA-CSDB: ~89k printed samples (from ChEMBL)
  • Synthetic Complexity Dataset: A custom split of ChEMBL data grouped by structural complexity (atoms + bonds + rings) to test generalization
• Ablation Studies: Tested the necessity of the MSD architecture vs. standard decoders and the impact of the [conn] token for filtering branch candidates

Outcomes and Conclusions

• SOTA Performance: The proposed method (MSD-RCGD) outperformed the previous state-of-the-art (RCGD) on both handwritten (95.38% Exact Match) and printed (95.58% Exact Match) datasets.
• Universal Improvement: Applying MSD/RFL to an older baseline (DenseWAP) improved its accuracy significantly (e.g., from 87.6% to 81.9% on complex sets), proving the method is model-agnostic.
• Complexity Handling: The method showed robust generalization on unseen high-complexity molecules, where standard 1D baselines failed completely (0% accuracy for standard DenseWAP vs. ~30% for MSD version on the hardest tier).

Reproducibility Details

Data

The authors utilized one handwritten and one printed dataset, plus a synthetic set for stress-testing complexity.

Algorithms

RFL Splitting (Encoding):

1. Detect Rings: Use DFS to find all non-nested rings $\mathcal{R}$.
2. Determine Adjacency ($\gamma$): Calculate shared edges between rings.
3. Merge:
   • If $\gamma(r_i) = 0$ (isolated), merge ring into a SuperAtom node.
   • If $\gamma(r_i) > 0$ (adjacent), merge ring into a SuperBond edge.
4. Update: Record connection info in $\mathcal{F}$ and remove ring details from the main graph to form Skeleton $\mathcal{S}$.

MSD Decoding:

• Hierarchical Prediction: The model predicts the Skeleton $\mathcal{S}$ first.
• Contextual Ring Prediction: When a SuperAtom/Bond token is predicted, its hidden state $f^s$ is stored. After the skeleton is finished, $f^s$ is used as a condition to autoregressively decode the specific ring structure.
• Token [conn]: A special token separates connected ring bonds from unconnected ones to sparsify the branch classification task.

Models

The architecture follows a standard Image-to-Sequence pattern but with a forked decoder.

• Encoder: DenseNet (Growth rate=24, Depth=32 per block)
• Decoder (MSD):
  • Core: GRU with Attention (Hidden dim=256, Embedding dim=256, Dropout=0.15)
  • Skeleton Module: Autoregressively predicts sequence tokens. Uses Maxout activation.
  • Branch Module: A binary classifier (MLP) taking concatenated features of skeleton bonds $f_{bs}$ and ring bonds $f_{br}$ to predict connectivity matrix $\mathcal{F}$.
• Loss Function: $\mathcal{L} = \lambda_1 \mathcal{L}_{ce} + \lambda_2 \mathcal{L}_{cls}$ (where $\lambda=1$)

Evaluation

Metrics focus on exact image reconstruction and structural validity.

Hardware

• Compute: 4 x NVIDIA Tesla V100 (32GB RAM)
• Training Configuration:
  • Batch size: 8 (Handwritten), 32 (Printed)
  • Epochs: 50
  • Optimizer: Adam ($lr=2\times10^{-4}$, decayed by 0.5 via MultiStepLR)

Citation

@misc{changRFLSimplifyingChemical2025,
  title = {RFL: Simplifying Chemical Structure Recognition with Ring-Free Language},
  shorttitle = {RFL},
  author = {Chang, Qikai and Chen, Mingjun and Pi, Changpeng and Hu, Pengfei and Zhang, Zhenrong and Ma, Jiefeng and Du, Jun and Yin, Baocai and Hu, Jinshui},
  year = {2024},
  month = dec,
  number = {arXiv:2412.07594},
  eprint = {2412.07594},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2412.07594},
  archiveprefix = {arXiv}
}

Citation: Fan, S., Xie, Y., Cai, B., Xie, A., Liu, G., Qiao, M., Xing, J., & Nie, Z. (2025). OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discovery. arXiv preprint arXiv:2501.15415. https://doi.org/10.48550/arXiv.2501.15415

Publication: arXiv 2025

Additional Resources:

• Code and Dataset (GitHub)

Multi-Level Chemical Understanding (Method and Resource)

This is primarily a Methodological Paper ($\Psi_{\text{Method}}$) with a significant Resource ($\Psi_{\text{Resource}}$) contribution.

• Methodological: It proposes two novel architectures, DoubleCheck (an enhanced recognition model) and Mol-VL (an end-to-end vision-language model), to solve the newly formulated OCSU task.
• Resource: It constructs and releases Vis-CheBI20, the first large-scale dataset specifically designed for optical chemical structure understanding, containing 29.7K images and 117.7K image-text pairs.

The Motivation for OCSU Beyond Basic Graph Recognition

Existing methods for processing molecular images focus narrowly on Optical Chemical Structure Recognition (OCSR), which translates an image solely into a machine-readable graph or SMILES string. However, SMILES strings are not chemist-friendly and lack high-level semantic context.

• Gap: There is a lack of systems that can translate chemical diagrams into human-readable descriptions (e.g., functional groups, IUPAC names) alongside the graph structure.
• Goal: To enable Optical Chemical Structure Understanding (OCSU), bridging the gap between visual representations and both machine/chemist-readable descriptions to support drug discovery and property prediction.

Key Innovations: DoubleCheck, Mol-VL, and the Vis-CheBI20 Dataset

The paper introduces the OCSU task, enabling multi-level understanding (motif, molecule, and abstract levels). To solve this, it introduces two distinct paradigms:

1. DoubleCheck (OCSR-based): An enhancement to standard OCSR models (like MolScribe) that performs a “second look” at locally ambiguous atoms. It uses attentive feature enhancement to fuse global molecular features with local features from ambiguous regions.
2. Mol-VL (OCSR-free): An end-to-end Vision-Language Model (VLM) based on Qwen2-VL. It uses multi-task learning to directly generate text descriptions from molecular images without an intermediate SMILES step.
3. Vis-CheBI20 Dataset: A new benchmark specifically constructed for OCSU, deriving captions and functional group data from ChEBI-20 and PubChem.

Methodology and Experimental Evaluation

The authors evaluated both paradigms on Vis-CheBI20 and existing benchmarks (USPTO, ACS) across four subtasks:

1. Functional Group Caption: Retrieval/F1 score evaluation.
2. Molecule Description: Natural language generation metrics (BLEU, ROUGE, METEOR).
3. IUPAC Naming: Text generation metrics (BLEU, ROUGE).
4. SMILES Naming (OCSR): Exact matching accuracy ($Acc_s$).

Baselines:

• Task-Specific: MolScribe, MolVec, OSRA.
• LLM/VLM: Qwen2-VL, BioT5+, Mol-Instructions.
• Ablation: DoubleCheck vs. MolScribe backbone to test the “feature enhancement” mechanism.

Results and Conclusions: Paradigm Trade-Offs

• DoubleCheck Superiority: DoubleCheck outperformed the MolScribe baseline on OCSR tasks, achieving 92.85% accuracy on Vis-CheBI20 (vs. 92.57%) and showing significant gains on chiral molecules in the ACS dataset (+3.12%).
• Paradigm Trade-offs:
  • Mol-VL (OCSR-free) excelled at semantic tasks like Functional Group Captioning, outperforming the strongest baseline by 7.7% in F1 score. It benefits from end-to-end learning of structural context.
  • DoubleCheck (OCSR-based) performed better on IUPAC naming recall and exact SMILES recovery, as explicit graph reconstruction is more precise for rigid nomenclature than VLM generation.
• Conclusion: Enhancing submodules improves OCSR-based paradigms, while end-to-end VLMs offer stronger semantic understanding but struggle with exact syntax generation (SMILES/IUPAC).

Reproducibility Details

Data

Vis-CheBI20 Dataset

• Source: Derived from ChEBI-20 and PubChem.
• Size: 29,700 molecular diagrams, 117,700 image-text pairs.
• Generation: Images generated from SMILES using RDKit to simulate real-world journal/patent styles.
• Splits:
  • Training: ~26,000 images (varies slightly by task).
  • Test: ~3,300 images.

Algorithms

DoubleCheck (Attentive Feature Enhancement)

1. Ambiguity Detection: Uses atom prediction confidence to identify “ambiguous atoms”.
2. Masking: Applies a 2D Gaussian mask to the image centered on the ambiguous atom.
3. Local Encoding: A Swin-B encoder ($\Phi_l$) encodes the masked image region.
4. Fusion: Aligns local features ($\mathcal{F}_l$) with global features ($\mathcal{F}_g$) using a 2-layer MLP and fuses them via weighted summation.

$$ \begin{aligned} \mathcal{F}_e = \mathcal{F}_g + \text{MLP}(\mathcal{F}_g \oplus \hat{\mathcal{F}}_l) \cdot \hat{\mathcal{F}}_l \end{aligned} $$

5. Two-Stage Training:
   • Stage 1: Train atom/bond predictors (30 epochs).
   • Stage 2: Train alignment/fusion modules with random Gaussian mask noise (10 epochs).

Mol-VL (Multi-Task VLM)

• Prompting: System prompt: “You are working as an excellent assistant in chemistry…”
• Tokens: Uses <image> and </image> special tokens.
• Auxiliary Task: Functional group recognition (identifying highlighted groups) added to training to improve context learning.

Models

• DoubleCheck:
  • Backbone: MolScribe architecture.
  • Encoders: Swin-B for both global and local atom encoding.
• Mol-VL:
  • Base Model: Qwen2-VL (2B and 7B versions).
  • Vision Encoder: ViT with naive dynamic resolution and M-ROPE.

Evaluation

Key Metrics:

• SMILES: Exact Match Accuracy ($Acc_s$), Chiral Accuracy ($Acc_c$).
• Functional Groups: F1 Score (Information Retrieval task).
• Text Generation: BLEU-2/4, METEOR, ROUGE-L.

Selected Results:

Hardware

• DoubleCheck: Trained on 4 NVIDIA A100 GPUs for 4 days.
  • Max LR: 4e-4.
• Mol-VL: Trained on 4 NVIDIA A100 GPUs for 10 days.
  • Max LR: 1e-5, 50 epochs.

Citation

@misc{fanOCSUOpticalChemical2025,
  title = {OCSU: Optical Chemical Structure Understanding for Molecule-centric Scientific Discovery},
  shorttitle = {OCSU},
  author = {Fan, Siqi and Xie, Yuguang and Cai, Bowen and Xie, Ailin and Liu, Gaochao and Qiao, Mu and Xing, Jie and Nie, Zaiqing},
  year = {2025},
  month = jan,
  number = {arXiv:2501.15415},
  eprint = {2501.15415},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2501.15415},
  archiveprefix = {arXiv}
}

Citation: Zhang, W., Wang, X., Feng, B., & Liu, W. (2025). MolSight: Optical Chemical Structure Recognition with SMILES Pretraining, Multi-Granularity Learning and Reinforcement Learning. arXiv preprint arXiv:2511.17300. https://doi.org/10.48550/arXiv.2511.17300

Publication: arXiv 2025

Additional Resources:

• Official Repository

Contribution: A Framework for Optical Chemical Structure Recognition

This is primarily a Method paper. It proposes a novel three-stage training framework (Pretraining → Fine-tuning → RL Post-training) to improve Optical Chemical Structure Recognition (OCSR). Specifically, it introduces the use of Group Relative Policy Optimization (GRPO) to solve non-differentiable chemical validity issues.

It also has a Resource component, as the authors construct and release Stereo-200k, a dataset specifically designed to train models on challenging stereoisomeric molecules.

Motivation: Resolving Stereochemical Cues

Existing OCSR systems struggle to accurately recognize stereochemical information (e.g., chirality, geometric isomerism) because the visual cues distinguishing stereoisomers (such as wedge and dash bonds) are subtle. Current methods often fail to capture the geometric relationships required to distinguish molecules with identical connectivity but different spatial arrangements. Accurate recognition is critical for downstream tasks like drug discovery where stereochemistry determines pharmacological effects.

Core Innovations: GRPO and Multi-Granularity Learning

MolSight introduces three key technical innovations:

1. Reinforcement Learning for OCSR: It is the first OCSR system to incorporate RL (specifically GRPO) to directly optimize for chemical semantic correctness.
2. Multi-Granularity Learning: It employs auxiliary heads for chemical bond classification and atom localization. Unlike previous approaches that optimize these jointly, MolSight decouples the coordinate head to prevent interference with SMILES generation.
3. SMILES-M Notation: A lightweight extension to SMILES to handle Markush structures (common in patents) without significant sequence length increase.

Experimental Methodology

The authors evaluated MolSight using a rigorous mix of real and synthetic benchmarks:

• Baselines: Compared against rule-based (OSRA, MolVec) and deep learning methods (MolScribe, MolGrapher, DECIMER).
• Benchmarks: Evaluated on real-world datasets (USPTO, Maybridge UoB, CLEF-2012, JPO) and synthetic datasets (Staker, ChemDraw, Indigo, Stereo-2K).
• Ablation Studies: Tested the impact of the bond head, coordinate head, and RL stages separately.
• Transfer Learning: Assessed the quality of learned representations by using the frozen encoder for molecular property prediction on MoleculeNet.

Results and Conclusions

• SOTA Performance: MolSight achieved 85.1% stereochemical accuracy on the USPTO dataset, significantly outperforming the previous SOTA (MolScribe) which achieved 69.0%.
• RL Effectiveness: Reinforcement learning post-training specifically improved performance on stereoisomers, raising Tanimoto similarity and exact match rates on the Stereo-2k test set.
• Robustness: The model maintained high performance even on perturbed (rotated/sheared) and low-resolution images, outperforming rule-based methods significantly in these scenarios.

Reproducibility Details

Data

The training pipeline uses three distinct data sources:

1. Pre-training: MolParser-7M. Contains diverse images but requires the SMILES-M extension to handle Markush structures.
2. Fine-tuning: PubChem-1M and USPTO-680K. Used for multi-granularity learning with bond and coordinate labels.
3. RL Post-training: Stereo-200k. A self-collected dataset from the first 2M compounds in PubChem, filtered for chirality (’@’) and cis-trans isomerism (’/’, ‘\’). It uses 5 different RDKit drawing styles to ensure robustness.

Algorithms

• Reinforcement Learning: Uses GRPO (Group Relative Policy Optimization).
  • Reward Function: A linear combination of Tanimoto similarity and Stereochemical exact match. $$ R = w_t \cdot \text{Tanimoto} + w_s \cdot \text{ExactMatch} $$ where $w_t=0.4$ and $w_s=0.6$.
  • Sampling: Samples 4 completions per image with temperature 1.0 during RL training.
• Auxiliary Tasks:
  • Bond Classification: Concatenates hidden states of two atom queries to predict bond type via MLP.
  • Atom Localization: Treated as a classification task (SimCC) but optimized using Maximum Likelihood Estimation (MLE) to account for uncertainty.

Models

• Architecture: Encoder-Decoder Transformer.
  • Encoder: EfficientViT-L1 (~53M params), chosen for linear attention efficiency.
  • Decoder: 6-layer Transformer with RoPE, SwiGLU, and RMSNorm. Randomly initialized (no LLM weights) due to vocabulary mismatch.
  • Coordinate Head: Separated from the main decoder. It adds 2 extra Transformer layers to process atom queries before prediction to improve accuracy.
• Parameter Tuning:
  • Stage 3 (RL) uses LoRA (Rank=8, Alpha=16) to optimize the decoder.

Evaluation

• Metrics:
  • Exact Match: String identity for standard SMILES.
  • Tanimoto Coefficient: Fingerprint similarity for chemical semantics.
  • OKS (Object Keypoint Similarity): Used specifically for evaluating atom localization accuracy.
• Perturbation: Robustness tested with random rotations [-5°, 5°] and xy-shearing [-0.1, 0.1].

Hardware

• Compute: Training and inference performed on a single node.
• Processors: Intel Xeon Silver 4210R CPU.
• Accelerators: 4x NVIDIA GeForce RTX 3090/4090 GPUs.
• Hyperparameters:
  • Stage 1: Batch size 512, LR $4 \x10^{-4}$.
  • Stage 2: Batch size 256, Bond head LR $4 \x10^{-4}$, Coord head LR $4 \x10^{-5}$.
  • Stage 3 (RL): Batch size 64, Base LR $1 \x10^{-4}$.

Citation

@misc{zhang2025molsight,
      title={MolSight: Optical Chemical Structure Recognition with SMILES Pretraining, Multi-Granularity Learning and Reinforcement Learning},
      author={Wenrui Zhang and Xinggang Wang and Bin Feng and Wenyu Liu},
      year={2025},
      eprint={2511.17300},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2511.17300},
}

Citation: Qian, Y., Guo, J., Tu, Z., Li, Z., Coley, C. W., & Barzilay, R. (2023). MolScribe: Robust Molecular Structure Recognition with Image-To-Graph Generation. Journal of Chemical Information and Modeling, 63(7), 1925-1934. https://doi.org/10.1021/acs.jcim.2c01480

Publication: Journal of Chemical Information and Modeling 2023

Additional Resources:

• Hugging Face Space

Contribution: Generative Image-to-Graph Modelling

This is a Methodological Paper ($\Psi_{\text{Method}}$) with a secondary contribution to Resources ($\Psi_{\text{Resource}}$).

It proposes a novel architecture (image-to-graph generation) to solve the Optical Chemical Structure Recognition (OCSR) task, validating it through extensive ablation studies and comparisons against strong baselines like MolVec and DECIMER. It also contributes a new benchmark dataset of annotated images from ACS journals.

Motivation: Limitations in Existing OCSR Pipelines

Translating molecular images into machine-readable graphs (OCSR) is challenging due to the high variance in drawing styles, stereochemistry conventions, and abbreviated structures found in literature.

Existing solutions face structural bottlenecks:

• Rule-based systems (e.g., OSRA) rely on rigid heuristics that fail on diverse styles.
• Image-to-SMILES neural models treat the problem as captioning. They struggle with geometric reasoning (which is strictly required for chirality) and struggle to incorporate chemical constraints or verify correctness because they omit explicit atom locations.

Core Innovation: Joint Graph and Coordinate Prediction

MolScribe introduces an Image-to-Graph generation paradigm that combines the flexibility of neural networks with the precision of symbolic constraints. It frames the task probabilistically as:

$$ P(G | I) = P(A | I) P(B | A, I) $$

Where the model predicts a sequence of atoms $A$ given an image $I$, followed by the bonds $B$ given both the atoms and the image.

1. Explicit Graph Prediction: It predicts a sequence of atoms (with 2D coordinates) and then predicts bonds between them.
2. Symbolic Constraints: It uses the predicted graph structure and coordinates to strictly determine chirality and cis/trans isomerism.
3. Abbreviation Expansion: It employs a greedy algorithm to parse and expand “superatoms” (e.g., “CO2Et”) into their full atomic structure.
4. Dynamic Augmentation: It introduces a data augmentation strategy that randomly substitutes functional groups with abbreviations and adds R-groups during training to improve generalization.

Methodology: Autoregressive Atoms and Pairwise Bonds

The authors evaluate MolScribe on synthetic and real-world datasets, focusing on Exact Match Accuracy of the canonical SMILES string. The model generates atom sequences autoregressively:

$$ P(A | I) = \prod_{i=1}^n P(a_i | A_{<i}, I) $$

To handle continuous spatial locations, atom coordinates map to discrete bins (e.g., $\hat{x}_i = \lfloor \frac{x_i}{W} \times n_{\text{bins}} \rfloor$), and decode alongside element labels. Bonds act on a pairwise classifier over the hidden states of every atom pair:

$$ P(B | A, I) = \prod_{i=1}^n \prod_{j=1}^n P(b_{i,j} | A, I) $$

• Baselines: Compared against rule-based (MolVec, OSRA) and neural (Img2Mol, DECIMER, SwinOCSR) systems.
• Benchmarks:
  • Synthetic: Indigo (in-domain) and ChemDraw (out-of-domain).
  • Realistic: Five public benchmarks (CLEF, JPO, UOB, USPTO, Staker).
  • New Dataset: 331 images from ACS Publications (journal articles).
• Ablations: Tested performance without data augmentation, with continuous vs. discrete coordinates, and without non-atom tokens.
• Human Eval: Measured the time reduction for chemists using MolScribe to digitize molecules vs. drawing from scratch.

Results: Robust Exact Match Accuracy

• State-of-the-Art Performance: MolScribe achieved 76-93% accuracy across diverse benchmarks, significantly outperforming baselines (e.g., on the difficult Staker dataset, MolScribe achieved 71.9% compared to the next best 46.5%).
• Chirality Verification: Explicit geometric reasoning allowed MolScribe to predict chiral molecules significantly better than image-to-SMILES baselines. When chirality is ignored, the performance gap narrows (e.g., on Indigo, baseline accuracy rises from 94.1% to 96.3%), isolating MolScribe’s primary advantage to geometric reasoning for stereochemistry.
• Hand-Drawn Generalization: The model achieved 11.2% exact match accuracy on the DECIMER-HDM dataset, despite lacking hand-drawn images in the training set, with many errors bounded to a few atomic mismatches.
• Robustness: The model maintained high performance on perturbed images (rotation/shear), whereas rule-based systems degraded severely.
• Usability: The atom-level alignment allows for confidence visualization, and human evaluation showed it reduced digitization time from 137s to 20s per molecule.

Reproducibility Details

Data

The model was trained on a mix of synthetic and patent data with extensive dynamic augmentation:

Molecule Augmentation:

• Functional Groups: Randomly substituted using 53 common substitution rules (e.g., replacing substructures with “Et” or “Ph”).
• R-Groups: Randomly added using vocabulary: [R, R1...R12, Ra, Rb, Rc, Rd, X, Y, Z, A, Ar].
• Styles: Random variation of aromaticity (circle vs. bonds) and explicit hydrogens.

Image Augmentation:

• Rendering: Randomized font (Arial, Times, Courier, Helvetica), line width, and label modes during synthetic generation.
• Perturbations: Applied rotation ($\pm 90^{\circ}$), cropping ($1\%$), padding ($40\%$), downscaling, blurring, and Salt-and-Pepper/Gaussian noise.

Preprocessing: Input images are resized to $384 \x384$.

Algorithms

• Atom Prediction (Pix2Seq-style):
  • The model generates a sequence of tokens: $S^A = [l_1, \hat{x}_1, \hat{y}_1, \dots, l_n, \hat{x}_n, \hat{y}_n]$.
  • Discretization: Coordinates are binned into integer tokens ($n_{bins} = 64$).
  • Tokenizer: Atom-wise tokenizer splits SMILES into atoms; non-atom tokens (parentheses, digits) are kept to help structure learning.
• Bond Prediction:
  • Format: Pairwise classification for every pair of predicted atoms.
  • Symmetry: For symmetric bonds (single/double), the probability is averaged as: $$ \hat{P}(b_{i,j} = t) = \frac{1}{2} \big( P(b_{i,j} = t) + P(b_{j,i} = t) \big) $$ For wedges, directional logic strictly applies instead.
• Abbreviation Expansion (Algorithm 1):
  • A greedy algorithm connects atoms within an expanded abbreviation (e.g., “COOH”) until valences are full, avoiding the need for a fixed dictionary.
  • Carbon Chains: Splits condensed chains like $C_aX_b$ into explicit sequences ($CX_q…CX_{q+r}$).
  • Nested Formulas: Recursively parses nested structures like $N(CH_3)_2$ by treating them as superatoms attached to the current backbone.
  • Valence Handling: Iterates through common valences first to resolve ambiguities.

Models

The architecture is an encoder-decoder with a classification head:

• Encoder: Swin Transformer (Swin-B), pre-trained on ImageNet-22K (88M params).
• Decoder: 6-layer Transformer, 8 heads, hidden dimension 256.
• Bond Predictor: 2-layer MLP (Feedforward) with ReLU, taking concatenated atom hidden states as input.
• Training: Teacher forcing, Cross-Entropy Loss, Batch size 128, 30 epochs.

Evaluation

Metric: Exact Match of Canonical SMILES.

• Stereochemistry: Must match tetrahedral chirality; cis-trans ignored.
• R-groups: Replaced with wildcards * or [d*] for evaluation.

Hardware

• Compute: Training performed on Linux server with 96 CPUs and 500GB RAM.
• GPUs: 4x NVIDIA A100 GPUs.
• Training Time: Unspecified; comparative models on large datasets took “more than one day”.
• Inference: Requires autoregressive decoding for atoms, followed by a single forward pass for bonds.

Citation

@article{qianMolScribeRobustMolecular2023,
  title = {{{MolScribe}}: {{Robust Molecular Structure Recognition}} with {{Image-To-Graph Generation}}},
  shorttitle = {{{MolScribe}}},
  author = {Qian, Yujie and Guo, Jiang and Tu, Zhengkai and Li, Zhening and Coley, Connor W. and Barzilay, Regina},
  year = 2023,
  month = apr,
  journal = {Journal of Chemical Information and Modeling},
  volume = {63},
  number = {7},
  pages = {1925--1934},
  doi = {10.1021/acs.jcim.2c01480},
  url = {https://pubs.acs.org/doi/10.1021/acs.jcim.2c01480}
}

Citation: Chun, S., Kim, J., Jo, A., Jo, Y., & Oh, S. (2025). MolMole: Molecule Mining from Scientific Literature. arXiv preprint arXiv:2505.03777. https://doi.org/10.48550/arXiv.2505.03777

Publication: arXiv 2025

Additional Resources:

• Project Page

MolMole’s Dual Contribution: Unified OCSR Method and Page-Level Benchmarks

This is primarily a Method paper, with a strong Resource contribution.

It functions as a Method paper because it introduces “MolMole,” a unified deep learning framework that integrates molecule detection, reaction diagram parsing, and optical chemical structure recognition (OCSR) into a single pipeline. It validates this method through extensive comparisons against state-of-the-art baselines like DECIMER and OpenChemIE.

It also serves as a Resource paper because the authors construct and release a novel page-level benchmark dataset of 550 annotated pages (patents and articles) to address the lack of standardized evaluation metrics for full-page chemical extraction.

Addressing the Limitations of Fragmented Processing

The rapid accumulation of chemical literature has trapped valuable molecular and reaction data in unstructured formats like images and PDFs. Extracting this manually is time-consuming, while existing AI frameworks have significant limitations:

• DECIMER: Lacks the ability to process reaction diagrams entirely.
• OpenChemIE: Relies on external layout parser models to crop elements before processing. This dependence often leads to detection failures in documents with complex layouts.
• Generative Hallucination: Existing generative OCSR models (like MolScribe) are prone to “hallucinating” structures or failing on complex notations like polymers.

A Unified Vision Pipeline for Robust Detection

MolMole introduces several architectural and workflow innovations:

• Direct Page-Level Processing: Unlike OpenChemIE, MolMole processes full document pages directly without requiring an external layout parser, which improves robustness on complex layouts like two-column patents.
• Unified Vision Pipeline: It integrates three specialized vision models into one workflow:
  • ViDetect: A DINO-based object detector for identifying molecular regions.
  • ViReact: An RxnScribe-based model adapted for full-page reaction parsing.
  • ViMore: A detection-based OCSR model that explicitly predicts atoms and bonds.
• Hallucination Mitigation: By using a detection-based approach (ViMore), the model avoids hallucinating chemical structures and provides confidence scores.
• Advanced Notation Support: The system explicitly handles “wavy bonds” (variable attachments in patents) and polymer bracket notations, which confuse standard SMILES-based models.

Page-Level Benchmark Evaluation and Unified Metrics

The authors evaluated the framework on both a newly curated benchmark and existing public datasets:

• New Benchmark Creation: They curated 550 pages (300 patents, 250 articles) fully annotated with bounding boxes, reaction roles (reactant, product, condition), and MOLfiles.
• Baselines: MolMole was compared against DECIMER 2.0, OpenChemIE, and ReactionDataExtractor 2.0.
• OCSR Benchmarking: ViMore was evaluated against DECIMER, MolScribe, and MolGrapher on four public datasets: USPTO, UOB, CLEF, and JPO.
• Metric Proposal: They introduced a combined “End-to-End” metric that modifies standard object detection Precision/Recall to strictly require correct SMILES conversion for a “True Positive”.

$$ \text{True Positive (End-to-End)} = ( \text{IoU} \geq 0.5 ) \land ( \text{SMILES}_{\text{gt}} == \text{SMILES}_{\text{pred}} ) $$

State-of-the-Art Extraction Outcomes

• SOTA Page-Level Performance: On the new benchmark, MolMole achieved F1 scores of 89.1% (Patents) and 86.8% (Articles) for the combined detection-to-conversion task, significantly outperforming DECIMER and OpenChemIE.
• Reaction Parsing: ViReact achieved an F1 score of 98.0% (soft match) on patents, compared to 82.2% for the next best model (RxnScribe).
• Public Benchmarks: ViMore outperformed competitors on 3 out of 4 public OCSR datasets (CLEF, JPO, USPTO).
• Qualitative Superiority: The authors demonstrated that MolMole successfully handles multi-column reaction diagrams where cropping-based models fail and faithfully preserves layout geometry in generated MOLfiles.

Reproducibility Details

Data

• Training Data: The models (ViDetect and ViMore) were trained on private/proprietary datasets, which is a limitation for full reproducibility from scratch.
• Benchmark Data: The authors introduce a test set of 550 pages (3,897 molecules, 1,022 reactions) derived from patents and scientific articles. This dataset is stated to be made “publicly available”.
• Public Evaluation Data: Standard OCSR datasets used include USPTO (5,719 images), UOB (5,740 images), CLEF (992 images), and JPO (450 images).

Algorithms

• Pipeline Workflow: PDF → PNG Images → Parallel execution of ViDetect and ViReact → Cropping of molecular regions → ViMore conversion → Output (JSON/Excel).
• Post-Processing:
  • ViDetect: Removes overlapping proposals based on confidence scores and size constraints.
  • ViReact: Refines predictions by correcting duplicates and removing empty entities.
  • ViMore: Assembles detected atom/bond information into structured representations (MOLfile).

Models

Evaluation

• Molecule Detection: Evaluated using COCO metrics (AP, AR, F1) at IoU thresholds 0.50-0.95.
• Molecule Conversion: Evaluated using SMILES exact match accuracy and Tanimoto similarity.
• Combined Metric: A custom metric where a True Positive requires both IoU $\geq$ 0.5 and a correct SMILES string match where $\text{SMILES}_{\text{gt}} == \text{SMILES}_{\text{pred}}$.
• Reaction Parsing: Evaluated using Hard Match (all components correct) and Soft Match (molecular entities only, ignoring text labels).

Title: MolGrapher: Graph-based Visual Recognition of Chemical Structures

Authors: Lucas Morin, Martin Danelljan, Maria Isabel Agea, Ahmed Nassar, Valery Weber, Ingmar Meijer, Peter Staar, Fisher Yu

Citation: Morin, L., Danelljan, M., Agea, M. I., Nassar, A., Weber, V., Meijer, I., Staar, P., & Yu, F. (2023). MolGrapher: Graph-based Visual Recognition of Chemical Structures. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 19552-19561.

Publication: ICCV 2023

Links:

• Paper
• GitHub Repository

1. Contribution / Type

This is primarily a Methodological paper that proposes a novel neural architecture (MolGrapher), shifting the paradigm of Optical Chemical Structure Recognition (OCSR) from image captioning back to graph reconstruction. It also has a significant Resource component, releasing a synthetic data generation pipeline and a new large-scale benchmark (USPTO-30K) to address the scarcity of annotated real-world data.

2. Motivation

The automatic analysis of chemical literature is critical for accelerating drug and material discovery, but much of this information is locked in 2D images of molecular structures.

• Problem: Existing rule-based methods are rigid, while recent deep learning methods based on “image captioning” (predicting SMILES strings) struggle with complex molecules and fail to exploit the natural graph structure of molecules.
• Gap: There is a lack of diverse, annotated real-world training data, and captioning models suffer from “hallucinations” where they predict valid SMILES that do not match the image.

3. Novelty / Core Innovation

MolGrapher introduces a graph-based deep learning pipeline that explicitly models the molecule’s geometry and topology.

• Supergraph Concept: It first detects all atom keypoints and builds a “supergraph” of all plausible bonds.
• Hybrid Approach: It combines a ResNet-based keypoint detector with a Graph Neural Network (GNN) that classifies nodes (atoms) and edges (bonds) within the supergraph context.
• Synthetic Pipeline: A robust data generation pipeline that renders molecules with varying styles (fonts, bond widths) and augmentations to simulate real document noise.

At the core of the Keypoint Detector’s performance is the Weight-Adaptive Heatmap Regression (WAHR) loss. Since pixels without an atom drastically outnumber pixels containing an atom, WAHR loss is designed to counter the class imbalance. For ground-truth heatmap $y$ and prediction $p$:

$$ L_{WAHR}(p, y) = \sum_i \alpha_y (p_i - y_i)^2 $$

where $\alpha_y$ dynamically down-weights easily classified background pixels.

4. Methodology & Experiments

The authors evaluated MolGrapher against both rule-based (OSRA, MolVec) and deep learning baselines (DECIMER, Img2Mol, Image2Graph).

• Benchmarks: Evaluated on standard datasets: USPTO, Maybridge UoB, CLEF-2012, and JPO.
• New Benchmark: Introduced and tested on USPTO-30K, split into clean, abbreviated, and large molecule subsets.
• Ablations: Analyzed the impact of synthetic augmentations, keypoint loss functions, supergraph connectivity radius, and GNN layers.
• Robustness: Tested on perturbed images (rotations, shearing) to mimic scanned patent quality.

The key mathematical formulation in the model involves the node context updates in the GNN mechanism. Messages $m_{j\to i}$ from neighboring nodes are aggregated to form a topological update $\Delta x_i$, which is then combined with visual updates.

5. Results & Conclusions

MolGrapher achieved state-of-the-art performance, significantly outperforming image captioning methods on standard benchmarks.

• Accuracy: It achieved 91.5% accuracy on USPTO compared to 61.0% for DECIMER 2.0 (the next best synthetic-only method).
• Large Molecules: It demonstrated superior scaling, correctly recognizing large molecules (USPTO-10K-L) where image captioning methods like Img2Mol failed completely (0.0% accuracy).
• Generalization: The method proved robust to image perturbations and style variations without requiring fine-tuning on real data. Important caveats: The paper acknowledges MolGrapher’s sensitivity to disconnected structures and lack of support for stereochemistry mapping.

Reproducibility Details

Data

The model relies on synthetic data for training due to the scarcity of annotated real-world images.

Algorithms

The pipeline consists of three distinct algorithmic stages:

1. Keypoint Detection:

   • Predicts a heatmap of atom locations using a CNN.
   • Thresholds heatmaps at the bottom 10th percentile and uses a $5\times5$ window for local maxima.
   • Uses Weight-Adaptive Heatmap Regression (WAHR) loss to handle class imbalance (background vs. atoms).

2. Supergraph Construction:

   • Connects every detected keypoint to neighbors within a radius of $3 \times$ the estimated bond length.
   • Prunes edges with no filled pixels or if obstructed by a third keypoint.
   • Keeps a maximum of 6 bond candidates per atom.

3. Superatom Recognition:

   • Detects “superatom” nodes (abbreviations like COOH).
   • Uses PP-OCR to transcribe the text at these node locations.

Models

The architecture utilizes standard backbones tailored for specific sub-tasks:

• Keypoint Detector: ResNet-18 backbone with $8\times$ dilation to preserve spatial resolution.
• Node Classifier: ResNet-50 backbone with $2\times$ dilation for extracting visual features at node locations.
• Graph Neural Network: A custom GNN that updates node embeddings based on visual features and neighborhood context. The initial node embedding combines the visual feature vector $v_i$ and a learnable type encoding $w_{t_i}$.
• Readout: MLPs classify nodes into atom types (e.g., C, O, N) and bond types (Single, Double, None).

Evaluation

Accuracy is defined strictly: the predicted molecule must have an identical InChI string to the ground truth. Stereochemistry and Markush structures are excluded from evaluation.

Hardware

• GPUs: Trained on 3 NVIDIA A100 GPUs.
• Training Time: 20 epochs.
• Optimization: ADAM optimizer, learning rate 0.0001, decayed by 0.8 after 5000 iterations.
• Loss Weighting: Atom classifier loss weighted by 1; bond classifier loss weighted by 3.

Citation

@inproceedings{morinMolGrapherGraphbasedVisual2023,
  title = {{{MolGrapher}}: {{Graph-based Visual Recognition}} of {{Chemical Structures}}},
  shorttitle = {{{MolGrapher}}},
  booktitle = {Proceedings of the {{IEEE}}/{{CVF International Conference}} on {{Computer Vision}}},
  author = {Morin, Lucas and Danelljan, Martin and Agea, Maria Isabel and Nassar, Ahmed and Weber, Valery and Meijer, Ingmar and Staar, Peter and Yu, Fisher},
  year = {2023},
  pages = {19552--19561},
  doi = {10.1109/ICCV51070.2023.01793},
  urldate = {2025-10-18},
  langid = {english}
}

Citation: Zhang, D., Zhao, D., Wang, Z., Li, J., & Li, J. (2024). MMSSC-Net: multi-stage sequence cognitive networks for drug molecule recognition. RSC Advances, 14(26), 18182-18191. https://doi.org/10.1039/D4RA02442G

Publication: RSC Advances 2024

Contribution: A Multi-Stage Architectural Pipeline

Methodological Paper ($\Psi_{\text{Method}}$). The paper proposes a novel deep learning architecture (MMSSC-Net) for Optical Chemical Structure Recognition (OCSR). It focuses on architectural innovation, specifically combining a SwinV2 visual encoder with a GPT-2 decoder, and validates this method through extensive benchmarking against existing rule-based and deep-learning baselines. It includes ablation studies to justify the choice of the visual encoder.

Motivation: Addressing Noise and Rigid Image Recognition

• Data Usage Gap: Drug discovery relies heavily on scientific literature, but molecular structures are often locked in vector graphics or images that computers cannot easily process.
• Limitations of Prior Work: Existing Rule-based methods are rigid and sensitive to noise. Previous Deep Learning approaches (Encoder-Decoder “Image Captioning” styles) often lack precision, interpretability, and struggle with varying image resolutions or large molecules.
• Need for “Cognition”: The authors argue that treating the image as a single isolated whole is insufficient; a model needs to “perceive” fine-grained details (atoms and bonds) to handle noise and varying pixel qualities effectively.

Novelty: A Fine-Grained Perception Pipeline

• Multi-Stage Cognitive Architecture: MMSSC-Net splits the task into stages:
  1. Fine-grained Perception: Detecting atom and bond sequences (including spatial coordinates) using SwinV2.
  2. Graph Construction: Assembling these into a molecular graph.
  3. Sequence Evolution: converting the graph into a machine-readable format (SMILES).
• Hybrid Transformer Model: It combines a hierarchical vision transformer (SwinV2) for encoding with a generative pre-trained transformer (GPT-2) and MLPs for decoding atomic and bond targets.
• Robustness Mechanisms: The inclusion of random noise sequences during training to improve generalization to new molecular targets.

Methodology and Benchmarks

• Baselines: compared against 7 other tools:
  • Rule-based: MolVec, OSRA.
  • Image-Smiles (DL): ABC-Net, Img2Mol, MolMiner.
  • Image-Graph-Smiles (DL): Image-To-Graph, MolScribe, ChemGrapher.
• Datasets: Evaluated on 5 diverse datasets: STAKER (synthetic), USPTO, CLEF, JPO, and UOB (real-world).
• Metrics:
  • Accuracy: Exact string match of the predicted SMILES.
  • Tanimoto Similarity: Chemical similarity using Morgan fingerprints.
• Ablation Study: Tested different visual encoders (Swin Transformer, ViT-B, ResNet-50) to validate the choice of SwinV2.
• Resolution Sensitivity: Tested model performance across image resolutions from 256px to 2048px.

Results and Core Outcomes

• State-of-the-Art Performance: MMSSC-Net achieved 75-94% accuracy across datasets, outperforming baselines on most benchmarks.
• Resolution Robustness: The model maintained high accuracy (approx 90-95%) across resolutions (256px to 2048px), whereas baselines like Img2Mol dropped significantly at higher resolutions.
• Efficiency: The SwinV2 encoder was noted to be more efficient than ViT-B in this context.
• Limitations: The model struggles with stereochemistry (virtual vs. solid wedge bonds) and “irrelevant text” noise (e.g., in JPO/DECIMER datasets).

Reproducibility Details

Data

The model was trained on a combination of PubChem and USPTO data, augmented to handle visual variability.

Augmentation:

• Image: Random perturbations using RDKit/Indigo (rotation, filling, cropping, bond thickness/length, font size, Gaussian noise).
• Molecular: Introduction of functional group abbreviations and R-substituents (dummy atoms) using SMARTS templates.

Algorithms

• Target Sequence Formulation: The model predicts a sequence containing bounding box coordinates and type labels: ${y_{\text{min}}, x_{\text{min}}, y_{\text{max}}, x_{\text{max}}, C_{\text{type}}}$.
• Loss Function: Cross-entropy loss with maximum likelihood estimation. $$ \max \sum_{i=1}^{N} \sum_{j=1}^{L} \omega_{j} \log P(t_{j}^{i}|x^{i}, t_{1}^{i}, \dots, t_{j-1}^{i}) $$
• Noise Injection: A random sequence $T_r$ is appended to the target sequence during training to improve generalization to new goals.
• Graph Construction: Atoms ($v$) and bonds ($e$) are recognized separately; bonds are defined by connecting spatial atomic coordinates.

Models

• Encoder: Swin Transformer V2.
  • Pre-trained on ImageNet-1K.
  • Window size: $16 \times 16$.
  • Parameters: 88M.
  • Input resolution: $256 \times 256$.
  • Features: Scaled cosine attention; log-space continuous position bias.
• Decoder: GPT-2 + MLP.
  • GPT-2: Used for recognizing atom types.
    • Layers: 24.
    • Attention Heads: 12.
    • Hidden Dimension: 768.
    • Dropout: 0.1.
  • MLP: Used for classifying bond types (single, double, triple, aromatic, wedge).
• Vocabulary:
  • Standard: 95 common numbers/characters ([0], [C], [=], etc.).
  • Extended: 2000 SMARTS-based characters for isomers/groups (e.g., “[C2F5]”, “[halo]”).

Evaluation

Metrics:

1. Accuracy: Exact match of the generated SMILES string.
2. Tanimoto Similarity: Similarity of Morgan fingerprints between predicted and ground truth molecules.

Key Results (Accuracy):