Surge is an open-source constitutional isomer generator that enumerates all possible molecular structures for a given molecular formula. It is built on the nauty package for graph automorphism computation and uses a three-stage canonical generation path method that decomposes the enumeration problem into progressively refined graph operations. Surge outperforms the previous state-of-the-art (MOLGEN 5.0) by orders of magnitude in speed while running in under 5 MB of RAM regardless of molecule size.

Motivation: The Need for Fast, Open Structure Generators

Chemical structure generators are essential for computer-assisted structure elucidation (CASE), virtual library creation, and chemical space enumeration (e.g., GDB-17’s 166.4 billion molecules). MOLGEN had been the gold standard for decades but is closed-source. The previous best open-source alternative, MAYGEN, was roughly 3x slower than MOLGEN. Reymond’s lab used an in-house nauty-based generator for GDB-17 but did not release it publicly. Surge fills this gap as a fast, open-source, and extensible alternative.

The Three-Stage Algorithm

Given a molecular formula (e.g., $\text{C}_9\text{H}_{18}\text{N}_2\text{O}_4$), Surge proceeds through three stages:

Stage 1 (geng): Simple graph generation. Computes all connected simple graphs with the appropriate number of non-hydrogen atoms and edges, subject to maximum degree constraints from the molecular formula. These graphs represent molecular topologies without atom types or bond orders. For Lysopine ($\text{C}_9\text{H}_{18}\text{N}_2\text{O}_4$), this produces 534,493 graphs in 1.3 seconds.

Stage 2 (vcolg): Vertex coloring (atom assignment). Assigns element types (C, N, O, S, etc.) to vertices in all distinct ways, using the automorphism group of each simple graph to avoid generating equivalent assignments. Given a fixed ordering of elements (e.g., $\text{C} < \text{O} < \text{S}$), element assignments are represented as lists $L$ and compared lexicographically. Exactly one representative from each equivalence class is selected by computing the canonical (lexicographically maximal) list:

$$ \text{canon}(L) = \max\{\gamma(L) \mid \gamma \in \text{Aut}(G)\} $$

A list $L$ is accepted if and only if $\text{canon}(L) = L$, i.e., no automorphism produces a lexicographically larger list. For Lysopine, this expands to 3.0 billion vertex-labeled graphs in 90 seconds.

Stage 3 (multig): Edge multiplicity (bond orders). Assigns bond multiplicities (single, double, triple) to edges, again using automorphism group factorization to avoid duplicates. For Lysopine, this produces 6.0 billion completed molecules in an additional 100 seconds.

Efficient Automorphism Handling via Group Factorization

The key algorithmic innovation is the factorization of the automorphism group:

$$ \text{Aut}(G) = NM = \{\gamma\delta \mid \gamma \in N,; \delta \in M\} $$

where $M$ is the “minor subgroup” generated by transpositions of leaves sharing a common neighbor (“flowers”), and $N$ is a complete set of coset representatives computed by nauty. A flower is a maximal set of degree-1 vertices (leaves) with the same neighbor. The minor subgroup $M$ is normal in $\text{Aut}(G)$, making the factorization well-defined.

Theorem. A list $L$ satisfies $L = \text{canon}(L)$ if and only if $L = \max\{\delta(L) \mid \delta \in M\}$ and $L = \max\{\gamma(L) \mid \gamma \in N\}$.

This factorization enables efficient canonicity testing. Maximality under $M$ reduces to enforcing decreasing element order within each flower (simple inequality constraints during recursive assignment). Maximality under $N$ requires explicit testing against the $N$ generators, but $N$ is trivial (identity only) 58% of the time in Stage 2 and 98% of the time in Stage 3.

Benchmark Results

Benchmarked against MOLGEN 5.0 on 30 natural product molecular formulas from the COCONUT database on a compute-optimized c2-standard-4 Google Cloud VM, Surge achieves 7-22 million molecules per second with a memory footprint of at most 5 MB regardless of molecule size. Representative results:

MOLGEN hit its built-in limit of $2^{31} - 1$ structures for most formulas; reported times were linearly extrapolated. Both generators were instructed to generate but not output structures. MOLGEN was run with -noaromaticity for fair comparison since Surge v1.0 lacks aromaticity detection.

Surge supports output in both SDfile and SMILES formats. SMILES output is produced efficiently by constructing a template for each simple graph at Stage 1, so that only atom types and bond multiplicities must be filled in before output.

Surge also supports built-in filters applied during generation (more efficient than post-hoc filtering):

• -p0:1: at most one cycle of length 5
• -P: the molecule must be planar
• -B5: no atom has two double bonds and otherwise only hydrogen neighbors
• -B9: no atom lies on more than one cycle of length 3 or 4

These filter options are inspired by corresponding features in MOLGEN. Surge’s open-source design also supports a plugin mechanism: users can write small code snippets to insert custom filters into any of the three stages, enabling efficient pruning of the generation tree.

Limitations

• Version 1.0 does not perform Hückel aromaticity detection, so it generates duplicate Kekulé structures for aromatic rings that are graph-theoretically distinct
• Benchmarking against MOLGEN required disabling MOLGEN’s aromaticity detection (-noaromaticity) for fair comparison
• Written in C (from the nauty suite), which limits accessibility compared to Python-based tools, though this is also the source of its speed

Reproducibility Details

• Status: Highly Reproducible. Source code, build instructions, and benchmark formulas are all publicly available.
• Hardware: Benchmarks used a compute-optimized c2-standard-4 Google Cloud VM. Surge runs in at most 5 MB of RAM regardless of molecule size.
• Build: Standard Unix Configure/Make scheme producing a standalone command-line executable. Written in portable C from the nauty suite.
• Dependencies: Requires the nauty package (bundled).

Paper Information

• Published: Journal of Cheminformatics, Volume 14, Article 24, April 23, 2022
• Preprint: ChemRxiv, December 7, 2021
• License: Apache 2.0 (software), Open Access (paper)

@article{mckay2022surge,
  title={Surge: a fast open-source chemical graph generator},
  author={McKay, Brendan D. and Yirik, Mehmet Aziz and Steinbeck, Christoph},
  journal={Journal of Cheminformatics},
  volume={14},
  number={1},
  pages={24},
  year={2022},
  publisher={BioMed Central},
  doi={10.1186/s13321-022-00604-9}
}

This is a Method paper that introduces Liquid-S4, a new state-space model combining the structured state-space framework (S4) with liquid time-constant (LTC) networks. The primary contribution is an input-dependent state transition mechanism that allows the model to adapt its dynamics based on incoming inputs, while retaining the efficient convolutional kernel computation of S4.

Scaling Liquid Networks to Long Sequences

Liquid time-constant (LTC) networks are continuous-time neural networks with input-dependent state transitions, giving them strong generalization and causal modeling properties. However, LTCs rely on ODE solvers that limit their scalability to long sequences. Structured state-space models (S4) solve this scalability problem through HiPPO initialization, diagonal plus low-rank (DPLR) parameterization, and efficient Cauchy kernel computation in the frequency domain, but they use fixed (input-independent) state transitions.

The key question this paper addresses: can the expressivity of LTC networks be combined with the efficiency and scalability of S4 to improve long-range sequence modeling?

The Liquid Kernel: Input-Dependent Convolutions

The core innovation is a linearized LTC state-space model that replaces the standard SSM dynamics:

$$\dot{x}(t) = \mathbf{A}x(t) + \mathbf{B}u(t)$$

with an input-dependent formulation:

$$\dot{x}(t) = \left[\mathbf{A} + \mathbf{B}u(t)\right]x(t) + \mathbf{B}u(t)$$

where $u(t)$ now modulates the state transition matrix itself. After discretization via the bilinear transform, the recurrence becomes:

$$x_{k} = \left(\overline{\mathbf{A}} + \overline{\mathbf{B}}u_{k}\right)x_{k-1} + \overline{\mathbf{B}}u_{k}$$

Unrolling this recurrence reveals that the output $y_{k}$ decomposes into two parts:

$$y = \overline{\mathbf{K}} * u + \overline{\mathbf{K}}_{\text{liquid}} * u_{\text{correlations}}$$

The first term is the standard S4 convolutional kernel $\overline{\mathbf{K}}$, mapping individual input time steps independently. The second term is a new “liquid kernel” $\overline{\mathbf{K}}_{\text{liquid}}$ that operates on auto-correlation terms of the input signal (products $u_{i}u_{j}$, $u_{i}u_{j}u_{k}$, etc., up to a chosen order $\mathcal{P}$).

Proposition 1 shows that each liquid kernel of order $p$ can be computed from the precomputed S4 kernel via a Hadamard product with $\overline{\mathbf{B}}^{p-1}$ followed by an anti-diagonal transformation (flip):

$$\overline{\mathbf{K}}_{\text{liquid}=p} = \left[\overline{\mathbf{K}}_{(L-\tilde{L},L)} \odot \overline{\mathbf{B}}_{(L-\tilde{L},L)}^{p-1}\right] * \mathbf{J}_{\tilde{L}}$$

This is the KB (Kernel $\times$ B) mode. The authors also propose a simplified PB (Powers of B) mode that sets the transition matrix $\overline{\mathbf{A}}$ to identity for the correlation terms:

$$\overline{\mathbf{K}}_{\text{liquid}=p} = \overline{\mathbf{C}} \odot \overline{\mathbf{B}}^{p-1}$$

The PB kernel is cheaper to compute and performs equally well or better in practice.

The computational complexity is $\tilde{\mathcal{O}}(N + L + p_{\text{max}}\tilde{L})$, where $N$ is the state size, $L$ the sequence length, $p_{\text{max}}$ the maximum liquid order, and $\tilde{L}$ the liquid kernel length (typically two orders of magnitude smaller than $L$).

Benchmarks Across Long-Range Sequence Tasks

Liquid-S4 is evaluated on four benchmark suites with the PB kernel using the S4-LegS (scaled Legendre) parameterization.

Long Range Arena (LRA)

The LRA benchmark contains six tasks with sequence lengths from 1K to 16K. Liquid-S4 achieves state-of-the-art on all six tasks with an average accuracy of 87.32%:

Liquid orders $p$ range from 2 to 6 across tasks.

BIDMC Vital Signs

On medical time-series regression (heart rate, respiratory rate, SpO2 prediction from length-4000 biomarker signals):

Sequential CIFAR (sCIFAR)

Liquid-S4 with $p=3$ achieves 92.02% accuracy on 1-D pixel-level image classification, improving over S4-LegS (91.80%).

Speech Commands (Full 35 Labels)

On the raw 16kHz speech recognition task, Liquid-S4 achieves 96.78% accuracy with only 224K parameters, a 30% reduction compared to S4’s 307K. On the zero-shot 8kHz experiment, performance drops to 90.00% (vs. 91.32% for S4-LegS), which the authors attribute to the liquid kernel’s sensitivity to input covariance structure at different sampling rates.

Consistent Improvements with Smaller Models

Liquid-S4 achieves state-of-the-art performance on every benchmark evaluated: all six LRA tasks (87.32% average), all three BIDMC vital signs tasks, sCIFAR, and full Speech Commands recognition. The gains are particularly large on tasks where input correlation structure matters (ListOps +3.15%, IMDB +2.20%, respiratory rate RMSE improvement of 36%).

A practical advantage is that Liquid-S4 works well with smaller state sizes (as low as 7 units for some tasks), reducing parameter counts. The PB kernel is recommended over KB for its simplicity and competitive performance. Higher liquid orders ($p$) consistently improve performance, though $p=3$ is recommended as a default.

Limitations include degraded performance in zero-shot frequency transfer (8kHz Speech Commands), suggesting the liquid kernel’s input covariance terms may not generalize well across sampling rate changes. The paper also does not compare against non-SSM approaches beyond the LRA benchmark. The causal (unidirectional) configuration works better than bidirectional for Liquid-S4, which may limit applicability to tasks that benefit from bidirectional context.

Reproducibility Details

Classification: Partially Reproducible. Code and all benchmark datasets are publicly available, with complete hyperparameters documented. No pre-trained weights are released and hardware requirements are not specified.

Artifacts

Data

Algorithms

The Liquid-S4 kernel computation builds on the S4 kernel pipeline:

1. Initialize $\mathbf{A}$ with HiPPO (scaled Legendre) matrix in DPLR form
2. Compute S4 kernel $\overline{\mathbf{K}}$ via Cauchy kernel and iFFT
3. For each liquid order $p \in {2, \ldots, \mathcal{P}}$, compute $\overline{\mathbf{K}}_{\text{liquid}=p}$ using either KB or PB mode
4. Convolve $\overline{\mathbf{K}}_{\text{liquid}}$ with input correlation vector $u_{\text{correlations}}$

The PB kernel mode is used in all reported experiments. The PyKeops package is used for large tensor computations.

Models

Liquid-S4 generally requires smaller learning rates than S4/S4D. $\Delta t_{\text{max}} = 0.2$ for all experiments; $\Delta t_{\text{min}} \propto 1/\text{seq_length}$.

Evaluation

All results report validation accuracy (except BIDMC, which reports test RMSE). Experiments use 2-3 random seeds with standard deviations reported.

Hardware

Not specified in the paper.

Paper Information

Citation: Hasani, R., Lechner, M., Wang, T.-H., Chahine, M., Amini, A., & Rus, D. (2022). Liquid Structural State-Space Models. arXiv preprint arXiv:2209.12951.

@misc{hasani2022liquid,
  title={Liquid Structural State-Space Models},
  author={Hasani, Ramin and Lechner, Mathias and Wang, Tsun-Hsuan and Chahine, Makram and Amini, Alexander and Rus, Daniela},
  year={2022},
  eprint={2209.12951},
  archiveprefix={arXiv},
  primaryclass={cs.LG}
}

This is a Method paper that introduces Lagrangian Neural Networks (LNNs), a neural network architecture that parameterizes arbitrary Lagrangians to learn energy-conserving dynamics from data. The key contribution is showing that neural networks can learn Lagrangian functions directly, and that the Euler-Lagrange equation can be solved numerically using automatic differentiation to produce physically consistent dynamics. The approach is strictly more general than prior methods: it does not require canonical coordinates (unlike Hamiltonian Neural Networks) and does not restrict the functional form of kinetic energy (unlike Deep Lagrangian Networks).

Why Standard Neural Networks Fail at Conservation Laws

Neural networks struggle to learn fundamental symmetries and conservation laws from data. A standard neural network trained on trajectories of a double pendulum will gradually dissipate energy over long rollouts, producing physically implausible behavior. This happens because unconstrained function approximators have no inductive bias toward conservation.

Hamiltonian Neural Networks (HNNs) addressed this by learning a Hamiltonian function, which automatically enforces energy conservation. However, the Hamiltonian formalism requires inputs in canonical coordinates $(q, p)$ satisfying strict Poisson bracket relations:

$$ p_i \equiv \frac{\partial \mathcal{L}}{\partial \dot{q}_i} \quad \Longleftrightarrow \quad {q_i, q_j} = 0, \quad {p_i, p_j} = 0, \quad {q_i, p_j} = \delta_{ij} $$

In many real-world settings, the canonical momenta are unknown or difficult to compute. For example, in special relativity the canonical momentum $\dot{q}(1 - \dot{q}^2)^{-3/2}$ is a complex nonlinear function of velocity. Deep Lagrangian Networks (DeLaNs) partially addressed this by learning Lagrangians, but they assumed kinetic energy takes the rigid-body form $T = \dot{q}^T M \dot{q}$, which excludes relativistic and other non-standard systems.

Solving Euler-Lagrange for a Black-Box Lagrangian

The core innovation of LNNs is a method for computing accelerations from a neural network that represents an arbitrary Lagrangian $\mathcal{L}(q, \dot{q})$. Starting from the Euler-Lagrange equation:

$$ \frac{d}{dt} \nabla_{\dot{q}} \mathcal{L} = \nabla_{q} \mathcal{L} $$

The authors expand the time derivative using the chain rule, yielding:

$$ \left(\nabla_{\dot{q}} \nabla_{\dot{q}}^{\top} \mathcal{L}\right) \ddot{q} + \left(\nabla_{q} \nabla_{\dot{q}}^{\top} \mathcal{L}\right) \dot{q} = \nabla_{q} \mathcal{L} $$

Solving for the accelerations gives:

$$ \ddot{q} = \left(\nabla_{\dot{q}} \nabla_{\dot{q}}^{\top} \mathcal{L}\right)^{-1} \left[ \nabla_{q} \mathcal{L} - \left(\nabla_{q} \nabla_{\dot{q}}^{\top} \mathcal{L}\right) \dot{q} \right] $$

This requires computing the Hessian of the neural network with respect to $\dot{q}$ and then inverting it (using a pseudoinverse for numerical stability). JAX’s automatic differentiation makes this feasible in just a few lines of code, despite the seemingly complex chain of second-order derivatives. The matrix inverse scales as $\mathcal{O}(d^3)$ with the number of coordinates $d$.

A critical implementation detail is the choice of activation function. Since the method takes second-order derivatives of the network, ReLU is unsuitable (its second derivative is zero everywhere). After a hyperparameter search over ReLU$^2$, ReLU$^3$, tanh, sigmoid, and softplus, the authors found softplus performed best.

The authors also developed a custom initialization scheme, using symbolic regression to find initialization variances that maintain well-conditioned gradients through the Hessian computation:

$$ \sigma = \frac{1}{\sqrt{n}} \begin{cases} 2.2 & \text{First layer} \\ 0.58i & \text{Hidden layer } i \\ n & \text{Output layer} \end{cases} $$

Extension to Graphs and Continuous Systems

LNNs extend naturally to graph-structured and continuous systems via Lagrangian Graph Networks. For a system with $n$ gridpoints, the total Lagrangian is decomposed into local densities:

$$ \mathcal{L} = \sum_{i=1}^{n} \mathcal{L}_i, \quad \text{where} \quad \mathcal{L}_i = \mathcal{L}_{\text{density}}\left({\phi_j, \dot{\phi}_j}_{j \in \mathcal{I}_i}\right) $$

Here $\mathcal{I}_i$ defines the neighborhood of node $i$ (e.g., ${i-1, i, i+1}$ for a 1D grid). The Lagrangian density is modeled as an MLP. The resulting Hessian matrix is sparse, with non-zero entries only at “neighbor of neighbor” positions, enabling efficient computation: in 1D, only 5 forward-over-backward autodiff passes are needed, and the tridiagonal inverse runs in linear time.

Experiments: Double Pendulum, Relativity, and Waves

All models used 4-layer MLPs with 500 hidden units, softplus activations, a decaying learning rate starting at $10^{-3}$, and batch size 32.

Double Pendulum

The LNN and baseline achieved similar instantaneous acceleration losses ($7.3$ vs. $7.4 \times 10^{-2}$). The key difference appeared in long-term energy conservation: averaged over 40 random initial conditions with 100 time steps, the mean energy discrepancy was 8% of max potential energy for the baseline but only 0.4% for the LNN.

Relativistic Particle

For a particle with Lagrangian $\mathcal{L} = ((1 - \dot{q}^2)^{-1/2} - 1) + gq$, the canonical momenta $\dot{q}(1 - \dot{q}^2)^{-3/2}$ are non-trivial. An HNN trained on non-canonical coordinates $(q, \dot{q})$ failed to learn the dynamics. The LNN succeeded using the same non-canonical coordinates, matching the performance of an HNN given the correct canonical coordinates.

1D Wave Equation

The Lagrangian Graph Network learned the wave equation dynamics ($\ddot{\phi} = \frac{\partial^2 \phi}{\partial x^2}$ with $c = 1$) on a 100-gridpoint domain with periodic boundary conditions. The network learned the Lagrangian density corresponding to the continuum form $\mathcal{L} = \int (\dot{\phi}^2 - (\partial \phi / \partial x)^2) dx$, accurately modeling wave propagation and conserving energy across the material.

Findings and Comparison to Prior Approaches

LNNs combine several desirable properties that no single prior method offers:

The main limitation is computational cost: the Hessian computation and inversion scale as $\mathcal{O}(d^3)$ in the number of coordinates. The Lagrangian Graph Network partially mitigates this for spatially extended systems through the sparsity of the resulting Hessian. The method also assumes access to state derivatives ($\dot{q}$) during training, which may not always be directly available from observations.

Reproducibility Details

Data

Algorithms

• Forward model: Euler-Lagrange equation solved via Equation 6 using JAX autodiff
• Pseudoinverse used for Hessian inversion to handle potential singular matrices
• Custom initialization scheme (Equation 16) derived via symbolic regression with eureqa
• Softplus activation selected via hyperparameter search

Models

• 4-layer MLP with 500 hidden units for all experiments
• Softplus activation function
• Code: github.com/MilesCranmer/lagrangian_nns (Apache-2.0)

Evaluation

Hardware

Not specified in the paper. JAX-based implementation supports CPU and GPU execution.

Reproducibility Status: Highly Reproducible

Artifacts

Paper Information

Citation: Cranmer, M., Greydanus, S., Hoyer, S., Battaglia, P., Spergel, D., & Ho, S. (2020). Lagrangian Neural Networks. ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations. arXiv: 2003.04630

Publication: ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations

@misc{cranmer2020lagrangian,
  title={Lagrangian Neural Networks},
  author={Cranmer, Miles and Greydanus, Sam and Hoyer, Stephan and Battaglia, Peter and Spergel, David and Ho, Shirley},
  year={2020},
  eprint={2003.04630},
  archiveprefix={arXiv},
  primaryclass={cs.LG}
}

This is a Method paper that introduces STONED (Superfast Traversal, Optimization, Novelty, Exploration and Discovery), a suite of algorithms for molecular generation and chemical space exploration. STONED operates entirely through string manipulations on the SELFIES molecular representation, avoiding the need for deep learning models, training data, or GPU resources. The key claim is that simple character-level mutations and interpolations in SELFIES can achieve results competitive with state-of-the-art deep generative models on standard benchmarks.

Why Deep Generative Models May Be Overkill

Deep generative models (VAEs, GANs, RNNs, reinforcement learning) have become popular for inverse molecular design, but they come with practical costs: large training datasets, expensive GPU compute, and long training times. Fragile representations like SMILES compound the problem, since large portions of a latent space can map to invalid molecules. Even with the introduction of SELFIES (a 100% valid string representation), prior work still embedded it within neural network architectures.

The authors argue that for tasks like local chemical space exploration and molecular interpolation, the guarantees of SELFIES alone may be sufficient. Because every SELFIES string maps to a valid molecule, random character mutations always produce valid structures. This observation eliminates the need for learned generation procedures entirely.

Core Innovation: SELFIES String Mutations as Molecular Operators

STONED relies on four key techniques built on SELFIES string manipulations:

1. Random character mutations. A point mutation in SELFIES (character replacement, deletion, or addition) always yields a valid molecule. The position of mutations serves as a hyperparameter controlling exploration vs. exploitation: terminal character mutations preserve more structural similarity to the seed, while random mutations explore more broadly.

2. Multiple SMILES orderings. A single molecule has many valid SMILES strings, each mapping to a different SELFIES. By generating 50,000 SMILES orderings and converting to SELFIES before mutation, the diversity of generated structures increases substantially.

3. Deterministic interpolation. Given two SELFIES strings (padded to equal length), characters at equivalent positions can be successively replaced from the start molecule to the target molecule. Every intermediate string is a valid molecule. A chemical path is extracted by keeping only those intermediates that increase fingerprint similarity to the target.

4. Fingerprint-based filtering. Since edit distance in SELFIES does not reflect molecular similarity, STONED uses fingerprint comparisons (ECFP4, FCFP4, atom-pair) to enforce structural similarity constraints.

The authors also propose a revised joint molecular similarity metric for evaluating median molecules. Given $n$ reference molecules $M = {m_1, m_2, \ldots, m_n}$, the joint similarity of a candidate molecule $m$ is:

$$ F(m) = \frac{1}{n} \sum_{i=1}^{n} \text{sim}(m_i, m) - \left[\max_{i} \text{sim}(m_i, m) - \min_{i} \text{sim}(m_i, m)\right] $$

This penalizes candidates that are similar to only a subset of references, unlike the geometric mean metric used in GuacaMol which can yield high scores even with lopsided similarities.

Experimental Setup and Applications

Local chemical subspace formation

Starting from a single seed molecule (aripiprazole, albuterol, mestranol, or celecoxib), the algorithm generates 50,000 SMILES orderings and performs 1-5 point mutations per ordering, producing 250,000 candidate strings. Unique valid molecules are filtered by fingerprint similarity thresholds.

Key finding: restricting mutations to terminal characters yields a 20x increase in high-similarity molecules compared to random positions. Compared to SMILES mutations (0.30% valid) and DeepSMILES (1.44% valid), SELFIES mutations are all valid by construction.

A two-step expansion (mutating all unique first-round neighbors) produced over 17 million unique molecules, with 120,000 having similarity greater than 0.4 to celecoxib.

Chemical path formation and drug design

Deterministic SELFIES interpolation between tadalafil and sildenafil generated paths where logP and QED values varied smoothly. A more challenging application docked intermediates between dihydroergotamine (5-HT1B binder) and prinomastat (CYP2D6 binder), finding molecules with non-trivial binding affinity to both proteins without any optimization routine.

Median molecules for photovoltaics

Using 100 triplets from the Harvard Clean Energy (HCE) dataset, each with one molecule optimized for high LUMO energy, one for high dipole moment, and one for high HOMO-LUMO gap, generalized chemical paths produced median molecules. These were evaluated with GFN2-xTB semiempirical calculations. The generated medians matched or exceeded the best molecules available in the HCE database in both structural similarity and target properties.

GuacaMol benchmarks

Without any training, STONED achieved an overall GuacaMol score of 14.70, competitive with several deep generative models. The approach simply identifies the single best molecule in the benchmark’s training set and generates its local chemical subspace. 38% of the top-100 molecules from each benchmark passed compound quality filters, comparable to Graph GA and SMILES GA.

Results Summary and Limitations

STONED demonstrates that SELFIES string mutations can match or approach deep generative models on standard molecular design benchmarks while being orders of magnitude faster and requiring no training. The most expensive benchmark (aripiprazole subspace) completed in 500 seconds on a laptop CPU.

The method comparison table from the paper highlights STONED’s unique position:

Limitations acknowledged by the authors:

• STONED lacks property-based navigation (gradient-guided optimization toward specific property targets). It can only do stochastic property optimization when wrapped in a genetic algorithm.
• The success rate of mutations leading to structurally similar molecules is relatively low (0.1-2% at high similarity thresholds), though speed compensates.
• Chemical paths can contain molecules with unstable functional groups or tautomerization issues, requiring post-hoc filtering with domain-specific rules.
• Fingerprint similarity does not capture all aspects of chemical similarity (3D geometry, reactivity, synthesizability).
• The penalized logP and QED benchmarks used by GuacaMol do not represent the full complexity of practical molecular design.

Reproducibility Details

Data

Algorithms

• Local subspace formation: 50,000 SMILES orderings per seed molecule, 1-5 SELFIES point mutations each, totaling 250,000 candidates per experiment.
• Chemical paths: Deterministic character-by-character interpolation between padded SELFIES strings, with monotonic fingerprint similarity filtering.
• Median molecules: Generalized paths between 3+ reference molecules using 10,000 paths per triplet with randomized SMILES orderings.
• Docking: SMINA with crystal structures from PDB (4IAQ for 5-HT1B, 3QM4 for CYP2D6). Top-5 binding poses averaged.
• Quantum chemistry: GFN2-xTB for dipole moments, LUMO energies, and HOMO-LUMO gaps.

Evaluation

Hardware

All experiments run on a laptop with Intel i7-8750H CPU at 2.20 GHz. No GPU required. Most expensive single experiment (aripiprazole subspace) completed in 500 seconds.

Artifacts

Paper Information

Citation: Nigam, A. K., Pollice, R., Krenn, M., dos Passos Gomes, G., & Aspuru-Guzik, A. (2021). Beyond generative models: superfast traversal, optimization, novelty, exploration and discovery (STONED) algorithm for molecules using SELFIES. Chemical Science, 12(20), 7079-7090. https://doi.org/10.1039/d1sc00231g

@article{nigam2021stoned,
  title={Beyond generative models: superfast traversal, optimization, novelty, exploration and discovery ({STONED}) algorithm for molecules using {SELFIES}},
  author={Nigam, AkshatKumar and Pollice, Robert and Krenn, Mario and dos Passos Gomes, Gabriel and Aspuru-Guzik, Al{\'a}n},
  journal={Chemical Science},
  volume={12},
  number={20},
  pages={7079--7090},
  year={2021},
  publisher={Royal Society of Chemistry},
  doi={10.1039/d1sc00231g}
}

This is a Method paper that introduces two graph-based approaches for exploring chemical space: a genetic algorithm (GB-GA) and a generative model combined with Monte Carlo tree search (GB-GM-MCTS). The primary contribution is demonstrating that these non-ML, graph-based methods can match or exceed the performance of contemporary ML-based generative models for molecular property optimization, while being several orders of magnitude faster. The paper provides open-source implementations built on the RDKit cheminformatics package. The two approaches explore chemical space using direct graph manipulations rather than string-based representations like SMILES.

Why Compare Simple Baselines to ML Generative Models?

By 2018, several ML-based generative models for molecules had been published, including VAEs, RNNs, and graph convolutional policy networks. However, these models were rarely compared against traditional optimization approaches such as genetic algorithms. Jensen identifies this gap explicitly: while ML generative model performance had been impressive, the lack of comparison to simpler baselines made it difficult to assess whether the complexity of ML approaches was justified.

A practical barrier to such comparisons was the absence of free, open-source GA implementations for molecular optimization (the existing ACSESS algorithm required proprietary OpenEye toolkits). This paper fills that gap by providing RDKit-based implementations of both the GB-GA and GB-GM-MCTS.

Graph-Based Crossovers, Mutations, and Monte Carlo Tree Search

GB-GA: Crossovers and Mutations on Molecular Graphs

The GB-GA operates directly on molecular graph representations (not string representations like SMILES). It combines ideas from Brown et al. (2004) and the ACSESS algorithm of Virshup et al. (2013).

Crossovers can occur at two types of positions with equal probability:

• Non-ring bonds: a molecule is cut at a non-ring bond, and fragments from two parent molecules are recombined
• Ring bonds: adjacent bonds or bonds separated by one bond are cut, and fragments are mated using single or double bonds

Mutations include seven operation types, each with specified probabilities:

• Append atom (15%): adds an atom with a single, double, or triple bond
• Insert atom (15%): inserts an atom into an existing bond
• Delete atom (14%): removes an atom, reconnecting neighbors
• Change atom type (14%): swaps element identity (C, N, O, F, S, Cl, Br)
• Change bond order (14%): toggles between single, double, and triple bonds
• Delete ring bond (14%): opens a ring
• Add ring bond (14%): closes a new ring

Molecules with macrocycles (seven or more atoms), allene centers in rings, fewer than five heavy atoms, incorrect valences, or more non-H atoms than the target size are discarded. The target size is sampled from a normal distribution with mean 39.15 and standard deviation 3.50 non-H atoms, calibrated to match the molecules found by Yang et al. (2017).

GB-GM-MCTS: A Probabilistic Growth Model with Tree Search

The GB-GM grows molecules one atom at a time, with the choice of bond order and atom type determined probabilistically from a bonding analysis of a reference dataset (the first 1000 molecules from ZINC). Since 63% of atoms in the reference set are ring atoms, ring-creation or ring-insertion mutations are chosen 63% of the time.

The generative model is combined with a Monte Carlo tree search where:

• Each node corresponds to an atom addition step
• Leaf parallelization uses a maximum of 25 leaf nodes
• The exploration factor is $1 / \sqrt{2}$
• Rollout terminates if the molecule exceeds the target size
• The reward function returns 1 if the predicted $J(\mathbf{m})$ value exceeds the largest value found so far, and 0 otherwise

The Penalized logP Objective

Both methods optimize the penalized logP score $J(\mathbf{m})$:

$$ J(\mathbf{m}) = \log P(\mathbf{m}) - \text{SA}(\mathbf{m}) - \text{RingPenalty}(\mathbf{m}) $$

where $\log P(\mathbf{m})$ is the octanol-water partition coefficient predicted by RDKit, $\text{SA}(\mathbf{m})$ is a synthetic accessibility score, and $\text{RingPenalty}(\mathbf{m})$ penalizes unrealistically large rings by reducing the score by $\text{RingSize} - 6$ for each oversized ring. Each property is normalized to zero mean and unit standard deviation across the ZINC dataset.

Experimental Setup and Comparisons to ML Methods

GB-GA Experiments

Ten GA simulations were performed with a population size of 20 over 50 generations (1000 $J(\mathbf{m})$ evaluations per run). The initial mating pool was 20 random molecules from the first 1000 molecules in ZINC. Two mutation rates were tested: 50% and 1%.

GB-GM-MCTS Experiments

Ten simulations used ethane as a seed molecule with 1000 tree traversals per run. Additional experiments used 5000 traversals and an adjusted probability of generating $\text{C}=\text{C}-\text{C}$ ring patterns (increased from 62% to 80%).

Baselines

Results were compared to those compiled by Yang et al. (2017):

• ChemTS (RNN + MCTS)
• RNN with and without Bayesian optimization
• Continuous VAE (CVAE)
• Grammar VAE (GVAE)
• Graph convolutional policy network (GCPN, from You et al. 2018)

Key Results

The GB-GA with 1% mutation rate achieved an average maximum $J(\mathbf{m})$ of 7.4, which is 1.8 units higher than the best ML result (ChemTS at 5.6) while using 20x fewer evaluations and completing in 30 seconds versus 8 hours. The two highest-scoring individual molecules found by GB-GA had $J(\mathbf{m})$ scores of 8.8 and 8.5, exceeding the 7.8-8.0 range found by the GCPN approach. These molecules bore little resemblance to the initial mating pool (Tanimoto similarities of 0.27 and 0.12 to the most similar ZINC molecules), indicating that the GA traversed a large distance in chemical space in just 50 generations.

The GB-GM-MCTS performed below ChemTS at equal evaluations (4.3 vs. 4.9 at 5000 evaluations) but was several orders of magnitude faster (9 minutes vs. 2 hours). The MCTS approach tended to extract the dominant hydrophobic structural motif (benzene rings) from the training set, making it more dependent on training set composition than the GA.

Simple Methods Set a High Bar for Molecular Optimization

The central finding is that a simple graph-based genetic algorithm outperforms all tested ML-based generative models on penalized logP optimization, both in terms of solution quality and computational efficiency. The GB-GA achieves higher $J(\mathbf{m})$ scores with 1000 evaluations in 30 seconds than ML methods achieve with 20,000 evaluations over 8 hours.

Several additional observations emerge:

1. Chemical space traversal: The GB-GA can reach high-scoring molecules that are structurally distant from the starting population, with Tanimoto similarity as low as 0.12 to the nearest ZINC molecule.
2. Mutation rate matters: A 1% mutation rate outperformed a 50% rate (7.4 vs. 6.8), suggesting that preserving more parental structure during crossover is beneficial for this objective.
3. Training set dependence: The GB-GM-MCTS is more sensitive to training set composition than the GA. Its preference for benzene-ring-containing molecules (the dominant ZINC motif) limits its ability to discover alternative structural solutions like the long aliphatic chains favored by the GA.
4. Generalizability caveat: Jensen explicitly notes that these comparisons cover only one property (penalized logP) and that similar comparisons for other properties are needed before drawing general conclusions.

The paper’s influence has been substantial: it helped establish the expectation that new molecular generative models should be benchmarked against genetic algorithm baselines, a position subsequently reinforced by Brown et al. (2019) in GuacaMol and by Tripp and Hernandez-Lobato (2023).

Reproducibility Details

Data

Algorithms

• GB-GA: Population size 20, 50 generations, mutation rates of 1% and 50% tested. Crossovers at ring and non-ring bonds with equal probability. Seven mutation types with specified probabilities. Molecules selected from mating pool based on normalized logP scores.
• GB-GM: Atom-by-atom growth using probabilistic rules derived from ZINC bonding analysis. Ring creation probability 63% (matching ZINC), with 80% variant also tested. Seed molecule: ethane.
• MCTS: Modified from haroldsultan/MCTS Python implementation. Leaf parallelization with max 25 leaf nodes. Exploration factor $1/\sqrt{2}$. Binary reward function (1 if new best, 0 otherwise).
• Property calculation: logP, SA score, and ring penalty all computed via RDKit. Each property normalized to zero mean and unit standard deviation across ZINC.

Models

No neural network models are used. The GB-GA and GB-GM are purely algorithmic approaches parameterized by bonding statistics from the ZINC dataset.

Evaluation

Hardware

All GB-GA and GB-GM experiments were run on a laptop. No GPU required. The GB-GA completes in 30 seconds per run and the GB-GM-MCTS in 90 seconds (1000 traversals) to 9 minutes (5000 traversals).

Artifacts

Paper Information

Citation: Jensen, J. H. (2019). A graph-based genetic algorithm and generative model/Monte Carlo tree search for the exploration of chemical space. Chemical Science, 10(12), 3567-3572. https://doi.org/10.1039/c8sc05372c

Publication: Chemical Science (Royal Society of Chemistry), 2019

Additional Resources:

• GB-GA Code (GitHub)
• GB-GM Code (GitHub)

Citation

@article{jensen2019graph,
  title={A graph-based genetic algorithm and generative model/Monte Carlo tree search for the exploration of chemical space},
  author={Jensen, Jan H.},
  journal={Chemical Science},
  volume={10},
  number={12},
  pages={3567--3572},
  year={2019},
  publisher={Royal Society of Chemistry},
  doi={10.1039/c8sc05372c}
}

This is a Position paper that argues genetic algorithms (GAs) are underused and underappreciated as baselines in the molecular generation community. The primary contribution is empirical evidence that a simple GA implementation (MOL_GA) matches or outperforms many sophisticated deep learning methods on standard benchmarks. The authors propose the “GA criterion” as a minimum bar for evaluating new molecular generation algorithms.

Why Molecular Generation May Be Easier Than Assumed

Drug discovery is fundamentally a molecular generation task, and many machine learning methods have been proposed for it (Du et al., 2022). The problem has many variants, from unconditional generation of novel molecules to directed optimization of specific molecular properties.

The authors observe that generating valid molecules is, in some respects, straightforward. The rules governing molecular validity are well-defined bond constraints that can be checked using standard cheminformatics software like RDKit. This means new molecules can be generated simply by adding, removing, or substituting fragments of known molecules. When applied iteratively, this is exactly what a genetic algorithm does. Despite this, many papers in the field propose complex deep learning methods without adequately comparing to simple GA baselines.

The GA Criterion for Evaluating New Methods

The core proposal is the GA criterion: new methods in molecular generation should offer some clear advantage over genetic algorithms. This advantage can be:

• Empirical: outperforming GAs on relevant benchmarks
• Conceptual: identifying and overcoming a specific limitation of randomly modifying known molecules

The authors argue that the current state of molecular generation research reflects poor empirical practices, where comprehensive baseline evaluation is treated as optional rather than essential.

Genetic Algorithm Framework and Benchmark Experiments

How Genetic Algorithms Work for Molecules

GAs operate through the following iterative procedure:

1. Start with an initial population $P$ of molecules
2. Sample a subset $S \subseteq P$ from the population (possibly biased toward better molecules)
3. Generate new molecules $N$ from $S$ via mutation and crossover operations
4. Select a new population $P’$ from $P \cup N$ (e.g., keep the highest-scoring molecules)
5. Set $P \leftarrow P’$ and repeat from step 2

The MOL_GA implementation uses:

• Quantile-based sampling (step 2): molecules are sampled from the top quantiles of the population using a log-uniform distribution over quantile thresholds:

$$ u \sim \mathcal{U}[-3, 0], \quad \epsilon = 10^{u} $$

A molecule is drawn uniformly from the top $\epsilon$ fraction of the population.

• Mutation and crossover (step 3): graph-based operations from Jensen (2019), as implemented in the GuacaMol benchmark (Brown et al., 2019)
• Greedy population selection (step 4): molecules with the highest scores are retained

Unconditional Generation on ZINC 250K

The first experiment evaluates unconditional molecule generation, where the task is to produce novel, valid, and unique molecules distinct from a reference set (ZINC 250K). Success is measured by validity, novelty (at 10,000 generated molecules), and uniqueness.

All methods perform near 100% on all metrics, demonstrating that unconditional molecule generation is not a particularly discriminative benchmark. The authors note that generation speed (molecules per second) is an important missing dimension from these comparisons, where simple methods like GAs have a clear advantage.

Molecule Optimization on the PMO Benchmark

The second experiment evaluates directed molecule optimization on the Practical Molecular Optimization (PMO) benchmark (Gao et al., 2022), which measures the ability to find molecules optimizing a scalar objective function $f: \mathcal{M} \mapsto \mathbb{R}$ with a budget of 10,000 evaluations.

A key insight is that previous GA implementations in PMO used large generation sizes ($\approx 100$), which limits the number of improvement iterations. The authors set the generation size to 5, allowing approximately 2,000 iterations of improvement within the same evaluation budget.

MOL_GA achieves the highest aggregate score across all 23 PMO tasks, outperforming both the previous best GA (Graph GA) and the previous best overall method (REINVENT). The authors attribute this partly to the tuning of the baselines in PMO rather than MOL_GA being an especially strong method, since MOL_GA is essentially the same algorithm as Graph GA with different hyperparameters.

Implications for Molecular Generation Research

The key findings and arguments are:

1. GAs match or outperform deep learning methods on standard molecular generation benchmarks, both for unconditional generation and directed optimization.

2. Hyperparameter choices matter significantly: MOL_GA’s strong performance on PMO comes partly from using a smaller generation size (5 vs. ~100), which allows more iterations of refinement within the same evaluation budget.

3. The GA criterion should be enforced in peer review: new molecular generation methods should demonstrate a clear advantage over GAs, whether empirical or conceptual.

4. Deep learning methods may implicitly do what GAs do explicitly: many generative models are trained on datasets of known molecules, so the novel molecules they produce may simply be variants of their training data. The authors consider this an important direction for future investigation.

5. Poor empirical practices are widespread: the paper argues that many experiments in molecule generation are conducted with an explicit desired outcome (that the novel algorithm is the best), leading to inadequate baseline comparisons.

The authors are careful to note that this result should not be interpreted as GAs being exceptional algorithms. Rather, it is an indication that more complex methods have made surprisingly little progress beyond what simple heuristic search can achieve.

Reproducibility Details

Data

Algorithms

• GA implementation: MOL_GA package, using graph-based mutation and crossover from Jensen (2019) via the GuacaMol implementation
• Generation size: 5 molecules per iteration (allowing ~2,000 iterations with 10,000 evaluations)
• Population selection: Greedy (highest-scoring molecules retained)
• Sampling: Quantile-based with log-uniform distribution over quantile thresholds

Evaluation

Hardware

The paper does not specify hardware requirements. Given that GAs are computationally lightweight compared to deep learning methods, standard CPU hardware is likely sufficient.

Artifacts

Paper Information

Citation: Tripp, A., & Hernández-Lobato, J. M. (2023). Genetic algorithms are strong baselines for molecule generation. arXiv preprint arXiv:2310.09267. https://arxiv.org/abs/2310.09267

Publication: arXiv preprint, 2023

Additional Resources:

• MOL_GA Python Package (GitHub)
• MOL_GA on PyPI

Citation

@article{tripp2023genetic,
  title={Genetic algorithms are strong baselines for molecule generation},
  author={Tripp, Austin and Hern{\'a}ndez-Lobato, Jos{\'e} Miguel},
  journal={arXiv preprint arXiv:2310.09267},
  year={2023}
}

Aligning two sets of corresponding points, finding the optimal rotation (and optionally translation and scale) that maps one onto the other, is a fundamental operation across scientific computing. It appears in molecular dynamics (superimposing protein conformations), robotics (sensor registration), and computer vision (shape matching). The two dominant algorithm families are the Kabsch (SVD-based) method and the Horn (quaternion-based) method.

The Kabsch-Horn Cookbook is a Python library that implements both algorithm families across five numerical frameworks: NumPy, PyTorch, JAX, TensorFlow, and MLX. Every backend shares the same API, supports N-dimensional point sets, per-point weights, and arbitrary batch dimensions. The PyTorch, JAX, TensorFlow, and MLX backends are fully differentiable, with custom autograd rules that bypass the numerically unstable gradient of the standard SVD near degenerate singular values.

Features

Algorithms

• Kabsch: SVD-based optimal rotation for rigid alignment
• Kabsch-Umeyama: Kabsch with an additional optimal scaling factor $c$, solving $Q \approx cRP + t$
• Horn: Quaternion-based optimal rotation via the eigendecomposition of a $4 \times 4$ key matrix
• Horn + Scale: Horn’s method extended with optimal isotropic scaling
• RMSD Wrappers: Convenience functions that return RMSD directly alongside the alignment parameters

Framework Support

torch.compile and jax.jit are the tested compile/JIT paths. MLX supports 3D inputs only; the Kabsch (SVD) path is N-dimensional on the other four backends.

Numerical Robustness

Standard SVD and eigendecomposition backward passes produce NaN gradients when singular values collide or are near-zero. The library provides custom autograd primitives to handle these cases:

• SafeSVD (PyTorch, JAX, TF, MLX): Custom backward pass that clamps the singular value gap, preventing division-by-zero in the gradient
• SafeEigh (PyTorch, JAX, TF, MLX): Analogous safe backward for the symmetric eigendecomposition used in Horn’s method
• Per-point weights: Weighted centroids and weighted cross-covariance for mass-weighted or confidence-weighted alignment
• Batch dimensions: All functions broadcast over leading batch dimensions without explicit loops
• Mixed-dtype promotion: Inputs are promoted to a common floating-point dtype automatically

Testing

The test suite uses Hypothesis-based property testing across 13 modules covering:

• Round-trip correctness (align then compare)
• Gradient finiteness and correctness (finite-difference checks)
• Reflection handling (proper vs. improper rotations)
• Weighted alignment consistency
• Batch broadcasting
• 4 differentiable backends $\times$ 4 precisions (float32, float64, and where supported, float16, bfloat16)

Usage

This is a reference cookbook, so you can copy the framework folder you need from src/kabsch_horn/<framework>/ directly into your project (the code has no runtime dependencies beyond the framework itself). To depend on it instead, install a pinned version from GitHub:

pip install "git+https://github.com/hunter-heidenreich/Kabsch-Cookbook.git@v0.4.1"

Basic alignment with NumPy:

import numpy as np
from kabsch_horn import numpy as kh

# Two sets of corresponding 3D points
P = np.random.randn(100, 3)
R_true = np.linalg.qr(np.random.randn(3, 3))[0]  # random rotation matrix
Q = (P @ R_true.T) + np.random.randn(1, 3)

R, t, rmsd = kh.kabsch(P, Q)
aligned = P @ R.T + t

RMSD loss for training in PyTorch:

import torch
from kabsch_horn import pytorch as kh

pred_coords = model(input_features)   # (B, N, 3), requires_grad=True
target_coords = batch["target"]       # (B, N, 3)

rmsd = kh.kabsch_rmsd(pred_coords, target_coords)  # (B,)
loss = rmsd.mean()
loss.backward()  # safe gradients via SafeSVD

For the full API reference and additional examples, see the documentation site.

Results

Gradient Stability

The standard SVD backward pass computes terms of the form $\frac{1}{\sigma_i^2 - \sigma_j^2}$, which diverges when two singular values are close. In molecular alignment this happens frequently: planar molecules, symmetric structures, and noisy coordinates can all produce near-degenerate singular values. The SafeSVD primitive floors the magnitude of that denominator at the dtype’s machine epsilon (finfo(dtype).eps), producing finite (if slightly biased) gradients in these edge cases. Property-based tests confirm that gradients remain finite across thousands of random rotations, scales, and noise levels for all four differentiable backends.

Framework Parity

All five backends produce numerically equivalent results (up to floating-point tolerance) on the same inputs. The shared API means switching from NumPy prototyping to PyTorch training requires changing only the import path.

Related Work

This project builds on the foundational alignment algorithms described in these papers:

• Kabsch (1976): the original SVD-based rotation alignment
• Arun et al. (1987): SVD formulation for 3D point set fitting
• Horn (1987): quaternion-based closed-form absolute orientation
• Horn et al. (1988): orthonormal matrix (polar decomposition) approach
• Umeyama (1991): extension to include optimal scaling

For a detailed walkthrough of the Kabsch algorithm with code examples, see the companion blog post: The Kabsch Algorithm.

This Method paper addresses a specific failure mode in prior SVD-based solutions to the point set registration problem. Both Arun et al. (1987) and Horn, Hilden, and Negahdaripour (1988) presented SVD-based methods for finding the optimal rotation between two point patterns. (Note: this is a different paper from Horn’s 1987 quaternion method, which does not suffer from this issue.) These SVD-based methods can produce a reflection ($\det(R) = -1$) instead of a proper rotation when the data is severely corrupted. Umeyama provides a corrected formulation that always yields a proper rotation matrix.

The Similarity Transformation Problem

Given two point sets ${\mathbf{x}_i}$ and ${\mathbf{y}_i}$ ($i = 1, \ldots, n$) in $m$-dimensional space, find the similarity transformation parameters (rotation $R$, translation $\mathbf{t}$, and scale $c$) minimizing the mean squared error:

$$ e^2(R, \mathbf{t}, c) = \frac{1}{n} \sum_{i=1}^{n} \lVert \mathbf{y}_i - (cR\mathbf{x}_i + \mathbf{t}) \rVert^2 $$

This generalizes the Kabsch problem (rotation only) and the absolute orientation problem (rotation + translation + scale) to arbitrary dimensions $m$.

The Core Lemma: Corrected SVD Rotation

The key contribution is a lemma for finding the rotation $R$ minimizing $\lVert A - RB \rVert^2$. Given the SVD of $AB^T = UDV^T$ (with $d_1 \geq d_2 \geq \cdots \geq d_m \geq 0$), define the correction matrix:

$$ S = \begin{cases} I & \text{if } \det(AB^T) \geq 0 \\ \operatorname{diag}(1, 1, \ldots, 1, -1) & \text{if } \det(AB^T) < 0 \end{cases} $$

The minimum value is:

$$ \min_{R} \lVert A - RB \rVert^2 = \lVert A \rVert^2 + \lVert B \rVert^2 - 2\operatorname{tr}(DS) $$

When $\operatorname{rank}(AB^T) \geq m - 1$, the optimal rotation is uniquely determined as:

$$ R = USV^T $$

The critical insight is that when $\det(AB^T) = 0$ (i.e., $\operatorname{rank}(AB^T) = m - 1$), the matrix $S$ must instead be chosen based on $\det(U)\det(V)$:

$$ S = \begin{cases} I & \text{if } \det(U)\det(V) = 1 \\ \operatorname{diag}(1, 1, \ldots, 1, -1) & \text{if } \det(U)\det(V) = -1 \end{cases} $$

This handles the degenerate case where the sign of $\det(AB^T)$ is unreliable.

Complete Similarity Transformation Solution

Umeyama derives the full solution using centered coordinates and the covariance matrix $\Sigma_{xy} = \frac{1}{n} \sum_i (\mathbf{y}_i - \boldsymbol{\mu}_y)(\mathbf{x}_i - \boldsymbol{\mu}_x)^T$.

Given the SVD $\Sigma_{xy} = UDV^T$:

Rotation:

$$ R = USV^T $$

Scale:

$$ c = \frac{1}{\sigma_x^2} \operatorname{tr}(DS) $$

Translation:

$$ \mathbf{t} = \boldsymbol{\mu}_y - cR\boldsymbol{\mu}_x $$

Minimum error:

$$ \varepsilon^2 = \sigma_y^2 - \frac{\operatorname{tr}(DS)^2}{\sigma_x^2} $$

where $\sigma_x^2$ and $\sigma_y^2$ are the variances of the respective point sets around their centroids.

Why Prior Methods Fail

The methods of Arun et al. and Horn et al. use $R = UV^T$ directly from the SVD. This works when $\det(UV^T) = 1$ (proper rotation). When $\det(UV^T) = -1$, these methods either produce a reflection or apply an ad hoc correction (flipping the sign of the last column of $U$). Umeyama shows that the correct fix depends on $\det(\Sigma_{xy})$:

• If $\det(\Sigma_{xy}) \geq 0$: set $S = I$, so $R = UV^T$
• If $\det(\Sigma_{xy}) < 0$: set $S = \operatorname{diag}(1, \ldots, 1, -1)$, flipping the last singular value’s contribution

This distinction matters because corrupted data can make $\det(UV^T) = -1$ even when the true transformation is a proper rotation. Simply flipping a column of $U$ does not always yield the correct least-squares solution.

Generality

The formulation works for any dimension $m$, covering both 2D and 3D registration problems. The proof uses Lagrange multipliers with explicit enforcement of both orthogonality ($R^T R = I$) and the proper rotation constraint ($\det(R) = 1$), which prior methods enforced only partially.

Paper Information

Citation: Umeyama, S. (1991). Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(4), 376-380. https://doi.org/10.1109/34.88573

Publication: IEEE TPAMI, 1991

Additional Resources:

• Kabsch Algorithm: NumPy, PyTorch, TensorFlow, and JAX (tutorial with implementations including the Kabsch-Umeyama scaling extension)
• Kabsch-Horn Cookbook (a differentiable, gradient-safe implementation of Kabsch, Horn, and Umeyama alignment across NumPy, PyTorch, JAX, TensorFlow, and MLX)

@article{umeyama1991least,
  title={Least-squares estimation of transformation parameters between two point patterns},
  author={Umeyama, Shinji},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  volume={13},
  number={4},
  pages={376--380},
  year={1991},
  publisher={IEEE},
  doi={10.1109/34.88573}
}

This Method paper presents a closed-form solution to the absolute orientation problem using $3 \times 3$ orthonormal matrices directly, complementing Horn’s earlier quaternion-based solution (1987). The authors note that while quaternions are more elegant, orthonormal matrices are more widely used in photogrammetry, graphics, and robotics. The solution relies on the polar decomposition of the cross-covariance matrix via its matrix square root.

The paper also compares two approaches: (1) directly finding the best-fit orthonormal matrix (the main result), and (2) finding an unconstrained best-fit linear transformation and then projecting it onto the nearest orthonormal matrix. These give different results, and only the first approach has the desired symmetry property.

The Rotation via Polar Decomposition

As in the quaternion paper, the problem reduces to finding the orthonormal matrix $R$ maximizing $\operatorname{Tr}(R^T M)$, where $M = \sum_{i=1}^{n} \mathbf{r}’_{r,i} (\mathbf{r}’_{l,i})^T$ is the cross-covariance matrix of the centered point sets.

The key insight is the polar decomposition: any matrix $M$ can be written as:

$$ M = U S $$

where $U$ is orthonormal and $S = (M^T M)^{1/2}$ is positive semidefinite. When $M$ is nonsingular:

$$ U = M (M^T M)^{-1/2} $$

The matrix square root $(M^T M)^{1/2}$ is computed via eigendecomposition. If $M^T M$ has eigenvalues $\lambda_1, \lambda_2, \lambda_3$ and eigenvectors $\hat{\mathbf{u}}_1, \hat{\mathbf{u}}_2, \hat{\mathbf{u}}_3$:

$$ (M^T M)^{1/2} = \sqrt{\lambda_1} , \hat{\mathbf{u}}_1 \hat{\mathbf{u}}_1^T + \sqrt{\lambda_2} , \hat{\mathbf{u}}_2 \hat{\mathbf{u}}_2^T + \sqrt{\lambda_3} , \hat{\mathbf{u}}_3 \hat{\mathbf{u}}_3^T $$

The sign of $\det(U)$ equals the sign of $\det(M)$, so $U$ is a proper rotation when $\det(M) > 0$ and a reflection when $\det(M) < 0$.

Handling the Coplanar Case

When one set of measurements is coplanar, $M$ is singular ($\operatorname{rank}(M) = 2$) and one eigenvalue of $M^T M$ is zero. The matrix square root still exists (positive semidefinite rather than positive definite), but $S$ is no longer invertible.

In this case, $U$ is determined only for two of its three columns. The third column (corresponding to the zero eigenvalue) is fixed by the orthonormality constraint, with a sign ambiguity resolved by requiring $\det(U) = +1$ (proper rotation).

The Nearest Orthonormal Matrix (Alternative Approach)

The paper also derives a closed-form solution for finding the orthonormal matrix nearest to an arbitrary matrix $A$ (minimizing $\lVert A - R \rVert^2$). This uses the same polar decomposition machinery: if $A = U_A S_A$, then $U_A$ is the nearest orthonormal matrix.

This approach (find unconstrained best-fit transform, then project to nearest orthonormal matrix) was used by some earlier methods. Horn et al. show it gives a different result from the direct least-squares solution and lacks the symmetry property: the inverse transformation from right-to-left is generally not the exact inverse of the left-to-right solution.

Relationship to Other Methods

The polar decomposition approach (this paper) and the SVD approach (Arun et al.) are closely related: the SVD $M = U \Lambda V^T$ gives the polar decomposition as $M = (UV^T)(V \Lambda V^T)$ where $UV^T$ is the orthonormal factor and $V \Lambda V^T$ is the positive semidefinite factor. Both methods can produce reflections under noisy data, which Umeyama (1991) later addressed.

Paper Information

Citation: Horn, B. K. P., Hilden, H. M., & Negahdaripour, S. (1988). Closed-form solution of absolute orientation using orthonormal matrices. Journal of the Optical Society of America A, 5(7), 1127-1135. https://doi.org/10.1364/josaa.5.001127

Publication: Journal of the Optical Society of America A, 1988

Additional Resources:

• Kabsch Algorithm: NumPy, PyTorch, TensorFlow, and JAX (tutorial with differentiable implementations)
• Kabsch-Horn Cookbook (a differentiable, gradient-safe implementation of Kabsch, Horn, and Umeyama alignment across NumPy, PyTorch, JAX, TensorFlow, and MLX)

@article{horn1988closed,
  title={Closed-form solution of absolute orientation using orthonormal matrices},
  author={Horn, Berthold K. P. and Hilden, Hugh M. and Negahdaripour, Shahriar},
  journal={Journal of the Optical Society of America A},
  volume={5},
  number={7},
  pages={1127--1135},
  year={1988},
  publisher={Optica Publishing Group},
  doi={10.1364/josaa.5.001127}
}

This Method paper presents a concise algorithm for finding the least-squares rotation and translation between two 3D point sets using the singular value decomposition (SVD) of a $3 \times 3$ cross-covariance matrix. The approach is closely related to the earlier Kabsch algorithm (1976), which used eigendecomposition, and was developed independently of Horn’s quaternion method (1987). The paper also identifies a reflection degeneracy that Umeyama later provided a complete fix for.

Problem Formulation

Given two 3D point sets ${p_i}$ and ${p’_i}$ ($i = 1, \ldots, N$) related by:

$$ p’_i = R p_i + T + N_i $$

where $R$ is a rotation matrix, $T$ is a translation vector, and $N_i$ is noise, find $\hat{R}$ and $\hat{T}$ minimizing:

$$ \Sigma^2 = \sum_{i=1}^{N} \lVert p’_i - (R p_i + T) \rVert^2 $$

Decoupling Translation and Rotation

The translation is eliminated by centering both point sets at their centroids $p$ and $p’$. Defining centered coordinates $q_i = p_i - p$ and $q’_i = p’_i - p’$, the problem reduces to:

$$ \Sigma^2 = \sum_{i=1}^{N} \lVert q’_i - R q_i \rVert^2 $$

Once $\hat{R}$ is found, the translation follows as $\hat{T} = p’ - \hat{R} p$.

The SVD Algorithm

The algorithm proceeds in five steps:

1. Center both point sets by subtracting centroids
2. Compute the $3 \times 3$ cross-covariance matrix: $H = \sum_{i=1}^{N} q_i q’^t_i$
3. Compute the SVD: $H = U \Lambda V^t$
4. Form the candidate rotation: $X = V U^t$
5. Check $\det(X)$: if $+1$, then $\hat{R} = X$; if $-1$, the result is a reflection

The key insight is that minimizing $\Sigma^2$ is equivalent to maximizing $\operatorname{Trace}(RH)$. Using a lemma based on the Cauchy-Schwarz inequality, Arun et al. show that $X = VU^t$ maximizes this trace over all orthonormal matrices.

The Reflection Problem

When $\det(VU^t) = -1$, the SVD produces a reflection rather than a proper rotation. Arun et al. analyze three cases:

Noiseless, non-coplanar points: The SVD always gives a proper rotation ($\det = +1$). No issue arises.

Coplanar points (including $N = 3$): One singular value of $H$ is zero. Both a rotation and a reflection achieve $\Sigma^2 = 0$. The fix is to flip the sign of the column of $V$ corresponding to the zero singular value:

$$ V’ = [v_1, v_2, -v_3], \quad X’ = V’ U^t $$

Noisy, non-coplanar points with $\det = -1$: The paper acknowledges this case cannot be handled by the algorithm. The reflection genuinely minimizes $\Sigma^2$ over all orthonormal matrices, meaning no rotation achieves a lower error. The authors suggest this only occurs with very large noise and recommend RANSAC-like approaches.

This last case is precisely what Umeyama (1991) later resolved with a corrected formulation using a sign matrix $S$ conditioned on $\det(\Sigma_{xy})$.

Computational Comparison

The paper includes VAX 11/780 benchmarks comparing three methods:

The SVD and quaternion methods have comparable speed, both significantly faster than the iterative approach. SVD becomes faster than quaternion for larger point sets since its core computation operates on a $3 \times 3$ matrix regardless of $N$.

Paper Information

Citation: Arun, K. S., Huang, T. S., & Blostein, S. D. (1987). Least-Squares Fitting of Two 3-D Point Sets. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-9(5), 698-700. https://doi.org/10.1109/TPAMI.1987.4767965

Publication: IEEE TPAMI, 1987

Additional Resources:

• Kabsch Algorithm: NumPy, PyTorch, TensorFlow, and JAX (tutorial with differentiable implementations)
• Kabsch-Horn Cookbook (a differentiable, gradient-safe implementation of Kabsch, Horn, and Umeyama alignment across NumPy, PyTorch, JAX, TensorFlow, and MLX)

@article{arun1987least,
  title={Least-Squares Fitting of Two 3-D Point Sets},
  author={Arun, K. S. and Huang, T. S. and Blostein, S. D.},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  volume={PAMI-9},
  number={5},
  pages={698--700},
  year={1987},
  publisher={IEEE},
  doi={10.1109/TPAMI.1987.4767965}
}

This is a Method paper. It introduces Latent Diffusion Models (LDMs), which train denoising diffusion models in the latent space of pretrained autoencoders rather than directly in pixel space. The key insight is that separating perceptual compression from generative learning enables high-resolution image synthesis at a fraction of the computational cost of pixel-based diffusion. The paper also introduces a cross-attention conditioning mechanism for flexible multi-modal generation.

Computational Cost of Pixel-Space Diffusion

Training diffusion models directly in pixel space is computationally expensive (150 to 1000 V100 GPU-days for leading models at the time) because the model must process high-dimensional RGB data at every denoising step. Much of this compute is spent modeling imperceptible high-frequency details. The authors observe that learning can be split into two stages: a perceptual compression stage that removes high-frequency detail, and a semantic compression stage where the generative model learns the conceptual composition. Prior two-stage approaches (VQGAN, DALL-E) relied on aggressive compression and autoregressive modeling in discrete latent spaces, trading off reconstruction quality for tractability.

Core Innovation: Diffusion in Latent Space

LDMs decompose image synthesis into two phases:

Phase 1: Perceptual Compression. A pretrained autoencoder (encoder $\mathcal{E}$, decoder $\mathcal{D}$) maps images $x \in \mathbb{R}^{H \times W \times 3}$ to a lower-dimensional latent representation $z = \mathcal{E}(x) \in \mathbb{R}^{h \times w \times c}$ with spatial downsampling factor $f = H/h$. The autoencoder is trained with a perceptual loss (matching deep features from a pretrained VGG network) and a patch-based adversarial objective, with either KL or VQ regularization on the latent space.

Phase 2: Latent Diffusion. A standard denoising diffusion model operates in this latent space. The training objective becomes:

$$L_{\text{LDM}} := \mathbb{E}_{\mathcal{E}(x), \epsilon \sim \mathcal{N}(0,1), t} \left[ \left| \epsilon - \epsilon_\theta(z_t, t) \right|_2^2 \right]$$

where $z_t$ is the noised latent at timestep $t$, and $\epsilon_\theta$ is a time-conditional UNet.

Cross-Attention Conditioning. To enable conditioning on text, semantic maps, or other modalities, the authors introduce cross-attention layers into the UNet. A domain-specific encoder $\tau_\theta$ maps conditioning input $y$ to an intermediate representation $\tau_\theta(y) \in \mathbb{R}^{M \times d_\tau}$, which interacts with the UNet features via:

$$Q = W_Q^{(i)} \cdot \varphi_i(z_t), \quad K = W_K^{(i)} \cdot \tau_\theta(y), \quad V = W_V^{(i)} \cdot \tau_\theta(y)$$

The conditional objective then becomes:

$$L_{\text{LDM}} := \mathbb{E}_{\mathcal{E}(x), y, \epsilon \sim \mathcal{N}(0,1), t} \left[ \left| \epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y)) \right|_2^2 \right]$$

Both $\tau_\theta$ and $\epsilon_\theta$ are optimized jointly.

Experimental Setup and Results

The authors evaluate across multiple tasks and datasets:

Perceptual compression tradeoffs. Downsampling factors $f \in {1, 2, 4, 8, 16, 32}$ are compared on ImageNet class-conditional generation. LDM-1 (pixel-based) trains slowly; LDM-32 loses too much information. LDM-4 and LDM-8 achieve the best balance, with LDM-8 outperforming pixel-based diffusion by 38 FID points after 2M training steps on a single A100.

Unconditional image synthesis on CelebA-HQ 256, FFHQ 256, LSUN Churches/Bedrooms 256: LDM-4 achieves FID 5.11 on CelebA-HQ (state of the art at the time), outperforming LSGM, GANs, and other likelihood-based models. On LSUN-Bedrooms, LDM-4 achieves FID 2.95, close to ADM (1.90) with half the parameters and roughly 4x less training compute (see Appendix E.3.5).

Text-to-image synthesis on MS-COCO: A 1.45B parameter LDM-KL-8 model trained on LAION-400M achieves FID 12.63 with classifier-free guidance (a technique that amplifies the conditioning signal at the cost of diversity, by interpolating between conditional and unconditional predictions) at scale s=1.5, on par with GLIDE (FID 12.24, 6B params) and Make-A-Scene (FID 11.84, 4B params) with substantially fewer parameters.

Class-conditional ImageNet 256: LDM-4-G achieves FID 3.60, IS 247.67, outperforming ADM-G (FID 4.59) with fewer parameters and less compute.

Super-resolution: LDM-4 (big) achieves FID 2.4 on ImageNet 64-to-256 upscaling (validation split), outperforming SR3 in FID.

Inpainting on Places: LDM-4 (big, w/ ft) achieves FID 1.50, setting a new state of the art on image inpainting.

Key Findings and Limitations

• LDM-4 and LDM-8 offer the best tradeoff between perceptual compression and generation quality.
• The autoencoder only needs to be trained once and can be reused across different diffusion models and tasks.
• Cross-attention conditioning generalizes to text, semantic layouts, and bounding boxes without architecture changes.
• Convolutional sampling enables generation at resolutions higher than the training resolution (up to 1024x1024).
• Sequential sampling remains slower than GANs. The autoencoder reconstruction can become a bottleneck for tasks requiring pixel-level precision.

Reproducibility Details

Data

Algorithms

• Autoencoder regularization: KL-reg (KL penalty toward standard normal, weighted by ~$10^{-6}$) or VQ-reg (vector quantization layer on the latent space with a learned codebook)
• Diffusion: Standard DDPM denoising with reweighted objective
• Sampling: DDIM sampler with configurable steps (100 to 500 depending on task)
• Guidance: Classifier-free diffusion guidance with scale $s$ (1.5 for class-conditional and text-to-image quantitative evaluation; 10.0 for qualitative text-to-image samples)

Models

• Autoencoder: Based on VQGAN architecture with perceptual + adversarial loss
• UNet backbone: Time-conditional with cross-attention layers at multiple resolutions
• Text encoder: BERT-tokenizer with transformer $\tau_\theta$ for LAION text-to-image model
• LDM-4-G: 400M parameters, $f=4$ downsampling
• LDM-KL-8 (text): 1.45B parameters, $f=8$ downsampling, KL-regularized

Evaluation

Hardware

• Perceptual compression tradeoff experiments: single NVIDIA A100
• Inpainting model trained on eight V100
• Training at least 2.7x faster than pixel-based diffusion at equal parameters

Artifacts

Paper Information

Citation: Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022. https://arxiv.org/abs/2112.10752

Publication: CVPR 2022

@inproceedings{rombach2022highresolution,
  title     = {High-Resolution Image Synthesis with Latent Diffusion Models},
  author    = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj{\"o}rn},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages     = {10684--10695},
  year      = {2022}
}

Additional Resources:

• GitHub Repository
• Score-Based Generative Modeling with SDEs

This short communication presents a Method paper: a direct, analytical solution to a constrained optimization problem. Given two sets of vectors, Kabsch derives the orthogonal matrix (rotation) that best superimposes one set onto the other by minimizing a weighted sum of squared deviations. Prior approaches either solved an unconstrained problem and factorized the result (Diamond, 1976) or used iterative methods (McLachlan, 1972). Kabsch shows that a direct, non-iterative solution exists despite the non-linear nature of the orthogonality constraint.

The Superposition Problem

The core problem arises frequently in crystallography and structural biology: given two sets of corresponding points (e.g., atomic coordinates from a known structure and experimentally measured coordinates), find the rigid rotation that best aligns them. Translations can be removed by centering both point sets at the origin, leaving only the rotational component.

Formally, given vector sets $\mathbf{x}_n$ and $\mathbf{y}_n$ ($n = 1, 2, \ldots, N$) with weights $w_n$, find the orthogonal matrix $\mathsf{U}$ minimizing:

$$ E = \frac{1}{2} \sum_{n} w_n (\mathsf{U} \mathbf{x}_n - \mathbf{y}_n)^2 $$

subject to orthogonality: $\tilde{\mathsf{U}} \mathsf{U} = \mathsf{I}$.

Derivation via Lagrange Multipliers

Kabsch introduces a symmetric matrix $\mathsf{L}$ of Lagrange multipliers to enforce orthogonality, forming the Lagrangian:

$$ G = E + \frac{1}{2} \sum_{i,j} l_{ij} \left( \sum_{k} u_{ki} u_{kj} - \delta_{ij} \right) $$

Setting $\partial G / \partial u_{ij} = 0$ and defining two key matrices:

$$ r_{ij} = \sum_{n} w_n , y_{ni} , x_{nj} \qquad s_{ij} = \sum_{n} w_n , x_{ni} , x_{nj} $$

where $\mathsf{R} = (r_{ij})$ is the weighted cross-covariance matrix and $\mathsf{S} = (s_{ij})$ is the weighted auto-covariance matrix, the stationarity condition becomes:

$$ \mathsf{U} \cdot (\mathsf{S} + \mathsf{L}) = \mathsf{R} $$

Eigendecomposition Solution

The key insight is that multiplying both sides by their transposes eliminates the unknown $\mathsf{U}$:

$$ (\mathsf{S} + \mathsf{L})(\mathsf{S} + \mathsf{L}) = \tilde{\mathsf{R}} \mathsf{R} $$

Since $\tilde{\mathsf{R}} \mathsf{R}$ is symmetric positive definite, it has positive eigenvalues $\mu_k$ and eigenvectors $\mathbf{a}_k$. The matrix $\mathsf{S} + \mathsf{L}$ shares the same eigenvectors with eigenvalues $\sqrt{\mu_k}$.

From the eigenvectors $\mathbf{a}_k$, a second set of unit vectors $\mathbf{b}_k$ is defined:

$$ \mathbf{b}_k = \frac{1}{\sqrt{\mu_k}} \mathsf{R} , \mathbf{a}_k $$

The optimal rotation matrix is then constructed directly:

$$ u_{ij} = \sum_{k} b_{ki} , a_{kj} $$

Handling Degeneracies and Generalizations

Kabsch addresses two extensions:

1. Planar point sets: When all vectors lie in a plane, one eigenvalue of $\tilde{\mathsf{R}} \mathsf{R}$ is zero. The missing eigenvectors are recovered via cross products: $\mathbf{a}_3 = \mathbf{a}_1 \times \mathbf{a}_2$ and $\mathbf{b}_3 = \mathbf{b}_1 \times \mathbf{b}_2$.

2. General metric constraints: The orthogonality constraint $\tilde{\mathsf{U}} \mathsf{U} = \mathsf{I}$ can be replaced by $\tilde{\mathsf{U}} \mathsf{U} = \mathsf{M}$ for any symmetric positive definite $\mathsf{M}$. By finding any specific solution $\mathsf{B}$ and transforming the input vectors as $\mathbf{x}’_n = \mathsf{B} \mathbf{x}_n$, the problem reduces back to the standard orthogonal case.

The method generalizes naturally to vector spaces of arbitrary dimension.

Legacy and Impact

This two-page communication became one of the most cited papers in structural biology. The “Kabsch algorithm” (or “Kabsch rotation”) is the standard method for computing the root-mean-square deviation (RMSD) between two molecular structures after optimal superposition. It underpins structure comparison tools across crystallography, NMR spectroscopy, cryo-EM, and computational chemistry.

Paper Information

Citation: Kabsch, W. (1976). A solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A, 32(5), 922-923. https://doi.org/10.1107/s0567739476001873

Publication: Acta Crystallographica Section A, 1976

Additional Resources:

• Kabsch Algorithm: NumPy, PyTorch, TensorFlow, and JAX (tutorial with differentiable implementations)
• Kabsch-Horn Cookbook (a differentiable, gradient-safe implementation of Kabsch, Horn, and Umeyama alignment across NumPy, PyTorch, JAX, TensorFlow, and MLX)

@article{kabsch1976solution,
  title={A solution for the best rotation to relate two sets of vectors},
  author={Kabsch, Wolfgang},
  journal={Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography},
  volume={32},
  number={5},
  pages={922--923},
  year={1976},
  publisher={International Union of Crystallography},
  doi={10.1107/s0567739476001873}
}

This Method paper presents a closed-form solution to the absolute orientation problem: given corresponding points measured in two different coordinate systems, find the optimal rotation, translation, and scale that maps one set onto the other. While the Kabsch algorithm (1976) solved the rotation subproblem via eigendecomposition of $\tilde{\mathsf{R}}\mathsf{R}$, Horn’s approach uses unit quaternions to represent rotation, reducing the problem to finding the eigenvector of a $4 \times 4$ symmetric matrix associated with its largest eigenvalue.

The Absolute Orientation Problem

Given $n$ point pairs ${\mathbf{r}_{l,i}}$ and ${\mathbf{r}_{r,i}}$ measured in “left” and “right” coordinate systems, find the transformation:

$$ \mathbf{r}_r = s , R(\mathbf{r}_l) + \mathbf{r}_0 $$

where $s$ is a scale factor, $R$ is a rotation, and $\mathbf{r}_0$ is a translation, minimizing the sum of squared residual errors:

$$ \sum_{i=1}^{n} \lVert \mathbf{r}_{r,i} - s , R(\mathbf{r}_{l,i}) - \mathbf{r}_0 \rVert^2 $$

Prior methods either used iterative numerical procedures or selectively discarded constraints (e.g., Thompson’s and Schut’s three-point methods). Horn derives a direct solution that uses all available information from all points simultaneously.

Decoupling Translation, Scale, and Rotation

Horn shows that the three components of the transformation can be solved sequentially.

Translation: After centering both point sets at their centroids ($\bar{\mathbf{r}}_l$ and $\bar{\mathbf{r}}_r$), the optimal translation is:

$$ \mathbf{r}_0 = \bar{\mathbf{r}}_r - s , R(\bar{\mathbf{r}}_l) $$

Scale: Horn derives three formulations (asymmetric left, asymmetric right, and symmetric). The symmetric version, which ensures the inverse transformation yields the reciprocal scale, is:

$$ s = \left( \frac{\sum_{i=1}^{n} \lVert \mathbf{r}’_{r,i} \rVert^2}{\sum_{i=1}^{n} \lVert \mathbf{r}’_{l,i} \rVert^2} \right)^{1/2} $$

the ratio of root-mean-square deviations from the respective centroids.

Rotation: After removing translation and scale, the remaining problem is to find the rotation $R$ that maximizes:

$$ \sum_{i=1}^{n} \mathbf{r}’_{r,i} \cdot R(\mathbf{r}’_{l,i}) $$

The Quaternion Eigenvector Solution

Horn represents rotation using unit quaternions $\dot{q} = q_0 + i q_x + j q_y + k q_z$ with $\lVert \dot{q} \rVert = 1$. A rotation acts on a vector (represented as a purely imaginary quaternion $\dot{r}$) via the composite product:

$$ \dot{r}’ = \dot{q} , \dot{r} , \dot{q}^* $$

Using the $4 \times 4$ matrix representations of quaternion products, the objective function becomes a quadratic form:

$$ \dot{q}^T N \dot{q} $$

where $N$ is a real symmetric $4 \times 4$ matrix whose elements are combinations of the sums of products $S_{xx}, S_{xy}, \ldots, S_{zz}$ from the $3 \times 3$ cross-covariance matrix $M = \sum_i \mathbf{r}’_{l,i} \mathbf{r}’^T_{r,i}$:

$$ N = \begin{bmatrix} (S_{xx} + S_{yy} + S_{zz}) & S_{yz} - S_{zy} & S_{zx} - S_{xz} & S_{xy} - S_{yx} \\ S_{yz} - S_{zy} & (S_{xx} - S_{yy} - S_{zz}) & S_{xy} + S_{yx} & S_{zx} + S_{xz} \\ S_{zx} - S_{xz} & S_{xy} + S_{yx} & (-S_{xx} + S_{yy} - S_{zz}) & S_{yz} + S_{zy} \\ S_{xy} - S_{yx} & S_{zx} + S_{xz} & S_{yz} + S_{zy} & (-S_{xx} - S_{yy} + S_{zz}) \end{bmatrix} $$

The trace of $N$ is always zero. The unit quaternion maximizing $\dot{q}^T N \dot{q}$ is the eigenvector corresponding to the most positive eigenvalue of $N$.

The Characteristic Polynomial

The eigenvalues satisfy a quartic $\lambda^4 + c_3 \lambda^3 + c_2 \lambda^2 + c_1 \lambda + c_0 = 0$ where:

• $c_3 = 0$ (trace of $N$ is zero, so the four roots sum to zero)
• $c_2 = -2 \operatorname{Tr}(M^T M)$ (always negative, guaranteeing both positive and negative roots)
• $c_1 = -8 \det(M)$
• $c_0 = \det(N)$

When points are coplanar (including the common case of exactly three points), $\det(M) = 0$, so $c_1 = 0$ and the quartic reduces to a biquadratic solvable in closed form.

Coplanar Points and the Three-Point Case

For coplanar measurements, the quartic simplifies to $\lambda^4 + c_2 \lambda^2 + c_0 = 0$, yielding:

$$ \lambda_m = \left[ \frac{1}{2} \left( (c_2^2 - 4c_0)^{1/2} - c_2 \right) \right]^{1/2} $$

Horn also provides a geometric interpretation for the coplanar case: first rotate one plane into the other (about their line of intersection), then solve a 2D least-squares rotation within the shared plane.

Comparison with the Kabsch Algorithm

Both methods solve the same underlying optimization problem but approach it differently:

Horn emphasizes a symmetry property: the inverse transformation should yield exactly the inverse parameters. This holds automatically for the quaternion rotation but requires a specific (symmetric) choice of scale formula.

Paper Information

Citation: Horn, B. K. P. (1987). Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America A, 4(4), 629-642. https://doi.org/10.1364/JOSAA.4.000629

Publication: Journal of the Optical Society of America A, 1987

Additional Resources:

• Kabsch Algorithm: NumPy, PyTorch, TensorFlow, and JAX (tutorial with differentiable implementations of the related SVD-based method)
• Kabsch-Horn Cookbook (a differentiable, gradient-safe implementation of Kabsch, Horn, and Umeyama alignment across NumPy, PyTorch, JAX, TensorFlow, and MLX)

@article{horn1987closed,
  title={Closed-form solution of absolute orientation using unit quaternions},
  author={Horn, Berthold K. P.},
  journal={Journal of the Optical Society of America A},
  volume={4},
  number={4},
  pages={629--642},
  year={1987},
  publisher={Optica Publishing Group},
  doi={10.1364/josaa.4.000629}
}

Citation: Wang, H., Yu, Y., & Liu, J.-C. (2026). GraphReco: Probabilistic Structure Recognition for Chemical Molecules. ChemistryOpen, e202500537. https://doi.org/10.1002/open.202500537

Publication: ChemistryOpen 2026 (Open Access)

A Rule-Based OCSR System with Probabilistic Graph Assembly

GraphReco tackles a challenge that is rarely addressed explicitly in rule-based OCSR: the ambiguity that arises during graph assembly when lower-level component extraction results are imprecise. Small deviations in bond endpoint locations, false positive detections, and spatial proximity between elements all create uncertainty about which atoms and bonds should be connected, merged, or discarded.

The system introduces two main contributions:

1. Fragment Merging (FM) line detection: An adaptive three-stage algorithm for precise bond line identification across images of variable resolution
2. Probabilistic ambiguity resolution: A Markov network that infers the most likely existence and merging state of atom and bond candidates

Three-Stage Pipeline

GraphReco follows a three-stage workflow:

1. Component Extraction: Detects circles (aromatic bonds), bond lines (via the FM algorithm), and chemical symbols (via Tesseract OCR). Includes detection of solid wedge, dashed wedge, dashed line, and wavy bond styles. A semi-open-loop correction step resolves cases where symbols are misclassified as bonds and vice versa.

2. Atom and Bond Ambiguity Resolution: Creates atom and bond candidates from detected components, builds a Markov network to infer their most probable states, and resolves candidates through existence and merging decisions.

3. Graph Reconstruction: Assembles resolved atoms and bonds into a molecule graph, selects the largest connected component, and exports as MDL Molfile.

Fragment Merging Line Detection

Classical Line Hough Transform (LHT) struggles with chemical structure images because bond lines suffer from pixelization, and algorithm parameters that work for one image resolution fail at others. The FM algorithm addresses this with three stages:

1. Fragment extraction: Apply LHT with high-resolution parameters (resolution $r = 2$, resolution $\theta = 2°$) to detect fine line fragments. Walk along detected theoretical lines to find actual black pixels and group them by connectivity.

2. Fragment grouping: Pair fragments that share similar angles, are close in the perpendicular direction, and are either overlapping or connected by a path of black pixels.

3. Fragment merging: Merge grouped fragments into single line segments using the two border pixels farthest from the centroid.

The FM algorithm effectively handles the tradeoff that plagues standard LHT: coarse parameters miss short lines and produce overlaps, while fine parameters return many fragments shorter than actual bonds.

Probabilistic Ambiguity Resolution via Markov Network

After component extraction, GraphReco creates atom and bond candidates rather than directly assembling the graph. Each bond endpoint generates an atom candidate with a circular bounding area of radius:

$$r_b = \min(l_{\text{bond}}, l_{\text{med}}) / 4$$

where $l_{\text{bond}}$ is the bond length and $l_{\text{med}}$ is the median bond length.

A Markov network is constructed with four types of nodes:

• Atom nodes: Boolean existence variables for each atom candidate
• Bond nodes: Boolean existence variables for each bond candidate
• Atom merge nodes: Boolean variables for pairs of overlapping atom candidates
• Bond merge nodes: Boolean variables for pairs of nearby bond candidates

Potential functions encode rules about when candidates should exist or merge, with merging likelihood between two bond-ending atom candidates defined as a piecewise function of center distance $d$:

$$P(a_1, a_2) = \begin{cases} 0.9, & \text{if } d \leq Q \\ 0.7 - 0.4(d - Q)/(R - Q), & \text{if } Q < d \leq R \\ 0.1, & \text{if } d > R \end{cases}$$

where $Q = \max(r_1, r_2)$ and $R = \min(1.5Q, r_1 + r_2)$. MAP inference determines the final state of all candidates.

Evaluation Results

GraphReco is evaluated on USPTO benchmarks with InChI string comparison (stereochemistry removed):

GraphReco outperforms all rule-based systems and most ML systems, with a particularly large margin on USPTO-10K-Abb (abbreviation-heavy molecules). MolGrapher achieves slightly higher accuracy on the USPTO dataset.

Robustness on Perturbed Images

On USPTO-perturbed (rotation and shearing applied), rule-based methods degrade substantially:

GraphReco performs better than other rule-based systems on perturbed inputs (40.6% vs. under 31%) thanks to its probabilistic assembly, but still falls far behind MolGrapher (86.7%), demonstrating the robustness advantage of learned approaches.

Ablation Study

Each component contributes substantially to overall performance on USPTO-10K:

The FM algorithm and atom candidate mechanism are both critical (accuracy drops below 10% without either). Bond candidates provide a moderate improvement (~15 percentage points), and the Markov network adds ~6 points over hard-threshold alternatives.

Limitations

• Deterministic expert rules limit robustness on perturbed or noisy images, as evidenced by the large accuracy gap with MolGrapher on USPTO-perturbed
• The system relies on Tesseract OCR for symbol recognition, which may struggle with unusual fonts or degraded image quality
• Only handles single 2D molecule structures per image
• Stereochemistry is removed during evaluation, so performance on stereo-bond recognition is not assessed

Reproducibility

GraphReco is implemented in Python and relies on Tesseract OCR, OpenCV, and RDKit. The authors provide an online demo for testing but have not released the source code or a public repository.

Missing components for full reproduction:

• Source code is not publicly available
• No pre-built package or installable library
• Hyperparameters for Markov network potential functions are given in the paper (Equations 8-11), but full implementation details are not released

Hardware/compute requirements: Not specified in the paper. The system uses classical computer vision (Hough transforms, thinning) and probabilistic inference (Markov networks), so GPU hardware is likely not required.

This is a Method paper. It extends denoising diffusion probabilistic models (DDPMs) from continuous to discrete state-spaces by introducing structured Markov transition matrices for the corruption process. The paper unifies several corruption strategies, draws a formal connection between absorbing-state diffusion and masked language models, and demonstrates competitive results on both image and text generation.

Diffusion Beyond Continuous Spaces

Standard DDPMs operate in continuous state-spaces (e.g., pixel values treated as real numbers) and use Gaussian noise for corruption. Many important data types are inherently discrete: text (tokens from a vocabulary), quantized images (discrete pixel values), molecular structures, and segmentation maps. Prior work by Hoogeboom et al. extended binary diffusion to multinomial diffusion with uniform transition probabilities, but this limits the structure of the corruption process. D3PMs generalize this by allowing arbitrary transition matrices that encode domain-specific inductive biases.

Core Innovation: Structured Transition Matrices

D3PMs define a forward corruption process over discrete variables $\mathbf{x} \in {1, \ldots, K}^D$ using transition matrices $\mathbf{Q}_t \in \mathbb{R}^{K \times K}$:

$$q(\mathbf{x}_t | \mathbf{x}_{t-1}) = \text{Cat}(\mathbf{x}_t; \mathbf{p} = \mathbf{x}_{t-1} \mathbf{Q}_t)$$

where $\mathbf{x}_{t-1}$ is a one-hot row vector. The cumulative transition after $t$ steps is $\overline{\mathbf{Q}}_t = \mathbf{Q}_1 \mathbf{Q}_2 \cdots \mathbf{Q}_t$, giving:

$$q(\mathbf{x}_t | \mathbf{x}_0) = \text{Cat}(\mathbf{x}_t; \mathbf{p} = \mathbf{x}_0 \overline{\mathbf{Q}}_t)$$

The paper explores several transition matrix designs:

Uniform diffusion: $[\mathbf{Q}_t]_{ij} = (1 - \beta_t) \mathbf{1}_{i=j} + \beta_t / K$. Transitions with equal probability to any state. Stationary distribution is uniform.

Absorbing state: In absorbing-state diffusion, each non-mask token transitions to the mask state with probability $\beta_t$ per step, while tokens already at the mask state remain there:

$[\mathbf{Q}_t]_{ij} = (1-\beta_t)\mathbf{1}_{i=j\neq m} + \beta_t \mathbf{1}_{j=m} + \mathbf{1}_{i=j=m}$. Each token transitions to a designated absorbing state $m$ (e.g., [MASK] for text, gray pixel for images) with probability $\beta_t$. This establishes a direct connection to masked language models like BERT.

Discretized Gaussian: Transition probabilities decay as a function of the distance $|i-j|$ between states, mimicking Gaussian diffusion on ordinal data like pixel values.

Embedding-based nearest neighbor: For text, transitions are weighted by proximity in a pretrained word embedding space, so corruption preferentially swaps words with semantically similar ones.

Training objective. The reverse process $p_\theta(\mathbf{x}_{t-1} | \mathbf{x}_t)$ is parameterized by predicting $\tilde{p}_\theta(\tilde{\mathbf{x}}_0 | \mathbf{x}_t)$ and computing the posterior:

$$p_\theta(\mathbf{x}_{t-1} | \mathbf{x}_t) \propto \sum_{\tilde{\mathbf{x}}_0} q(\mathbf{x}_{t-1} | \mathbf{x}_t, \tilde{\mathbf{x}}_0) , \tilde{p}_\theta(\tilde{\mathbf{x}}_0 | \mathbf{x}_t)$$

The loss combines the variational lower bound (VLB) with an auxiliary cross-entropy loss $L_\lambda$:

$$L = L_{\text{VLB}} + \lambda , L_{\text{CE}}$$

where $L_{\text{CE}}$ is a reweighted cross-entropy loss on the $\mathbf{x}_0$ prediction that stabilizes training and improves sample quality. The VLB decomposes into per-timestep KL divergences between the true and predicted reverse transitions.

Experiments and Results

Image generation (CIFAR-10):

The best discrete D3PM variant is D3PM Gauss + logistic, which achieves FID 7.34 and NLL 3.44 bpd using the combined $L_\lambda$ loss with a truncated logistic parameterization. The truncated logistic parameterization replaces the standard softmax output with a discretized logistic distribution over pixel values, assigning probability mass to each discrete bin based on a continuous logistic CDF. This provides a smoother output distribution that better captures the ordinal structure of pixel intensities. This variant exceeds the continuous DDPM in log-likelihood (3.44 vs. 3.75 bpd) while approaching its sample quality (FID 7.34 vs. 3.17).

Text generation (text8, character-level, 1000 steps):

Among the D3PM variants and baselines evaluated, D3PM absorbing achieves the best bpc on text8 apart from Discrete Flow (Tran et al., 2019). On LM1B (sentencepiece vocabulary of 8192 tokens), D3PM absorbing achieves a perplexity of 76.9 at 1000 steps, compared to 137.9 for D3PM uniform and 43.6 for a comparable autoregressive transformer, demonstrating that discrete diffusion scales to large vocabularies.

Ablation findings:

• The auxiliary cross-entropy loss $L_\lambda$ is critical: for D3PM Gauss, it improves FID from 15.30 ($L_{\text{VLB}}$) to 8.34 ($L_\lambda$, $\lambda{=}0.001$). Adding the truncated logistic parameterization further improves FID to 7.34.
• Discretized Gaussian transitions outperform both uniform and absorbing-state transitions on CIFAR-10 across all metrics.
• For text, the absorbing-state (mask) model outperforms uniform and nearest-neighbor models. Nearest-neighbor diffusion provides only marginal improvement over uniform, a surprising negative result.
• The $\mathbf{x}_0$-parameterization ensures the learned reverse distribution has the correct sparsity pattern dictated by the transition matrix $\mathbf{Q}_t$.

Findings and Limitations

• The choice of transition matrix is an important design decision that encodes domain-specific inductive biases. Discretized Gaussian transitions work best for ordinal image data; absorbing-state transitions work best for text.
• D3PMs formally unify diffusion models and masked language models: absorbing-state diffusion with a [MASK] token is equivalent to a reweighted BERT-style training objective.
• The combined VLB + auxiliary loss ($L_\lambda$) achieves better density estimation (3.44 bpd) than continuous DDPMs (3.75 bpd) while producing competitive samples.
• Sample quality (best FID 7.34 for D3PM Gauss + logistic) still lags behind continuous-space DDPMs (FID 3.17) on CIFAR-10, though the gap narrows with structured transitions and the auxiliary loss.
• Scaling to very large numbers of categories $K$ requires special techniques (low-rank corruption or matrix exponentials) to manage the $O(K^2 T)$ memory cost of storing transition matrices.

Reproducibility Details

Data

Algorithms

• Noise schedules: Linear schedule for D3PM Gauss, cosine schedule for D3PM uniform, and a novel mutual information schedule for absorbing and nearest-neighbor models
• Reverse parameterization: $\mathbf{x}_0$-parameterization with posterior computation via Bayes’ rule
• Loss: $L_{\text{VLB}} + \lambda L_{\text{CE}}$ with $\lambda = 0.001$ for images and $\lambda = 0.01$ for text absorbing models
• Scaling: Low-rank corruption (absorbing, uniform) scales as $O(r^2 T)$; matrix exponentials for nearest-neighbor transitions

Models

• Image models: Modified U-Net architecture from Ho et al. (2020) adapted for categorical output via softmax over $K$ classes
• Text models: 12-layer T5-style transformer encoder with 70M parameters (12 heads, MLP dim 3072, QKV dim 768)
• Timesteps: $T = 1000$ for both images and text, though text models can be evaluated with fewer steps (e.g., 256 or 20)

Evaluation

Hardware

• All models trained for 1M steps with batch size 512 on TPUv2 or TPUv3
• Text models: 12-layer transformer encoder (T5 architecture), 70M parameters
• Image models: Modified U-Net architecture from Ho et al. (2020)

Artifacts

Paper Information

Citation: Austin, J., Johnson, D. D., Ho, J., Tarlow, D., & van den Berg, R. (2021). Structured Denoising Diffusion Models in Discrete State-Spaces. NeurIPS 2021. https://arxiv.org/abs/2107.03006

Publication: NeurIPS 2021

@inproceedings{austin2021structured,
  title     = {Structured Denoising Diffusion Models in Discrete State-Spaces},
  author    = {Austin, Jacob and Johnson, Daniel D. and Ho, Jonathan and Tarlow, Daniel and van den Berg, Rianne},
  booktitle = {Advances in Neural Information Processing Systems},
  volume    = {34},
  year      = {2021}
}

Additional Resources:

• Score-Based Generative Modeling with SDEs

This is a Method paper. It proposes consistency models, a new class of generative models designed for fast one-step (or few-step) generation. The models can be trained either by distilling pretrained diffusion models (consistency distillation) or as standalone generative models from scratch (consistency training). The paper provides theoretical analysis of both training modes and achieves FID 3.55 on CIFAR-10 for single-step non-adversarial generation (state of the art at the time of publication).

The Slow Sampling Problem in Diffusion

Diffusion models produce high-quality samples but require iterating through many denoising steps (often tens to hundreds), making generation slow compared to GANs or VAEs. Previous approaches to speed up sampling include faster ODE/SDE solvers (DDIM, DPM-Solver) and progressive distillation. These either still require multiple steps or depend on a complex multi-stage distillation pipeline. The goal is a model that can generate high-quality samples in a single forward pass while optionally allowing more steps for better quality.

Core Innovation: The Self-Consistency Property

The key idea builds on the Probability Flow (PF) ODE from the score-based SDE framework. The PF ODE describes a deterministic trajectory that converts noise into data, governed by the learned score function. For the VE-SDE parameterization used by EDM (Karras et al., 2022), this takes the form:

$$\frac{d\mathbf{x}_t}{dt} = -t , s_\phi(\mathbf{x}_t, t)$$

where $s_\phi$ is a pretrained score model, a consistency function $f(\mathbf{x}_t, t)$ maps any point on an ODE trajectory to the trajectory’s origin $\mathbf{x}_\epsilon$. The defining property is self-consistency:

$$f(\mathbf{x}_t, t) = f(\mathbf{x}_{t’}, t’) \quad \text{for all } t, t’ \in [\epsilon, T]$$

for any points $\mathbf{x}_t$ and $\mathbf{x}_{t’}$ on the same PF ODE trajectory.

Parameterization. The model enforces the boundary condition $f(\mathbf{x}_\epsilon, \epsilon) = \mathbf{x}_\epsilon$ using skip connections:

$$f_\theta(\mathbf{x}, t) = c_{\text{skip}}(t) , \mathbf{x} + c_{\text{out}}(t) , F_\theta(\mathbf{x}, t)$$

where $c_{\text{skip}}(\epsilon) = 1$ and $c_{\text{out}}(\epsilon) = 0$, ensuring the boundary condition is satisfied by construction.

Consistency Distillation (CD). Given a pretrained diffusion model, CD trains a consistency model by enforcing self-consistency between adjacent timesteps:

$$\mathcal{L}_{\text{CD}}^N(\theta, \theta^-; \phi) = \mathbb{E}\left[\lambda(t_n) , d!\left(f_\theta(\mathbf{x}_{t_{n+1}}, t_{n+1}), , f_{\theta^-}(\hat{\mathbf{x}}_{t_n}^\phi, t_n)\right)\right]$$

where $\hat{\mathbf{x}}_{t_n}^\phi$ is obtained by running one step of the ODE solver using the pretrained score model, $\theta^-$ is an exponential moving average (EMA) of $\theta$, and $d(\cdot, \cdot)$ is a distance metric. The use of a target network $\theta^-$ (updated via EMA) parallels techniques from deep Q-learning and momentum contrastive learning.

Consistency Training (CT). CT eliminates the need for a pretrained diffusion model. It replaces the ODE solver step with a score estimate derived from the denoising score matching identity:

$$\nabla_{\mathbf{x}_t} \log p_t(\mathbf{x}_t) = \mathbb{E}\left[\frac{\mathbf{x} - \mathbf{x}_t}{t^2} ,\middle|, \mathbf{x}_t\right]$$

Because this identity lets us estimate the score from noisy data alone (without a pretrained model), we can compute the ODE update directly from training samples. This allows training directly on data pairs $(\mathbf{x}, \mathbf{x} + t\mathbf{z})$ where $\mathbf{z} \sim \mathcal{N}(0, I)$.

Theoretical guarantee. If CD achieves zero loss, the consistency model error is bounded by $O((\Delta t)^p)$ where $\Delta t$ is the maximum timestep gap and $p$ is the order of the ODE solver.

Experiments and Benchmarks

Datasets: CIFAR-10 (32x32), ImageNet 64x64, LSUN Bedroom 256x256, LSUN Cat 256x256.

Architecture: All models use the NCSN++/EDM architecture. CD distills from pretrained EDM models.

Key results for consistency distillation (CD):

CD outperforms progressive distillation (PD) across all datasets and sampling steps, with the exception of single-step generation on Bedroom 256x256 where CD with $\ell_2$ slightly underperforms PD with $\ell_2$.

Key results for consistency training (CT):

CT outperforms existing single-step non-adversarial models (VAEs, normalizing flows), e.g., improving over DC-VAE’s FID of 17.90 on CIFAR-10. Samples from CT share structural similarity with EDM samples from the same initial noise, suggesting CT does not suffer from mode collapse.

Zero-shot editing: Consistency models support colorization, super-resolution, inpainting, stroke-guided generation, interpolation, and denoising at test time without task-specific training, by modifying the multi-step sampling algorithm.

Findings and Limitations

• Consistency distillation achieves state-of-the-art FID for one-step generation (3.55 on CIFAR-10, 6.20 on ImageNet 64x64).
• Multi-step sampling provides a smooth quality-compute tradeoff: more steps yield better FID.
• CT produces competitive results without any pretrained diffusion model, making consistency models a standalone generative model family.
• The LPIPS distance metric $d(\cdot, \cdot)$ generally outperforms $\ell_1$ and $\ell_2$ for training consistency models.
• At higher resolutions (LSUN 256x256), the gap between CD/CT and full EDM sampling widens.
• CT currently underperforms CD, suggesting room for improvement in the standalone training paradigm.

Reproducibility Details

Data

Algorithms

• ODE solver for CD: Euler and Heun (2nd order) solvers on the empirical PF ODE
• EMA for target network: Decay rate $\mu$ scheduled as a function of training step
• Schedule functions: $N$ (number of discretization steps) and $\mu$ (EMA rate) increase over training following specific schedules (see Appendix C of the paper)
• Distance metric: LPIPS performs best; $\ell_2$ and $\ell_1$ also evaluated

Models

• Architecture: NCSN++/EDM architecture from Karras et al. (2022)
• CD teacher: Pretrained EDM models
• Parameterization: Skip-connection formulation with $c_{\text{skip}}(t)$ and $c_{\text{out}}(t)$ from EDM

Evaluation

Hardware

• Training details follow EDM conventions
• CD and CT use the same batch sizes and learning rate schedules as EDM training

Artifacts

Paper Information

Citation: Song, Y., Dhariwal, P., Chen, M., & Sutskever, I. (2023). Consistency Models. ICML 2023. https://arxiv.org/abs/2303.01469

Publication: ICML 2023

@inproceedings{song2023consistency,
  title     = {Consistency Models},
  author    = {Song, Yang and Dhariwal, Prafulla and Chen, Mark and Sutskever, Ilya},
  booktitle = {International Conference on Machine Learning},
  volume    = {202},
  year      = {2023},
  url       = {https://arxiv.org/abs/2303.01469}
}

Additional Resources:

• GitHub Repository
• Score-Based Generative Modeling with SDEs

This is a theory paper that takes a reductionist approach to attention mechanisms. It classifies all possible fundamental building blocks of attention (“quarks”) within a formal neural network framework, then proves capacity theorems for circuits built from these primitives using linear and polynomial threshold gates.

Why decompose attention into primitives?

Descriptions of attention in deep learning often seem complex and obscure the underlying neural architecture. Despite the widespread use of attention in transformers and beyond, there has been little formal theory about the computational nature and capacity of attention mechanisms. Baldi and Vershynin address this by identifying the smallest building blocks and rigorously analyzing what they can compute.

The Standard Model and its extensions

The paper defines the “Standard Model” (SM) as the class of all neural networks built from McCulloch-Pitt neurons: directed weighted graphs where neuron $i$ computes $O_i = f_i(S_i)$ with activation $S_i = \sum_j w_{ij} O_j$. The SM already has universal approximation properties, so extensions should be evaluated on efficiency (circuit size, depth, learning), not on what functions can be represented.

Three variable types exist in the SM: activations ($S$), outputs ($O$), and synaptic weights ($w$). Cross these with two mechanisms (addition, multiplication) and the constraint that attending signals originate from neuronal outputs, and you get six possible attention primitives.

The six quarks, reduced to three

The paper shows these reduce to three cases worth studying:

Multiplexing (additive activation attention)

The attending signal $S_2$ is added to the normal activation $S_1$, producing $O_i = f_i(S_1 + S_2)$. With sigmoid or threshold activations, a large negative $S_2$ forces the output to zero regardless of $S_1$, suppressing unattended stimuli. This mechanism lives entirely within the SM and plays a central role in proving capacity lower bounds.

Output gating

Neuron $j$ multiplies the output of neuron $i$, producing $O_i O_j$. This quadratic term is new to the SM. The gated signal $O_i O_j$ propagates to all downstream neurons of $i$. When $O_j \approx 0$, the attended neuron is silenced; when $O_j$ is large, it is enhanced.

Synaptic gating

Neuron $j$ multiplies a synaptic weight $w_{ki}$, creating a dynamic weight $w_{ki} O_j$. This produces the same local term $w_{ki} O_i O_j$ at neuron $k$ as output gating, but affects only the single downstream connection rather than all of neuron $i$’s outputs. Synaptic gating is a fast weight mechanism: the attending network dynamically changes the program executed by the attended network.

Transformers are built entirely from gating

The paper shows that transformer encoder modules decompose into:

1. Output gating ($mn^2$ operations): computing all $n^2$ pairwise dot products of $Q$ and $K$ vectors, each requiring $m$ element-wise multiplications
2. Softmax: a standard SM extension
3. Synaptic gating ($n^2$ operations): weighting $V$ vectors by the softmax outputs to form convex combinations

The entire attention mechanism uses $O(mn^2)$ gating operations. The permutation invariance of transformers follows directly from the weight sharing across input positions.

Relationship to polynomial neural networks

Gating is a special case of polynomial activation. A neuron with full quadratic activation over $n$ inputs has the form $S_i = \sum_{jk} w_{ijk} O_j O_k$, requiring $O(n^2)$ three-way synaptic weights for all possible pairs. Gating introduces only one new quadratic term per operation. The same gating concepts can also be applied to more complex units with polynomial activations of degree $d$, where one polynomial threshold unit gates the output or synapse of another.

Functional properties of gating

Several examples illustrate what gating enables:

• Shaping activation functions: When a unit with activation function $f$ is output-gated by a unit with activation function $g$ (both having the same inputs), the result is $f(S)g(S) = fg(S)$. This changes the effective activation function from $f$ to $fg$. For instance, a linear unit gated by a $(0,1)$ threshold function produces the ReLU activation.
• XOR without hidden layers: The XOR function cannot be computed by a single linear threshold gate. However, gating the OR function by the NAND function (both implementable by single linear threshold gates) produces XOR in a shallow network with no hidden layers.
• Universal approximation: Every continuous function on a compact set can be approximated to arbitrary precision by a shallow attention network of linear units gated by linear threshold gates (Theorem 4.3).

Attention as sparse quadratic terms

Both output and synaptic gating introduce quadratic terms of the form $w_{ki} O_i O_j$. A neuron with full quadratic activation over $n$ inputs would require $O(n^2)$ parameters. Gating introduces only one new quadratic term per operation. This is the key insight: attention mechanisms gain some of the expressiveness of quadratic activations while avoiding the combinatorial parameter explosion.

Capacity results

Using cardinal capacity (the base-2 logarithm of the number of distinct Boolean functions a class of circuits can implement), the paper proves bounds for attentional circuits with linear and polynomial threshold gates:

• Single unit with output gating: a gated pair of linear threshold gates on $n$ inputs has capacity $2n^2(1 + o(1))$, compared to $n^2(1 + o(1))$ for a single gate (Theorem 6.1). This represents a doubling of capacity with a doubling of parameters (from $n$ to $2n$), a sign of efficiency.
• Multiplexing technique: additive activation attention enables a “multiplexing” proof strategy where one unit in a layer is selected as a function of the attending units while driving remaining units to saturation. This is the key tool for proving lower bounds.
• Attention layers: extending to layers of $m$ gated units with $n$ inputs, the capacity is $2mn^2(1 + o(1))$ for output gating (Theorem 7.1), confirming that gating approximately doubles the capacity relative to ungated layers.
• Depth reduction: gating operations available as physical primitives in a neural network can reduce the depth required for certain basic circuits.

Limitations and future work

The authors note several open directions:

• The capacity estimates for some configurations (e.g., single-weight synaptic gating in Proposition 6.9) have gaps between the lower and upper bounds that remain to be tightened.
• The analysis uses Boolean neurons (linear and polynomial threshold gates) as approximations. Extending results to other activation functions (sigmoid, ReLU) is left for future work.
• The paper focuses on single layers and pairs of units. Capacity analysis for deeper attention architectures with multiple stacked layers is not addressed.
• The theory treats attention on the time scale of individual inputs. The paper briefly notes that fast synaptic mechanisms operating on different time scales raise interesting architectural questions but does not develop this direction.

Reproducibility

This is a purely theoretical paper with no associated code, datasets, or pretrained models. All results are mathematical theorems and proofs that can be verified from the paper itself. The paper is freely available on arXiv under a CC BY-NC-ND 4.0 license.

Paper Information

Citation: Baldi, P. & Vershynin, R. (2023). The quarks of attention: Structure and capacity of neural attention building blocks. Artificial Intelligence, 319, 103901.

Publication: Artificial Intelligence 2023

Additional Resources:

• Journal (ScienceDirect)
• arXiv

@article{baldi2023quarks,
  title={The quarks of attention: Structure and capacity of neural attention building blocks},
  author={Baldi, Pierre and Vershynin, Roman},
  journal={Artificial Intelligence},
  volume={319},
  pages={103901},
  year={2023},
  doi={10.1016/j.artint.2023.103901}
}

This is a method paper that introduces the theory and implementation of convolutional neural networks on the sphere. The key contribution is defining spherical cross-correlation that is SO(3)-equivariant and can be computed efficiently using generalized Fast Fourier Transforms from non-commutative harmonic analysis.

Why planar convolutions fail on spherical data

Many problems require analyzing spherical signals: omnidirectional vision for robots and autonomous vehicles, molecular regression, and global weather modeling. A naive approach of projecting spherical data to a plane introduces space-varying distortions that break translational weight sharing. Rotating a spherical signal cannot be emulated by translating its planar projection.

The fundamental issue is geometric: patterns on a plane move via translations, but patterns on a sphere move via 3D rotations. A spherical CNN should detect patterns regardless of how they are rotated over the sphere. The relevant symmetry group is SO(3) (the group of all 3D rotations).

Spherical cross-correlation and the SO(3) output space

The paper defines spherical cross-correlation by replacing filter translations with rotations. For spherical signals $f$ on $S^2$ (the unit sphere) and filter $\psi$, the correlation is:

$$\lbrack\psi \star f\rbrack(R) = \langle L_R \psi, f \rangle = \int_{S^2} \sum_{k=1}^{K} \psi_k(R^{-1}x) f_k(x) , dx$$

where $L_R$ is the rotation operator $\lbrack L_R f\rbrack(x) = f(R^{-1}x)$.

A crucial subtlety: whereas the space of moves for the plane (2D translations) is isomorphic to the plane itself, the space of moves for the sphere (3D rotations) is SO(3), a different three-dimensional manifold. The output of a spherical correlation is therefore a function on SO(3), not on $S^2$. This means subsequent layers must use SO(3) correlation:

$$\lbrack\psi \star f\rbrack(R) = \int_{\text{SO}(3)} \sum_{k=1}^{K} \psi_k(R^{-1}Q) f_k(Q) , dQ$$

Equivariance proof

Equivariance follows from the unitarity of $L_R$ in a single line:

$$\lbrack\psi \star \lbrack L_Q f\rbrack\rbrack(R) = \langle L_R \psi, L_Q f \rangle = \langle L_{Q^{-1}R} \psi, f \rangle = \lbrack\psi \star f\rbrack(Q^{-1}R) = \lbrack L_Q\lbrack\psi \star f\rbrack\rbrack(R)$$

This holds for both $S^2$ and SO(3) correlation.

Efficient computation via generalized FFT

A naive SO(3) correlation is $O(n^6)$. The paper addresses this using the generalized Fourier transform (GFT) from non-commutative harmonic analysis.

The GFT projects functions onto orthogonal basis functions: spherical harmonics $Y_m^l(x)$ for $S^2$, and Wigner D-functions $D_{mn}^l(R)$ for SO(3). Both satisfy generalized Fourier theorems:

• SO(3) convolution theorem: $\widehat{\psi \star f} = \hat{f} \cdot \hat{\psi}^\dagger$ (matrix multiplication of block Fourier coefficients)
• $S^2$ convolution theorem: $\widehat{\psi \star f}^l = \hat{f}^l \cdot \hat{\psi}^{l\dagger}$ (outer product of $S^2$ Fourier coefficient vectors)

The SO(3) FFT works in two steps: (1) standard 2D FFT over the $\alpha$ and $\gamma$ Euler angles, then (2) linear contraction of the $\beta$ axis with precomputed Wigner-d function samples, implemented as a custom GPU kernel.

Experiments

Equivariance error

Since the theory applies to continuous functions but the implementation is discretized, the authors rigorously measure equivariance error. The approximation error grows with resolution and depth but stays manageable for practical bandwidths. With ReLU activations, the error is higher but stays flat across layers, indicating the error comes from feature map rotation (exact only for bandlimited functions) rather than accumulating through the network.

Spherical MNIST

MNIST digits projected onto the sphere, tested in non-rotated (NR) and rotated (R) settings with ~165K parameters per model:

The planar CNN collapses to chance when trained on non-rotated data and tested on rotated data. The spherical CNN maintains strong performance across all settings.

3D shape recognition (SHREC17)

3D meshes projected onto an enclosing sphere via ray casting. For each point on the sphere, a ray is cast toward the origin, collecting three types of information from the intersection: ray length and cos/sin of the surface angle. The same three channels are computed for the convex hull, giving 6 channels total. The network (~1.4M parameters) placed 2nd on recall, mAP, and NDCG, and 3rd on precision and F1 in the SHREC17 competition, competing against methods with highly task-specialized architectures.

Molecular atomization energy (QM7)

Molecules represented as spherical potential functions around each atom (generalizing the Coulomb matrix). A deep ResNet-style $S^2$CNN with DeepSets-style permutation-invariant aggregation over atoms achieved 8.47 RMSE, outperforming all kernel-based approaches and sorted Coulomb matrix methods.

Discussion and future directions

The authors highlight several avenues for future work. For volumetric tasks like 3D model recognition, extending beyond SO(3) to the roto-translation group SE(3) could improve results. They also note that a Steerable CNN for the sphere would enable analysis of vector fields (e.g., global wind directions). Omnidirectional vision is mentioned as a compelling application as 360-degree sensors become more prevalent.

Reproducibility

The official PyTorch implementation is publicly available. The code does not support recent PyTorch versions due to changes in the FFT interface.

Hardware requirements from the paper: the SHREC17 model uses 8GB GPU memory at batch size 16 and takes 50 hours to train. The QM7 model uses 7GB at batch size 20 and takes 3 hours to train. Datasets used (Spherical MNIST, SHREC17, QM7) are all publicly available.

Paper Information

Citation: Cohen, T. S., Geiger, M., Köhler, J., & Welling, M. (2018). Spherical CNNs. International Conference on Learning Representations. https://arxiv.org/abs/1801.10130

Publication: ICLR 2018

Additional Resources:

• OpenReview
• arXiv
• GitHub

@inproceedings{cohen2018spherical,
  title={Spherical {CNNs}},
  author={Cohen, Taco S. and Geiger, Mario and K{\"o}hler, Jonas and Welling, Max},
  booktitle={International Conference on Learning Representations},
  year={2018}
}

This is a method paper that introduces the SE(3)-Transformer, a self-attention mechanism for 3D point clouds and graphs that is equivariant under continuous 3D rotations and translations. It builds on tensor field networks (TFNs) by adding data-dependent attention weights, resolving a known expressiveness limitation of equivariant convolutions.

Why equivariant attention for point clouds?

Point cloud data appears in 3D object scans, molecular structures, and particle simulations. Two properties are essential: handling varying numbers of irregularly sampled points, and invariance to global changes in pose (rotations and translations).

Self-attention handles variable-size inputs naturally and has proven effective across many domains. Tensor field networks provide SE(3)-equivariant convolutions but suffer from a key limitation: their filter kernels are decomposed into learnable radial functions and fixed angular components (spherical harmonics). The angular dependence is completely constrained by the equivariance condition, leaving no learnable degrees of freedom in the angular direction. This has been identified in the literature as severely limiting performance.

The SE(3)-Transformer resolves this by introducing data-dependent attention weights that modulate the angular profile of the kernels while maintaining equivariance.

Architecture: invariant attention meets equivariant values

The core layer combines three components:

$$\mathbf{f}_{\text{out},i}^{\ell} = \underbrace{\mathbf{W}_V^{\ell\ell} \mathbf{f}_{\text{in},i}^{\ell}}_{\text{self-interaction}} + \sum_{k \geq 0} \sum_{j \in \mathcal{N}_i \setminus i} \underbrace{\alpha_{ij}}_{\text{attention}} \underbrace{\mathbf{W}_V^{\ell k}(\mathbf{x}_j - \mathbf{x}_i) \mathbf{f}_{\text{in},j}^k}_{\text{value message}}$$

Invariant attention weights

The attention weights use dot-product attention between equivariant queries and keys:

$$\alpha_{ij} = \frac{\exp(\mathbf{q}_i^\top \mathbf{k}_{ij})}{\sum_{j’ \in \mathcal{N}_i \setminus i} \exp(\mathbf{q}_i^\top \mathbf{k}_{ij’})}$$

Both $\mathbf{q}_i$ and $\mathbf{k}_{ij}$ are constructed using TFN-type linear embeddings, making them SE(3)-equivariant. Their inner product is invariant because SO(3) representations are orthogonal: $\mathbf{q}^\top \mathbf{S}_g^\top \mathbf{S}_g \mathbf{k} = \mathbf{q}^\top \mathbf{k}$.

Equivariant value messages

The value messages use the same TFN kernel structure as tensor field networks: weight kernels $\mathbf{W}_V^{\ell k}(\mathbf{x})$ decomposed into learnable radial functions and Clebsch-Gordan/spherical harmonic angular components. Features are typed by irreducible representation degree $\ell$ (the independent matrix blocks into which SO(3) group actions decompose): type-0 vectors are rotation-invariant scalars, type-1 vectors transform as 3D vectors, and so on.

Angular modulation

The attention weights $\alpha_{ij}$ multiply the value messages, creating data-dependent kernels $\alpha_{ij} \mathbf{W}_V^{\ell k}(\mathbf{x})$. This effectively modulates the angular profile of the fixed spherical harmonic components, adding learnable angular degrees of freedom while preserving equivariance. The authors describe this as one of the first examples of a nonlinear equivariant layer.

Attentive self-interaction

The paper also introduces attentive self-interaction as an alternative to the standard linear self-interaction (analogous to 1x1 convolutions). Instead of fixed learned weights across all points, the weights are generated by an MLP operating on invariant inner products of the input features:

$$w_{i,c’c}^{\ell\ell} = \text{MLP}\left(\bigoplus_{c,c’} \mathbf{f}_{\text{in},i,c’}^{\ell\top} \mathbf{f}_{\text{in},i,c}^{\ell}\right)$$

Experiments

N-body particle simulation

Five charged particles carrying positive or negative charges, exerting repulsive or attractive forces on each other. The task is predicting positions and velocities 500 timesteps ahead. The SE(3)-Transformer achieves 0.0076 MSE on position (vs. 0.0139 for Set Transformer and 0.0151 for TFN), with equivariance error on the order of $10^{-7}$, confirming exact equivariance up to numerical precision.

ScanObjectNN (real-world 3D object classification)

2902 real-world scanned objects across 15 categories. This task is only SO(2)-invariant (gravity axis matters), so the authors provide the z-component as an additional scalar input. With only 128 input points, the SE(3)-Transformer+z achieves 85.0% accuracy, competitive with methods using 1024 points and task-specific architectures. The model learns to ignore the symmetry-breaking z-input when trained on rotation-augmented data.

QM9 molecular property regression

134k molecules with up to 29 atoms, predicting 6 quantum chemical properties. The SE(3)-Transformer achieves competitive results against other equivariant models (TFN, Cormorant), with improvements over TFN on all six targets. Across all three experiments, the SE(3)-Transformer outperforms both a non-equivariant attention baseline (Set Transformer) and equivariant models without attention (TFN).

Practical contributions

The paper includes a PyTorch spherical harmonics implementation that is 10x faster than Scipy on CPU and 100-1000x faster on GPU. For a ScanObjectNN model, this yields roughly 22x speedup of the forward pass compared to the lie-learn library, directly addressing a major bottleneck of TFN-based architectures.

Conclusions and limitations

Adding attention to a roto-translation-equivariant model consistently led to higher accuracy and increased training stability across all three experiments. For large neighbourhoods, attention proved essential for model convergence. The equivariance constraints also improved performance compared to conventional (non-equivariant) attention in all experiments.

The authors note that the SE(3)-Transformer is inherently suited for classification and regression on molecular data and discuss applications in drug research, including early-stage suitability classification of molecules for inhibiting viral reproductive cycles.

Reproducibility

The repository includes code for N-body simulations and QM9 experiments. Hyperparameters and architecture details are provided in the paper’s appendix (4 equivariant layers, representation degrees, channels per degree, learning rates, batch sizes). Hardware requirements are not explicitly stated in the paper.

Paper Information

Citation: Fuchs, F. B., Worrall, D. E., Fischer, V., & Welling, M. (2020). SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks. Advances in Neural Information Processing Systems, 33. https://arxiv.org/abs/2006.10503

Publication: NeurIPS 2020

Additional Resources:

• arXiv
• GitHub

@inproceedings{fuchs2020se3,
  title={{SE(3)-Transformers}: 3D Roto-Translation Equivariant Attention Networks},
  author={Fuchs, Fabian B. and Worrall, Daniel E. and Fischer, Volker and Welling, Max},
  booktitle={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
}

This is a systematization paper that is part position paper, part review, and part unification. It argues that combinatorial generalization, the ability to construct new inferences from known building blocks, is a top priority for AI. It frames relational inductive biases as the key design principle connecting standard deep learning architectures, presents the graph network (GN) as a general framework subsuming prior graph neural network variants, and advocates for combining structured approaches with deep learning rather than choosing between them.

The case for relational inductive biases

Human intelligence relies on representing the world as compositions of entities, relations, and rules. We understand complex systems by decomposing them into parts and their interactions. Modern deep learning’s “end-to-end” philosophy minimizes structural assumptions, relying on data and compute to learn representations from scratch. The paper argues this approach struggles with combinatorial generalization: generalizing beyond one’s experiences by composing known elements in new ways.

The authors reject the false dichotomy between “hand-engineering” and “end-to-end” learning. Just as biology uses both nature and nurture, they advocate for architectures that bake in useful structural assumptions (inductive biases) while still learning flexibly from data.

Inductive biases across standard architectures

The paper provides a systematic analysis of how existing architectures encode relational structure:

Fully connected networks (MLPs): The weakest relational inductive bias. All input units can interact with all others, with no reuse of parameters. No assumptions about the structure of the input.

Convolutional networks (CNNs): Encode locality (nearby elements interact) and translation invariance (the same local function is applied everywhere). The entities are individual units or grid elements (e.g., pixels), the relations are defined by the grid neighborhood, and the rule (convolution kernel) is shared across all positions.

Recurrent networks (RNNs): Encode sequential structure and temporal invariance. The entities are time steps, each step relates to the previous one through a shared transition function. This imposes a Markovian bias (the future depends on the present state, not the full history directly).

Sets and self-attention: Permutation invariant architectures impose no ordering on entities. Self-attention (as in Transformers) allows all pairwise interactions but with no structural prior on which interactions matter.

Each architecture can be understood as making specific commitments about what the entities are, what the relations between them are, and what rules govern their interactions.

The graph network framework

The paper defines a general “graph network” (GN) block that operates on graphs with attributes on nodes, edges, and the global graph level. A GN block performs three update steps and three aggregation steps:

1. Edge update: For each edge, compute updated edge attributes using the current edge attributes, the sender node attributes, the receiver node attributes, and the global attributes
2. Node update: For each node, aggregate incoming updated edge attributes, then compute updated node attributes using the aggregated edges, current node attributes, and global attributes
3. Global update: Aggregate all updated edge and node attributes, then compute updated global attributes

Each update function is learned (typically a small neural network), and each aggregation function must be permutation invariant (typically sum, mean, or max).

This framework generalizes prior work:

• Message Passing Neural Networks (Gilmer et al., 2017): edge and node updates with a readout function but no explicit global attribute in message passing
• Non-local Neural Networks (Wang et al., 2018): attention-weighted edge interactions
• Interaction Networks (Battaglia et al., 2016): physics-inspired message passing
• Relation Networks (Santoro et al., 2017): a simple neural network module for relational reasoning
• Discovering objects and their relations (Raposo et al., 2017): discovering objects and their relations from entangled scene representations
• Deep Sets (Zaheer et al., 2017): node-only aggregation without edge structure
• CommNet, Structure2Vec, GGNNs, and others

The paper shows how each prior approach corresponds to a specific configuration of which GN components are used and how they are connected.

Design principles for graph networks

The paper identifies several key design choices:

Flexible representations: GN blocks can output graphs with different structure than their input (e.g., predicting edge existence), enabling tasks like link prediction, clustering, or property regression.

Configurable within-block structure: The internal update and aggregation functions can be swapped freely. The framework separates what is computed (the relational structure) from how it is computed (the function approximators).

Composable multi-block architectures: GN blocks can be stacked, sharing or not sharing weights across layers. They can be composed with other architectures (e.g., an encoder-GN-decoder pattern) or arranged in recurrent configurations.

Combinatorial generalization: Because GN blocks share functions across edges and nodes, they can generalize to graphs of different sizes and topologies than those seen during training. A GN trained on small graphs can, in principle, be applied to larger ones.

Connections to broader AI themes

The paper frames graph networks as supporting:

• Relational reasoning: Learning about entities and their interactions
• Combinatorial generalization: Applying learned rules to novel combinations
• Structured prediction: Producing complex, structured outputs including graphs and sequences
• Interpretable representations: Graph structure provides a natural vocabulary for understanding what the model has learned

The authors also discuss connections to classical AI (logic, planning, causal reasoning) and argue that graph networks provide a bridge between the flexibility of deep learning and the compositionality of symbolic approaches.

Limitations and open questions

The paper identifies several limitations of graph networks:

• Graph isomorphism: Learned message-passing cannot be guaranteed to discriminate between certain non-isomorphic graphs. Kondor et al. (2018) suggested that covariance, rather than invariance to permutations, may be preferable.
• Expressivity limits of graphs: Notions like recursion, control flow, and conditional iteration are not straightforward to represent with graphs. Programs and more “computer-like” processing may offer greater representational and computational expressivity for these concepts.
• Where do graphs come from?: Converting raw sensory data (images, text) into graph-structured representations remains an open problem. Fully connected graphs between spatial or linguistic entities are a common workaround but may not reflect the true underlying structure.
• Adaptive graph structure: How to modify graph topology during computation (e.g., splitting a node when an object fractures, or adding/removing edges based on contact) is an active research direction.

Reproducibility

The authors released an open-source software library for building graph networks in TensorFlow/Sonnet, including demos for shortest-path finding, sorting, and physical prediction tasks.

This is a position/systematization paper rather than an empirical one, so reproducibility pertains to the accompanying library rather than experimental results. The library and demos are publicly available, making the framework highly accessible.

Paper Information

Citation: Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M., … & Pascanu, R. (2018). Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261.

Publication: arXiv 2018

Additional Resources:

• arXiv
• Graph Nets library (GitHub)

@article{battaglia2018relational,
  title={Relational inductive biases, deep learning, and graph networks},
  author={Battaglia, Peter W. and Hamrick, Jessica B. and Bapst, Victor and Sanchez-Gonzalez, Alvaro and Zambaldi, Vinicius and Malinowski, Mateusz and Tacchetti, Andrea and Raposo, David and Santoro, Adam and Faulkner, Ryan and others},
  journal={arXiv preprint arXiv:1806.01261},
  year={2018}
}

This is a theory paper that provides a principled derivation of gating mechanisms in recurrent neural networks from an axiom of invariance to time transformations. The theoretical insights also yield a practical contribution: the chrono initialization for LSTM gate biases.

Why time warping invariance matters for recurrent models

Standard recurrent neural networks are highly sensitive to changes in the time scale of their input data. Inserting a fixed number of blank steps between elements of an input sequence can make an otherwise easy task impossible for a vanilla RNN to learn. This fragility arises because the class of functions representable by an ordinary RNN is not closed under time rescaling.

The vanishing gradient problem compounds this issue: learning long-term dependencies requires gradient signals to persist across many time steps, but stability of the dynamical system causes these signals to decay exponentially. Prior solutions include gating mechanisms (LSTMs, GRUs) introduced on engineering grounds, and orthogonal weight constraints that limit representational power and make forgetting difficult.

Tallec and Ollivier ask a clean theoretical question: what structural properties must a recurrent model have to be invariant to arbitrary time transformations in its input?

Deriving gates from time warping invariance

The core insight starts from the continuous-time formulation of a basic RNN:

$$\frac{\mathrm{d}h(t)}{\mathrm{d}t} = \tanh(W_x x(t) + W_h h(t) + b) - h(t)$$

Applying a time warping $t \gets c(t)$ (any increasing differentiable function) to the input data $x(c(t))$ transforms this equation into:

$$\frac{\mathrm{d}h(t)}{\mathrm{d}t} = \frac{\mathrm{d}c(t)}{\mathrm{d}t} \tanh(W_x x(t) + W_h h(t) + b) - \frac{\mathrm{d}c(t)}{\mathrm{d}t} h(t)$$

The derivative $\frac{\mathrm{d}c(t)}{\mathrm{d}t}$ of the time warping appears as a multiplicative factor. For the model class to represent this equation for any time warping, a learnable function $g(t)$ must replace the unknown derivative:

$$\frac{\mathrm{d}h(t)}{\mathrm{d}t} = g(t) \tanh(W_x x(t) + W_h h(t) + b) - g(t) h(t)$$

Discretizing with a Taylor expansion ($\delta t = 1$) yields:

$$h_{t+1} = g_t \tanh(W_x x_t + W_h h_t + b) + (1 - g_t) h_t$$

This is a gated recurrent network with input gate $g_t$ and forget gate $(1 - g_t)$, where $g_t$ is computed by a sigmoid function of the inputs. The value $1/g(t_0)$ represents the local forgetting time of the network at time $t_0$.

The special case of linear time rescaling

For the simpler case of a constant time rescaling $c(t) = \alpha t$, the same derivation produces a leaky RNN:

$$h_{t+1} = \alpha \tanh(W_x x_t + W_h h_t + b) + (1 - \alpha) h_t$$

Leaky RNNs are invariant to global time rescalings but fail with variable warpings. Full gating (where $g_t$ depends on the input) is required for invariance to general time warpings.

Per-unit gates and the connection to LSTMs

Extending to per-unit gates $g_t^i$ allows different units to operate at different characteristic timescales:

$$h_{t+1}^i = g_t^i \tanh(W_x^i x_t + W_h^i h_t + b^i) + (1 - g_t^i) h_t^i$$

This closely resembles the LSTM cell update equation, where $(1 - g_t^i)$ corresponds to the forget gate $f_t$ and $g_t^i$ corresponds to the input gate $i_t$. The derivation naturally ties these two gates (they sum to 1), a constraint that has been used successfully in practice.

Chrono initialization for gate biases

The theoretical framework provides a principled initialization strategy. If the sequential data has temporal dependencies in a range $[T_{\text{min}}, T_{\text{max}}]$, then gate values $g$ should lie in $[1/T_{\text{max}}, 1/T_{\text{min}}]$. Since gate values center around $\sigma(b_g)$ when inputs are centered, the biases should be initialized as:

$$b_g \sim -\log(\mathcal{U}([T_{\text{min}}, T_{\text{max}}]) - 1)$$

For LSTMs specifically, the chrono initialization sets:

$$b_f \sim \log(\mathcal{U}([1, T_{\text{max}} - 1]))$$ $$b_i = -b_f$$

where $T_{\text{max}}$ is the expected range of long-term dependencies. This contrasts with the standard practice of setting forget gate biases to 1 or 2.

Experimental validation

Time warping robustness

On a character recall task with artificially warped sequences, three architectures are compared (64 units each):

• Vanilla RNNs fail with even moderate warping coefficients
• Leaky RNNs perfectly solve uniform warpings but fail with variable warpings
• Gated RNNs achieve perfect performance under both uniform and variable warpings for all tested warping factors

This directly validates the theory: leaky RNNs handle constant time rescalings, but only gated models handle general time warpings.

Synthetic tasks (copy and adding)

Using 128-unit LSTMs:

• Copy task ($T = 500, 2000$): Chrono initialization converges to the solution while standard initialization plateaus at the memoryless baseline
• Variable copy ($T = 500, 1000$): Chrono matches standard for smaller $T$ but outperforms for $T = 1000$
• Adding task ($T = 200, 750$): Chrono converges significantly faster, approximately 7x faster for $T = 750$

Real-world tasks

• Permuted MNIST (512-unit LSTM): Chrono achieves 96.3% vs. 95.4% for standard initialization
• Character-level text8 (2000-unit LSTM): Slight improvement (1.37 vs. 1.38 bits-per-character)
• Word-level Penn Treebank (10-layer RHN): Comparable results to the baseline (65.4 test perplexity)

Short-term dependency tasks show minimal differences, consistent with the theory that chrono initialization primarily helps when long-term dependencies dominate.

Limitations

The continuous-to-discrete time correspondence relies on a Taylor expansion with step size $\delta t = 1$. This approximation holds when the derivative of the time warping is not too large ($g_t \lesssim 1$). Discrete-time gated models are therefore invariant to time warpings that stretch time (such as interspersing data with blanks or introducing long-term dependencies), but they cannot handle warpings that compress events faster than the model’s time step. Additionally, the chrono initialization requires specifying $T_{\text{max}}$, the expected range of long-term dependencies, which may not be known in advance.

Reproducibility

Status: Partially Reproducible.

The paper describes all hyperparameters, architectures, and training procedures in sufficient detail to reproduce the experiments. The synthetic tasks (copy, adding, time warping) follow standard setups from prior work with clearly specified parameters. The real-world experiments (permuted MNIST, text8, Penn Treebank) use established benchmarks with referenced codebases (the text8 setup reuses code from Cooijmans et al. 2016).

The chrono initialization itself requires minimal implementation effort: it only changes the bias initialization of gate units, with no modifications to the model architecture or training procedure.

No official code repository is provided by the authors. No pre-trained models or datasets beyond standard benchmarks are released.

Paper Information

Citation: Tallec, C. & Ollivier, Y. (2018). Can recurrent neural networks warp time? International Conference on Learning Representations (ICLR 2018).

Publication: ICLR 2018

Additional Resources:

• OpenReview
• arXiv

@inproceedings{tallec2018can,
  title={Can recurrent neural networks warp time?},
  author={Tallec, Corentin and Ollivier, Yann},
  booktitle={International Conference on Learning Representations},
  year={2018}
}

This is primarily a Method paper with significant Resource contributions.

• Methodological Basis: The paper introduces a training pipeline (“mix-sourced distillation”) and domain-specific reinforcement learning to improve reasoning capabilities in chemical LLMs. It validates the approach through ablation studies across training stages.
• Resource Contribution: The authors constructed ChemFG, a 101 billion-token corpus annotated with “atomized” knowledge regarding functional groups and reaction centers.

Bridging the Chemical Reasoning Gap

Current chemical LLMs struggle to reason logically for two main reasons:

1. Shallow Domain Understanding: Models generally learn molecule-level properties directly, bypassing the intermediate “atomized” characteristics (e.g., functional groups) that ultimately dictate chemical behavior.
2. Specialized Reasoning Logic: Chemical logic differs fundamentally from math or code. Distilling reasoning from general teacher models like DeepSeek-R1 frequently fails because the teachers lack the domain intuition required to generate valid chemical rationales.

Atomized Knowledge and Mixed-Source Distillation

The authors introduce three structural innovations to solve the reasoning gap:

1. Atomized Knowledge Enhancement (ChemFG): A toolkit was built leveraging SMARTS notations to identify functional group changes during reactions. A critique of this approach is that it relies heavily on 2D cheminformatics abstractions, potentially missing deeper 3D stereochemical interactions.
2. Mix-Sourced Distillation: General models (DeepSeek-R1/o3-mini) are fed “pseudo-reasoning” prompts that include ground truth answers and functional group data. While this forces the teacher to generate high-quality rationales for the student to learn, it introduces a layer of hindsight bias into the generated reasoning chains. During inference, the student model lacks both the pre-calculated functional group metadata and the ground truth, forcing it to bridge an artificially steep generalization gap.
3. Chemical Reinforcement Learning: The intermediate model undergoes domain-specific reinforcement learning. The RL details are described in the paper’s Appendix D, with the authors citing the open-source DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) framework. The optimization relies on rule-based rewards (format adherence and canonicalized SMILES accuracy) across a variety of chemical tasks.

Benchmark Evaluation and Ablation Studies

The model was evaluated on comprehensive chemical benchmarks: SciKnowEval (19 tasks) and ChemEval (36 tasks).

• Baselines: Compared against similarly sized open models (Qwen2.5-14B-Instruct, Qwen3-14B), domain models (ChemLLM, MolInst), and frontier models (GPT-4o, DeepSeek-R1).
• Ablation: Evaluated across training stages (Base → ChemDFM-I → ChemDFM-R) to measure the specific impact of the instruction tuning versus the reasoning stages.
• Qualitative Analysis: The paper includes case studies demonstrating the model’s step-by-step chemical reasoning and its potential for human-AI collaboration (Sections 4.2 and 4.3).

Performance Outcomes and Numerical Limitations

• Performance vs. Baselines: ChemDFM-R outperforms similarly sized open models and domain models on molecule-centric and reaction-centric tasks, and surpasses the much larger DeepSeek-R1 on ChemEval (0.78 vs. 0.58 overall). It shows competitive results relative to o4-mini, though o4-mini leads on SciKnowEval (0.74 vs. 0.70).
• Reasoning Interactivity: The model generates readable rationales that allow users to catch structural errors or identify reaction mechanisms accurately. Section 4.3 of the paper demonstrates human-AI collaboration scenarios.
• Quantitative Limitations: The model struggles with tasks involving numerical prediction and calculation (e.g., yield extraction, molecular property calculation). The paper notes that all molecule-centric and reaction-centric tasks where ChemDFM-R falls short of Qwen2.5-14B-Instruct involve numerical reasoning.

Reproducibility Details

Data

The training data is constructed in three phases:

1. Domain Pre-training (ChemFG):

• Size: 101 billion tokens
• Composition:
  • 12M literature documents (79B tokens)
  • 30M molecules from PubChem/PubChemQC
  • 7M reactions from USPTO-FULL
• Augmentation: SMILES augmentation (10x) using R-SMILES
• Atomized Features: Annotated with a custom “Functional Group Identification Toolkit” that identifies 241 functional group types and tracks changes in reaction centers. Note: Data and toolkit are partially reproduced; while the toolkit (ChemFG-Tool) was open-sourced on GitHub, the 101 billion-token ChemFG dataset itself has not been publicly released.

2. Instruction Tuning:

• Sources: Molecule-centric (PubChem, MoleculeNet), Reaction-centric (USPTO), and Knowledge-centric (Exams, Literature QA) tasks
• Mixing: Mixed with general instruction data in a 1:2 ratio

3. Distillation Dataset:

• Sources:
  • ~70% ChemDFM-R instruction data
  • ~22% constructed pseudo-reasoning (functional group descriptions)
  • ~8% teacher rationales (from DeepSeek-R1/o3-mini)
• Mixing: Mixed with general data (including AM-Deepseek-R1-Distill-1.4M) in a 1:2 ratio

Algorithms

Functional Group Identification:

• Extends the thermo library’s SMARTS list
• For reactions, identifies “reacting functional groups” by finding reactants containing atoms involved in bond changes (reaction centers) that do not appear in the product

Mix-Sourced Distillation:

• Teacher models (DeepSeek-R1, o3-mini) are prompted with Question + Ground Truth + Functional Group Info to generate high-quality “Thoughts”
• These rationales are distilled into the student model using a supervised fine-tuning loss across target tokens $y_t$: $$ \mathcal{L}_{\text{SFT}} = - \sum_{t=1}^T \log P_\theta(y_t \mid x, y_{<t}) $$

Reinforcement Learning:

• Algorithm: The paper cites DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization) as the RL framework; full details are in Appendix D of the paper. Note: While the underlying DAPO framework is open-source, the specific chemistry-oriented RL pipeline and environment used for ChemDFM-R has not been publicly released.
• Hyperparameters (from paper appendix): Learning rate 5e-7, rollout batch size 512, training batch size 128
• Rewards: The reward system applies rule-based constraints focusing on physical form and chemical validity. The total reward $R(y, y^*)$ for a generated response $y$ given target $y^*$ combines a format adherence reward ($R_{\text{format}}$) and an accuracy reward ($R_{\text{acc}}$) evaluated on canonicalized SMILES: $$ R(y, y^*) = R_{\text{format}}(y) + R_{\text{acc}}(\text{canonicalize}(y), \text{canonicalize}(y^*)) $$

Models

• Base Model: Qwen2.5-14B
• ChemDFM-I: Result of instruction tuning the domain-pretrained model for 2 epochs
• ChemDFM-R: Result of applying mix-sourced distillation (1 epoch) followed by RL on ChemDFM-I. Note: Model weights are publicly available on Hugging Face.

Hardware

Hardware and training time details are described in the paper’s appendices, which are not available in the extracted text. The details below are reported from the paper but could not be independently cross-verified against the main text:

• Compute: NVIDIA A800 Tensor Core GPUs
• Training Time: 30,840 GPU hours total (Domain Pretraining: 24,728 hours; Instruction Tuning: 3,785 hours; Distillation: 2,059 hours; Reinforcement Learning: 268 hours)

Evaluation

Benchmarks:

• SciKnowEval: 19 tasks (text-centric, molecule-centric, reaction-centric)
• ChemEval: 36 tasks, categorized similarly

Key Metrics: Accuracy, F1 Score, BLEU score (with PRS normalization for ChemEval)

Artifacts

Missing components: The 101B-token ChemFG pretraining dataset is not publicly released. The chemistry-oriented RL pipeline and training code are not open-sourced. The instruction tuning and distillation datasets are not available.

Paper Information

Citation: Zhao, Z., Chen, B., Wan, Z., Chen, L., Lin, X., Yu, S., Zhang, S., Ma, D., Zhu, Z., Zhang, D., Wang, H., Dai, Z., Wen, L., Chen, X., & Yu, K. (2025). ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledge. arXiv preprint arXiv:2507.21990. https://doi.org/10.48550/arXiv.2507.21990

Publication: arXiv 2025

@misc{zhao2025chemdfmr,
  title={ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledge},
  author={Zihan Zhao and Bo Chen and Ziping Wan and Lu Chen and Xuanze Lin and Shiyang Yu and Situo Zhang and Da Ma and Zichen Zhu and Danyang Zhang and Huayang Wang and Zhongyang Dai and Liyang Wen and Xin Chen and Kai Yu},
  year={2025},
  eprint={2507.21990},
  archivePrefix={arXiv},
  primaryClass={cs.CE},
  url={https://arxiv.org/abs/2507.21990}
}

This is primarily a Method paper. It proposes a unified framework that generalizes previous discrete score-based models (SMLD and DDPM) into continuous-time Stochastic Differential Equations (SDEs). The paper introduces algorithms for sampling (Predictor-Corrector) and likelihood computation (Probability Flow ODE), validated by setting new records on CIFAR-10 (FID 2.20, IS 9.89 at the time of publication). It also contains elements of Systematization by showing how existing methods are special cases of this broader framework.

What is the motivation?

Prior successful generative models, specifically Score Matching with Langevin Dynamics (SMLD) and Denoising Diffusion Probabilistic Models (DDPM), operate by sequentially corrupting data with slowly increasing noise and learning to reverse the process. Both methods treat the noise scales as a finite set of discrete steps. The authors aim to generalize this to a continuum of noise scales by modeling the diffusion process as a Stochastic Differential Equation (SDE). This continuous formulation enables:

• Flexible sampling: Use of general-purpose SDE solvers.
• Exact likelihood computation: Via connection to Neural ODEs.
• Controllable generation: Solving inverse problems (inpainting, colorization) without retraining.

What is the novelty here?

The core novelty is the SDE framework for score-based generative modeling:

• Continuous Generalization: Proving that SMLD and DDPM noise perturbations correspond to discretizations of Variance Exploding (VE) SDEs and Variance Preserving (VP) SDEs, respectively.
• Reverse-Time SDE: Leveraging Anderson’s result (Anderson, 1982: a result on time-reversal of diffusion processes showing that the reverse is also a diffusion, with the forward drift reversed and a correction term involving the score of the marginal density) that the reverse of a diffusion process is also a diffusion process, governed by the score (gradient of log density).
• Predictor-Corrector (PC) Samplers: A hybrid sampling strategy where a numerical SDE solver (Predictor) estimates the next step, and a score-based MCMC approach (Corrector) corrects the marginal distribution.
• Probability Flow ODE: Deriving a deterministic ODE that shares the same marginal densities as the SDE, enabling near-exact likelihood computation (accuracy is limited by both numerical ODE solver discretization and variance of the unbiased Hutchinson trace estimator) and latent space manipulation.
• Sub-VP SDE: A new SDE class proposed to improve likelihoods by bounding variance tighter than the VP SDE.

What experiments were performed?

The authors validated the framework on standard image benchmarks:

• Datasets: CIFAR-10 (32x32), CelebA (64x64), LSUN (Bedroom, Church), and CelebA-HQ (256x256 and 1024x1024).
• Ablation Studies: Comparing samplers (Ancestral vs. Reverse Diffusion vs. Probability Flow vs. PC) and SDE types (VE, VP, sub-VP).
• Architecture Search: Exploring improvements like FIR up/downsampling, rescaling skip connections, and increasing depth (leading to NCSN++ and DDPM++ architectures).
• Likelihood Evaluation: Computing Negative Log-Likelihood (NLL) in bits/dim using the Probability Flow ODE.
• Inverse Problems: Testing class-conditional generation, inpainting, and colorization using the conditional reverse-time SDE.

What outcomes/conclusions?

• Record Performance: The NCSN++ cont. (deep, VE) model achieved an Inception Score of 9.89 and FID of 2.20 on CIFAR-10 (as of ICLR 2021).
• High-Fidelity Generation: First score-based model to generate 1024x1024 images (CelebA-HQ).
• Competitive Likelihoods: The DDPM++ cont. (deep, sub-VP) model achieved 2.99 bits/dim on uniformly dequantized CIFAR-10, a record at the time.
• Sampling Efficiency: PC samplers consistently outperformed predictor-only methods (like standard ancestral sampling) for the same computational cost.
• Controllable Generation: Successful application to inpainting and colorization using a single unconditional model.
• Limitations: Sampling remains slower than GANs on the same datasets. The breadth of available samplers introduces many hyperparameters (SDE type, predictor, corrector, signal-to-noise ratio, number of steps) that require tuning.

Reproducibility Details

Data

• CIFAR-10: Used for main benchmarking (FID, Inception Score, NLL).
• CelebA-HQ: Used for high-resolution experiments at 256x256 and 1024x1024.
• LSUN: Bedroom and Church Outdoor categories (256x256) used for sampler comparison and controllable generation (inpainting, colorization).
• Preprocessing: CIFAR-10 images are 32x32; CelebA pre-processed to 64x64 following Song & Ermon (2020). Data is typically scaled to $[0, 1]$ or standardized depending on the specific SDE config.

Algorithms

Forward SDEs:

Here $dw$ denotes a Wiener process increment (a small, independent Gaussian noise burst at each timestep).

• VE SDE (Variance Exploding): $dx = \sqrt{\frac{d[\sigma^2(t)]}{dt}} dw$. Corresponds to SMLD. Used with $\sigma_{\min}=0.01$ and $\sigma_{\max}$ chosen via heuristics.
• VP SDE (Variance Preserving): $dx = -\frac{1}{2}\beta(t)x dt + \sqrt{\beta(t)} dw$. Corresponds to DDPM.
• Sub-VP SDE: $dx = -\frac{1}{2}\beta(t)x dt + \sqrt{\beta(t)(1 - e^{-2\int_0^t \beta(s)ds})} dw$. Bounded variance, good for likelihoods.

Reverse-Time SDE Solver (Predictor):

• Discretized via Reverse Diffusion Sampling, which matches the forward discretization.
• Euler-Maruyama solver used for continuously-trained models.

Corrector Algorithm:

• Langevin MCMC: Applies annealed Langevin dynamics: adds noise and takes a score-guided gradient step to correct the marginal distribution at each timestep.
• PC Sampling: Alternates between one step of the Predictor and one step of the Corrector.
• Signal-to-Noise Ratio ($r$): A hyperparameter for the corrector step size. Tuned values: $r \approx 0.16$ for VE SDEs on CIFAR-10.

Models

• NCSN++: Optimized architecture for VE SDEs. Key features:
  • 4 residual blocks per resolution.
  • BigGAN-type residual blocks.
  • Rescaling skip connections by $1/\sqrt{2}$.
  • FIR (Finite Impulse Response) up/downsampling.
  • “Residual” progressive architecture for input, no progressive growing for output.
• DDPM++: Optimized architecture for VP/sub-VP SDEs. Similar to NCSN++ but without FIR upsampling and no progressive growing.
• Deep Variants: “cont. (deep)” models double the depth (from 4 to 8 blocks per resolution) for the best reported results.
• Conditioning: Time $t$ is conditioned via random Fourier feature embeddings (scale 16) for continuous models.

Evaluation

Metrics:

• FID (Fréchet Inception Distance): Computed on 50k samples.
• Inception Score: Reported for CIFAR-10.
• NLL (Negative Log-Likelihood): Reported in bits/dim on uniformly dequantized data using the Probability Flow ODE.

Denoising: A single denoising step using Tweedie’s formula is applied at the end of sampling to remove residual noise, which significantly improves FID.

Hardware

Training:

• Batch size: 128 for CIFAR-10, 64 for LSUN, 8 for high-res CelebA-HQ.
• Iterations: Discrete-objective models trained for 1.3M iterations during architecture exploration. Continuous-objective models (cont.) trained for 0.95M iterations. High-res CelebA-HQ (1024x1024) trained for approximately 2.4M iterations.
• EMA: Exponential Moving Average rate of 0.999 used for VE models, 0.9999 for VP models.

Artifacts

All datasets used (CIFAR-10, CelebA-HQ, LSUN) are publicly available. Pretrained model checkpoints for CIFAR-10, CelebA-HQ, and FFHQ are provided in the repository. Specific hardware requirements (GPU type, training time) are not detailed in the paper.

Paper Information

Citation: Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., & Poole, B. (2021). Score-Based Generative Modeling through Stochastic Differential Equations. ICLR 2021. https://arxiv.org/abs/2011.13456

Publication: ICLR 2021

@inproceedings{song2021scorebased,
  title     = {Score-Based Generative Modeling through Stochastic Differential Equations},
  author    = {Song, Yang and Sohl-Dickstein, Jascha and Kingma, Diederik P and Kumar, Abhishek and Ermon, Stefano and Poole, Ben},
  booktitle = {International Conference on Learning Representations},
  year      = {2021},
  url       = {https://openreview.net/forum?id=PxTIG12RRHS}
}

Additional Resources:

• GitHub Repository
• Score Matching and Denoising Autoencoders

This is a Theory Paper.

Its primary contribution is a formal mathematical derivation connecting two previously distinct techniques: Score Matching (SM) and Denoising Autoencoders (DAE). It provides the “why” behind the empirical success of DAEs by grounding them in the probabilistic framework of energy-based models. It relies on proofs and equivalence relations (e.g., $J_{ESMq_{\sigma}} \sim J_{DSMq_{\sigma}}$).

What is the motivation?

The paper bridges a gap between two successful but disconnected approaches in unsupervised learning:

1. Denoising Autoencoders (DAE): Empirically successful for pre-training deep networks. They previously lacked a clear probabilistic interpretation.
2. Score Matching (SM): A theoretically sound method for estimating unnormalized density models that avoids the partition function problem but requires computing expensive second derivatives.

By connecting them, the authors aim to define a proper probabilistic model for DAEs (allowing sampling/ranking) and find a simpler way to apply score matching that avoids second derivatives.

What is the novelty here?

The core novelty is the Denoising Score Matching (DSM) framework and the proof of its equivalence to DAEs. Key contributions include:

• Equivalence Proof: Showing that training a DAE with Gaussian noise is equivalent to matching the score of a model against a non-parametric Parzen density estimator of the data.
• Denoising Score Matching ($J_{DSM}$): A new objective that learns a score function by trying to denoise corrupted samples. This avoids the explicit second derivatives required by standard Implicit Score Matching ($J_{ISM}$).
• Explicit Energy Function: Deriving the specific energy function $E(x;\theta)$ that corresponds to the standard sigmoid DAE architecture.
• Justification for Tied Weights: Providing a theoretical justification for tying encoder and decoder weights, which arises naturally from differentiating the energy function.

What experiments were performed?

The validation in this theoretical paper is purely mathematical and focuses on formal proofs:

• Derivation of Equivalence: The paper formally proves the chain of equivalences: $$J_{ISMq_{\sigma}} \sim J_{ESMq_{\sigma}} \sim J_{DSMq_{\sigma}} \sim J_{DAE\sigma}$$ where $q_{\sigma}$ is the Parzen density estimate.
• Appendix Proof: A detailed proof is provided to show that Explicit Score Matching ($J_{ESM}$) on the Parzen density is equivalent to the proposed Denoising Score Matching ($J_{DSM}$) objective.

What outcomes/conclusions?

• Theoretical Unification: DAE training is formally equivalent to Score Matching on a smoothed data distribution ($q_{\sigma}$).
• New Training Objective: The $J_{DSM}$ objective offers a computationally efficient way to perform score matching (no Hessian required) by using a denoising objective.
• Probabilistic Interpretation: DAEs can now be understood as Energy-Based Models (EBMs), allowing for operations like sampling (via Hybrid Monte Carlo) and likelihood ranking, which were previously ill-defined for standard autoencoders.
• Regularization Insight: The smoothing kernel width $\sigma$ in the Parzen estimator corresponds to the noise level in the DAE. This suggests that DAEs are learning a regularized version of the score, which may explain their robustness.
• Connection to Regularized Score Matching: The paper notes that Kingma and LeCun (2010) independently proposed a regularized score matching criterion $J_{ISMreg}$ derived by approximating $J_{ISMq_{\sigma}}$. The four $q_{\sigma}$-based objectives in this work (including the DAE objective) can be seen as approximation-free forms of regularized score matching, with the additional advantage that $J_{DSMq_{\sigma}}$ does not require second derivatives.

Key Concepts Explained

1. “Score” and “Score Matching”

What does “score” actually mean?

In this paper (and probabilistic modeling generally), the score is the gradient of the log-density with respect to the data vector $x$.

• Definition: $\psi(x) = \nabla_x \log p(x)$.
• Intuition: It is a vector field pointing in the direction of highest probability increase. Crucially, calculating the score avoids the intractable partition function $Z$, because $\nabla_x \log p(x) = \nabla_x \log \tilde{p}(x) - \nabla_x \log Z = \nabla_x \log \tilde{p}(x)$. The constant $Z$ vanishes upon differentiation.

What is Score Matching?

Score Matching is a training objective for unnormalized models. It minimizes the squared Euclidean distance between the model’s score $\psi(x;\theta)$ and the data’s true score $\nabla_x \log q(x)$.

2. The Parzen Density Estimator

What is it?

It is a non-parametric method for estimating a probability density function from finite data. It places a smooth kernel (here, a Gaussian) centered at every data point in the training set $D_n$.

• Formula: $q_{\sigma}(\tilde{x}) = \frac{1}{n} \sum_{t=1}^n \mathcal{N}(\tilde{x}; x^{(t)}, \sigma^2 I)$.

Why smooth the data?

1. To define the score: The empirical data distribution is a set of Dirac deltas (spikes). The gradient (score) of a Dirac delta is undefined. Smoothing creates a differentiable surface, allowing a valid target score $\nabla_{\tilde{x}} \log q_{\sigma}(\tilde{x})$ to be computed.

2. To model corruption: The Parzen estimator with Gaussian kernels mathematically models the process of taking a clean data point $x$ and adding Gaussian noise - the exact procedure used in Denoising Autoencoders.

3. Why avoiding second derivatives matters

Standard Implicit Score Matching (ISM) eliminates the need for the unknown data score, but introduces a new cost: it requires computing the trace of the Hessian (the sum of second partial derivatives) of the log-density.

• The Cost: For high-dimensional data (like images) and deep networks, computing second derivatives of the log-density is computationally expensive.
• This paper shows that Denoising Score Matching (DSM) allows you to bypass Hessian computation entirely. By using the Parzen target, the objective simplifies to matching a first-order vector, making it scalable to deep neural networks.

4. The equivalence chain - why each step?

The chain $J_{ISMq_{\sigma}} \sim J_{ESMq_{\sigma}} \sim J_{DSMq_{\sigma}} \sim J_{DAE\sigma}$ connects the concepts.

• $J_{ISMq_{\sigma}} \sim J_{ESMq_{\sigma}}$ (Implicit $\to$ Explicit): Why: Integration by parts. This is Hyvärinen’s original proof (2005): integration by parts moves the derivative from $\psi$ onto the data density $q$, producing a term involving $q$’s gradient (the score). The boundary term vanishes because $q_{\sigma}$ decays to zero at infinity (Hyvärinen’s 2005 regularity condition for Implicit Score Matching). The result allows replacing the unknown data score with a computable term involving only the model’s score and its Jacobian.

• $J_{ESMq_{\sigma}} \sim J_{DSMq_{\sigma}}$ (Explicit $\to$ Denoising): Why: The explicit score of the Parzen density is known. When $x$ is perturbed to $\tilde{x}$ by Gaussian noise $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$, the gradient of the log-density pointing back to the mean is exactly $\frac{1}{\sigma^2}(x - \tilde{x})$. Minimizing the error against the true score becomes minimizing the error against this restoration vector.

• $J_{DSMq_{\sigma}} \sim J_{DAE\sigma}$ (Denoising $\to$ Autoencoder): Why: Algebraic substitution. If you define the model’s score $\psi(\tilde{x};\theta)$ to be proportional to the reconstruction error ($\propto x^r - \tilde{x}$), the score matching loss $J_{DSM}$ becomes proportional to the standard autoencoder squared loss $|x^r - x|^2$.

5. Energy-Based Models (EBMs) connection

What is an EBM?

An EBM defines a probability distribution via an energy function $E(x;\theta)$, where $p(x;\theta) \propto e^{-E(x;\theta)}$.

Why standard autoencoders lack probabilistic interpretation:

A standard autoencoder acts as a deterministic map $x \to x^r$, providing a reconstruction error. It lacks a normalization constant or a defined density function to support sampling or probability queries.

What does this enable?

By proving the equivalence, the DAE is formally defined as an EBM. This enables:

1. Sampling: Using MCMC methods (like Hybrid Monte Carlo) to generate new data from the DAE.
2. Ranking: Calculating the energy of inputs to determine which are more “likely” or “normal” (useful for anomaly detection).

6. The specific energy function form

The function is:

$$E(x; W, b, c) = - \frac{1}{\sigma^2} \left( \langle c, x \rangle - \frac{1}{2}|x|^2 + \sum_{j=1}^{d_h} \text{softplus}(\langle W_j, x \rangle + b_j) \right)$$

Why does it have that specific form?

It was derived via integration to ensure its derivative matches the DAE architecture. The authors worked backward from the DAE’s reconstruction function (sigmoid + linear) to find the scalar field that generates it.

Where does the quadratic term come from?

The score (negative energy gradient) needs to look like $\psi(x) \propto c - x + W^T\text{sigmoid}(Wx + b)$.

• The term $-x$ in the score arises because $\nabla_x(-\frac{1}{2}|x|^2) = -x$. Including $-\frac{1}{2}|x|^2$ inside the energy’s numerator produces this linear term after differentiation.

How does differentiating it recover the DAE reconstruction?

• $\nabla_x \sum_j \text{softplus}(\langle W_j, x \rangle + b_j) = W^T \sigma(Wx + b)$ (The encoder part).
• $\nabla_x \langle c, x \rangle = c$ (The bias).
• $\nabla_x (-\frac{1}{2}|x|^2) = -x$ (The input subtraction).
• Result: $-\nabla_x E \propto c + W^T h - x = x^r - x$.

7. “Tied weights” justification

What does it mean for weights to be “tied”?

The decoder matrix is the transpose of the encoder matrix ($W^T$).

Why is this theoretically justified?

Because the reconstruction function is interpreted as the gradient of an energy function. A vector field can only be the gradient of a scalar field if its Jacobian is symmetric.

• In the DAE energy derivative, the encoder contributes $W^T \sigma(Wx + b)$. If the decoder used a separate matrix $U$, the resulting vector field would not be a valid gradient of any scalar energy function (unless $U = W^T$).
• Therefore, for a DAE to correspond to a valid probabilistic Energy-Based Model, the weights must be tied.

The necessity of tied weights:

Within this parametrization, tied weights are a mathematical necessity: a separate decoder matrix $U \neq W^T$ would make the reconstruction function an invalid gradient of any scalar energy, breaking the EBM correspondence.

Reproducibility Details

Since this is a theoretical paper, the “reproducibility” lies in the mathematical formulations derived.

Data

• Input Data ($D_n$): The theory assumes a set of training examples $D_n = {x^{(1)}, …, x^{(n)}}$ drawn from an unknown true pdf $q(x)$.
• Parzen Density Estimate ($q_{\sigma}$): The theoretical targets are derived from a kernel-smoothed empirical distribution: $$q_{\sigma}(\tilde{x}) = \frac{1}{n} \sum_{t=1}^n q_{\sigma}(\tilde{x}|x^{(t)})$$ where the kernel is an isotropic Gaussian of variance $\sigma^2$.

Algorithms

1. Denoising Score Matching (DSM) Objective

The paper proposes this objective as a tractable alternative to standard score matching. It minimizes the distance between the model score and the gradient of the log-noise density:

$$J_{DSMq_{\sigma}}(\theta) = \mathbb{E}_{q_{\sigma}(x,\tilde{x})} \left[ \frac{1}{2} \left| \psi(\tilde{x};\theta) - \frac{\partial \log q_{\sigma}(\tilde{x}|x)}{\partial \tilde{x}} \right|^2 \right]$$

For Gaussian noise, the target score is simply $\frac{1}{\sigma^2}(x - \tilde{x})$.

2. Equivalence Chain

The central result connects four objectives:

$$J_{ISMq_{\sigma}} \sim J_{ESMq_{\sigma}} \sim J_{DSMq_{\sigma}} \sim J_{DAE\sigma}$$

This implies optimizing the DAE reconstruction error is minimizing a score matching objective.

Models

1. The Denoising Autoencoder (DAE)

• Corruption: Additive isotropic Gaussian noise $\tilde{x} = x + \epsilon, \epsilon \sim \mathcal{N}(0, \sigma^2 I)$.
• Encoder: $h = \text{sigmoid}(W\tilde{x} + b)$.
• Decoder: $x^r = W^T h + c$ (Tied weights $W$).
• Loss: Squared reconstruction error $|x^r - x|^2$. (The equivalence with DSM introduces a $\frac{1}{2\sigma^4}$ scaling factor.)

2. The Corresponding Energy Function

To make the DAE equivalent to Score Matching, the underlying Energy-Based Model $p(x;\theta) \propto e^{-E(x;\theta)}$ must have the following energy function:

$$E(x; W, b, c) = - \frac{1}{\sigma^2} \left( \langle c, x \rangle - \frac{1}{2}|x|^2 + \sum_{j=1}^{d_h} \text{softplus}(\langle W_j, x \rangle + b_j) \right)$$

Note the scaling by $1/\sigma^2$ and the quadratic term $|x|^2$.

Evaluation

• Metric: Theoretical Equivalence ($\sim$).
• Condition: The equivalence holds provided $\sigma > 0$ and the density $q_{\sigma}$ is differentiable and vanishes at infinity (Hyvärinen’s 2005 regularity condition for Implicit Score Matching).

Paper Information

Citation: Vincent, P. (2011). A Connection Between Score Matching and Denoising Autoencoders. Neural Computation, 23(7), 1661-1674. https://doi.org/10.1162/NECO_a_00142

Publication: Neural Computation 2011

@article{vincentConnectionScoreMatching2011,
  title = {A {{Connection Between Score Matching}} and {{Denoising Autoencoders}}},
  author = {Vincent, Pascal},
  year = 2011,
  month = jul,
  journal = {Neural Computation},
  volume = {23},
  number = {7},
  pages = {1661--1674},
  doi = {10.1162/NECO_a_00142}
}

Additional Resources:

• Official PDF

This is primarily a Method paper, with a significant Theory component.

• Method: It proposes “Rectified Flow,” a novel generative framework that learns ordinary differential equations (ODEs) to transport distributions via straight paths. It introduces the “Reflow” algorithm to iteratively straighten these paths.
• Theory: It provides rigorous proofs connecting the method to Optimal Transport, showing that the rectification process yields a coupling with non-increasing convex transport costs and that recursive reflow reduces the curvature of trajectories.

What is the motivation?

The work addresses two main challenges in unsupervised learning: generative modeling (generating data from noise) and domain transfer (mapping between two observed distributions).

• Inefficiency of ODE/SDE Models: Continuous-time models (like Score-based Generative Models and DDPMs) require simulating diffusions over many steps, resulting in high computational costs during inference.
• Complexity of GANs: GANs provide fast (one-step) generation alongside challenges with training instability and mode collapse.
• Disconnection: Generative modeling and domain transfer are often treated as separate tasks requiring different techniques.

The authors aim to unify these tasks into a single “transport mapping” problem while bridging the gap between high-quality continuous models and fast one-step models.

What is the novelty here?

The core novelty is the Rectified Flow framework and the Reflow procedure.

• Straight-Line ODEs: Rectified Flow learns an ODE drift $v$ to follow the straight line connecting data pairs $(X_0, X_1)$, providing an alternative to diffusion models that rely on stochastic paths or specific forward processes. This is achieved via a simple least-squares optimization problem.
• Reflow (Iterative Straightening): The authors introduce a recursive training procedure where a new flow is trained on the data pairs $(Z_0, Z_1)$ generated by the previous flow. Theoretical analysis shows this reduces the “transport cost” and straightens the trajectories, allowing for accurate 1-step simulation (effectively converting the ODE into a one-step model).
• Unified Framework: The method uses the exact same algorithm for generation ($\pi_0$ is Gaussian) and domain transfer ($\pi_0$ is a source dataset), removing the need for adversarial losses or cycle-consistency constraints.

What experiments were performed?

The authors validated the method across image generation, translation, and domain adaptation tasks.

• Unconditioned Image Generation:
  • Dataset: CIFAR-10 ($32\times32$).
  • Baselines: Compared against GANs (StyleGAN2, TDPM), Diffusion/SDE Models (VP SDE, sub-VP SDE, VE SDE), ODE methods (VP ODE, sub-VP ODE, VE ODE), and distilled methods (DDIM Distillation).
  • High-Res: Validated on LSUN Bedroom/Church, CelebA-HQ, and AFHQ ($256\times256$).
• Image-to-Image Translation:
  • Datasets: AFHQ (Cat $\leftrightarrow$ Dog/Wild), MetFace $\leftrightarrow$ CelebA-HQ.
  • Setup: Transferring styles while preserving semantic identity (using a classifier-based feature mapping metric).
• Domain Adaptation:
  • Datasets: DomainNet, Office-Home.
  • Metric: Classification accuracy on the transferred testing data.

What outcomes/conclusions?

• Superior 1-Step Generation: On CIFAR-10 with a single Euler step (as of ICLR 2023), the distilled 2-Rectified Flow achieved an FID of 4.85, beating the best one-step U-Net model TDPM (FID 8.91, a truncated diffusion model using a GAN). The distilled 3-Rectified Flow reached a Recall of 0.51, beating the GAN baseline StyleGAN2+ADA (Recall 0.49).
• Straightening Effect: The “Reflow” procedure was empirically shown to reduce the “straightness” error and transport costs, validating the theoretical claims. “Straightness” is measured as $S(Z) = \mathbb{E}[\int_0^1 |\dot{Z}_t - (Z_1 - Z_0)|^2, dt]$ (zero means perfectly straight); “transport cost” is $\mathbb{E}[c(Z_1 - Z_0)]$ for a convex cost $c$, and Reflow reduces this for all convex costs.
• High-Quality Transfer: The model successfully performed image translation (e.g., Cat to Wild Animal) without paired data or cycle-consistency losses.
• Strong Full-Simulation Results: With RK45 adaptive ODE solving, 1-Rectified Flow achieves FID 2.58 and Recall 0.57 on CIFAR-10 (Table 1a), the best among ODE methods and comparable to fully simulated SDEs (VP SDE: FID 2.55).
• Fast Simulation: The method allows for extremely coarse time discretization (e.g., $N=1$) without significant quality loss after reflow, effectively solving the slow inference speed of standard ODE models.
• Domain Adaptation: On Office-Home, Rectified Flow achieves 69.2% accuracy, outperforming Deep CORAL (68.7%) and other baselines. On DomainNet, it achieves 41.4%, comparable to Deep CORAL (41.5%) and MLDG (41.2%).

Reproducibility Details

Data

The paper utilizes several standard computer vision benchmarks.

Algorithms

• Objective Function: The drift $v(Z_t, t)$ is trained by minimizing a least-squares regression objective: $$\min_{v} \int_{0}^{1} \mathbb{E}[|(X_1 - X_0) - v(X_t, t)|^2] dt$$ where $X_t = tX_1 + (1-t)X_0$ is the linear interpolation.

• Reflow Procedure: Iteratively updates the flow. Let $Z^k$ be the $k$-th rectified flow.

  1. Generate 4 million data pairs $(Z_0^k, Z_1^k)$ by simulating the current flow.
  2. Fine-tune the $i$-rectified flow model for 300,000 steps on these pairs to obtain the $(i+1)$-rectified flow.

• Distillation: For 1-step distillation ($k=1$), the L2 loss is replaced with LPIPS perceptual similarity, which empirically yields better image quality. For multi-step distillation, training samples $t$ from ${0, 1/k, \ldots, (k-1)/k}$ rather than the full $[0, 1]$ interval.

• ODE Solver:

  • Training: Analytical linear interpolation.
  • Inference: Euler method (constant step size $1/N$) or RK45 (adaptive).

Models

• Architecture:

  • Uses the DDPM++ U-Net architecture (from Song et al., 2020) across experiments. Implementation is modified from the open-source code of Song et al.

• Optimization:

  • Optimizer: Adam (CIFAR-10) or AdamW (Transfer/Adaptation).
  • Hyperparameters:
    • LR: $2 \times 10^{-4}$ (CIFAR), Grid search for transfer.
    • EMA: 0.999999 (CIFAR), 0.9999 (Transfer).
    • Batch Size: 4 (Transfer), 16 (Domain Adaptation).
    • Dropout: 0.15 (CIFAR), 0.1 (Transfer).

Evaluation

Hardware

The paper does not specify GPU models or training times. The DDPM++ U-Net architecture used in the experiments typically requires multi-GPU setups for training on high-resolution datasets.

Artifacts

Paper Information

Citation: Liu, X., Gong, C., & Liu, Q. (2023). Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=XVjTT1nw5z

Publication: ICLR 2023

@inproceedings{liuFlowStraightFast2023,
  title = {Flow {{Straight}} and {{Fast}}: {{Learning}} to {{Generate}} and {{Transfer Data}} with {{Rectified Flow}}},
  booktitle = {International Conference on Learning Representations},
  author = {Liu, Xingchao and Gong, Chengyue and Liu, Qiang},
  year = 2023,
  url = {https://openreview.net/forum?id=XVjTT1nw5z}
}

Additional Resources:

• Official Code Repository
• OpenReview Page

Key Prerequisites: Before diving in, note that for the ODE solver to guarantee a unique solution, the neural network $f(h(t), t, \theta)$ parameterizing the dynamics must be Lipschitz continuous. This ensures the Picard-Lindelöf theorem holds, preventing trajectories from crossing and guaranteeing a well-defined backward pass.

What kind of paper is this?

This is primarily a Method paper, with a strong secondary Theory component.

• Method: It proposes a novel family of deep neural network models where the derivative of the hidden state is parameterized by a neural network. It provides specific algorithms (Algorithm 1) for training these models scalably.
• Theory: It derives the adjoint sensitivity method for backpropagating through black-box ODE solvers and proves the “Instantaneous Change of Variables” theorem (Theorem 1) for continuous normalizing flows.

What is the motivation?

The authors aim to address limitations in discrete deep learning architectures:

• Discrete vs. Continuous: Existing models like Residual Networks build transformations by composing discrete steps, which can be seen as an Euler discretization of a continuous transformation. The authors investigate the limit as step sizes go to zero.
• Memory Efficiency: Backpropagating through deep discrete networks requires storing intermediate activations, leading to linear memory cost in terms of depth, which is a major bottleneck.
• Irregular Data: Recurrent Neural Networks (RNNs) struggle with data arriving at arbitrary times, typically requiring discretization into fixed bins.
• Normalizing Flow Costs: Standard normalizing flows have a bottleneck in computing the determinant of the Jacobian, which is computationally expensive ($O(D^3)$).

What is the novelty here?

The core contribution is the Neural ODE formulation: $$\frac{dh(t)}{dt} = f(h(t), t, \theta)$$ where the output is computed using a black-box differential equation solver.

Key technical innovations include:

1. Adjoint Sensitivity Method for Backprop: The authors treat the solver as a black box and compute gradients by solving a second, augmented ODE backwards in time. This allows for constant memory cost regardless of depth.
2. Adaptive Computation: The model uses modern ODE solvers that adapt evaluation steps based on error tolerance, allowing the model to trade precision for speed explicitly.
3. Continuous Normalizing Flows (CNF): By moving to continuous time, the change of variables formula simplifies from a log-determinant (cubic cost) to a trace operation (linear cost), enabling scalable generative modeling.
4. Latent ODEs: A generative time-series model that represents time-series as latent trajectories determined by a local initial state and global shared dynamics, handling irregular sampling naturally.

What experiments were performed?

The authors validated the method across three distinct domains:

1. Supervised Learning (MNIST):
   • Compared ODE-Net against a standard ResNet and a Runge-Kutta network (RK-Net).
   • Measured test error, parameter count, and memory usage.
   • Analyzed the trade-off between numerical precision (tolerance) and speed (NFE).
2. Continuous Normalizing Flows (Generative):
   • Compared CNF against standard Normalizing Flows (NF) on density matching and maximum likelihood estimation tasks using toy 2D datasets (Two Circles, Two Moons, and other target distributions).
   • Evaluated training loss (KL divergence) and maximum likelihood estimation.
3. Time-Series Modeling (Latent ODE):
   • Tested on a dataset of bi-directional spirals with irregular timestamps and Gaussian noise.
   • Compared Latent ODEs against an RNN baseline on predictive RMSE. A second RNN variant with time-difference concatenation was also trained.

What outcomes/conclusions?

• Efficiency: ODE-Nets achieved roughly equivalent accuracy to ResNets on MNIST (0.42% vs 0.41% error) but with constant memory cost ($O(1)$) compared to ResNet’s linear cost ($O(L)$).
• Adaptive Depth: The number of function evaluations (NFE) in ODE-Nets increases with training epoch, suggesting the model adapts its complexity as it learns. The backward pass NFE is roughly half the forward pass NFE, indicating that the adjoint method is also more computationally efficient than direct backpropagation through the integrator.
• Generative Performance: Continuous Normalizing Flows (CNF) achieved lower KL divergence loss than standard Normalizing Flows (NF), trained with only 10,000 iterations (Adam) compared to 500,000 iterations (RMSprop) for NF. Note that the two models used different optimizers, so the comparison is not fully controlled. CNF can also expand capacity by increasing width ($M$) without architectural constraints.
• Irregular Time-Series: Latent ODEs significantly outperformed RNNs across all observation counts on irregular spiral data. The advantage is most pronounced with sparse observations (0.1642 vs 0.3937 RMSE at 30 obs), and the model learns interpretable latent trajectories that switch direction smoothly.

Reproducibility Details

Data

• MNIST: Standard handwritten digit dataset used for supervised learning benchmarks.
• Toy 2D Densities: “Two Circles” and “Two Moons” distributions used for visualizing normalizing flows.
• Bi-directional Spirals: A generated dataset of 1,000 2D spirals (half clockwise, half counter-clockwise). Each spiral is sampled at 100 equally-spaced timesteps with added Gaussian noise. For training, each spiral is then subsampled without replacement to $n \in {30, 50, 100}$ irregularly-spaced observations, simulating realistic missing data.

Algorithms

1. Adjoint Sensitivity Method (Backpropagation)

To optimize the parameters of the ODE-Net, the authors use the adjoint sensitivity method to compute gradients. Standard backpropagation would require storing the activations at every step of the ODE solver, incurring a high memory cost that scales linearly with the number of steps.

Instead, this method treats the ODE solver as a “black box” and computes gradients by solving a second, augmented ODE backwards in time from the final state $t_1$ to the initial state $t_0$.

The augmented state contains three components that are solved simultaneously:

1. The State: The original hidden state $z(t)$, which is reconstructed backwards.
2. The Adjoint: The sensitivity of the loss with respect to the state, $a(t) = \partial L / \partial z(t)$.
3. The Gradient: The accumulating gradients with respect to parameters, $\partial L / \partial \theta$.

The dynamics of this augmented system are defined as: $$\frac{d}{dt}\begin{bmatrix} z(t) \ a(t) \ \partial L/\partial \theta \end{bmatrix} = \begin{bmatrix} f(z(t), t, \theta) \ -a(t)^T \frac{\partial f}{\partial z} \ -a(t)^T \frac{\partial f}{\partial \theta} \end{bmatrix}$$

Using this approach, the vector-Jacobian products (e.g., $a(t)^T \frac{\partial f}{\partial z}$) are evaluated efficiently using automatic differentiation.

Why: Reconstructing $z(t)$ backwards avoids storing the forward pass, enabling constant memory cost ($O(1)$) regardless of depth.

Origin: Adapted from Pontryagin’s maximum principle (1962) for optimal control.

import torch
import torch.nn as nn
from torchdiffeq import odeint_adjoint

class ODEFunc(nn.Module):
    def __init__(self, dim):
        super(ODEFunc, self).__init__()
        self.net = nn.Sequential(
            nn.Linear(dim, 50),
            nn.Tanh(),
            nn.Linear(50, dim),
        )

    def forward(self, t, y):
        # Defines dy/dt = f(y, t)
        return self.net(y)

# Usage with adjoint method for O(1) memory backprop
func = ODEFunc(dim=2)
y0 = torch.tensor([[1., 0.]]) # Initial state
t = torch.linspace(0., 1., 10) # Time points to solve for

# 'odeint_adjoint' automatically handles the augmented state backward pass
out = odeint_adjoint(func, y0, t, method='dopri5')

2. Instantaneous Change of Variables (CNF)

For generative modeling, the authors introduce Continuous Normalizing Flows (CNF). In discrete normalizing flows, the probability density of a transformed variable is calculated using the change of variables theorem, which requires computing the log-determinant of the Jacobian: $\log p(z_1) = \log p(z_0) - \log |\det \frac{\partial z_1}{\partial z_0}|$. This operation is computationally expensive ($O(D^3)$) and often restricts model architectures to ensure the Jacobian is easy to compute (e.g., triangular).

Moving to continuous time simplifies this requirement. The paper proves that if the transformation is defined by an ODE, the change in log-probability follows a differential equation determined by the trace of the Jacobian: $$\frac{\partial \log p(z(t))}{\partial t} = -\text{tr}\left( \frac{\partial f}{\partial z(t)} \right)$$

The total change in log-density is obtained by integrating this value over time.

def get_trace(y, f):
    """
    Computes trace of Jacobian df/dy.
    For high dimensions, use Hutchinson's trace estimator (approximate).
    """
    tr = 0.
    for i in range(y.size(1)):
        # Gradients of f's i-th component w.r.t y's i-th component
        tr += torch.autograd.grad(f[:, i].sum(), y, create_graph=True)[0][:, i]
    return tr

# In the ODE function:
# d(log_p)/dt = -trace(df/dy)

Why: The trace operator has linear cost ($O(D)$), whereas the determinant has cubic cost ($O(D^3)$). This allows for unrestricted, “wide” architectures that are automatically bijective.

Origin: This is the “Instantaneous Change of Variables” theorem (Theorem 1), derived in Appendix A of the paper.

Models

ODE-Net (MNIST Classification):

• Input: Downsamples input twice.
• Core: 6 standard residual blocks replaced by a single ODESolve module.
• Output: Global average pooling + Fully connected layer.
• Solver: Implicit Adams method.

class ODEBlock(nn.Module):
    def __init__(self, odefunc):
        super(ODEBlock, self).__init__()
        self.odefunc = odefunc
        self.integration_time = torch.tensor([0, 1]).float()

    def forward(self, x):
        self.integration_time = self.integration_time.type_as(x)
        # Returns [x(t0), x(t1)]; we only want final state x(t1)
        out = odeint_adjoint(self.odefunc, x, self.integration_time)
        return out[1]

# ResNet-like architecture with ODE block
model = nn.Sequential(
    nn.Conv2d(1, 64, 3, 1),
    nn.ReLU(inplace=True),
    ODEBlock(ODEFunc(64)), # Continuous-depth layer replacement
    nn.BatchNorm2d(64),
    nn.AdaptiveAvgPool2d((1, 1)),
    nn.Flatten(),
    nn.Linear(64, 10)
)

Latent ODE (Time-Series):

• Encoder: RNN with 25 hidden units processing data backwards to produce $q(z_0|x)$. It runs backwards so the final RNN state summarizes the entire sequence at $t_0$, parameterizing the initial latent state $z_0$ for the forward-running ODE.
• Latent Space: 4-dimensional latent state $z_0$.
• Dynamics ($f$): Neural network with one hidden layer of 20 units.
• Decoder: Neural network with one hidden layer of 20 units computing $p(x_{t_i}|z_{t_i})$.
• Likelihood: Gaussian log-likelihood for the spiral reconstruction task. The paper also describes an optional Poisson process likelihood $\lambda(z(t))$ for event-time data (e.g., medical records), but this is not used in the spiral experiment.

Evaluation

Hardware

• Implementation: Hidden state dynamics evaluated on GPU using TensorFlow.
• Solvers: Fortran ODE solvers (LSODE, VODE) from scipy.integrate were used for the actual integration.
• Note: While the original paper used TensorFlow/Scipy, the authors later released torchdiffeq (PyTorch), which has become the standard implementation for this architecture. The code samples above reflect this modern standard.
• Interface: Python’s autograd framework bridged the TensorFlow dynamics and Scipy solvers.

Limitations

The paper identifies several practical limitations of Neural ODEs:

• Minibatching: Batching requires concatenating states of each batch element into a combined ODE of dimension $D \times K$. Controlling error on all batch elements together can require more evaluations than solving each system individually, though in practice this overhead was not substantial.
• Tolerance tuning: Users must choose error tolerances for both the forward and reverse passes. The paper used 1.5e-8 for sequence modeling, 1e-3 for classification, and 1e-5 for density estimation.
• Backward trajectory reconstruction: Running the dynamics backwards to reconstruct the forward state trajectory can introduce extra numerical error if the reconstructed trajectory diverges from the original. Checkpointing (storing intermediate states) can address this, though the authors did not find it necessary in practice.
• Uniqueness requirements: The neural network $f$ must be Lipschitz continuous (e.g., using tanh or ReLU activations with finite weights) to guarantee a unique solution via Picard’s existence theorem.

Artifacts

Paper Information

Citation: Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018). Neural ordinary differential equations. Proceedings of the 32nd International Conference on Neural Information Processing Systems, 6572-6583.

Publication: NeurIPS 2018

@inproceedings{chen2018neural,
  title={Neural ordinary differential equations},
  author={Chen, Ricky T. Q. and Rubanova, Yulia and Bettencourt, Jesse and Duvenaud, David},
  booktitle={Proceedings of the 32nd International Conference on Neural Information Processing Systems},
  pages={6572--6583},
  year={2018}
}

Additional Resources:

• Official PyTorch Implementation

This is primarily a Method paper, as it introduces “Flow Matching” (FM), a novel simulation-free paradigm for training Continuous Normalizing Flows (CNFs) at scale. It is supported by a strong Theory basis, providing formal theorems that allow the intractable marginal vector field regression to be solved via a tractable conditional objective. It also touches on Systematization by showing that existing diffusion paths are specific instances of the proposed Gaussian probability path framework.

What is the motivation?

The paper aims to overcome the scaling limitations of Continuous Normalizing Flows (CNFs).

• Problem: Standard Maximum Likelihood training for CNFs requires expensive numerical ODE simulations during training, which scales poorly. Existing simulation-free methods often involve intractable integrals or result in biased gradients.
• Gap: Diffusion models scale well, yet they are restricted to specific, curved probability paths (e.g., VP, VE) that can result in slow sampling and long training times.
• Goal: To develop an efficient, simulation-free training method for CNFs that supports arbitrary probability paths, specifically allowing for straighter, more efficient trajectories like those from Optimal Transport.

What is the novelty here?

The core novelty is Flow Matching (FM) and specifically the Conditional Flow Matching (CFM) objective.

• Direct Vector Field Regression: The model regresses a target vector field $u_t$ that generates a desired probability path $p_t$.
• Conditional Flow Matching (CFM): The authors prove that regressing the vector field of conditional paths (e.g., $p_t(x|x_1)$ given a single data point) yields the same gradients as regressing the intractable marginal vector field. This bypasses the need to know the marginal score or vector field.
• Optimal Transport Paths: The framework enables the use of Optimal Transport (OT) displacement interpolation for probability paths. OT paths are straight lines with constant speed, leading to faster training and easier sampling.

Concurrent work note: Rectified Flow (Liu et al., 2023) and Stochastic Interpolants (Albergo & Vanden-Eijnden, 2023) were published concurrently at ICLR 2023 with structurally similar contributions under different names. All three independently propose simulation-free training of continuous flows via direct vector field regression; the differences lie in the specific interpolation schemes, theoretical framing, and experimental focus.

What experiments were performed?

• Domains: 2D Checkerboard data, CIFAR-10, and ImageNet at resolutions $32 \times 32$, $64 \times 64$, and $128 \times 128$.
• Task: Unconditional generative modeling (density estimation and sample quality) and conditional super-resolution ($64 \times 64 \to 256 \times 256$).
• Baselines: Compared against Diffusion-based methods on the same architecture (U-Net): DDPM, Score Matching (SM), and ScoreFlow.
• Ablations: Specifically compared FM with Diffusion paths vs. FM with Optimal Transport (OT) paths to isolate the benefit of the training objective vs. the path choice.

What outcomes/conclusions?

• Outperforms diffusion baselines: FM-OT consistently outperforms all diffusion-based methods (DDPM, Score Matching, ScoreFlow) in both Likelihood (NLL) and Sample Quality (FID) across CIFAR-10 and ImageNet, using the same U-Net architecture and training budget. Selected rows from Table 1 (NLL in bits per dimension, BPD; lower is better for all three metrics; “FM w/ OT” and “FM w/ Diffusion” refer to FM trained with OT paths and Diffusion paths respectively):

• Training stability: FM with diffusion paths (FM w/ Diffusion) is itself a more stable alternative to diffusion training than DDPM and Score Matching, as shown by training curves in the paper (Figure 5), even before switching to OT paths. The OT path then provides further gains.
• Sampling speed: The straight trajectories of OT paths allow accurate sampling with significantly fewer function evaluations (NFE) compared to diffusion paths.
• Generality: Diffusion is a specific instance of Gaussian probability paths within FM. OT paths are a better-optimized alternative available within the same framework.
• Downstream adoption: Flow matching has been adopted beyond image generation. DynamicFlow uses it as the generative backbone for simultaneously generating ligand molecules and transforming protein pockets, extending flow matching to structure-based drug design.

Reproducibility Details

Data

• Datasets: CIFAR-10, ImageNet ($32 \times 32$, $64 \times 64$, $128 \times 128$).
• Preprocessing:
  • Images are center-cropped and resized.
  • For $32 \times 32$ and $64 \times 64$, the preprocessing follows Chrabaszcz et al. (2017).
  • Data is transformed via $\varphi(y) = 2^7(y+1)$ mapping $[-1, 1]$ pixel values to $[0, 256]$ for BPD computation.

Algorithms

1. Conditional Flow Matching (CFM) Objective

The practical training objective used is the CFM loss, which bypasses intractable marginalization:

$$\mathcal{L}_{CFM}(\theta) = \mathbb{E}_{t, q(x_1), p(x_0)} | v_t(\psi_t(x_0)) - u_t(\psi_t(x_0) | x_1) |^2$$

Where $t \sim \mathcal{U}[0,1]$, $x_1 \sim q(x_1)$ (data), and $x_0 \sim p(x_0)$ (noise).

2. Optimal Transport (OT) Probability Path

The authors recommend the OT path for efficiency.

• Mean/Std Schedule: $\mu_t(x) = t x_1$ and $\sigma_t(x) = 1 - (1 - \sigma_{min})t$.
• Conditional Flow Map: $\psi_t(x) = (1 - (1 - \sigma_{min})t)x + t x_1$.
• Target Vector Field: The closed-form regression target for OT is: $$u_t(x|x_1) = \frac{x_1 - (1 - \sigma_{min})x}{1 - (1 - \sigma_{min})t}$$

3. Sampling

Sampling is performed by solving the ODE $\frac{d}{dt}\phi_t(x) = v_t(\phi_t(x))$ from $t=0$ to $t=1$ using the learned vector field $v_t$.

• Solver: dopri5 (adaptive) is used for robust evaluation. Fixed-step solvers (Euler, Midpoint) are used for low-NFE efficiency tests.

Models

• Architecture: U-Net architecture from Dhariwal & Nichol (2021) is used for all image experiments.
• Toy Data: 5-layer MLP with 512 neurons.
• Hyperparameters:
  • Optimizer: Adam ($\beta_1=0.9, \beta_2=0.999$, weight decay=0.0).
  • Learning Rate: Polynomial decay or constant (see Table 3 in paper).
  • $\sigma_{min}$: Set to a small value (e.g., $1e-5$).

Evaluation

• Metrics:
  • NLL (BPD): Computed using the continuous change of variables formula, estimated via the Hutchinson trace estimator to bypass $O(d^3)$ divergence computation.
  • FID: Frechet Inception Distance for sample quality.
  • NFE: Number of Function Evaluations required by the solver.
• Likelihood Computation: Requires solving an augmented ODE to track the log-density change: $$\frac{d}{dt} \begin{bmatrix} \phi_t(x) \ f(t) \end{bmatrix} = \begin{bmatrix} v_t(\phi_t(x)) \ -\text{div}(v_t(\phi_t(x))) \end{bmatrix}$$

Hardware

• CIFAR-10: 2 GPUs.
• ImageNet-32: 4 GPUs.
• ImageNet-64: 16 GPUs.
• ImageNet-128: 32 GPUs.
• Precision: Full 32-bit for CIFAR/IM-32; 16-bit mixed precision for IM-64/128.

Artifacts

The paper does not release the original training code or model weights used in the experiments. The facebookresearch/flow_matching library was released later as a general-purpose PyTorch implementation of flow matching algorithms. Standard benchmark datasets (CIFAR-10, ImageNet) are publicly available.

Theoretical Notes: Why CFM Works

The paper relies on three key theorems to make training tractable.

Theorem 1 (Marginal Generation):

Marginalizing conditional vector fields $u_t(x|x_1)$ yields the correct marginal vector field $u_t(x)$ that generates the marginal probability path $p_t(x)$.

$$u_t(x) = \int u_t(x|x_1) \frac{p_t(x|x_1)q(x_1)}{p_t(x)} dx_1$$

Understanding the Proof:

To understand why this theorem holds, we have to look at the Continuity Equation, which is the fundamental partial differential equation (PDE) that links a probability density path $p_t$ to a vector field $u_t$.

A vector field $u_t$ is said to “generate” a probability path $p_t$ if and only if they satisfy the continuity equation:

$$\frac{\partial p_t(x)}{\partial t} + \nabla \cdot (p_t(x) u_t(x)) = 0$$

The proof of Theorem 1 relies on substituting the definitions of the marginal path and vector field into this equation to see if they balance out.

Step-by-Step Proof:

1. Start with the time derivative of the marginal path: We begin by differentiating the marginal probability path $p_t(x)$ with respect to time. By definition, the marginal path is the integral of the conditional paths over the data distribution: $$\frac{\partial p_t(x)}{\partial t} = \frac{\partial}{\partial t} \int p_t(x|x_1) q(x_1) dx_1$$

2. Swap derivative and integral: Assuming standard regularity conditions (Leibniz Rule), we can move the time derivative inside the integral: $$\frac{\partial p_t(x)}{\partial t} = \int \frac{\partial p_t(x|x_1)}{\partial t} q(x_1) dx_1$$

3. Apply the Conditional Continuity Equation: This is the critical step. We know that the conditional vector field $u_t(x|x_1)$ generates the conditional path $p_t(x|x_1)$. Therefore, for every single sample $x_1$, the pair satisfies the continuity equation: $$\frac{\partial p_t(x|x_1)}{\partial t} = -\nabla \cdot (p_t(x|x_1) u_t(x|x_1))$$

   Substituting this into our integral gives: $$\frac{\partial p_t(x)}{\partial t} = -\int \nabla \cdot (p_t(x|x_1) u_t(x|x_1)) q(x_1) dx_1$$

4. Pull the Divergence out: Since the divergence operator ($\nabla \cdot$) acts on $x$ and the integral is over $x_1$, we can pull the divergence operator outside the integral (by linearity): $$\frac{\partial p_t(x)}{\partial t} = -\nabla \cdot \left( \int p_t(x|x_1) u_t(x|x_1) q(x_1) dx_1 \right)$$

5. Match with the Marginal Vector Field Definition: Now, look at the term inside the parentheses. The paper defines the marginal vector field $u_t(x)$ specifically to make this term simpler. Rearranging the definition of $u_t(x)$ provided in the theorem: $$p_t(x) u_t(x) = \int p_t(x|x_1) u_t(x|x_1) q(x_1) dx_1$$

   Substitute $p_t(x) u_t(x)$ back into our equation from Step 4: $$\frac{\partial p_t(x)}{\partial t} = -\nabla \cdot (p_t(x) u_t(x))$$

Conclusion: We have just shown that $\frac{\partial p_t(x)}{\partial t} + \nabla \cdot (p_t(x) u_t(x)) = 0$. This is exactly the continuity equation. Because the marginal path and the aggregated marginal vector field satisfy this equation, the vector field is proven to generate the path.

Theorem 2 (Gradient Equivalence):

The intractable Flow Matching objective $\mathcal{L}_{FM}$ (which requires $u_t(x)$) has the same gradients as the tractable Conditional Flow Matching objective $\mathcal{L}_{CFM}$.

$$\nabla_\theta \mathcal{L}_{FM}(\theta) = \nabla_\theta \mathcal{L}_{CFM}(\theta)$$

This allows the model to learn the marginal vector field by only seeing conditional sample paths.

Understanding the Proof:

The reason Theorem 2 holds is that the “Conditional Flow Matching” (CFM) objective is essentially an unbiased estimator of the “Flow Matching” (FM) objective (up to a constant). When we average over all the conditional data points $x_1$, the “cross-term” in the loss function aligns perfectly with the marginal vector field.

1. Expand the Loss Functions

First, let’s look at the squared error in both objectives. Recall that $v_t$ is our neural network (parameterized by $\theta$), $u_t$ is the intractable marginal target, and $u_t(x|x_1)$ is the tractable conditional target.

Expanding the squared norms:

• FM Objective: $$\mathcal{L}_{FM}(\theta) = \mathbb{E}_{t, p_t(x)} \left[ |v_t(x)|^2 - 2v_t(x) \cdot u_t(x) + |u_t(x)|^2 \right]$$

• CFM Objective: $$\mathcal{L}_{CFM}(\theta) = \mathbb{E}_{t, q(x_1), p_t(x|x_1)} \left[ |v_t(x)|^2 - 2v_t(x) \cdot u_t(x|x_1) + |u_t(x|x_1)|^2 \right]$$

Key Insight: When we take the gradient $\nabla_\theta$, the last term in both equations disappears because the targets ($u_t$) are independent of the network weights $\theta$. We only need to show that the expectations of the first two terms match.

2. Matching the First Term ($|v_t(x)|^2$)

This part is straightforward. The expectation of $|v_t(x)|^2$ is the same in both cases because of how the marginal density $p_t(x)$ is defined.

• FM: averages over $p_t(x)$.
• CFM: averages over $p_t(x|x_1)q(x_1)$.

Since $p_t(x) = \int p_t(x|x_1) q(x_1) dx_1$ (by definition), averaging over the joint distribution is mathematically identical to averaging over the marginal $p_t(x)$.

3. Matching the Cross Term (The “Trick”)

This is the critical part of the proof. We need to show that the interaction between the network and the marginal field equals the interaction between the network and the conditional field.

The Goal: Show $\mathbb{E}_{t, p_t(x)} [v_t(x) \cdot u_t(x)] = \mathbb{E}_{t, q(x_1), p_t(x|x_1)} [v_t(x) \cdot u_t(x|x_1)]$.

The Proof:

1. Start with the FM cross-term (marginal): $$\mathbb{E}_{t, p_t(x)} [v_t(x) \cdot u_t(x)]$$

2. Substitute the definition of the marginal vector field $u_t(x)$ derived in Theorem 1: $$u_t(x) = \int u_t(x|x_1) \frac{p_t(x|x_1) q(x_1)}{p_t(x)} dx_1$$

3. Plug this into the integral. The $p_t(x)$ terms cancel: $$\mathbb{E}_{t, p_t(x)} [v_t(x) \cdot u_t(x)] = \int_t \int_x p_t(x) v_t(x) \cdot \left[ \int_{x_1} u_t(x|x_1) \frac{p_t(x|x_1) q(x_1)}{p_t(x)} dx_1 \right] dx$$

4. This simplifies to: $$= \int_t \int_x \int_{x_1} v_t(x) \cdot u_t(x|x_1) p_t(x|x_1) q(x_1) dx_1 dx dt$$

5. This is exactly the definition of the expectation in the CFM objective: $$= \mathbb{E}_{t, q(x_1), p_t(x|x_1)} [v_t(x) \cdot u_t(x|x_1)]$$

Conclusion: Because the expectations of all terms involving $\theta$ are identical, the gradients must be identical.

Intuitively, this works like Denoising Score Matching or Stochastic Gradient Descent: even though each individual conditional vector field $u_t(x|x_1)$ points to a specific data point $x_1$ (which may differ from the true marginal direction), the average of all these pulls equals the true marginal vector field $u_t(x)$.

Theorem 3 (Gaussian Conditional VFs):

For any Gaussian probability path $p_t(x|x_1) = \mathcal{N}(x | \mu_t(x_1), \sigma_t(x_1)^2 I)$, the unique vector field generating it is available in closed form:

$$u_t(x|x_1) = \frac{\sigma’_t(x_1)}{\sigma_t(x_1)}(x - \mu_t(x_1)) + \mu’_t(x_1)$$

This theorem allows explicitly defining targets for both Diffusion (curved) and Optimal Transport (straight) paths.

Understanding the Proof:

The derivation of Theorem 3 comes from the direct relationship between a flow map $\psi_t$ and its generating vector field. Because we chose a specific, simple path (Gaussian), we can invert the flow map to find the vector field in closed form.

1. Define the Flow Map $\psi_t$

We start by defining the conditional probability path as a Gaussian:

$$p_t(x|x_1) = \mathcal{N}(x | \mu_t(x_1), \sigma_t(x_1)^2 I)$$

The simplest way to “push” a standard normal distribution (noise) $p_0 = \mathcal{N}(0, I)$ to this Gaussian is using an affine transformation (scaling and shifting). We define the flow map $\psi_t$ as:

$$\psi_t(x_0) = \sigma_t(x_1) x_0 + \mu_t(x_1)$$

This map takes a noise sample $x_0$ and transforms it into a sample $x$ at time $t$.

2. The Definition of a Generating Vector Field

By definition, a vector field $u_t$ generates a flow $\psi_t$ if the vector field describes the instantaneous velocity of the flow at any point. Mathematically:

$$u_t(\psi_t(x_0)) = \frac{d}{dt}\psi_t(x_0)$$

Let $x = \psi_t(x_0)$ be the position of the particle at time $t$. We want to find $u_t(x)$.

3. Invert the Flow Map

To find $u_t(x)$, we must express the equation in terms of $x$ rather than $x_0$. Since our flow map is a simple affine transformation (multiply and add), it is easily invertible (assuming $\sigma_t(x_1) \neq 0$):

$$x_0 = \frac{x - \mu_t(x_1)}{\sigma_t(x_1)}$$

We will call this inverse map $\psi_t^{-1}(x)$.

4. Differentiate the Flow Map

Now we calculate the left side of our definition equation (velocity): $\frac{d}{dt}\psi_t(x_0)$.

Taking the time derivative of $\psi_t(x_0) = \sigma_t(x_1) x_0 + \mu_t(x_1)$:

$$\frac{d}{dt}\psi_t(x_0) = \sigma’_t(x_1) x_0 + \mu’_t(x_1)$$

(Note: $\sigma’_t$ and $\mu’_t$ denote time derivatives).

5. Substitute and Solve

Now we combine everything. We know $u_t(\psi_t(x_0)) = \frac{d}{dt}\psi_t(x_0)$.

Substitute the result from Step 4 into this equation:

$$u_t(\psi_t(x_0)) = \sigma’_t(x_1) x_0 + \mu’_t(x_1)$$

This expresses the vector field in terms of the initial point $x_0$. We must express it in terms of the current point $x$. So, we plug in the inverse formula for $x_0$ derived in Step 3:

$$u_t(x|x_1) = \sigma’_t(x_1) \frac{x - \mu_t(x_1)}{\sigma_t(x_1)} + \mu’_t(x_1)$$

Rearranging terms gives the final closed form:

$$u_t(x|x_1) = \frac{\sigma’_t(x_1)}{\sigma_t(x_1)}(x - \mu_t(x_1)) + \mu’_t(x_1)$$

Why is this useful?

This formula means that as long as you can define a mean schedule $\mu_t(x_1)$ and a standard deviation schedule $\sigma_t(x_1)$ (which is easy to do for both Diffusion and Optimal Transport), you immediately get the exact vector field target $u_t(x|x_1)$ needed to train your neural network, bypassing complex ODE solving or score matching approximations.

Paper Information

Citation: Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., & Le, M. (2023). Flow Matching for Generative Modeling. International Conference on Learning Representations (ICLR).

Publication: ICLR 2023

@inproceedings{lipmanFlowMatchingGenerative2023,
  title = {Flow Matching for Generative Modeling},
  author = {Lipman, Yaron and Chen, Ricky T. Q. and Ben-Hamu, Heli and Nickel, Maximilian and Le, Matt},
  booktitle = {International Conference on Learning Representations},
  year = {2023}
}

Additional Resources:

• ArXiv

This is primarily a Method paper, with significant Theory contributions.

The authors propose a specific algorithm (“InterFlow”) for constructing generative models based on continuous-time normalizing flows. The work is characterized by the derivation of a new training objective (a simple quadratic loss) that bypasses the computational bottlenecks of previous methods. It includes prominent baseline comparisons against continuous flow methods (FFJORD, OT-Flow) and diffusion models. The theoretical component establishes the validity of the interpolant density satisfying the continuity equation (a conservation law governing how probability mass flows) and bounds the Wasserstein-2 distance (a measure of transport cost between distributions, penalizing squared displacement) of the transport.

What is the motivation?

The primary motivation is to overcome the computational inefficiency of training Continuous Normalizing Flows (CNFs) using Maximum Likelihood Estimation (MLE). Standard CNF training requires backpropagating through numerical ODE solvers, which is costly and limits scalability.

Additionally, while score-based diffusion models (SDEs) have achieved high sample quality, they theoretically require infinite time integration and rely on specific noise schedules. The authors aim to establish a method that works strictly with Probability Flow ODEs on finite time intervals, retaining the flexibility to connect arbitrary densities without the complexity of SDEs or the cost of standard ODE adjoint methods.

What is the novelty here?

The core novelty is the Stochastic Interpolant framework:

• Explicit Interpolant Construction: The method defines a time-dependent interpolant $x_t = I_t(x_0, x_1)$ (e.g., trigonometric interpolation) that connects samples from the base density $\rho_0$ and target $\rho_1$.
• Simulation-Free Training: The velocity field $v_t(x)$ of the probability flow is learned by minimizing a simple quadratic objective: $G(\hat{v}) = \mathbb{E}[|\hat{v}_t(x_t)|^2 - 2\partial_t x_t \cdot \hat{v}_t(x_t)]$. Because $\partial_t I_t$ is known analytically from the interpolant definition, the expectation can be estimated by sampling $(x_0, x_1, t)$ directly. This avoids ODE integration during training (ODE integration is still required at inference).
• Decoupling Path and Optimization: The choice of path (interpolant) is separated from the optimization of the velocity field. MLE methods couple the path and objective.
• Connection to Score-Based Models: The authors show that for Gaussian base densities and trigonometric interpolants, the learned velocity field is explicitly related to the score function $\nabla \log \rho_t$, providing a theoretical bridge between CNFs and diffusion models.

What experiments were performed?

The authors performed validation across synthetic, tabular, and image domains:

• 2D Density Estimation: Benchmarked on “Checkerboard”, “8 Gaussians”, and anisotropic curved densities to visualize mode coverage and transport smoothness.
• High-Dimensional Tabular Data: Evaluated on standard benchmarks (POWER, GAS, HEPMASS, MINIBOONE, BSDS300) comparing Negative Log Likelihood (NLL) against FFJORD, OT-Flow, and others.
• Image Generation: Trained models on CIFAR-10 ($32 \times 32$), ImageNet ($32 \times 32$), and Oxford Flowers ($128 \times 128$) to test scalability.
• Ablations: Investigated optimizing the interpolant path itself (e.g., learning Fourier coefficients for the path) to approach optimal transport and minimize path length.

What outcomes/conclusions?

• Performance: The method matches or supersedes conventional ODE flows (like FFJORD) in terms of NLL while being significantly cheaper to train.
• Efficiency: The training cost per epoch is constant (simulation-free), whereas MLE-based ODE methods see growing costs as the dynamics become more complex.
• Scalability: The method successfully scales to $128 \times 128$ resolution on a single GPU, a resolution that prior ab-initio ODE flows had not demonstrated.
• Flexibility: The framework can connect any two arbitrary densities (e.g., connecting two different complex 2D distributions) without needing one to be Gaussian.
• Optimal Transport: For a fixed interpolant, minimizing $G(\hat{v})$ over the velocity field recovers the velocity for that specific path. Additionally optimizing over the interpolant family yields a solution to the Benamou-Brenier optimal transport problem.
• Limitations: The authors acknowledge that image FID scores trail dedicated diffusion models, noting that InterFlow was not optimized with standard training tricks such as exponential moving averages, truncation, or learning rate warm-ups. The framework’s sample quality could likely improve with these additions.

Reproducibility Details

Data

• Tabular Datasets: POWER (6D), GAS (8D), HEPMASS (21D), MINIBOONE (43D), BSDS300 (63D).
  • Training points range from ~30k (MINIBOONE) to ~1.6M (POWER).
• Image Datasets:
  • CIFAR-10 ($32 \times 32$, 50k training points).
  • ImageNet ($32 \times 32$, ~1.28M training points).
  • Oxford Flowers ($128 \times 128$, ~315k training points).
• Time Sampling: Time $t$ is sampled from a Beta distribution during training (reweighting) to focus learning near the target.

Algorithms

• Interpolant: The primary interpolant used is trigonometric: $I_t(x_0, x_1) = \cos(\frac{\pi t}{2})x_0 + \sin(\frac{\pi t}{2})x_1$.
  • Alternative linear interpolant: $I_t = a_t x_0 + b_t x_1$.
• Loss Function: $$G(\hat{v}) = \mathbb{E}_{t, x_0, x_1}[|\hat{v}_t(x_t)|^2 - 2\partial_t I_t(x_0, x_1) \cdot \hat{v}_t(x_t)]$$
  • The expectation is amenable to empirical estimation using batches of $x_0, x_1, t$.
• Sampling: Numerical integration using Dormand-Prince (Runge-Kutta 4/5).
• Optimization: SGD/Adam variants used for optimization.

Models

• Tabular Architectures:
  • Feed-forward networks with 4-5 hidden layers.
  • Hidden widths: 512 (POWER, GAS, HEPMASS, MINIBOONE) or 1024 (BSDS300).
  • Activation: ReLU (general) or ELU (BSDS300).
• Image Architectures:
  • U-Net based on the DDPM implementation.
  • Dimensions: 256 hidden dimension.
  • Sinusoidal time embeddings used.

Evaluation

• Metrics: Negative Log Likelihood (NLL) in nats (tabular) or bits per dim (images), Frechet Inception Distance (FID) for images.
• Baselines: FFJORD, Glow, Real NVP, OT-Flow, ScoreFlow, DDPM.

Tabular NLL (nats, lower is better; Table 2 Left):

Image Generation NLL and FID (Table 2 Right; NLL in bits per dim, lower is better):

Note: DDPM++ is from Song et al. (2021), the same work as ScoreSDE (it is the architecture optimized for VP/sub-VP SDEs). InterFlow matches ScoreSDE on CIFAR-10 NLL (2.99 bits per dim) while being simulation-free. FID is weaker than dedicated image models (10.27 vs 2.92 for ScoreSDE), reflecting the paper’s primary focus on tractable likelihood rather than sample quality.

Hardware

• Compute: All models were trained on a single NVIDIA A100 GPU.
• Training Time:
  • Tabular: $10^5$ steps.
  • Images: $1.5 \times 10^5$ to $6 \times 10^5$ steps.
  • Speedup: Demonstrated ~400x speedup compared to FFJORD on MiniBooNE dataset.

Artifacts

No official code release accompanies this paper. All tabular datasets (POWER, GAS, HEPMASS, MINIBOONE, BSDS300) are publicly available from prior work. CIFAR-10 and ImageNet are standard public benchmarks. Oxford Flowers 102 is also publicly available. Hyperparameters and architectures are fully specified in Tables 3 and 4 of the paper.

Paper Information

Citation: Albergo, M. S., & Vanden-Eijnden, E. (2023). Building Normalizing Flows with Stochastic Interpolants. The Eleventh International Conference on Learning Representations.

Publication: ICLR 2023

@inproceedings{albergoBuildingNormalizingFlows2022,
  title = {Building {{Normalizing Flows}} with {{Stochastic Interpolants}}},
  booktitle = {The {{Eleventh International Conference}} on {{Learning Representations}}},
  author = {Albergo, Michael Samuel and {Vanden-Eijnden}, Eric},
  year = 2023,
  url = {https://openreview.net/forum?id=li7qeBbCR1t}
}

Additional Resources:

• OpenReview
• arXiv

This is a Theory paper. It relies entirely on formal mathematical derivation to establish existence and uniqueness properties for energy functionals. It introduces a new mathematical structure (displacement interpolation) to analyze the geometry of probability measures.

What is the motivation?

The paper addresses the uniqueness of stationary configurations (ground states) for a gas model where particles interact via attractive forces while resisting compression.

The total energy functional $E(\rho)$ includes an interaction term $G(\rho)$ that lacks convexity under standard linear interpolation ($(1-t)\rho + t\rho’$), making it difficult to prove that a unique minimizer exists. Standard convexity tools and rearrangement inequalities are also insufficient for cases without specific symmetries (like spherical symmetry) or when convexity of the potential fails.

What is the novelty here?

The core novelty is the introduction of Displacement Interpolation.

• New Interpolant: The paper defines an interpolant $\rho_t$ by moving mass along the gradient of a convex potential $\psi$ (transport map).
• Displacement Convexity: It proves that the internal energy $U(\rho)$ and potential energy $G(\rho)$ become convex functions of $t$ along this displacement path. This is a property specific to displacement interpolation.
• Generalization: This framework generalizes the classical Brunn-Minkowski inequality from sets to measures.

Theoretical Framework

Mathematical Setup

Probability Measures

The gas state is represented by absolutely continuous probability measures $\rho \in \mathcal{P}_{ac}(\mathbb{R}^d)$ with finite second moments.

Energy Functional

The gas model is defined by the total energy functional $E(\rho)$: $$E(\rho) := \underbrace{\int_{\mathbb{R}^d} A(\rho(x))dx}_{\text{Internal Energy } U(\rho)} + \underbrace{\frac{1}{2} \iint d\rho(x)V(x-y)d\rho(y)}_{\text{Potential Energy } G(\rho)}$$

Key Construction: Displacement Interpolation

The core theoretical tool is the construction of the interpolant $\rho_t$ between two probability measures $\rho$ and $\rho’$:

1. Transport Map: By Brenier’s theorem, there exists a convex function $\psi$ such that $\nabla\psi_{\#}\rho = \rho’$ (push-forward).
2. Interpolation: The interpolant at time $t \in [0,1]$ is defined as the push-forward of $\rho$ under the linear interpolation of the identity and the transport map: $$\rho_t := [(1-t)\text{id} + t\nabla\psi]_{\#}\rho$$

This is the “displacement interpolation” where mass moves along straight lines from initial to final positions.

Assumptions for Uniqueness

The main existence and uniqueness theorem (Theorem 3.1) requires one condition on the interaction potential, two conditions on the equation of state, and one regularity condition:

1. Interaction: $V(x)$ is strictly convex.
2. (P1) Equation of State: $P(\rho) / \rho^{(d-1)/d}$ is non-decreasing. This is equivalent to convexity of $U$ under mass-preserving dilations, and is satisfied by polytropic gases $P(\rho) = \rho^q$ with $q > 1$.
3. (P2) Growth Condition: $P(\rho) \cdot \rho^{-2}$ is not integrable at $\infty$. This ensures the energy minimizer has no singular part with respect to Lebesgue measure.
4. Regularity: $\rho \in \mathcal{P}_{ac}(\mathbb{R}^d)$ (absolutely continuous probability measures).

Main Results

Theorem 2.2 (Displacement Convexity of Internal Energy): Under condition (A1) (that $\lambda^d A(\lambda^{-d})$ is convex non-increasing on $(0, \infty)$ with $A(0) = 0$, ensuring internal energy decreases as the gas dilates), the internal energy $U(\rho)$ is convex along displacement interpolation paths. Strict convexity follows unless $\nabla^2\psi(x) = I$ holds $\rho$-a.e., i.e., $\rho’$ is a translate of $\rho$.

Theorem 3.1 (Existence and Uniqueness of Ground State): For any equation of state satisfying (P1) and (P2) with a strictly convex interaction potential $V$, the total energy $E(\rho)$ attains a unique minimizer up to translation. The minimizer can be taken to be even, meaning $\rho_g(x) = \rho_g(-x)$.

Theorem 3.3 (Uniqueness for Spherically Symmetric Potentials): When the strict convexity of $V(x)$ is relaxed to spherical symmetry (with $V$ not constant), uniqueness up to translation still holds provided (P1) holds strictly. This extends the main result to cases like Coulomb-type interactions.

Lemma 3.2: A decomposition lemma for convex functions. Let $\psi$ and $\phi$ be convex on $\mathbb{R}^d$, and let $\Omega \subset \mathbb{R}^d$ be an open convex set on which both are finite. Suppose $\phi$ is differentiable on $\Omega$ with a locally Lipschitz gradient. If their Aleksandrov second derivatives agree almost everywhere on $\Omega$, then $\psi - \phi$ is convex on $\Omega$. This underpins the proof of Theorem 3.3.

What experiments were performed?

The validation consists entirely of rigorous mathematical proofs:

• Convexity Proofs: Deriving inequalities to show $E(\rho_t) \le (1-t)E(\rho) + tE(\rho’)$.
• Existence/Uniqueness: Using the new convexity principle to prove that the energy minimizer is unique up to translation.

What outcomes/conclusions?

• Uniqueness of Ground State: For equations of state satisfying specific monotonicity conditions (e.g., polytropic gases), the energy minimizing state is unique up to translation.
• Brunn-Minkowski Extension: The internal energy convexity implies the Brunn-Minkowski inequality as a special case ($A(\rho) = -\rho^{(d-1)/d}$).
• Norm Concavity: The functional $|\rho_t|_q^{-p/d}$ is shown to be concave along the interpolation path for conjugate $p, q$ with $q \geq (d-1)/d$.

Relevance to Machine Learning

This 1997 paper establishes the mathematical foundations of displacement convexity in optimal transport theory, which underpins several modern generative modeling techniques. The displacement interpolation framework introduced here is used in:

• Flow Matching: Uses optimal transport probability paths (straight-line interpolations with constant speed) to generate samples. See the Flow Matching note for details on how OT paths differ from diffusion paths.
• Wasserstein GANs: Use the Wasserstein distance (optimal transport metric) for training stability.
• Continuous Normalizing Flows: Use OT-inspired transport maps for probability density transformation.

McCann’s convexity principle proves that energy functionals become convex along displacement paths, a mathematical structure that underpins the geometry used in flow matching and optimal transport-based generative modeling.

Paper Information

Citation: McCann, R. J. (1997). A Convexity Principle for Interacting Gases. Advances in Mathematics, 128(1), 153-179. https://doi.org/10.1006/aima.1997.1634

Publication: Advances in Mathematics 1997

@article{mccannConvexityPrincipleInteracting1997,
  title = {A {{Convexity Principle}} for {{Interacting Gases}}},
  author = {McCann, Robert J.},
  year = 1997,
  month = jun,
  journal = {Advances in Mathematics},
  volume = {128},
  number = {1},
  pages = {153--179},
  issn = {00018708},
  doi = {10.1006/aima.1997.1634},
  urldate = {2025-12-21}
}

This is primarily a Methodological Paper ($\Psi_{\text{Method}}$) with a strong Resource ($\Psi_{\text{Resource}}$) component.

• Method: It proposes DynamicFlow, a novel multiscale architecture combining atom-level SE(3)-equivariant GNNs (SE(3) is the special Euclidean group in 3D: the set of all 3D rotations and translations, and equivariance means predictions transform consistently under those symmetries) and residue-level Transformers within a flow matching framework to model the joint distribution of ligand generation and protein conformational change.
• Resource: It curates a significant dataset derived from MISATO, pairing AlphaFold2-predicted apo structures with multiple MD-simulated holo states, specifically filtered for flow matching tasks.

What is the motivation?

Traditional Structure-Based Drug Design (SBDD) methods typically assume the protein target is rigid, which limits their applicability because proteins are dynamic and undergo conformational changes (induced fit) upon ligand binding.

• Biological Reality: Proteins exist as ensembles of states; binding often involves transitions from “apo” (unbound) to “holo” (bound) conformational changes, sometimes revealing cryptic pockets.
• Computational Bottleneck: Molecular Dynamics (MD) simulates these changes but incurs high computational costs due to energy barriers.
• Gap: Existing generative models for SBDD mostly condition on a fixed pocket structure, ignoring the co-adaptation of the protein and ligand.

What is the novelty here?

The core novelty is the simultaneous modeling of ligand generation and protein conformational dynamics using a unified flow matching framework.

• DynamicFlow Architecture: A multiscale model that treats the protein as both full-atom (for interaction) and residue-level frames (for large-scale dynamics), utilizing separate flow matching objectives for backbone frames, side-chain torsions, and ligand atoms.
• Stochastic Flow (SDE): Introduction of a stochastic variant (DynamicFlow-SDE) that improves robustness and diversity compared to the deterministic ODE flow.
• Coupled Generation: The model learns to transport the apo pocket distribution to the holo pocket distribution while simultaneously denoising the ligand, advancing beyond rigid pocket docking methods.

What experiments were performed?

The authors validated the method on a curated dataset of 5,692 protein-ligand complexes.

• Baselines: Compared against rigid-pocket SBDD methods: Pocket2Mol, TargetDiff, and IPDiff (adapted as TargetDiff* and IPDiff* for fair comparison of atom numbers). Also compared against conformation sampling baselines (Str2Str).
• Metrics:
  • Ligand Quality: Vina Score (binding affinity), QED (drug-likeness), SA (synthesizability), Lipinski’s rule of 5.
  • Pocket Quality: RMSD between generated and ground-truth holo pockets, Cover Ratio (percentage of holo states successfully retrieved), and Pocket Volume distributions.
  • Interaction: Protein-Ligand Interaction Profiler (PLIP) to measure specific non-covalent interactions.
• Ablations: Tested the impact of the interaction loss, residue-level Transformer, and SDE vs. ODE formulations.

What outcomes/conclusions?

• Improved Affinity: DynamicFlow-SDE achieved the best (lowest) Vina scores ($-7.65$) compared to baselines like TargetDiff ($-5.09$) and Pocket2Mol ($-5.50$). Note that Vina scores are a computational proxy and do not directly predict experimental binding affinity. Moreover, Vina score optimization is gameable: molecules can achieve strong computed binding energies while remaining synthetically inaccessible. QED and SA scores, which assess drug-likeness and synthesizability respectively, were reported but were not primary optimization targets in the paper, which limits the strength of this affinity claim.
• Realistic Dynamics: The model successfully generated holo-like pocket conformations with volume distributions and interaction profiles closer to ground-truth MD simulations than the initial apo structures.
• Enhancing Rigid Methods: Holo pockets generated by DynamicFlow served as better inputs for rigid-SBDD baselines (e.g., TargetDiff improved from $-5.09$ to $-9.00$ and IPDiff improved from $-7.55$ to $-11.04$ when using “Our Pocket”), suggesting the method can act as a “pocket refiner”.
• ODE vs. SDE Trade-off: The deterministic ODE variant achieves better pocket RMSD, while the stochastic SDE variant achieves better Cover Ratio (diversity of holo states captured) and binding affinity. Neither dominates uniformly.
• Conformation Sampling Baseline: Str2Str, a dedicated conformation sampling baseline, performed worse than simply perturbing the apo structure with noise. One interpretation is that this highlights the difficulty of the apo-to-holo prediction task; another is that Str2Str was not designed specifically for apo-to-holo prediction, making it a limited test of its capabilities.

Reproducibility Details

Data

The dataset is derived from MISATO, which contains MD trajectories for PDBbind complexes.

Preprocessing:

• Clustering: Holo-ligand conformations clustered with RMSD threshold $1.0\text{\AA}$; top 10 clusters kept per complex.
• Pocket Definition: Residues within $7\text{\AA}$ of the ligand.
• Alignment: AlphaFold predicted structures (apo) aligned to MISATO holo structures using sequence alignment (Smith-Waterman) to identify pocket residues.

Algorithms

Flow Matching Framework:

• Continuous Variables (Pocket translation/rotation/torsions, Ligand positions): Modeled using Conditional Flow Matching (CFM).
  • Prior: Apo state for pocket; Normal distribution for ligand positions.
  • Target: Holo state from MD; Ground truth ligand.
  • Interpolant: Linear interpolation for Euclidean variables; Geodesic for rotations ($SO(3)$, the rotation-only subgroup of SE(3) containing all 3D rotations but not translations); Wrapped linear interpolation for torsions (Torus).
• Discrete Variables (Ligand atom/bond types): Modeled using Discrete Flow Matching based on Continuous-Time Markov Chains (CTMC).
  • Rate Matrix: Interpolates between mask token and data distribution.
• Loss Function: Weighted sum of 7 losses:
  1. Translation CFM (Eq 5)
  2. Rotation CFM (Eq 7)
  3. Torsion CFM (Eq 11)
  4. Ligand Position CFM
  5. Ligand Atom Type CTMC (Eq 14)
  6. Ligand Bond Type CTMC
  7. Interaction Loss (Eq 18): Explicitly penalizes deviations in pairwise distances between protein and ligand atoms for pairs $\leq 3.5\text{\AA}$.

Models

Architecture: DynamicFlow is a multiscale model with 15.9M parameters.

1. Atom-Level SE(3)-Equivariant GNN:
   • Input: Complex graph (k-NN) and Ligand graph (fully connected).
   • Layers: 6 EGNN blocks modified to maintain node and edge hidden states.
   • Function: Updates ligand positions and predicts ligand atom/bond types.
2. Residue-Level Transformer:
   • Input: Aggregated atom features from the GNN + Residue frames/torsions.
   • Layers: 4 Transformer blocks with Invariant Point Attention (IPA).
   • Function: Updates protein residue frames (translation/rotation) and predicts side-chain torsions.

Evaluation

Metrics:

• Vina Score: vina_minimize mode used for binding affinity.
• RMSD: Minimum RMSD between generated pocket and ground-truth holo conformations.
• Cover Ratio: % of ground-truth holo conformations covered by at least one generated sample (threshold $1.42\text{\AA}$).
• POVME 3: For pocket volume calculation.

Hardware

• Inference Benchmark: 1x Tesla V100-SXM2-32GB.
• Speed: Generates 10 ligands in ~35-36 seconds (100 NFE), significantly faster than diffusion baselines like Pocket2Mol (980s) or TargetDiff (156s).

Paper Information

Citation: Zhou, X., Xiao, Y., Lin, H., He, X., Guan, J., Wang, Y., Liu, Q., Zhou, F., Wang, L., & Ma, J. (2025). Integrating Protein Dynamics into Structure-Based Drug Design via Full-Atom Stochastic Flows. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2503.03989

Publication: ICLR 2025

@inproceedings{zhouIntegratingProteinDynamics2025,
  title = {Integrating Protein Dynamics into Structure-Based Drug Design via Full-Atom Stochastic Flows},
  author = {Zhou, Xiangxin and Xiao, Yi and Lin, Haowei and He, Xinheng and Guan, Jiaqi and Wang, Yang and Liu, Qiang and Zhou, Feng and Wang, Liang and Ma, Jianzhu},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://arxiv.org/abs/2503.03989}
}

Additional Resources:

• arXiv Page
• Code: no public repository available at time of writing

This is a Methodological Paper ($\Psi_{\text{Method}}$) from Microsoft Research Asia that addresses the challenge of recognizing complex 2D organic chemistry structures. By 2009, math expression recognition had seen significant commercial progress, but chemical expression recognition remained less developed.

The specific gap addressed is the geometric complexity of organic formulas. While inorganic formulas typically follow a linear, equation-like structure, organic formulas present complex 2D diagrammatic structures with various bond types and rings. Existing work often relied on strong assumptions (like single-stroke symbols) or failed to handle arbitrary compounds. There was a clear need for a unified solution capable of handling both inorganic and organic domains consistently.

The Chemical Expression Structure Graph (CESG)

The core innovation is a unified statistical framework that processes inorganic and organic expressions within the same pipeline. Key technical novelties include:

1. Unified Bond Modeling: Bonds are treated as special symbols. The framework detects “extended bond symbols” (multi-stroke bonds) and splits them into single, double, or triple bonds using corner detection for consistent processing.
2. Chemical Expression Structure Graph (CESG): A defined graph representation for generic chemical expressions where nodes represent symbols and edges represent bonds or spatial relations.
3. Non-Symbol Modeling: During the symbol grouping phase, the system explicitly models “invalid groups” to reduce over-grouping errors.
4. Global Graph Search: Structure analysis is formulated as finding the optimal CESG by searching over a Weighted Direction Graph ($G_{WD}$).

Graph Search and Statistical Validation

The authors validated the framework on a proprietary database of 35,932 handwritten chemical expressions collected from 300 writers.

• Setup: The data was split into roughly 26,000 training and 6,400 testing samples.
• Metric: Recognition accuracy was measured strictly by expression (all symbols and the complete structure must be correct).
• Ablations: The team evaluated the performance contribution of symbol grouping, structure analysis, and full semantic verification.

Recognition Accuracy and Outcomes

The full framework achieved a Top-1 accuracy of 75.4% and a Top-5 accuracy of 83.1%.

• Component Contribution: Structure analysis is the primary bottleneck. Adding it drops the theoretical “perfect grouping” performance from 85.9% to 74.1% (Top-1) due to structural errors.
• Semantic Verification: Checking valence and grammar improved relative accuracy by 1.7%.

The unified framework effectively handles the variance in 2D space for chemical expressions, demonstrating that delayed decision-making (keeping top-N candidates) improves robustness.

Reproducibility Details

Artifacts

Data

The study used a private Microsoft Research Asia dataset, making direct reproduction difficult.

• Content: 2,000 unique expressions from high school/college textbooks.
• Composition: ~25% of samples are organic expressions.
• Vocabulary: 163 symbol classes (elements, digits, +, ↑, %, bonds, etc.).

Algorithms

1. Symbol Grouping (Dynamic Programming)

• Objective: Find the optimal symbol sequence $G_{max}$ maximizing the posterior probability given the ink strokes: $$ G_{max} = \arg\max_{G} P(G | \text{Ink}) $$
• Non-symbol modeling: Iteratively trained models on “incorrect grouping results” to learn to reject invalid strokes.
• Inter-group modeling: Uses Gaussian Mixture Models (GMM) to model spatial relations ($R_j$) between groups.

2. Bond Processing

• Extended Bond Symbol: Recognizes connected strokes (e.g., a messy double bond written in one stroke) as a single “extended” symbol.
• Splitting: Uses Curvature Scale Space (CSS) corner detection to split extended symbols into primitive lines.
• Classification: A Neural Network verifies if the split lines form valid single, double, or triple bonds.

3. Structure Analysis (Graph Search)

• Graph Construction: Builds a Weighted Direction Graph ($G_{WD}$) where nodes are symbol candidates and edges are potential relationships ($E_{c}, E_{nc}, E_{peer}, E_{sub}$).
• Edge Weights: Calculated as the product of observation, spatial, and contextual probabilities: $$ W(S, O, R) = P(O|S) \times P(\text{Spatial}|R) \times P(\text{Context}|S, R) $$
  • Spatial probability uses rectangular control regions and distance functions.
  • Contextual probability uses statistical co-occurrence (e.g., ‘C’ often appears with ‘H’).
• Search: Breadth-first search with pruning to find the top-N optimal CESGs.

Models

• Symbol Recognition: Implementation details not specified, but likely HMM or NN based on the era. Bond verification explicitly uses a Neural Network.
• Spatial Models: Gaussian Mixture Models (GMM) are used to model the 9 spatial relations (e.g., Left-super, Above, Subscript).
• Semantic Model: A Context-Free Grammar (CFG) parser is used for final verification (e.g., ensuring digits aren’t reactants).

Evaluation

Evaluation is performed using “Expression-level accuracy”.

Paper Information

Citation: Chang, M., Han, S., & Zhang, D. (2009). A Unified Framework for Recognizing Handwritten Chemical Expressions. 2009 10th International Conference on Document Analysis and Recognition, 1345–1349. https://doi.org/10.1109/ICDAR.2009.64

Publication: ICDAR 2009

@inproceedings{changUnifiedFrameworkRecognizing2009,
  title = {A {{Unified Framework}} for {{Recognizing Handwritten Chemical Expressions}}},
  booktitle = {2009 10th {{International Conference}} on {{Document Analysis}} and {{Recognition}}},
  author = {Chang, Ming and Han, Shi and Zhang, Dongmei},
  year = 2009,
  volume = {3},
  pages = {1345--1349},
  publisher = {IEEE},
  address = {Barcelona, Spain},
  doi = {10.1109/ICDAR.2009.64}
}

Method. This paper is a methodological contribution that proposes a novel “double-stage classifier” architecture. It fits the taxonomy by introducing a specific algorithmic pipeline (SVM rough classification followed by HMM fine classification) and a novel pre-processing algorithm (Point Sequence Reordering) to solve technical limitations in recognizing organic ring structures. The contribution is validated through ablation studies (comparing SVM kernels and HMM state/Gaussian counts) and performance benchmarks.

Motivation: Recognizing Complex Organic Ring Structures

The primary motivation is the complexity of recognizing handwritten chemical symbols, specifically the distinction between Organic Ring Structures (ORS) and Non-Ring Structures (NRS). Existing single-stage classifiers are unreliable for ORS because these symbols have arbitrary writing styles, variable stroke numbers, and inconsistent stroke orders due to their 2D hexagonal structure. A robust system is needed to handle this uncertainty and achieve high accuracy.

Core Innovation: Point Sequence Reordering (PSR)

The authors introduce two main novelties:

1. Double-Stage Architecture: A hybrid system where an SVM (using RBF kernel) first roughly classifies inputs as either ORS or NRS, followed by specialized HMMs for fine-grained recognition.
2. Point Sequence Reordering (PSR) Algorithm: A stroke-order independent algorithm designed specifically for ORS. It reorders the point sequence of a symbol based on a counter-clockwise scan from the centroid, effectively eliminating the uncertainty caused by variations in stroke number and writing order.

Methodology & Experimental Design

The authors collected a custom dataset and performed sequential optimizations:

• SVM Optimization: Compared Polynomial, RBF, and Sigmoid kernels to find the best rough classifier.
• HMM Optimization: Tested multiple combinations of states (4, 6, 8) and Gaussians (3, 4, 6, 8, 9, 12) to maximize fine classification accuracy.
• PSR Validation: Conducted an ablation study comparing HMM accuracy on ORS symbols “Before PSR” vs “After PSR” to quantify the algorithm’s impact.

Results & Final Conclusions

• Architecture Performance: The RBF-based SVM achieved 99.88% accuracy in differentiating ORS from NRS.
• HMM Configuration: The optimal HMM topology was found to be 8-states and 12-Gaussians for both symbol types.
• PSR Impact: The PSR algorithm improved ORS recognition. Top-1 accuracy shifted from 49.84% (Before PSR) to 98.36% (After PSR).
• Overall Accuracy: The final integrated system achieved a Top-1 accuracy of 93.10% and Top-3 accuracy of 98.08% on the test set.

Reproducibility Details

Data

The study defined 101 chemical symbols split into two categories.

• Collection: 12,322 total samples (122 per symbol) collected from 20 writers (teachers and students).
• Split: 9,090 training samples and 3,232 test samples.
• Constraints: Three specifications were used: normal, standard, and freestyle.

Algorithms

1. SVM Feature Extraction (Rough Classification) The input strokes are scaled, and a 58-dimensional feature vector is calculated:

• Mesh ($4 \times 4$): Ratio of points in 16 grids (16 features).
• Outline: Normalized scan distance from 4 edges with 5 scan lines each (20 features).
• Projection: Point density in 5 bins per edge (20 features).
• Aspect Ratio: Height/Width ratios (2 features).

2. Point Sequence Reordering (PSR) Used strictly for ORS preprocessing:

1. Calculate the centroid $(x_c, y_c)$ of the symbol.
2. Initialize a scan line at angle $\theta = 0$.
3. Traverse points; if a point $p_i = (x_i, y_i)$ satisfies the distance threshold to the scan line, add it to the reordered list. Distance $d_i$ is calculated as: $$ d_i = |(y_i - y_c)\cos(\theta) - (x_i - x_c)\sin(\theta)| $$
4. Increment $\theta$ by $\Delta\theta$ and repeat until a full circle ($2\pi$) is completed.

Models

• SVM (Stage 1): RBF Kernel was selected as optimal with parameters $C=512$ and $\gamma=0.5$.
• HMM (Stage 2): Left-right continuous HMM trained via Baum-Welch algorithm. The topology is one model per symbol using 8 states and 12 Gaussians.

Evaluation

Metrics reported are Top-1, Top-2, and Top-3 accuracy on the held-out test set.

Hardware

• Device: HP Pavilion tx1000 Tablet PC.
• Processor: 2.00GHz CPU.

Paper Information

Citation: Zhang, Y., Shi, G., & Wang, K. (2010). A SVM-HMM Based Online Classifier for Handwritten Chemical Symbols. 2010 International Conference on Pattern Recognition, 1888–1891. https://doi.org/10.1109/ICPR.2010.465

Publication: ICPR 2010

@inproceedings{zhang2010svm,
  title = {A SVM-HMM Based Online Classifier for Handwritten Chemical Symbols},
  booktitle = {2010 International Conference on Pattern Recognition},
  author = {Zhang, Yang and Shi, Guangshun and Wang, Kai},
  year = {2010},
  pages = {1888--1891},
  publisher = {IEEE},
  doi = {10.1109/ICPR.2010.465}
}

This is a Method paper. It proposes a novel architectural framework for processing chemical formulas by decomposing them into three hierarchical levels (Formula, Molecule, Text). The contribution is defined by a specific set of formal grammatical rules and parsing algorithms used to construct a “grammar spanning tree” and “molecule spanning graph” from online handwritten strokes.

Motivation for Online Formula Recognition

The primary motivation is the application of mobile computing in chemistry education, where precise comprehension of casual, online handwritten formulas is a significant challenge.

• 2D Complexity: Unlike 1D text, chemical formulas utilize complex 2D spatial relationships that convey specific chemical meaning (e.g., bonds, rings).
• Format Limitations: Existing storage formats like CML (Chemical Markup Language) or MDL MOLFILE do not natively record the layout or abbreviated information necessary for recognizing handwritten input.
• Online Gap: Previous research focused heavily on offline (image-based) recognition, lacking solutions for online (stroke-based) handwritten chemical formulas (OHCF).

Core Novelty in Three-Level Grammatical Analysis

The core novelty is the Three-Level Grammatical Analysis approach:

1. Formula Level (1D): Treats the reaction equation as a linear sequence of components (Reactants, Products, Separators), parsed via a context-free grammar to build a spanning tree.
2. Molecule Level (2D): Treats molecules as graphs where “text groups” are vertices and “bonds” are edges. It introduces specific handling for “hidden Carbon dots” (intersections of bonds without text).
3. Text Level (1D): Analyzes the internal structure of text groups (atoms, subscripts).

Unique to this approach is the formal definition of the chemical grammar as a 5-tuple $G=(T,N,P,M,S)$ and the generation of an Adjacency Matrix directly from the handwritten sketch to represent chemical connectivity.

Experimental Validation on Handwritten Strokes

The authors validated their model using a custom dataset of online handwritten formulas.

• Data Source: 25 formulas were randomly selected from a larger pool of 1,250 samples.
• Scope: The test set included 484 total symbols, comprising generators, separators, text symbols, rings, and various bond types.
• Granular Validation: The system was tested at multiple distinct stages:
  • Key Symbol Extraction (Formula Level)
  • Text Localization (Molecule Level)
  • Bond End Grouping (Molecule Level)
  • Text Recognition (Text Level)

Downstream Impact and Parsing Accuracy

The system achieved high accuracy across all sub-tasks, demonstrating that the hierarchical grammar approach is effective for both inorganic and organic formulas.

• Formula Level: 98.3% accuracy for Key Symbols; 100% for State-assisted symbols.
• Molecule Level: 98.8% accuracy for Bond End Grouping; 100% for Free End-Text connection detection.
• Text Recognition: 98.7% accuracy (Top-3) using HMMs.
• Impact: The method successfully preserves the writer’s “online information” (habits/intentions) while converting the handwritten input into standard formats suitable for visual editing or data retrieval.

Reproducibility Details

To replicate this work, one would need to implement the specific grammatical production rules and the geometric thresholds defined for bond analysis.

Data

Algorithms

1. Formula Level Parsing

• HBL Analysis: Identify the “Horizontal Baseline” (HBL) containing the most symbols to locate key operators (e.g., $+$, $\rightarrow$).
• Grammar: Use the productions defined in Figure 4. Example rules include:
  • $Reaction ::= ReactantList \ Generator \ ProductList$
  • $Reactant ::= BalancingNum \ Molecule \ IonicCharacter$

2. Molecule Level Analysis (Bond Grouping)

• Endpoint Classification: Points are classified as free ends, junctions (3+ bonds), or connections (2 bonds).
• Grouping Equation: An endpoint $(x_k, y_k)$ belongs to Group A based on distance thresholding: $$ \begin{aligned} Include(x_0, y_0) = \begin{cases} 1, & d_0 < t \cdot \max d_x + \partial \\ 0, & \text{else} \end{cases} \end{aligned} $$ Where $d_k$ is the Euclidean distance to the group center $(x_a, y_a)$.

3. Connection Detection

• Text-Bond Connection: A text group is connected to a bond if the free end falls within a bounding box expanded by thresholds $t_W$ and $t_H$: $$ \begin{aligned} Con(x,y) = \begin{cases} 1, & \min x - t_W < x < \max x + t_W \text{ AND } \min y - t_H < y < \max y + t_H \\ 0, & \text{else} \end{cases} \end{aligned} $$

Models

• Text Recognition: Hidden Markov Models (HMM) are used for recognizing individual text symbols.
• Grammar: Context-Free Grammar (CFG) designed with ambiguity elimination to ensure a single valid parse tree for any valid formula.

Evaluation

Performance is measured by recognition accuracy at specific processing stages:

Paper Information

Citation: Wang, X., Shi, G., & Yang, J. (2009). The Understanding and Structure Analyzing for Online Handwritten Chemical Formulas. 2009 10th International Conference on Document Analysis and Recognition, 1056–1060. https://doi.org/10.1109/ICDAR.2009.70

Publication: ICDAR 2009

@inproceedings{wangUnderstandingStructureAnalyzing2009,
  title = {The {{Understanding}} and {{Structure Analyzing}} for {{Online Handwritten Chemical Formulas}}},
  booktitle = {2009 10th {{International Conference}} on {{Document Analysis}} and {{Recognition}}},
  author = {Wang, Xin and Shi, Guangshun and Yang, Jufeng},
  year = {2009},
  pages = {1056--1060},
  publisher = {IEEE},
  address = {Barcelona, Spain},
  doi = {10.1109/ICDAR.2009.70},
  isbn = {978-1-4244-4500-4},
  langid = {english}
}

Methodological Paper ($\Psi_{\text{Method}}$)

This paper focuses entirely on the architecture and workflow of the chemoCR system. It proposes specific algorithmic innovations (texture-based vectorization, graph constraint exploration) and defines a comprehensive pipeline for converting raster images into semantic chemical graphs. The primary contribution is the system design and its operational efficacy.

Motivation: Digitizing Image-Locked Chemical Structures

Chemical structures are the preferred language of chemistry, yet they are frequently locked in non-machine-readable formats (bitmap images like GIF, BMP) within patents and journals.

• The Problem: Once published as images, chemical structure information is “dead” to analysis software.
• The Gap: Manual reconstruction is slow and error-prone. Existing tools struggled with the diversity of drawing styles (e.g., varying line thickness, font types, and non-standard bond representations).
• The Goal: To automate the conversion of these depictions into connection tables (SDF/MOL files) to make the data accessible for computational chemistry applications.

Core Innovation: Rule-Based Semantic Object Identification

The system is based on a “Semantic Entity” approach that identifies chemically significant objects (chiral bonds, superatoms, reaction arrows) from structural formula depictions. Key technical innovations include:

1. Texture-based Vectorization: A new algorithm that computes local directions to vectorize lines, robust against varying drawing styles.
2. Expert System Integration: A graph constraint exploration algorithm that applies an XML-based rule set to classify geometric primitives into chemical classes such as BOND, DOUBLEBOND, TRIPLEBOND, BONDSET, DOTTED CHIRAL, STRINGASSOCIATION, DOT, RADICAL, REACTION, REACTION ARROW, REACTION PLUS, CHARGE, and UNKNOWN.
3. Validation Scoring: A built-in validation module that tests valences, bond lengths and angles, typical atom types, and fragments to assign a confidence score (0 to 1) to the reconstruction.

Experiments: The TREC 2011 Image-to-Structure Task

The system was evaluated as part of the TREC 2011 Image-to-Structure (I2S) Task.

• Dataset: 1,000 unique chemical structure images provided by USPTO.
• Configuration: The authors used chemoCR v0.93 in batch mode with a single pre-configured parameter set (“Houben-Weyl”), originally developed for the Houben-Weyl book series of organic chemistry reactions published by Thieme.
• Process: The workflow included image binarization, connected component analysis, OCR for atom labels, and final molecule assembly.
• Metric: Perfect match recall against ground-truth MOL files.

Results and Conclusions: Expert Systems vs. “Dirty” Data

• Performance: The system achieved a perfect match for 656 out of 1,000 structures (65.6%).
• Error Analysis: Failures were primarily attributed to “unclear semantics” in drawing styles, such as:
  • Overlapping objects (e.g., atom labels clashing with bonds).
  • Ambiguous primitives (dots interpreted as both radicals and chiral centers).
  • Markush structures (variable groups), which were excluded from the I2S task definition. A prototype for Markush detection existed but was not used.
• Limitations: The vectorizer cannot recognize curves and circles, only straight lines. Aromatic ring detection (via a heuristic that looks for a large “O” character in the center of a ring system) was switched off for the I2S task. The system maintained 12 different parameter sets for various drawing styles, and selecting the correct set was critical.
• Impact: Demonstrated that rule-based expert systems combined with standard pattern recognition could handle high-quality datasets effectively, though non-standard drawing styles remain a challenge.

Reproducibility Details

Data

The paper relies on the TREC 2011 I2S dataset, comprising images extracted from USPTO patents.

Algorithms

The paper describes three main workflow phases (preprocessing, semantic entity recognition, and molecule reconstruction plus validation), organized into four pipeline sections:

1. Preprocessing:

   • Vaporizer Unit: Erases parts of the image that are presumably not structure diagrams (e.g., text or other human-readable information), isolating the chemical depictions.
   • Connected Components: Groups all foreground pixels that are 8-connected into components.
   • Text Tagging and OCR: Identifies components that map to text areas and converts bitmap letters into characters.

2. Vectorization:

   • Algorithm: Compute Local Directions. It analyzes segment clusters to detect ascending, descending, horizontal, and vertical trends in pixel data, converting them into vectors.
   • Feature: Explicitly handles “thick chirals” (wedges) by computing orientation.

3. Reconstruction (Expert System):

   • Core Logic: Graph Constraint Exploration. It visits connected components and evaluates them against an XML Rule Set.
   • Classification: Objects are tagged with chemical keywords (e.g., BONDSET for ring systems and chains, STRINGASSOCIATION for atom labels, DOTTED CHIRAL for chiral bonds).
   • Rules: Configurable via chemoCRSettings.xml. The successful rule with the highest priority value defines the annotation for each component.

4. Assembly & Validation:

   • Combines classified vectors and OCR text into a semantic graph.
   • Superatoms: Matches text groups against a loaded superatom database (e.g., “COOH”, “Boc”).
   • Validation: Calculates a score (0-1) based on chemical feasibility (valences, bond lengths and angles, typical atom types, and fragments).

Models

The system is primarily rule-based but utilizes machine learning components for specific sub-tasks:

• OCR: A trainable OCR module using supervised machine learning to recognize atom labels ($H, C, N, O$). The specific classifier is not detailed in the paper.
• Rule Base: An XML file containing the expert system logic. This is the “model” for structural interpretation.

Evaluation

Evaluation was performed strictly within the context of the TREC competition.

Hardware

• Software Stack: Platform-independent JAVA libraries.
• Compute: Batch mode processing supported; specific hardware requirements (CPU/RAM) were not disclosed.

Artifacts

No source code was publicly released. The chemoCR system was a proprietary tool from Fraunhofer SCAI. The TREC 2011 I2S dataset was distributed to competition participants and is not independently hosted.

Paper Information

Citation: Zimmermann, M. (2011). Chemical Structure Reconstruction with chemoCR. TREC 2011 Proceedings.

Publication: Text REtrieval Conference (TREC) 2011

@inproceedings{zimmermannChemicalStructureReconstruction2011,
  title = {Chemical Structure Reconstruction with {{chemoCR}}},
  booktitle = {Text {{REtrieval Conference}} ({{TREC}}) 2011},
  author = {Zimmermann, Marc},
  year = {2011},
  langid = {english}
}

Method. This paper proposes a system architecture for document analysis. It introduces a specific pipeline (Global Perception followed by Incremental Extraction) and validates this strategy with recognition rates on specific tasks. The core contribution is the shift from bitmap-based processing to a structural graph representation of graphical primitives.

Motivation: Overcoming Bitmap Limitations in Freehand Drawings

• Complexity of Freehand: Freehand drawings contain fluctuating lines and noise that make standard vectorization techniques difficult to apply directly.
• Limitation of Bitmap Analysis: Most existing systems at the time attempted to interpret the document by working directly on the static bitmap image throughout the process.
• Need for Context: Interpretation requires a dynamic resource that can evolve as knowledge is extracted (e.g., recognizing a polygon changes the context for its neighbors).

Novelty: Dynamic Structural Graphs and Recursive Specialists

The authors propose a Structural Representation as the unique resource for interpretation.

• Quadrilateral Primitives: The system builds Quadrilaterals (pairs of vectors) to represent thin shapes, which are robust to handwriting fluctuations.
• Structural Graph: These primitives are organized into a graph where arcs represent geometric relationships (T-junctions, L-junctions, parallels).
• Specialist Agents: Interpretation is driven by independent modules (specialists) that browse this graph recursively to identify high-level chemical entities like rings (polygons) or chains.

Experimental Setup and Outcomes

• Validation Set: The system was tested on 20 handwritten off-line documents containing chemical formulas at 300 dpi resolution.
• Text Database: A separate base of 328 models was used for the text recognition component.
• High Graphical Accuracy: The system achieved a $\approx 97%$ recognition rate for graphical parts (chemical elements like rings and bonds).
• Text Recognition: The text recognition module achieved a $\approx 93%$ success rate.
• Robustness: The structural graph approach successfully handled multiple liaisons, polygons, chains and allowed for the progressive construction of a solution consistent with the context.

Reproducibility Details

Data

Algorithms

The interpretation process is divided into two distinct phases:

1. Global Perception (Graph Construction)

• Vectorization: Contour tracking produces a chain of vectors, which are simplified via iterative polygonal approximation until fusion stabilizes (2-5 iterations).
• Quadrilateral Formation: Vectors are paired to form quadrilaterals based on Euclidean distance and “empirical” alignment criteria.
• Graph Generation: Quadrilaterals become nodes. Arcs are created based on “zones of influence” and classified into 5 types: T-junction, Intersection (X), Parallel (//), L-junction, and Successive (S).
• Redraw Heuristic: A pre-processing step transforms T, X, and S junctions into L or // relations, as chemical drawings primarily consist of L-junctions and parallels.

2. Specialists (Interpretation)

• Liaison Specialist: Scans the graph for // arcs or quadrilaterals with free extremities to identify bonds.
• Polygon/Chain Specialist: Uses recursive look-left and look-right procedures. If a search returns to the start node after $n$ steps, a polygon is detected.
• Text Localization: Clusters “short” quadrilaterals by physical proximity into “focus zones”. Zones are classified as text/non-text based on connected components.

Models

Text Recognition Hybrid:

1. Normalization & Pattern Matching: A classic method using the database of 328 models.
2. Structural Rule Base: Uses “significant” quadrilaterals (length $\ge 1/3$ of zone dimension) to verify characters. A rule base defines the expected count of horizontal, vertical, right-diagonal, and left-diagonal lines for each character.

Evaluation

Paper Information

Citation: Ramel, J.-Y., Boissier, G., & Emptoz, H. (1999). Automatic Reading of Handwritten Chemical Formulas from a Structural Representation of the Image. Proceedings of the Fifth International Conference on Document Analysis and Recognition (ICDAR ‘99), 83-86. https://doi.org/10.1109/ICDAR.1999.791730

Publication: ICDAR 1999

@inproceedings{ramelAutomaticReadingHandwritten1999,
  title = {Automatic Reading of Handwritten Chemical Formulas from a Structural Representation of the Image},
  booktitle = {Proceedings of the {{Fifth International Conference}} on {{Document Analysis}} and {{Recognition}}. {{ICDAR}} '99 ({{Cat}}. {{No}}.{{PR00318}})},
  author = {Ramel, J.-Y. and Boissier, G. and Emptoz, H.},
  year = 1999,
  pages = {83--86},
  publisher = {IEEE},
  address = {Bangalore, India},
  doi = {10.1109/ICDAR.1999.791730},
  isbn = {978-0-7695-0318-9}
}

This is primarily a Resource ($\Psi_{\text{Resource}}$) paper, with a secondary Method ($\Psi_{\text{Method}}$) component.

Resource: The paper’s main contribution is the release of the “Imago” open-source toolkit. It emphasizes infrastructure utility: cross-platform C++ implementation, a core written from scratch without third-party code, and the inclusion of both GUI and command-line tools.

Method: It provides a detailed description of the recognition pipeline (filtering, segmentation, vectorization) to document the resource.

Motivation: The Deep Web of Chemical Structures

Chemical databases (like PubChem or PubMed) allow text and substructure searches, but a vast amount of chemical data remains “locked” in the images of scientific articles and patents. This is described as a “Deep Web indexing problem”. To bridge this gap, the authors identify a need for efficient, accurate, and portable algorithms to convert 2D raster images of molecules into graph representations suitable for indexing and search.

Core Innovation: A Dependency-Free C++ Architecture

The novelty lies in the open-source, dependency-free implementation.

Portability: The core of the toolkit is written from scratch in modern C++ without third-party libraries, specifically targeting portability to mobile devices and various platforms.

Integration: It combines optical character recognition (OCR) with specific chemical heuristics (like identifying stereochemistry and abbreviations) into a single usable workflow.

Methodology and Experimental Validation at TREC-CHEM

The paper describes the algorithm used in Imago and reflects on its participation in the Image2Structure task at TREC-CHEM 2011. No quantitative results are reported; the “Discussion” section instead reflects on qualitative performance issues observed during the task, such as handling low resolution, noise, and connected atom labels.

Outcomes, Limitations, and Future Directions

Release: The authors successfully released Imago under the GPLv3 license, including an API for developers. The toolkit outputs recognized structures in MDL Molfile format.

Limitations Identified: The straightforward pipeline fails when images have low resolution (atom labels merge with bonds), high noise, or tight character spacing (symbols rendered without space pixels between them). Additionally, when few symbols are present, the average bond length estimate can have large error, causing atom symbols to be misidentified as bond chains.

Future Directions: The authors propose moving from a linear pipeline to an “optimization procedure” that maximizes a confidence score, using probabilistic mapping of image primitives to chemical entities. They also argue that recognition programs should output a confidence score to enable automatic batch processing (only images with low confidence need manual review). They suggest a multi-pass workflow where each iteration adjusts parameters to improve the confidence level, and they note the additional challenge of separating molecule images from text in real articles and patents.

Reproducibility Details

Data

The paper does not specify a training dataset for the core logic (which appears heuristic-based), but references testing context:

• Domain: Images from scientific articles and patents.
• Validation: TREC-CHEM 2011 Image2Structure task data.
• Databases: Mentions PubMed and PubChem as context for the type of data being indexed.

Algorithms

The recognition pipeline follows a strict linear sequence:

1. Preprocessing:

   • Binarization: Threshold-based.
   • Supersegmentation: Locates the chemical structure using a $15 \times 15$ window neighbor search.
   • Filtering: Removes single-down stereo bonds (dashed triangles) early to prevent incorrect recognition of the small line segments during classification.

2. Separation (Symbols vs. Graphics):

   • Heuristic: Estimates “capital letter height”.
   • Criteria: Groups segments by height and aspect ratio range $[\text{MinSymRatio}, \text{MaxSymRatio}]$.

3. Skeleton Construction (Vectorization):

   • Thinning: Uses neighborhood maps to reduce lines to 1-pixel thickness.
   • De-crossing: Each black pixel with more than 2 black pixels in its 8-neighborhood becomes white, isolating polylines.
   • Smoothing: Uses the Douglas-Peucker algorithm.
   • Graph Adjustment: Merges close vertices and detects bond orders based on parallel edges.

4. Symbol Recognition:

   • Grouping: Uses a Relative Neighborhood Graph to group characters into superatoms/labels.
   • OCR: Classification based on Fourier descriptors of outer/inner contours.

5. Chemical Expansion:

   • Abbreviation: Expands common groups (e.g., Ph, COOH) stored as SMILES notation, using the Indigo toolkit for 2D coordinate generation of the expanded structures.

Models

• Type: Heuristic-based computer vision pipeline; no learned deep learning weights mentioned.
• Stereo Recognition:
  • Single Down: Identified as $k \ge 3$ parallel equidistant lines.
  • Single Up: Identified by checking if a bond was a solid triangle before thinning.

Evaluation

• Metrics: None quantitatively reported in the text; discussion focuses on qualitative failure modes (low resolution, noise).

Artifacts

Hardware

• Requirements: Designed to be lightweight and portable (mobile-device capable), written in C++. No specific GPU/TPU requirements.

Paper Information

Citation: Smolov, V., Zentsev, F., & Rybalkin, M. (2011). Imago: Open-Source Toolkit for 2D Chemical Structure Image Recognition. TREC-CHEM 2011.

Publication: TREC-CHEM 2011

Additional Resources:

• TREC-CHEM 2011 Proceedings
• Project Website
• GitHub Repository

@techreport{smolovImagoOpenSourceToolkit2011,
  title = {Imago: {{Open-Source Toolkit}} for {{2D Chemical Structure Image Recognition}}},
  author = {Smolov, Viktor and Zentsev, Fedor and Rybalkin, Mikhail},
  year = {2011},
  institution = {{GGA Software Services LLC}},
  note = {TREC-CHEM 2011}
}

Citation: Valko, A. T., & Johnson, A. P. (2009). CLiDE Pro: The Latest Generation of CLiDE, a Tool for Optical Chemical Structure Recognition. Journal of Chemical Information and Modeling, 49(4), 780-787. https://doi.org/10.1021/ci800449t

Publication: Journal of Chemical Information and Modeling 2009

Contribution: Robust Algorithmic Pipeline for OCSR

This is primarily a Method ($\Psi_{\text{Method}}$) paper, as it proposes a specific algorithmic architecture (CLiDE Pro) for converting raster images of chemical structures into connection tables. It details the procedural steps for segmentation, vectorization, and graph reconstruction.

It also has a secondary Resource ($\Psi_{\text{Resource}}$) contribution, as the authors compile and release a validation set of 454 real-world images to serve as a community benchmark for OCSR systems.

Motivation: Bridging the Gap Between Legacy Document Images and Machine-Readable Chemistry

While modern chemical drawing software captures structural information explicitly, the vast majority of legacy and current chemical literature (journals, patents, reports) exists as images or PDF documents. These images are human-readable but lack the semantic “connection table” data required for chemical databases and software. Manual redrawing is time-consuming and error-prone. Therefore, there is a critical need for efficient Optical Chemical Structure Recognition (OCSR) systems to automate this extraction.

Novelty: Integrated Document Segmentation and Ambiguity Resolution Heuristics

CLiDE Pro introduces several algorithmic improvements over its predecessor (CLiDE) and contemporary tools:

• Integrated Document Segmentation: Unlike page-oriented systems, it processes whole documents to link information across pages.
• Robust “Difficult Feature” Handling: It implements specific heuristic rules to resolve ambiguities in crossing bonds (bridged structures), which are often misinterpreted as carbon atoms in other systems.
• Generic Structure Interpretation: It includes a module to parse “generic” (Markush) structures by matching R-group labels in the diagram with text-based definitions found in the document.
• Ambiguity Resolution: It uses context-aware rules to distinguish between geometrically similar features, such as vertical lines representing bonds vs. the letter ’l’ in ‘Cl’.

Methodology and Benchmarking on Real-World Data

The authors conducted a systematic validation on a dataset of 454 images containing 519 structure diagrams.

• Source Material: Images were extracted from published materials (journals, patents), ensuring “real artifacts” like noise and scanning distortions were present.
• Automation: The test was fully automated without human intervention.
• Metrics: The primary metric was the “success rate,” defined as the correct reconstruction of the molecule’s connection table. They also performed fine-grained error analysis on specific features (e.g., atom labels, dashed bonds, wavy bonds).

Results: High Topological Accuracy and Persistent OCR Challenges

• High Accuracy: The system achieved a 89.79% retrieval rate (466/519 molecules correctly reconstructed).
• Robustness on Primitives: Solid straight bonds were recognized with 99.92% accuracy.
• Key Failure Modes: The majority of errors (58 cases) occurred in atom label construction, specifically when labels touched nearby bonds or other artifacts, causing OCR failures.
• Impact: The study demonstrated that handling “difficult” drawing features like crossing bonds and bridged structures significantly reduces topological errors. The authors released the test set to encourage standardized benchmarking in the OCSR field.

Reproducibility Details

Data

The authors utilized a custom dataset designed to reflect real-world noise.

Algorithms

The CLiDE Pro pipeline consists of five distinct phases. To replicate this system, one would need to implement:

1. Image Binarization:

   • Input images are binarized using a threshold-based technique to separate foreground (molecule) from background.
   • Connected Component Analysis (CCA): A non-recursive scan identifies connected components (CCs) and generates interpixel contours (using N, S, E, W directions).

2. Document Segmentation:

   • Layout Analysis: Uses a bottom-up approach building a tree structure. It treats CCs as graph vertices and distances as edges.
   • Clustering: A minimal-cost spanning tree (Kruskal’s algorithm) groups CCs into words, lines, and blocks.
   • Classification: CCs are classified (Character, Dash, Line, Graphics, Noise) based on size thresholds derived from statistical image analysis.

3. Vectorization:

   • Contour Approximation: Uses a method similar to Sklansky and Gonzalez to approximate contours into polygons.
   • Vector Formation: Long polygon sides become straight lines; short consecutive sides become curves. Opposing borders of a line are matched to define the bond vector.
   • Wavy Bonds: Detected by finding groups of short vectors lying on a straight line.
   • Dashed Bonds: Detected using the Hough transform to find collinear or parallel dashes.

4. Atom Label Construction:

   • OCR: An OCR engine (filtering + topological analysis) interprets characters.
   • Grouping: Characters are grouped into words based on horizontal and vertical proximity (for vertical labels).
   • Superatom Lookup: Labels are matched against a database of elements, functional groups, and R-groups. Unknown linear formulas (e.g., $\text{CH}_2\text{CH}_2\text{OH}$) are parsed.

5. Graph Reconstruction:

   • Connection Logic: Bond endpoints are joined to atoms if they are within a distance threshold and “point toward” the label.
   • Implicit Carbons: Unconnected bond ends are joined if close; parallel bonds merge into double/triple bonds.
   • Crossing Bonds: Rules check proximity, length, and ring membership to determine if crossing lines are valid atoms or 3D visual artifacts.

6. Generic Structure Interpretation:

   • Text Mining: A lexical/syntactic analyzer extracts R-group definitions (e.g., “R = Me or H”) from text blocks.
   • Matching: The system attempts to match R-group labels in the diagram with the parsed text definitions.

Models

• OCR Engine: The system relies on a customized OCR engine capable of handling rotation and chemical symbols, though the specific architecture (neural vs. feature-based) is not detailed beyond “topological and geometrical feature analysis”.
• Superatom Database: A lookup table containing elements, common functional groups, and R-group labels.

Evaluation

The evaluation focused on the topological correctness of the output.

Hardware

• Requirements: Unspecified; described as efficient.
• Performance: The system processed the complex Palytoxin structure “within a few seconds”. This implies low computational overhead suitable for standard desktop hardware of the 2009 era.

Citation

@article{valkoCLiDEProLatest2009,
  title = {CLiDE Pro: The Latest Generation of CLiDE, a Tool for Optical Chemical Structure Recognition},
  author = {Valko, Aniko T. and Johnson, A. Peter},
  journal = {Journal of Chemical Information and Modeling},
  volume = {49},
  number = {4},
  pages = {780--787},
  year = {2009},
  doi = {10.1021/ci800449t}
}

This is a Method paper. It proposes a novel architectural framework for sketch recognition that integrates visual features at three distinct levels (inkpoints, segments, symbols) into a single probabilistic model. The rhetorical structure centers on the proposal of this new architecture, the introduction of a specific “trainable corner detector” algorithm, and the validation of these methods against existing benchmarks and alternative toolsets (ChemDraw).

Motivation: Bridging the Gap Between Sketching and CAD

The primary motivation is to bridge the gap between the natural, efficient process of drawing chemical diagrams by hand and the cumbersome “point-click-and-drag” interactions required by CAD tools like ChemDraw. While chemists prefer sketching for communication, existing digital tools do not offer the same speed or ease of use. The goal is to build an intelligent system that understands freehand sketches in real-time, converting them into structured data suitable for analysis or search.

Core Innovation: Hierarchical Joint CRF Model

The core novelty lies in the hierarchical joint model. Unlike previous approaches that might treat stroke segmentation and symbol recognition as separate, isolated steps, ChemInk uses a Conditional Random Field (CRF) to jointly model dependencies across three levels:

1. Inkpoints: Local visual appearance.
2. Segments: Stroke fragments separated by corners.
3. Candidates: Potential symbol groupings.

Additionally, the paper introduces a trainable corner detector that learns domain-specific corner definitions from data.

Experimental Design and Baselines

The authors conducted two primary evaluations:

1. Off-line Accuracy Evaluation:
   • Dataset: 12 real-world organic compounds drawn by 10 participants.
   • Metric: Recognition accuracy (Recall and Precision).
   • Baseline: Comparison against their own previous work (O&D 2009) and ablations (with/without context).
2. On-line User Study:
   • Task: 9 participants (chemistry students) drew 5 diagrams using both ChemInk (Tablet PC) and ChemDraw (Mouse/Keyboard).
   • Metric: Time to completion and subjective user ratings (speed/ease of use).

Results: Accuracy and User Study Outcomes

• Accuracy: The system achieved 97.4% symbol recognition accuracy, slightly outperforming the best prior result (97.1%). The trainable corner detector achieved 99.91% recall.
• Speed: Users were twice as fast using ChemInk (avg. 36s) compared to ChemDraw (avg. 79s).
• Usability: Participants rated ChemInk significantly higher for speed (6.3 vs 4.5) and ease of use (6.3 vs 4.7) on a 7-point scale.
• Conclusion: Sketch recognition is a viable, superior alternative to standard CAD tools for authoring chemical diagrams.

Reproducibility Details

Data

• Training/Test Data: 12 real-world organic compounds (e.g., Aspirin, Penicillin) drawn by 10 participants (organic chemistry familiar).
• Evaluation Split: User-independent cross-validation (training on 9 users, testing on 1).
• Input: Raw digital ink (strokes) collected on a Tablet PC.

Algorithms

1. Corner Detection (Trainable)

• Method: Iterative vertex elimination.
• Cost Function: $cost(p_{i}) = \sqrt{mse(s_{i}; p_{i-1}, p_{i+1})} \cdot dist(p_{i}; p_{i-1}, p_{i+1})$
• Procedure: Repeatedly remove the vertex with the lowest cost until the classifier (trained on features like cost, diagonal length, ink density) predicts the remaining vertices are corners.

2. Feature Extraction

• Inkpoints: Sampled at regular intervals. Features = $10 \times 10$ pixel orientation filters (0, 45, 90, 135 degrees) at two scales ($L/2$, $L$), smoothed and downsampled to $5 \times 5$. Total 400 features.
• Segments: Similar image features centered at segment midpoint, plus geometric features (length, ink density).
• Candidates: 5 feature images ($20 \times 20$) including an “endpoint” image, stretched to normalize aspect ratio.
• Dimensionality Reduction: PCA used to compress feature images to 256 components.

3. Structure Generation

• Clustering: Agglomerative clustering with a complete-link metric to connect symbols.
• Threshold: Stop clustering at distance $0.4L$.