This is a Position paper that argues large language models (LLMs) capable of generating code from natural language prompts, specifically OpenAI’s Codex and GPT-3, are poised to transform both chemistry research and chemistry education. Published in the inaugural volume of Digital Discovery (RSC), the paper combines a brief history of NLP developments with concrete demonstrations of code generation for computational chemistry tasks, then offers a forward-looking perspective on challenges and opportunities.

Bridging the Gap Between Natural Language and Scientific Software

The authors identify a core friction in modern computational chemistry: while the number of available software packages has grown dramatically, researchers spend a large fraction of their time learning interfaces to these packages rather than doing science. Tasks like searching documentation, following tutorials, and trial-and-error experimentation with APIs consume effort that could be directed at research itself.

At the same time, programming assignments in chemistry courses serve dual pedagogical purposes (reinforcing physical intuition and teaching marketable skills), but are constrained by students’ median programming experience. The emergence of code-generating NLP models opens the possibility of reducing both barriers simultaneously.

Code Generation as a Chemistry Interface

The paper’s core thesis is that NLP models trained on code can serve as a natural language interface to the entire ecosystem of scientific computing tools. The authors demonstrate this with several concrete examples using OpenAI Codex:

1. Quantum chemistry: Prompting Codex to “compute the dissociation curve of H2 using pyscf” produced correct, runnable code that selected Hartree-Fock with STO-3G. A follow-up prompt requesting “the most accurate method” caused it to switch to CCSD in a large basis set.

2. Chemical entity recognition: Using GPT-3 with only three training examples, the authors demonstrated extraction of chemical entity names from published text, a task that previously required thousands of labeled examples.

3. Molecular visualization: Drawing caffeine from its SMILES string, generating Gaussian input files from SMILES, implementing random walks, and downloading and analyzing PDB structures with MDTraj.

4. Voice-controlled molecular dynamics: The authors previously built MARVIS, a voice-controlled molecular dynamics analysis tool that uses GPT-3 to convert natural language into VMD commands. Only about a dozen examples were needed to teach GPT-3 to render proteins, change representations, and select atoms.

An important caveat: the authors emphasize that all chemistry “knowledge” (including the SMILES string for caffeine) is entirely contained in the model’s learned floating-point weights. The model has no access to databases or curated lists of chemical concepts.

Demonstrations and Practical Evaluation

Rather than a formal experimental evaluation with benchmarks and metrics, this perspective paper relies on qualitative demonstrations. The key examples, with full details provided in the ESI, include:

The authors note that Codex generates correct code at about a 30% rate on a single attempt for standard problems, improving to above 50% when multiple solutions are tried. Mistakes tend to occur when complex algorithms are requested with little specificity, and the code rarely has syntax errors but may fail in obvious ways (missing imports, wrong data types).

Challenges: Access, Correctness, and Bias

The paper identifies three ongoing challenges:

Access and price. Advanced models from OpenAI were, at the time of writing, limited to early testers. Per-query costs (1-3 cents for GPT-3) would become prohibitive at the scale needed for parsing academic literature or supporting medium-sized courses. The authors advocate for open-source models and equitable deployment by researchers with computational resources.

Correctness. Code generation does not guarantee correctness. The authors raise a subtle point: Codex may produce code that executes successfully but does not follow best scientific practice for a particular computational task. Over-reliance on AI-generated code without verification could erode trust in scientific software. However, they argue that strategies for assessing code correctness apply equally to human-written and AI-generated code.

Fairness and bias. The authors flag several concerns: AI-generated code trained on its own outputs could narrow the range of packages, methods, or programming languages used in chemistry. They observed Codex’s preference for Python and for specific popular libraries (e.g., defaulting to Psi4 for single-point energy calculations). GPT-3 has also been shown to reflect racism, sexism, and other biases present in its training data.

Implications for Research and Education

The authors conclude with an optimistic but measured outlook:

• For research: NLP code generation will increase accessibility of software tools and expand what a single research group can accomplish. Better tools have historically not reduced the need for scientists but expanded the complexity of problems that can be tackled.
• For programming skills: Using Codex will make chemists better programmers, not worse. The process of crafting prompts, mentally checking outputs, testing on sample inputs, and iterating develops algorithmic thinking. The authors report discovering chemistry software libraries they would not have found otherwise through iterative prompt creation.
• For education: Instructors should rethink programming assignments. The authors suggest moving toward more difficult compound assignments, treating code exercises as laboratory explorations of scientific concepts rather than syntax drills, and aligning coursework with the tools students will have access to in their careers.
• For accessibility: NLP models can reduce barriers for non-native English speakers (though accuracy with non-English prompts was not fully explored) and for users who have difficulty with keyboard-and-mouse interfaces (via voice control).

The paper acknowledges that these capabilities were, in early 2022, just beginning, with Codex being the first capable code-generation model. Already at the time of writing, models surpassing GPT-3 in language tasks had appeared, and models matching GPT-3 with 1/20th the parameters had been demonstrated.

Reproducibility Details

This is a perspective paper with qualitative demonstrations rather than a reproducible experimental study. The authors provide all prompts and multiple responses in the ESI.

Data

All prompts and code outputs are provided in the Electronic Supplementary Information (ESI) available from the RSC.

Algorithms

The paper does not introduce new algorithms. It evaluates existing models (GPT-3, Codex) on chemistry-related code generation tasks.

Models

Evaluation

No formal metrics are reported for the chemistry demonstrations. The authors cite the Codex paper’s reported ~30% pass rate on single attempts and >50% with multiple attempts on standard programming problems.

Hardware

No hardware requirements are specified for the demonstrations (API-based inference).

Artifacts

Paper Information

Citation: Hocky, G. M., & White, A. D. (2022). Natural language processing models that automate programming will transform chemistry research and teaching. Digital Discovery, 1(2), 79-83. https://doi.org/10.1039/d1dd00009h

@article{hocky2022natural,
  title={Natural language processing models that automate programming will transform chemistry research and teaching},
  author={Hocky, Glen M. and White, Andrew D.},
  journal={Digital Discovery},
  volume={1},
  number={2},
  pages={79--83},
  year={2022},
  publisher={Royal Society of Chemistry},
  doi={10.1039/d1dd00009h}
}

Aligning two sets of corresponding points, finding the optimal rotation (and optionally translation and scale) that maps one onto the other, is a fundamental operation across scientific computing. It appears in molecular dynamics (superimposing protein conformations), robotics (sensor registration), and computer vision (shape matching). The two dominant algorithm families are the Kabsch (SVD-based) method and the Horn (quaternion-based) method.

The Kabsch-Horn Cookbook is a Python library that implements both algorithm families across five numerical frameworks: NumPy, PyTorch, JAX, TensorFlow, and MLX. Every backend shares the same API, supports N-dimensional point sets, per-point weights, and arbitrary batch dimensions. The PyTorch, JAX, TensorFlow, and MLX backends are fully differentiable, with custom autograd rules that bypass the numerically unstable gradient of the standard SVD near degenerate singular values.

Features

Algorithms

• Kabsch: SVD-based optimal rotation for rigid alignment
• Kabsch-Umeyama: Kabsch with an additional optimal scaling factor $c$, solving $Q \approx cRP + t$
• Horn: Quaternion-based optimal rotation via the eigendecomposition of a $4 \times 4$ key matrix
• Horn + Scale: Horn’s method extended with optimal isotropic scaling
• RMSD Wrappers: Convenience functions that return RMSD directly alongside the alignment parameters

Framework Support

torch.compile and jax.jit are the tested compile/JIT paths. MLX supports 3D inputs only; the Kabsch (SVD) path is N-dimensional on the other four backends.

Numerical Robustness

Standard SVD and eigendecomposition backward passes produce NaN gradients when singular values collide or are near-zero. The library provides custom autograd primitives to handle these cases:

• SafeSVD (PyTorch, JAX, TF, MLX): Custom backward pass that clamps the singular value gap, preventing division-by-zero in the gradient
• SafeEigh (PyTorch, JAX, TF, MLX): Analogous safe backward for the symmetric eigendecomposition used in Horn’s method
• Per-point weights: Weighted centroids and weighted cross-covariance for mass-weighted or confidence-weighted alignment
• Batch dimensions: All functions broadcast over leading batch dimensions without explicit loops
• Mixed-dtype promotion: Inputs are promoted to a common floating-point dtype automatically

Testing

The test suite uses Hypothesis-based property testing across 13 modules covering:

• Round-trip correctness (align then compare)
• Gradient finiteness and correctness (finite-difference checks)
• Reflection handling (proper vs. improper rotations)
• Weighted alignment consistency
• Batch broadcasting
• 4 differentiable backends $\times$ 4 precisions (float32, float64, and where supported, float16, bfloat16)

Usage

This is a reference cookbook, so you can copy the framework folder you need from src/kabsch_horn/<framework>/ directly into your project (the code has no runtime dependencies beyond the framework itself). To depend on it instead, install a pinned version from GitHub:

pip install "git+https://github.com/hunter-heidenreich/Kabsch-Cookbook.git@v0.4.1"

Basic alignment with NumPy:

import numpy as np
from kabsch_horn import numpy as kh

# Two sets of corresponding 3D points
P = np.random.randn(100, 3)
R_true = np.linalg.qr(np.random.randn(3, 3))[0]  # random rotation matrix
Q = (P @ R_true.T) + np.random.randn(1, 3)

R, t, rmsd = kh.kabsch(P, Q)
aligned = P @ R.T + t

RMSD loss for training in PyTorch:

import torch
from kabsch_horn import pytorch as kh

pred_coords = model(input_features)   # (B, N, 3), requires_grad=True
target_coords = batch["target"]       # (B, N, 3)

rmsd = kh.kabsch_rmsd(pred_coords, target_coords)  # (B,)
loss = rmsd.mean()
loss.backward()  # safe gradients via SafeSVD

For the full API reference and additional examples, see the documentation site.

Results

Gradient Stability

The standard SVD backward pass computes terms of the form $\frac{1}{\sigma_i^2 - \sigma_j^2}$, which diverges when two singular values are close. In molecular alignment this happens frequently: planar molecules, symmetric structures, and noisy coordinates can all produce near-degenerate singular values. The SafeSVD primitive floors the magnitude of that denominator at the dtype’s machine epsilon (finfo(dtype).eps), producing finite (if slightly biased) gradients in these edge cases. Property-based tests confirm that gradients remain finite across thousands of random rotations, scales, and noise levels for all four differentiable backends.

Framework Parity

All five backends produce numerically equivalent results (up to floating-point tolerance) on the same inputs. The shared API means switching from NumPy prototyping to PyTorch training requires changing only the import path.

Related Work

This project builds on the foundational alignment algorithms described in these papers:

• Kabsch (1976): the original SVD-based rotation alignment
• Arun et al. (1987): SVD formulation for 3D point set fitting
• Horn (1987): quaternion-based closed-form absolute orientation
• Horn et al. (1988): orthonormal matrix (polar decomposition) approach
• Umeyama (1991): extension to include optimal scaling

For a detailed walkthrough of the Kabsch algorithm with code examples, see the companion blog post: The Kabsch Algorithm.

This is primarily a Resource ($\Psi_{\text{Resource}}$) paper, with secondary Method ($\Psi_{\text{Method}}$) contributions.

• Resource Basis: The core contribution is “ChemBERTa-3,” an open-source framework integrated into DeepChem that standardizes the pretraining and benchmarking of chemical foundation models. The authors focus heavily on infrastructure (AWS/Ray integration) and correcting benchmarking inconsistencies in the field.
• Method Basis: It trains models like “c3-MoLFormer” to reproduce and validate the infrastructure.

The Pretraining Scalability Challenge

• Scalability Challenges: Building robust molecular models is difficult due to the vast size of chemical space and the computational intensity of pretraining on large datasets.
• Proprietary Barriers: Many high-performing chemical foundation models (e.g., the full MoLFormer-XL) are partially closed-source or difficult to reproduce.
• Benchmarking Inconsistencies: There is a lack of systematic comparison between architectures (e.g., Graph vs. Transformer) using unified protocols. Specifically, previous comparisons relied on reported results that used differing scaffold splitting algorithms, making them inaccurate.

Unified Infrastructure & Standardized Benchmarking

• Unified Infrastructure: Integration of DeepChem with Ray for distributed, scalable pretraining and fine-tuning of both graph and transformer models.
• Standardized Benchmarking: Identification that MoLFormer’s scaffold splitting algorithm differs from the standard DeepChem/MoleculeNet splitter, and the subsequent standardization of these benchmarks for fair comparison.
• New DeepChem Tools: Introduction of the ModularTorchModel class for flexible loss computation and HuggingFaceModel wrappers to bridge ecosystems.

Benchmarking Transformers vs. Graph Models

• Architecture Comparison: Benchmarked Transformers (ChemBERTa, MoLFormer) against Graph models (GROVER, InfoGraph, InfoMax3D, DMPNN, GCN) and baselines (Random Forest).
• Pretraining Scale Disparity:
  • Transformers were pretrained on ZINC20 subsets ranging from 10M to 1.1B molecules (combining ZINC and PubChem).
  • Graph models were limited to 250K molecule subsets due to memory and computational overhead of message passing on large graphs. While this highlights the superior scalability of Transformer architectures, comparing a 1.1B-trained Transformer to a 250K-trained Graph model provides an unbalanced evaluation of architectural capacity.
• Reproducibility Validation: Trained “c3-MoLFormer” (a reproduction of MoLFormer) on 1.1B molecules using two distinct hardware setups: AWS spot instances (Ray) and a local HPC cluster.
• Scaffold Split Analysis: Compared performance metrics using “DeepChem scaffold splits” vs. “MoLFormer scaffold splits” to quantify the impact of data leakage/overlap.

Overcoming Scaffold Splitting Inconsistencies

• Scaling Transformers vs. Graphs: Transformer-based models are significantly easier to scale to large datasets than current graph-based approaches, though performance is comparable at small scales.
• Benchmarking sensitivity: MoLFormer’s reported superiority over baselines was partly inflated by its specific scaffold splitting method, which had higher structural overlap between train and test sets (yielding a lower Tanimoto distance, generally quantified via $1 - \frac{|A \cap B|}{|A \cup B|}$) than DeepChem splits. When standardized, baselines like DMPNN perform more competitively.
• Infrastructure Viability: The framework successfully replicated large-scale training (MoLFormer-1.1B) on both cloud and on-premise HPC, confirming reproducibility.
• Open Source Release: All code, configurations, and the c3-MoLFormer-1.1B model weights are released to facilitate future research.

Reproducibility Details

Data

• Pretraining:
  • Source: ZINC20 (1.4B compounds) and PubChem.
  • Scale: Subsets of 10M, 100M, and 1.1B (100% ZINC20 + 100% PubChem) were used for Transformers. Graph models used a 250K subset.
• Fine-tuning:
  • Suite: MoleculeNet.
  • Tasks: Classification (BACE, BBBP, Tox21, HIV, SIDER, ClinTox) and Regression (ESOL, FreeSolv, Lipophilicity, QM9).
  • Splits: Critical distinction made between “DeepChem scaffold splits” (80/10/10) and “MoLFormer scaffold splits” (which can be downloaded from https://ibm.ent.box.com/v/MoLFormer-data). The paper notes these algorithms differ.

Algorithms

• Framework: DeepChem integrated with Ray for distributed training. To recreate the environment, the repository relies on a nightly version of DeepChem (pip install --pre deepchem) and specific dependencies found within the requirements.txt. Pretraining scripts are available in the chemberta3_benchmarking/pretraining directory of the repository.
• Data Preparation: Featurization workflows (e.g., CircularFingerprint, RDKitConformer) are documented under chemberta3_benchmarking/data/data_preprocessing/ in the codebase.
• Modular Training: Uses ModularTorchModel to allow loss computation from intermediate values and flexible component connection.
• Training Brittleness:
  • Optimizer: Linear learning rate scheduler with warmup.
  • Instability Handling: The authors observed significant loss spikes during warmup. Their primary mitigation strategy involved checkpointing frequently and restarting from the last stable state upon a spike, highlighting a persistent brittleness in optimizing these large chemical foundation models.
  • Numerical Issues: Addressed NaN values by pretraining on a small dataset with low LR before scaling up.

Models

• ChemBERTa: RoBERTa-based architecture trained with Masked Language Modeling (MLM) and Multitask Regression (MTR). Specific model identifiers (e.g., DeepChem/ChemBERTa-100M-MLM) are hosted on Hugging Face so researchers can pull them directly via the transformers library. The core pretraining objective minimized the standard MLM loss: $$ \mathcal{L}_{\text{MLM}} = - \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \log \hat{y}_{i} $$ where $\mathcal{M}$ represents the set of masked SMILES token indices, and $\hat{y}_{i}$ is the model’s predicted probability for the correct token given the corrupted sequence context.
• MoLFormer (c3-MoLFormer): Re-implementation of the MoLFormer architecture (Rotary embeddings, linear attention). Specific model identifiers (e.g., DeepChem/MoLFormer-c3-1.1B) are similarly available on Hugging Face.
  • Tokenizer: ibm/MoLFormer-XL-both-10pct tokenizer.
• Graph Models:
  • GROVER: Graph Transformer with node/edge/graph level self-supervision.
  • InfoGraph: Maximizes mutual information between graph-level and substructure representations.
  • InfoMax3D: Incorporates 3D conformer data (via RDKit ETKDGv2) into contrastive pretraining.
  • DMPNN: Directed Message Passing Neural Network (Chemprop variant).

Evaluation

• Metrics: ROC-AUC for classification; RMSE for regression (MAE for QM9).
• Baselines: Random Forest, GCN, DMPNN trained on fine-tuning splits only.
• Protocol: Three independent runs per configuration to report mean and range (not a confidence interval), with the exception of the compute-heavy QM9 dataset, which only received a single run. Benchmarking execution scripts (e.g., GCN, RF, DMPNN, ChemBERTa) are stored in the repo under chemberta3_benchmarking/models_benchmarking/ and contain the specific fine-tuning hyperparameters and optimizer configurations used for each downstream task.
• Key Results:
  • c3-MoLFormer-1.1B achieved ~0.848 ROC-AUC on BACE and ~0.900 on BBBP (using MoLFormer splits). This closely matches the original IBM MoLFormer metrics, validating the reproducibility of the open-source framework.
  • When constrained to the equivalent 250K subset, Graph models (InfoGraph, GROVER) performed comparably to Transformers, indicating that Transformer superiority in chemistry is largely driven by data scalability rather than an inherent architectural advantage at small scales.

Hardware

• Cloud (AWS):
  • Compute: 40 NVIDIA T4 GPUs (g4dn.12xlarge spot instances for pretraining, g4dn.2xlarge for benchmarking).
  • Cost: ~$4000 for MoLFormer 1.1B pretraining.
  • Time: ~10 days (260 hours) for 1.1B model pretraining.
  • Setup: Setup scripts for single-node and multi-node spot EC2 clusters are provided in the GitHub repository’s infra/ and spot/ folders.
• On-Premise HPC:
  • Compute: 16 nodes (AMD EPYC), each with 4 AMD MI300A APUs.
  • Environment: Ray multi-node multi-GPU framework.

Artifacts

Paper Information

Citation: Singh, R. et al. (2026). ChemBERTa-3: an open source training framework for chemical foundation models. Digital Discovery, 5, 662-685. https://doi.org/10.1039/D5DD00348B

Publication: Digital Discovery 2026

Additional Resources:

• ChemBERTa-3 GitHub Repository
• DeepChem Project
• DeepChem Hugging Face Models

@article{singhChemBERTa3OpenSource2026,
  author = {Singh, Riya and Barsainyan, Aryan Amit and Irfan, Rida and Amorin, Connor Joseph and He, Stewart and Davis, Tony and Thiagarajan, Arun and Sankaran, Shiva and Chithrananda, Seyone and Ahmad, Walid and Jones, Derek and McLoughlin, Kevin and Kim, Hyojin and Bhutani, Anoushka and Sathyanarayana, Shreyas Vinaya and Viswanathan, Venkat and Allen, Jonathan E. and Ramsundar, Bharath},
  title = {{{ChemBERTa-3}}: an open source training framework for chemical foundation models},
  journal = {Digital Discovery},
  year = {2026},
  volume = {5},
  pages = {662-685},
  publisher = {The Royal Society of Chemistry},
  doi = {10.1039/D5DD00348B},
  url = {https://doi.org/10.1039/D5DD00348B}
}

This is primarily a Methodological paper ($\Psi_{\text{Method}}$) that introduces a novel pipeline (MERMaid) for extracting structured chemical data from unstructured PDF documents. It proposes a specific architecture combining fine-tuned vision models (VisualHeist) with vision-language models (DataRaider) and a retrieval-augmented generation system (KGWizard) to solve the problem of multimodal data ingestion.

Secondarily, it is a Resource paper ($\Psi_{\text{Resource}}$) as it releases the source code, prompts, and a new benchmark dataset (MERMaid-100) consisting of annotated reaction data across three chemical domains.

The Inaccessibility of Diagrammatic Reaction Data

• Data Inaccessibility: A significant volume of chemical knowledge currently resides in “print-optimized” PDF formats, specifically within graphical elements like figures, schemes, and tables, which resist standard text mining.
• Limitations of Prior Work: Existing tools (e.g., ChemDataExtractor, OpenChemIE) focus primarily on text, struggle with multimodal parsing, or lack the “contextual awareness” needed to interpret implicit information (e.g., “standard conditions” with modifications in optimization tables).
• Need for Structured Data: To enable self-driving laboratories and data-driven discovery, this unstructured literature must be converted into machine-actionable formats like knowledge graphs.

The MERMaid Pipeline: Vision Models and LLM RAG

• VisualHeist (Fine-tuned Segmentation): A custom fine-tuned model based on Microsoft’s Florence-2 that accurately segments figures, captions, and footnotes, even in messy supplementary materials.
• DataRaider (Context-Aware Extraction): A VLM-powered module (using GPT-4o) with a two-step prompt framework that performs “self-directed context completion.” It can infer missing reaction parameters from context and resolve footnote labels (e.g., linking “condition a” in a table to its footnote description).
• KGWizard (Schema-Adaptive Graph Construction): A text-to-graph engine that uses LLMs as higher-order functions to synthesize parsers dynamically. It employs Retrieval-Augmented Generation (RAG) to check for existing nodes during creation, implicitly resolving coreferences (e.g., unifying “MeCN” and “Acetonitrile”).
• Topic-Agnostic Design: MERMaid features a flexible design that works across three distinct domains: organic electrosynthesis, photocatalysis, and organic synthesis.

Benchmarking Segmentation and Extraction Accuracy

• Segmentation Benchmarking: The authors compared VisualHeist against OpenChemIE (LayoutParser) and PDFigCapX using a dataset of 121 PDFs from 5 publishers.
• End-to-End Extraction: Evaluated the full pipeline on MERMaid-100, a curated dataset of 100 articles across three domains (organic electrosynthesis, photocatalysis, organic synthesis).
  • Validating extraction of specific parameters (e.g., catalysts, solvents, yields) using “hard-match” accuracy.
• Knowledge Graph Construction: Automatically generated knowledge graphs for the three domains and assessed the structural integrity and coreference resolution accuracy.

End-to-End Extraction Performance

• Segmentation Results: VisualHeist achieved >93% F1 score across all document types (including pre-2000 papers and supplementary materials), outperforming OpenChemIE by 15-75% and PDFigCapX by 28-75% across all metrics.
• Extraction Accuracy: DataRaider achieved >92% accuracy for VLM-based parameter extraction and near-unity accuracy for domain-specific reaction parameters (e.g., anode, cathode, photocatalyst).
• Graph Building: KGWizard achieved 96% accuracy in node creation and coreference resolution.
• Overall Performance: The pipeline demonstrated an 87% end-to-end overall accuracy.
• Limitations: The architecture relies heavily on closed-weight models (GPT-4o) for reasoning and graph construction, which risks future reproducibility if API snapshots are deprecated. Additionally, the system remains vulnerable to cumulative error propagation from upstream OCR/OCSR tools like RxnScribe.
• Availability: The authors provide a modular, extensible framework that can be adapted to other scientific domains.

Reproducibility Details

Data

• Training Data (VisualHeist):
  • Dataset of 3,435 figures and 1,716 tables annotated from 3,518 PDF pages.
  • Includes main text, supplementary materials, and unformatted archive papers.
• Evaluation Data (MERMaid-100):
  • 100 PDF articles curated from three domains: organic electrosynthesis, photocatalysis, and organic synthesis.
  • Includes 104 image-caption/table-heading pairs relevant to reaction optimization.
  • Available for download at Zenodo (DOI: 10.5281/zenodo.14917752).

Algorithms

• Two-Step Prompt Framework (DataRaider):
  • Step 1: Generic base prompt + domain keys to extract “reaction dictionaries” and “footnote dictionaries”. Uses “fill-in-the-blank” inference for missing details.
  • Step 2: Safety check prompt where the VLM updates the reaction dictionary using the footnote dictionary to resolve entry-specific modifications.
• LLM-Synthesized Parsers (KGWizard):
  • Uses LLM as a function $g_{A,B}: A \times B \rightarrow (X \rightarrow Y)$ to generate Python code (parsers) dynamically based on input schema instructions.
• RAG for Coreference:
  • During graph construction, the system queries the existing database for matching values (e.g., “MeCN”) before creating new nodes to prevent duplication.
• Batching:
  • Articles processed in dynamic batch sizes (starting at 1, increasing to 30) to balance speed and redundancy checks.

Models

• VisualHeist: Fine-tuned Florence-2-large (Microsoft vision foundation model).
  • Hyperparameters: 12 epochs, learning rate $5 \times 10^{-6}$, batch size 4.
• DataRaider & KGWizard: GPT-4o (version gpt-4o-2024-08-06). Note: Requires an active OpenAI API key. The pipeline’s long-term reproducibility is currently tied to the continued availability of this specific closed-source endpoint.
• RxnScribe: Used for Optical Chemical Structure Recognition (OCSR) to convert reactant/product images to SMILES.

Evaluation

• Metrics:
  • Segmentation: Precision, Recall, F1, Accuracy.
  • Caption Extraction: Evaluated via Jaccard similarity, mapping predicted token sets $A$ and true token sets $B$ to a threshold condition: $$J(A, B) = \frac{|A \cap B|}{|A \cup B|} \ge 0.70$$
  • Data Extraction: Evaluated via Hard-Match accuracy, requiring exact correspondence between predicted sets ($\hat{Y}$) and ground-truth parameters ($Y$) for specific roles (e.g., anode vs. cathode): $$\text{HMA} = \frac{1}{|N|} \sum_{i=1}^{N} \mathbb{1}[y_i = \hat{y}_i]$$
• Baselines: OpenChemIE (LayoutParser + EasyOCR) and PDFigCapX.

Hardware

• Training (VisualHeist): 2x NVLINK Nvidia RTX A6000 GPUs (48GB VRAM) + Intel Xeon w7-2495X CPU (48 cores).
• DataRaider Evaluation: 13th Gen Intel Core i7-1360P CPU (12 cores).
• Inference Costs:
  • DataRaider: ~$0.051 per image.
  • KGWizard: ~$0.40 per JSON.
• Timing:
  • VisualHeist inference: ~4.5 seconds/image.
  • DataRaider inference: ~41.3 seconds/image.
  • KGWizard processing: ~110.6 seconds/file.

Paper Information

Citation: Leong, S. X., Pablo-García, S., Wong, B., & Aspuru-Guzik, A. (2025). MERMaid: Universal multimodal mining of chemical reactions from PDFs using vision-language models. Matter, 8(12), 102331. https://doi.org/10.1016/j.matt.2025.102331

Publication: Matter, 2025

Artifacts:

@article{leong2025mermaid,
  title={MERMaid: Universal multimodal mining of chemical reactions from PDFs using vision-language models},
  author={Leong, Shi Xuan and Pablo-Garc{\'i}a, Sergio and Wong, Brandon and Aspuru-Guzik, Al{\'a}n},
  journal={Matter},
  volume={8},
  number={12},
  pages={102331},
  year={2025},
  doi={10.1016/j.matt.2025.102331}
}

In computational chemistry and AI drug discovery, visualization pipelines are often brittle; breaking on edge cases or failing silently when processing millions of molecules for training data.

I built molecular-string-renderer to treat molecular visualization as a strict software engineering problem. It is a highly configurable wrapper around RDKit that standardizes the conversion of text-based chemical representations (SMILES, InChI, SELFIES) into raster and vector graphics, degrading gracefully on inputs RDKit cannot vectorize.

Features

This library differentiates itself from standard plotting scripts through strict architectural patterns designed for reliability:

1. Strategy Pattern for SVG Generation

RDKit’s vector rendering can sometimes fail on complex molecular topologies. I implemented a Hybrid Strategy so that a single molecule RDKit cannot vectorize does not fail the batch:

• Vector Strategy: Attempts to generate a true, scalable vector graphic.
• Raster Fallback: If the vector engine fails, the system automatically renders a high-res PNG and embeds it transparently into the SVG container.

2. Native Generative AI Support

With the rise of Large Language Models in chemistry, SELFIES (Self-Referencing Embedded Strings) has become a standard output format. This library handles SELFIES natively, managing the decoding and sanitization lifecycle internally so that ML training loops can simply “pass strings and get images.”

3. Strict Configuration Contracts

The library uses Pydantic models (RenderConfig, ParserConfig, OutputConfig) to enforce strict data contracts. This ensures that visualization parameters are validated before any heavy computation begins, preventing runtime errors deep in a batch job.

Usage

The library provides a simple Python API for rendering single molecules or batches of molecules from various string formats.

Results

• Type Safety: The codebase runs with strict mypy settings, ensuring type safety across the entire pipeline.
• Grid Auto-Fitting: Implemented smart layout algorithms that automatically adjust grid dimensions based on the input batch size.
• Format Agnostic: Decouples the parsing logic (SMILES vs. MolBlock vs. SELFIES) from the rendering logic, making it trivial to add support for new proprietary formats.

Reliability

When rendering large batches of generated molecules, a single hard-to-draw structure should not fail the whole job. The raster fallback and the strict Pydantic and mypy contracts exist so the pipeline degrades gracefully on edge cases rather than crashing or failing silently, the common failure mode of ad hoc RDKit plotting scripts.

Related Work

• Visualizing SMILES and SELFIES Strings: walkthrough of the visualization pipeline this library implements
• Isomer Dataset Generation: related project generating molecular datasets using SMILES/SELFIES representations

This software update paper documents major improvements to the SELFIES Python library (version 2.1.1), covering its history, underlying algorithms, design, and performance.

Limitations in the Original SELFIES Implementation

While the original SELFIES concept was promising, the initial 2019 implementation had critical limitations that prevented widespread adoption:

1. Performance: Too slow for production ML workflows
2. Limited chemistry: Couldn’t represent aromatic molecules, stereochemistry, or many other important chemical features
3. Poor usability: Lacked user-friendly APIs for common tasks

These barriers meant that despite SELFIES’ theoretical advantages (100% validity guarantee), researchers couldn’t practically use it for real-world applications like drug discovery or materials science.

Architectural Refactoring and New ML Integrations

The 2023 update refactors the underlying SELFIES engine with improvements to design, efficiency, and supported features. The key updates include:

1. Streamlined Grammar: The underlying context-free grammar has been generalized and streamlined, improving execution speed and extensibility while maintaining the 100% validity guarantee.

2. Expanded Chemical Support: Adds support for aromatic systems (via internal kekulization), stereochemistry (chirality, cis/trans), charged species, and isotopic data, covering nearly all features supported by SMILES while preserving the validity guarantee.

3. Semantic Constraint API: Introduces the set_semantic_constraints() function, allowing specification of custom valence definitions useful for theoretical studies or hypervalent states.

4. ML Utility Functions: Provides tokenization (split_selfies), length estimation (len_selfies), label/one-hot encoding (selfies_to_encoding), vocabulary extraction, and attribution tracking for integration with neural network pipelines.

Performance Benchmarks & Validity Testing

The authors validated the library through several benchmarks:

Performance testing: Roundtrip conversion (SMILES to SELFIES to SMILES) on the DTP open compound collection (slightly over 300K molecules) completed in 252 seconds total (136s encoding, 116s decoding), using pure Python with no external dependencies.

Random SELFIES generation: Demonstrated that random SELFIES strings of varying lengths always decode to valid molecules, with the size distribution of generated molecules controllable by filtering the sampling alphabet (e.g., removing multi-bond and low-valence atom symbols shifts the distribution toward larger molecules).

Validity guarantee: By construction, every SELFIES string decodes to a valid molecule. The grammar’s bond demotion and deferred ring closure mechanisms make it impossible to generate chemically invalid structures.

Attribution system: Showed both encoder and decoder can track which input symbols produce which output symbols, useful for property alignment.

Future Trajectories for General Chemical Representations

The 2023 update successfully addresses the main adoption barriers:

1. Fast enough for large-scale ML applications (300K molecules in ~4 minutes)
2. Chemically comprehensive enough for drug discovery and materials science
3. User-friendly enough for straightforward integration into existing workflows

The validity guarantee, SELFIES’ core advantage, is now practically accessible for real-world research. The roadmap includes future extensions for polymers, crystals, chemical reactions, and non-covalent interactions, which would expand SELFIES’ applicability beyond small-molecule chemistry.

Limitations acknowledged: The paper focuses on implementation improvements. Some advanced chemical systems (polymers, crystals) still need future work.

Reproducibility Details

Artifacts

Code

The selfies library is completely open-source and written in pure Python. It requires no extra dependencies and is available on GitHub, installable via pip install selfies. The repository includes testing suites (tox) and example benchmarking scripts to reproduce the translation speeds reported in the paper.

Hardware

Performance benchmarks (e.g., the 252-second roundtrip conversion on 300K molecules) were executed on Google Colaboratory using two 2.20GHz Intel Xeon CPUs.

Algorithms

Technical Specification: The Grammar

The core innovation of SELFIES is a Context-Free Grammar (CFG) augmented with state-machine logic to ensure that every derived string represents a valid molecule. While the software features are important, understanding the underlying derivation rules is essential for replication or extension of the system.

1. Derivation Rules: The Atom State Machine

The fundamental mechanism that guarantees validity is a state machine that tracks the remaining valence of the most recently added atom:

• State Tracking: The derivation maintains a non-terminal state $X_l$, where $l$ represents the current atom’s remaining valence (number of bonds it can still form)
• Standard Derivation: An atom symbol $[\beta \alpha]$ (bond order + atom type) transitions the state from $S$ (start) to $X_l$, where $l$ is calculated from the atom’s standard valence minus the incoming bond order
• Bond Demotion (The Key Rule): When deriving atom symbol $[\beta \alpha]$ in state $X_i$, the actual bond order used is $d_0 = \min(\ell, i, d(\beta))$, where $\ell$ is the new atom’s valence, $i$ is the previous atom’s remaining capacity, and $d(\beta)$ is the requested bond order. This automatic downward adjustment is the mathematical core of the validity guarantee.

This state machine ensures that no atom ever exceeds its allowed valence, making it impossible to generate chemically invalid structures.

2. Control Symbols: Branches and Rings

Branch length calculation: SELFIES uses a hexadecimal encoding to determine branch lengths. A branch symbol [Branch l] consumes the next $\ell$ symbols from the queue and converts them to integer indices $c_1, \dots, c_\ell$ via a fixed mapping (Table III in the paper). The number of symbols $N$ to include in the branch is then:

$$ N = 1 + \sum_{k=1}^{\ell} 16^{\ell - k} , c_k $$

This formula interprets the indices as hexadecimal digits, allowing compact specification of branches up to hundreds of symbols long.

Ring closure queue system: Ring formation uses a deferred evaluation strategy to maintain validity. Ring symbols don’t create bonds immediately; instead, they push closure candidates into a queue $R$. These candidates are resolved after the main derivation completes. A ring closure candidate is rejected if either ring atom has no remaining valence ($m_1 = 0$ or $m_2 = 0$), or if the left and right ring atoms are not distinct (to avoid self-loops). If a prior bond already exists between the two atoms, the bond order is incremented rather than duplicated. This deferred validation prevents invalid ring structures while keeping the grammar context-free during the main derivation.

3. Symbol Structure and Standardization

SELFIES enforces a strict, standardized format for atom symbols to eliminate ambiguity:

• Canonical Format: Atom symbols follow the structure [Bond, Isotope, Element, Chirality, H-count, Charge]
• No Variation: There is only one way to write each symbol (e.g., [Fe++] and [Fe+2] are standardized to a single form)
• Order Matters: The components must appear in the specified order

4. Default Semantic Constraints

By default, the library enforces standard organic chemistry valence rules:

• Charge-Dependent Valences: Default constraints specify maximum bonds per charge state (e.g., C: 4/5/3 for neutral/+1/-1; S: 6/7/5). Unlisted atom types default to 8 maximum bonds as a catch-all.
• Preset Options: Three preset constraint sets are available: default, octet_rule, and hypervalent.
• Customizable: Constraints can be modified via set_semantic_constraints() for specialized applications (hypervalent compounds, theoretical studies, etc.)

The combination of these grammar rules with the state machine ensures that every valid SELFIES string decodes to a chemically valid molecule, regardless of how the string was generated (random, ML model output, manual construction, etc.).

Data

Benchmark dataset: DTP (Developmental Therapeutics Program) open compound collection with slightly over 300K SMILES strings, a set of molecules tested experimentally for potential treatment against cancer and AIDS.

Random generation testing: Random SELFIES strings of varying lengths (10, 100, 250 symbols) generated from both basic and filtered alphabets to test decoding validity and molecule size distributions.

Evaluation

Performance metric: Roundtrip conversion time (SMILES to SELFIES to SMILES) is 252 seconds for 300K+ molecules (136s encoding, 116s decoding). Times averaged over 3 replicate trials on Google Colaboratory.

Validity testing: Random SELFIES strings of lengths 10, 100, and 250 all decode to valid molecules. Decoding 1000 random strings of length 250 from the basic alphabet takes 0.341s; from the filtered alphabet, 1.633s.

Attribution system: Both encoder() and decoder() support an attribute flag that returns AttributionMap objects, tracing which input symbols produce which output symbols for property alignment.

Paper Information

Citation: Lo, A., Pollice, R., Nigam, A., White, A. D., Krenn, M., & Aspuru-Guzik, A. (2023). Recent advances in the self-referencing embedded strings (SELFIES) library. Digital Discovery, 2(4), 897-908. https://doi.org/10.1039/D3DD00044C

Publication: Digital Discovery 2023

@article{lo2023recent,
  title={Recent advances in the self-referencing embedded strings (SELFIES) library},
  author={Lo, Alston and Pollice, Robert and Nigam, AkshatKumar and White, Andrew D and Krenn, Mario and Aspuru-Guzik, Al{\'a}n},
  journal={Digital Discovery},
  volume={2},
  number={4},
  pages={897--908},
  year={2023},
  publisher={Royal Society of Chemistry},
  doi={10.1039/D3DD00044C}
}

Additional Resources:

• SELFIES GitHub Repository
• Original SELFIES Paper (2020)
• SELFIES Format Overview

I ran into this recently while debugging a generative model. Sometimes the grammar of the string provides the clue as to what is going wrong. Other times, actually seeing the molecule is what helps. I had a terminal full of generated strings and needed to verify their structures visually. I needed a streamlined way to generate these images locally. A lightweight script turns that text into a properly formatted image directly from the terminal.

SMILES vs. SELFIES

There are two primary string representations you will encounter in modern cheminformatics:

1. SMILES: The industry standard. It uses simple rules (C for carbon, = for double bonds, parentheses for branches). It is compact and machine-parseable. However, random SMILES strings are often invalid (e.g., unclosed rings or invalid valences).
2. SELFIES: Designed specifically for machine learning. It is a robust representation where every string corresponds to a valid molecular graph. This makes it ideal for generative models. Note that it is more verbose than SMILES.

I often need to visualize both formats. Let’s build a single, robust Python tool to handle them.

The Quick Win: Native RDKit

If you just need a quick image from a SMILES string and don’t care about the image dimensions or adding a legend, RDKit can do this in three lines:

from rdkit import Chem
from rdkit.Chem import Draw

mol = Chem.MolFromSmiles("CCO")
Draw.MolToFile(mol, "ethanol.png")

The native RDKit method is fast for quick checks. However, custom rendering provides necessary control over image dimensions, formula subscripts, and handling multiple input formats like SELFIES.

Building a Custom Renderer for Precise Control

Let’s build a fuller tool using RDKit for chemical processing, the selfies library for decoding, and PIL for image manipulation.

Core Dependencies

import selfies as sf
from rdkit import Chem
from rdkit.Chem import Draw, rdDepictor, rdMolDescriptors
from PIL import Image, ImageDraw, ImageFont

RDKit handles the chemical logic, selfies translates SELFIES to SMILES, and PIL gives us fine control over the final image appearance.

The Main Conversion Function

Here is the core conversion logic. Notice the Python type hints on the signature.

def string_to_png(mol_string: str, output_file: str, size: int = 500, is_selfies: bool = False) -> None:
    """Generates a 2D molecule image with a chemical formula legend from SMILES or SELFIES."""

    # Decode SELFIES to SMILES if necessary
    if is_selfies:
        try:
            smiles = sf.decoder(mol_string)
        except Exception as e:
            raise ValueError(f"Invalid SELFIES string: {mol_string}") from e
    else:
        smiles = mol_string

    mol = Chem.MolFromSmiles(smiles)
    if not mol:
        raise ValueError(f"Could not generate molecule from SMILES: {smiles}")

    # Generate 2D coordinates and formula
    rdDepictor.Compute2DCoords(mol)
    formula = rdMolDescriptors.CalcMolFormula(mol)

    # Render the molecule
    img = Draw.MolToImage(mol, size=(size, size)).convert("RGBA")

    # Create a canvas with extra space at the bottom for the legend
    legend_height = int(size * 0.1)
    canvas = Image.new("RGBA", (size, size + legend_height), "white")
    canvas.paste(img, (0, 0))

    draw = ImageDraw.Draw(canvas)

    # Define dynamic font sizes
    font_reg = get_font(int(size * 0.03))
    font_sub = get_font(int(size * 0.02))

    # Draw the legend
    x = int(size * 0.02)
    y = size + int(size * 0.02)

    # Draw "Formula: " label
    draw.text((x, y), "Formula: ", fill="black", font=font_reg)
    x += draw.textlength("Formula: ", font=font_reg)

    # Draw formula with subscript handling for numbers
    for char in formula:
        # Use smaller font and lower y-offset for numbers (subscripts)
        font = font_sub if char.isdigit() else font_reg
        y_offset = int(size * 0.005) if char.isdigit() else 0

        draw.text((x, y + y_offset), char, fill="black", font=font)
        x += draw.textlength(char, font=font)

    # Draw original string
    label = "SELFIES" if is_selfies else "SMILES"
    draw.text((x, y), f" | {label}: {mol_string}", fill="black", font=font_reg)

    canvas.save(output_file)
    print(f"Saved: {output_file}")

This function handles everything: SELFIES decoding, validation, coordinate generation, image creation, and legend drawing.

Font Handling

We need a helper to handle fonts robustly across systems:

def get_font(size: int, font_name: str = "arial.ttf"):
    """Attempts to load a TTF font, falls back to default if unavailable."""
    try:
        return ImageFont.truetype(font_name, size)
    except IOError:
        return ImageFont.load_default()

Examples in Action

Let’s see the tool in action with some common molecules, comparing the SMILES and SELFIES inputs.

Simple Molecules

Aromatic Compounds

Complex Pharmaceuticals

Going Further: Vector Graphics (SVG)

Use vector graphics (SVG/PDF) for true publication-quality figures. Vector graphics scale infinitely without pixelation.

RDKit handles this natively with rdMolDraw2D:

from rdkit import Chem
from rdkit.Chem.Draw import rdMolDraw2D

def string_to_svg(mol_string: str, output_file: str, size: int = 500, is_selfies: bool = False) -> None:
    """Generates a 2D molecule SVG image."""
    if is_selfies:
        try:
            mol_string = sf.decoder(mol_string)
        except Exception as e:
            raise ValueError(f"Invalid SELFIES string: {mol_string}") from e

    mol = Chem.MolFromSmiles(mol_string)
    if not mol:
        raise ValueError(f"Invalid string: {mol_string}")

    rdDepictor.Compute2DCoords(mol)

    d = rdMolDraw2D.MolDraw2DSVG(size, size)
    d.DrawMolecule(mol)
    d.FinishDrawing()

    with open(output_file, "w") as f:
        f.write(d.GetDrawingText())
    print(f"Saved: {output_file}")

This provides a perfect vector image. Note that this method omits the custom PIL-based legend. Choose the right tool for the job: PNG for quick checks and slides, SVG for journal submissions.

Command-Line Interface

The tool uses Python’s standard argparse library for the command-line interface. It automatically detects if you want an SVG based on the file extension and includes a --selfies flag.

# Basic SMILES usage
python mol2img.py "CCO" -o ethanol.png

# SELFIES usage
python mol2img.py "[C][C][O]" -o ethanol.png --selfies

# Generate SVG for publication
python mol2img.py "CCO" -o ethanol.svg

Download the Complete Script

You can copy the complete mol2img.py script directly from the code block below. For a fuller version with an SVG fallback, type hints, and batch (grid) rendering, see the Molecular String Renderer project.

Installation and Setup

Before using the script, install the required dependencies:

pip install rdkit pillow selfies

Complete Script

import argparse
import sys
import os
import selfies as sf
from rdkit import Chem
from rdkit.Chem import Draw, rdDepictor, rdMolDescriptors
from rdkit.Chem.Draw import rdMolDraw2D
from PIL import Image, ImageDraw, ImageFont

def get_font(size: int, font_name: str = "arial.ttf"):
    """Attempts to load a TTF font, falls back to default if unavailable."""
    try:
        return ImageFont.truetype(font_name, size)
    except IOError:
        return ImageFont.load_default()

def string_to_svg(mol_string: str, output_file: str, size: int = 500, is_selfies: bool = False) -> None:
    """Generates a 2D molecule SVG image."""
    if is_selfies:
        try:
            mol_string = sf.decoder(mol_string)
        except Exception as e:
            raise ValueError(f"Invalid SELFIES string: {mol_string}") from e

    mol = Chem.MolFromSmiles(mol_string)
    if not mol:
        raise ValueError(f"Invalid string: {mol_string}")

    rdDepictor.Compute2DCoords(mol)

    d = rdMolDraw2D.MolDraw2DSVG(size, size)
    d.DrawMolecule(mol)
    d.FinishDrawing()

    with open(output_file, "w") as f:
        f.write(d.GetDrawingText())
    print(f"Saved: {output_file}")

def string_to_png(mol_string: str, output_file: str, size: int = 500, is_selfies: bool = False) -> None:
    """Generates a 2D molecule image with a chemical formula legend."""
    if is_selfies:
        try:
            smiles = sf.decoder(mol_string)
        except Exception as e:
            raise ValueError(f"Invalid SELFIES string: {mol_string}") from e
    else:
        smiles = mol_string

    mol = Chem.MolFromSmiles(smiles)
    if not mol:
        raise ValueError(f"Could not generate molecule from string: {mol_string}")

    # Generate 2D coordinates and formula
    rdDepictor.Compute2DCoords(mol)
    formula = rdMolDescriptors.CalcMolFormula(mol)

    # Render the molecule
    img = Draw.MolToImage(mol, size=(size, size)).convert("RGBA")

    # Create a canvas with extra space at the bottom for the legend
    legend_height = int(size * 0.1)
    canvas = Image.new("RGBA", (size, size + legend_height), "white")
    canvas.paste(img, (0, 0))

    draw = ImageDraw.Draw(canvas)

    # Define dynamic font sizes
    font_reg = get_font(int(size * 0.03))
    font_sub = get_font(int(size * 0.02))

    # Draw the legend
    x = int(size * 0.02)
    y = size + int(size * 0.02)

    # Draw "Formula: " label
    draw.text((x, y), "Formula: ", fill="black", font=font_reg)
    x += draw.textlength("Formula: ", font=font_reg)

    # Draw formula with subscript handling for numbers
    for char in formula:
        # Use smaller font and lower y-offset for numbers (subscripts)
        font = font_sub if char.isdigit() else font_reg
        y_offset = int(size * 0.005) if char.isdigit() else 0

        draw.text((x, y + y_offset), char, fill="black", font=font)
        x += draw.textlength(char, font=font)

    # Draw original string
    label = "SELFIES" if is_selfies else "SMILES"
    draw.text((x, y), f" | {label}: {mol_string}", fill="black", font=font_reg)

    canvas.save(output_file)
    print(f"Saved: {output_file}")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Convert a SMILES or SELFIES string to a 2D molecular image.")
    parser.add_argument("string", help="The molecular string to convert")
    parser.add_argument("-o", "--output", default="molecule.png", help="Output filename (default: molecule.png)")
    parser.add_argument("--size", type=int, default=500, help="Image width/height in pixels (default: 500)")
    parser.add_argument("--svg", action="store_true", help="Force SVG output (overrides filename extension)")
    parser.add_argument("--selfies", action="store_true", help="Treat the input string as SELFIES.")

    args = parser.parse_args()

    try:
        # Determine format based on flag or file extension
        is_svg = args.svg or args.output.lower().endswith(".svg")

        if is_svg:
            # Ensure extension is correct if not present
            if not args.output.lower().endswith(".svg"):
                args.output = os.path.splitext(args.output)[0] + ".svg"
            string_to_svg(args.string, args.output, args.size, args.selfies)
        else:
            string_to_png(args.string, args.output, args.size, args.selfies)

    except Exception as e:
        print(f"Error: {e}")
        sys.exit(1)

In the early days of computing, generating random numbers was a significant computational challenge. In a landmark 1951 paper, mathematician John von Neumann detailed various “cooking recipes” for producing and using random numbers on machines like the ENIAC. While much of the paper focuses on generating uniform random digits, he also described ingenious methods for generating numbers from more complex, non-uniform probability distributions.

One of the most fundamental needs in scientific simulation (from modeling radioactive decay to calculating particle free-paths in molecular dynamics) is sampling from an exponential distribution with probability density function:

$$f(x) = e^{-x} \quad \text{for } x \ge 0$$

Today’s standard approach is elegant and direct, but it requires computing a natural logarithm (a computationally expensive operation on early hardware). To sidestep this limitation, von Neumann described a fascinating alternative that uses only basic comparisons, resembling what he called “a well known game of chance Twenty-One, or Black Jack.”

In this post, we’ll explore both methods: the modern inverse transform approach and von Neumann’s ingenious comparison-based algorithm. We’ll implement them in Python, verify their correctness, and compare their performance, empirically testing the trade-offs von Neumann identified nearly 75 years ago.

Method 1: The Standard Approach (Inverse Transform Sampling)

The most common method for sampling from a given distribution is inverse transform sampling. This method relies on a fundamental principle: if you have a uniform random variable $U$ on the interval (0, 1), you can transform it into a random variable $X$ with any desired cumulative distribution function (CDF) $F(x)$ by applying:

$$X = F^{-1}(U)$$

For the exponential distribution, the CDF is $F(x) = 1 - e^{-x}$. To find the inverse, we set $U = 1 - e^{-X}$ and solve for $X$:

$$ \begin{align} e^{-X} &= 1 - U \ -X &= \ln(1 - U) \ X &= -\ln(1 - U) \end{align} $$

Here’s a useful simplification: since $U$ is uniformly distributed on (0, 1), the quantity $(1 - U)$ is also uniformly distributed on (0, 1). Therefore, we can use the simpler formula:

$$X = -\ln(U)$$

This gives us an efficient method for generating exponentially distributed numbers, provided the logarithm function is computationally accessible.

Python Implementation

Here’s a straightforward implementation using NumPy:

import numpy as np

def exponential_inverse_transform(n_samples=1):
    """
    Generate samples from an exponential distribution using inverse transform sampling.

    Args:
        n_samples (int): Number of samples to generate.

    Returns:
        np.ndarray: Array of exponentially distributed samples.
    """
    # Generate uniform random numbers
    U = np.random.rand(n_samples)

    # Apply the inverse transform
    X = -np.log(U)

    return X

# Generate 100,000 samples for testing
n_samples = 100000
inverse_samples = exponential_inverse_transform(n_samples)

Method 2: Von Neumann’s Ingenious Trick (Rejection Sampling)

Von Neumann proposed a clever alternative that avoids transcendental functions entirely. His procedure, which he noted “resembles a well known game of chance Twenty-One, or Black Jack,” generates sequences of uniform random numbers and accepts or rejects them based on simple comparison rules.

The algorithm works as follows to generate a single exponential sample $X$:

1. Initialize: Start with an integer offset k = 0, which will form the integer part of the final result.

2. Generate a trial sequence:

   • Generate uniform random numbers $Y_1, Y_2, Y_3, \ldots$ from (0, 1)
   • Find the smallest integer n such that the sequence is no longer strictly decreasing
   • That is, find n where $Y_1 > Y_2 > \cdots > Y_n$ but $Y_n \leq Y_{n+1}$

3. Accept or reject:

   • If n is odd: Accept the trial. Return $X = Y_1 + k$ and terminate.
   • If n is even: Reject the trial. Increment k by 1 and start a new trial.

This process is guaranteed to terminate and produces samples that follow the exponential distribution exactly. As von Neumann elegantly put it, the machine has “in effect computed a logarithm by performing only discriminations on the relative magnitude of numbers.”

Python Implementation

This implementation requires more careful state management due to the nested trial structure:

import numpy as np

def exponential_von_neumann(n_samples=1):
    """
    Generate samples from an exponential distribution using von Neumann's
    comparison-based rejection sampling method.

    Args:
        n_samples (int): Number of samples to generate.

    Returns:
        tuple[np.ndarray, float]: Array of samples and average uniform draws per sample.
    """
    samples = []
    total_uniform_draws = 0

    for _ in range(n_samples):
        k = 0  # Integer offset

        while True:  # Trial loop
            # Generate decreasing sequence
            y_prev = np.random.rand()
            total_uniform_draws += 1
            y1 = y_prev  # Store first value
            n = 1

            # Find length of decreasing sequence
            while True:
                y_curr = np.random.rand()
                total_uniform_draws += 1
                if y_prev <= y_curr:
                    break  # Sequence no longer decreasing
                y_prev = y_curr
                n += 1

            # Accept if n is odd, reject if even
            if n % 2 == 1:  # Accept
                samples.append(y1 + k)
                break
            else:  # Reject
                k += 1

    avg_draws = total_uniform_draws / n_samples
    return np.array(samples), avg_draws

# Generate samples using von Neumann's method
von_neumann_samples, avg_draws = exponential_von_neumann(n_samples)
print(f"Von Neumann method used {avg_draws:.2f} uniform draws per sample on average.")

Von Neumann method used 4.30 uniform draws per sample on average.

The algorithm requires approximately 4.3 uniform draws per exponential sample, matching the theoretical value $e^2/(e-1) = 4.30$.

Verification and Comparison

The critical test: do both methods actually produce the same distribution? And how do their performance characteristics compare?

Visual Verification

Let’s plot histograms of samples from both methods alongside the theoretical probability density function $f(x) = e^{-x}$:

import matplotlib.pyplot as plt
import seaborn as sns

# Configure plot aesthetics
sns.set_style("whitegrid")
plt.figure(figsize=(12, 7))

# Plot histograms for both methods
plt.hist(inverse_samples, bins=50, density=True, alpha=0.7,
         label='Inverse Transform', color='skyblue')
plt.hist(von_neumann_samples, bins=50, density=True, alpha=0.7,
         label="Von Neumann's Method", color='lightcoral')

# Overlay theoretical PDF
x = np.linspace(0, 8, 400)
pdf = np.exp(-x)
plt.plot(x, pdf, 'r-', linewidth=2, label='Theoretical PDF ($e^{-x}$)')

plt.title('Exponential Sampling Methods vs. Theoretical Distribution', fontsize=16)
plt.xlabel('x', fontsize=12)
plt.ylabel('Density', fontsize=12)
plt.legend()
plt.xlim(0, 8)
plt.tight_layout()
plt.show()

The visualization confirms that both methods accurately reproduce the target exponential distribution. The empirical histograms match the theoretical curve, confirming both algorithms sample the target distribution.

Performance Analysis

Mathematical elegance often diverges from computational efficiency. Von Neumann himself observed that on the ENIAC, it was actually “slightly quicker to use a truncated power series for log(1-T)” than to perform all the comparisons his method required.

Let’s benchmark both approaches in a modern Python environment:

import time

# Benchmark inverse transform method
start_time = time.time()
_ = exponential_inverse_transform(n_samples)
inverse_time = time.time() - start_time

# Benchmark von Neumann method
start_time = time.time()
_ = exponential_von_neumann(n_samples)
vn_time = time.time() - start_time

print(f"Inverse Transform:  {inverse_time:.4f} seconds")
print(f"Von Neumann Method: {vn_time:.4f} seconds")
print(f"Speedup factor: {vn_time / inverse_time:.1f}x")

Inverse Transform:  0.0018 seconds
Von Neumann Method: 0.1860 seconds
Speedup factor: 103.3x

The gap is large. The vectorized NumPy implementation of inverse transform sampling, leveraging a highly optimized C-backed logarithm function, outperforms the Python-looped von Neumann implementation by more than two orders of magnitude. While a vectorized or JIT-compiled version of von Neumann’s method would close this gap by removing Python interpreter overhead, the inverse transform remains the practical winner on modern hardware with fast floating-point units. This confirms von Neumann’s prescient observation: the “theoretically elegant” method avoiding transcendental functions often yields to direct computation.

Conclusion

This exploration offers a window into the ingenuity of early computational mathematics. Von Neumann’s comparison-based algorithm demonstrates remarkable mathematical creativity (showing how to “compute a logarithm” using only basic machine operations). Our implementation reproduces the algorithm, producing samples whose histogram and moments match the exponential distribution.

The performance comparison validates von Neumann’s own pragmatic assessment. His rejection sampling method is intellectually elegant and historically significant. The direct logarithmic approach proves far more efficient on both early and modern hardware. It serves as a timeless reminder in scientific computing: theoretical beauty often diverges from computational practicality.

The enduring value of von Neumann’s work lies in the fundamental insight that creative mathematical thinking can circumvent apparent computational limitations. Understanding alternative methods deepens our appreciation for the rich landscape of algorithmic possibilities, even when the direct approach proves superior.

The physics of liquid argon is a solved problem. We know the answer.

So, why replicate it in 2025? To apply modern engineering standards to legacy science.

This project served as an exercise in software archaeology: taking a vintage scientific workflow and rebuilding it with a modular Python analysis pipeline. I wanted to see if I could replace Rahman’s “write-once” Fortran mentality with modern reproducibility, type safety, and intelligent caching.

The full source code is available on GitHub. The complete project overview, including analysis results and pipeline architecture, is on the Rahman 1964 Replication project page.

Engineering the Pipeline

The most interesting part of this project isn’t the simulation engine (LAMMPS handles that); it’s the architecture of the analysis suite. MD analysis is computationally expensive ($O(N^2)$), and iterating on plots can be painfully slow if you re-compute trajectory data every time.

Why bother? Don’t modern MD packages come with analysis tools? Well, some say that writing is thinking. Sometimes getting into the weeds of how an algorithm works or an analysis is performed, you gain insights and a deeper understanding that might be obscured by a plug-and-play tool.

Intelligent Caching

I built the argon_sim package with a decorator-based caching layer. The system hashes the source file’s modification time and the function’s arguments to avoid re-calculating the Radial Distribution Function (RDF) or Van Hove correlations on every script run.

@cached_computation("gr")
def compute_radial_distribution(filename: str, dr: float = 0.05):
    # ... expensive O(N^2) distance calculations ...
    return r_values, g_r, density

If I tweak a plot axis, the script runs instantly, loading pre-computed arrays from disk instead of re-running the $O(N^2)$ computation. If I change the simulation trajectory, the cache invalidates automatically.

Vectorization & Memory Management

Rahman likely relied on nested loops. Python is too slow for that. I utilized NumPy broadcasting to vectorize the calculation of atomic displacements.

However, calculating an $864 \times 864$ distance matrix for 5,000 frames consumes significant RAM. I implemented a chunked MSD (Mean Square Displacement) algorithm that processes the trajectory in blocks, balancing vectorization speed with memory constraints. The chunking trades some vectorization speed for a bounded memory footprint, so the analysis is not capped by holding the full distance matrix in RAM.

Reproducibility as a Feature

Academic code is notorious for “it works on my machine.” To combat this, I used uv for dependency management, locking the exact environment state. The entire workflow (from simulation to final figure generation) is abstracted into a Makefile.

# One command to run the physics, analyze data, and generate plots
make workflow

The Simulation: 1964 vs. 2025

I preserved Rahman’s physical parameters exactly to ensure a fair comparison:

• System: 864 Argon atoms
• Potential: Lennard-Jones ($\sigma = 3.4$ Å, $\epsilon/k_B = 120$ K)
• Target: 94.4 K, 1.374 g/cm³

However, I modernized the numerical methods to ensure stability:

The production run lasted 10 ps in the NVE ensemble, generating 5,001 frames. Temperature remained within 1% of target with an RMS fluctuation of 0.0165.

Validation Results

The replication was quantitatively successful. The analysis pipeline faithfully reproduced every key signature of liquid argon.

The Cage Effect

This is the paper’s crown jewel. In a gas, velocity correlations decay exponentially. In a liquid, Rahman discovered that atoms get trapped by their neighbors and bounce back, causing the correlation to go negative.

My simulation captures this minimum at -0.083, matching Rahman’s observation. The Fourier transform of this data (the frequency spectrum) reveals a peak at $\beta \approx 0.25$, physically representing the frequency of atomic collisions within the cage.

Structural Fingerprints

The Radial Distribution Function $g(r)$ and its Fourier transform, the Structure Factor $S(k)$, are the “fingerprints” of a liquid’s structure.

The agreement here is striking. My first peak appeared at 3.82 Å (Rahman: 3.7 Å). The slight discrepancy is likely due to my improved equilibration method, which allowed the system to relax into a more natural liquid state than Rahman’s 1960s hardware allowed.

Diffusion and Non-Gaussian Behavior

By calculating the Mean Square Displacement (MSD), I derived a diffusion coefficient of $D = 2.47 \times 10^{-5}$ cm²/s, which deviates only 2% from Rahman’s reported $2.43 \times 10^{-5}$.

More interestingly, I reproduced the “Non-Gaussian” parameters. Standard diffusion assumes a Gaussian distribution of displacements. Rahman found (and I confirmed) that liquid atoms deviate from this. They exhibit “jump” and “wait” dynamics, a behavior that standard Brownian motion models fail to capture.

Advanced Analysis: Van Hove Functions

Rahman also explored advanced properties like the Van Hove correlation function $G(r,t)$, which describes how liquid structure evolves over time.

At 1.0 ps, the structure remains well-defined with clear shells. By 2.5 ps, it becomes increasingly diffuse. Rahman compared this evolution to theoretical predictions (the Vineyard approximation) and found that theory predicted overly rapid structural decay. My results confirm this finding.

System Validation

Before analyzing physics, basic sanity checks confirmed proper thermal equilibrium.

Mean temperature was 94.73 K (0.33 K off target) with a standard deviation of 1.56 K.

The velocity distribution from 12.9 million velocity components produces a clean Maxwell-Boltzmann distribution, as expected for thermal equilibrium. The distribution widths at various heights closely match Rahman’s results: 1.77, 2.48, and 3.56 compared to his 1.77, 2.52, and 3.52.

Conclusion

Replicating a 60-year-old paper might seem like a solved puzzle, but it teaches a valuable lesson in computational science. Rahman relied on brilliance and raw mathematical intuition because he lacked compute power. Today, pairing modern compute with disciplined software practices makes the same result reproducible and auditable.

Applying modern software engineering (modular architecture, caching, and automated workflows) to classical physics reproduces the past and builds a foundation that makes the next discovery easier, faster, and more reliable.

The quantitative agreement is striking: diffusion coefficients within 2%, structural peaks within 0.1 Å, velocity distributions matching to three significant figures. This level of reproducibility, achieved with completely different hardware and software, validates something fundamental: Rahman’s physical model was remarkably sound, and his computational methodology was scientifically rigorous despite 1960s constraints.

The cage effect, velocity correlations, and structural evolution are fundamental characteristics of how matter behaves at the atomic scale, as relevant today as they were six decades ago.

This project is a “digital restoration” of Aneesur Rahman’s seminal 1964 paper, Correlations in the Motion of Atoms in Liquid Argon. While the physics of liquid argon is a solved problem, the challenge lies in bridging the gap between 1960s mainframe constraints and 2025 software architecture.

I replicated the simulation using LAMMPS and built a Python analysis pipeline to process the trajectory data. The project demonstrates how modern tooling (uv, type hinting, vectorized NumPy) can transform academic “write-once” scripts into a reproducible research toolkit.

Features

The Analysis Pipeline

I architected a modular Python package (argon_sim) designed for performance and maintainability.

• Intelligent Caching System: MD analysis is compute-intensive ($O(N^2)$). I implemented a decorator-based caching layer (@cached_computation) that hashes source file modification times and function arguments. This ensures expensive calculations (like RDF or Van Hove correlations) are only re-run when the underlying trajectory or parameters actually change.
• Vectorization & Optimization: To handle the $N^2$ complexity of pair-wise interactions without C++ extensions, I utilized NumPy broadcasting. For example, the Mean Square Displacement (MSD) calculation is fully vectorized, with a fallback “chunked” implementation to handle memory overflows on smaller machines.
• Modern Python Tooling:
  • Dependency Management: Used uv for deterministic environment locking (sub-second resolution).
  • Type Safety: Fully type-hinted codebase for static analysis compliance.
  • Automation: A Makefile abstracts the workflow (simulation → analysis → figure generation) into single commands (e.g., make figure-5).

The Simulation Strategy

I used LAMMPS for the MD engine but strictly adhered to Rahman’s physical parameters while modernizing the stability mechanisms.

• Integration: Replaced Rahman’s predictor-corrector method with the modern standard Velocity Verlet algorithm (2 fs timestep).
• Equilibration: I implemented a 1 ns NVT equilibration phase (500,000 steps at the 2 fs timestep) to properly melt the FCC crystal structure before the NVE production run.
• Intellectual Honesty: The in.argon script explicitly documents every deviation from the original methodology (e.g., energy minimization) and the justification for ensuring numerical stability.

Usage

The project uses a Makefile to automate the workflow. Run make all to execute the LAMMPS simulation and generate all analysis figures.

Results

The replication achieved high quantitative agreement with the historical data, validating both the simulation parameters and the custom analysis code.

Visual Replication

I used Matplotlib to digitally recreate Rahman’s hand-drawn plots, confirming signatures like the negative region in the Velocity Autocorrelation Function (VACF), which provided the first evidence of the “cage effect” in simple liquids.

Challenges & Learnings

• Unit Hell: Rahman’s paper uses a mix of reduced units and CGS. Mapping these to LAMMPS’s real units required a dedicated constants.py module and rigorous unit testing to prevent dimensional errors.
• Fourier Transforms: Calculating the Structure Factor $S(k)$ from $g(r)$ required implementing a manual 3D Fourier transform for spherical symmetry, as standard FFT packages do not account for the radial shell integration implicit in liquid structure analysis.
• Code as a Liability: Early in the project, I realized that re-running analysis scripts was becoming a bottleneck. This drove the decision to build the caching infrastructure, reinforcing the lesson that investing in developer tooling pays off even in small-scale scientific projects.

Related Work

The full methodology and physics are documented in the companion blog post:

• Replicating Rahman’s 1964 Liquid Argon Simulation

In computational drug discovery, data scarcity is often the bottleneck. This project builds a synthetic data generator that creates labeled 3D molecular datasets starting from nothing but a raw chemical formula (e.g., $C_6H_{14}$).

The pipeline bridges the gap between 1D Chemical Information (stoichiometry) and 3D Geometric Data (conformers), effectively serving as a “data factory” for training molecular machine learning models.

Features

1. Graph Enumeration & 3D Embedding

The core of the project is pysomer/data/gen.py, which orchestrates a multi-step generation process:

• Structural Isomerism: Uses MAYGEN (via a Java bridge) to mathematically enumerate all valid graph connectivities for a given formula
• Conformer Sampling: Uses RDKit to embed these graphs into 3D space, generating multiple conformers (rotamers) per isomer to capture flexibility
• IUPAC Labeling: Automatically queries PubChem APIs to assign human-readable labels (e.g., “2-methylpentane”) to the generated structures

2. Physics-Aware Featurization

The pipeline computes Coulomb Matrices, ensuring the input respects physical invariants:

$$C_{ij} = \begin{cases} 0.5 Z_i^{2.4} & i = j \ \frac{Z_i Z_j}{|R_i - R_j|} & i \neq j \end{cases}$$

This representation encodes the electrostatic potential of the molecule, providing a more informative signal for the neural network than raw Cartesian coordinates.

3. HDF5 Data Storage

To handle the large volume of generated conformers, the system writes to hierarchical HDF5 files. This allows for efficient, chunked I/O during training, a critical pattern for scaling to larger chemical spaces.

Usage

The pipeline is executed via a CLI, taking a chemical formula as input and outputting an HDF5 dataset of 3D conformers.

Results

This project serves as a “vertical slice” of a cheminformatics workflow.

• The Good: The separation of concerns is clean: dataclasses for configuration and HDF5 for storage keep the data-engineering layer tidy and extensible.
• The “Old School”: The model used is a simple Multi-Layer Perceptron (MLP) on flattened Coulomb Matrices. In a modern production setting (post-2020), I would replace this with an E(3)-Equivariant GNN (like SchNet or E3NN) to handle rotational symmetry natively, eliminating manual feature engineering.
• Dependency Management: The reliance on an external Java JAR (MAYGEN) for graph enumeration makes the environment brittle. Today, I would likely swap this for a pure Python enumerator or a containerized microservice to improve portability.

Related Work

This data pipeline powers the analysis in my comprehensive guide on molecular representation:

• Coulomb Matrix Eigenvalues: Can You Hear the Shape of a Molecule?: A deep dive into data generation, unsupervised clustering, and supervised classification of alkane isomers.

See also:

• The Coulomb Matrix: Deep dive into the physics-based featurization used here
• The Number of Isomeric Hydrocarbons: The foundational 1931 paper on alkane enumeration

When working with machine learning in chemistry, one of the first challenges you encounter is how to represent molecules in a way that algorithms can understand. You can’t just feed raw atomic coordinates into a model. The representation needs to be invariant to rotation, translation, and atom ordering, since these operations don’t change the molecule’s fundamental properties.

The Coulomb matrix, introduced by Rupp et al. in 2012 [1], provides a straightforward solution to this problem. While newer methods have largely superseded it for practical applications, the Coulomb matrix remains an excellent starting point for understanding how molecular descriptors work.

The key insight is simple: we encode pairwise relationships between atoms in a way that captures the essential physics while maintaining the required invariances.

The Coulomb Matrix: Theory and Intuition

The Coulomb matrix encodes molecular structure in a symmetric $N \times N$ matrix, where $N$ is the number of atoms. Each element $C_{ij}$ is defined as:

$$ C_{ij} = \begin{cases} 0.5 Z_i^{2.4} & \text{if } i = j, \\ \frac{Z_i Z_j}{|\mathbf{R}_i - \mathbf{R}_j|} & \text{if } i \neq j, \end{cases} $$

Here, $Z_i$ is the atomic number of atom $i$, and $\mathbf{R}_i$ is its position in 3D space. The diagonal elements ($0.5 Z_i^{2.4}$) represent atomic self-energies, derived from fitting atomic numbers to experimental data. The off-diagonal elements mimic Coulombic interactions between atoms. They’re inversely proportional to distance, just like electrostatic potential energy [3].

This construction gives us several useful properties:

• Rotation and translation invariant: Only relative distances matter
• Symmetric: $C_{ij} = C_{ji}$, which is physically sensible
• Size-extensive: Larger molecules have larger matrix elements
• Captures 3D structure: Nearby atoms have larger interaction terms

While more sophisticated methods exist today [2], the Coulomb matrix’s simplicity makes it ideal for understanding the fundamentals of molecular representation.

Hands-on Example: Bicyclobutane

Let’s calculate the Coulomb matrix for bicyclobutane, a strained but stable bicyclic system (bicyclo[1.1.0]butane, C4H6, two cis-fused cyclopropane rings). This example will show you exactly how the theory translates to practice.

I’ll use Python with the Atomic Simulation Environment (ase) for molecular structure [4] and dscribe for the Coulomb matrix calculation [2]:

from ase.build import molecule
from ase.visualize import view

# Load the bicyclobutane structure
bicyclobutane = molecule('bicyclobutane')

# Optional: visualize the structure
view(bicyclobutane, viewer='x3d')

Now we calculate the Coulomb matrix using DScribe:

from dscribe.descriptors import CoulombMatrix

# Set up the descriptor
cm = CoulombMatrix(n_atoms_max=len(bicyclobutane))

# Calculate and reshape into matrix form
cm_bicyclobutane = cm.create(bicyclobutane)
cm_bicyclobutane = cm_bicyclobutane.reshape(len(bicyclobutane), len(bicyclobutane))

Visualizing the Results

The Coulomb matrix can be visualized as a heatmap. Let’s look at both the raw matrix and its logarithm:

import matplotlib.pyplot as plt
import numpy as np

# Raw Coulomb matrix
plt.figure(figsize=(8, 8), dpi=150)
plt.imshow(cm_bicyclobutane, cmap='coolwarm')
plt.colorbar(label='Magnitude')
plt.title('Coulomb Matrix for Bicyclobutane')
plt.show()

The raw matrix shows clear patterns:

• Large diagonal elements: Carbon atoms (Z=6) dominate due to their higher atomic numbers
• Smaller off-diagonal elements: Represent pairwise interactions
• Minimal hydrogen contribution: Hydrogen atoms (Z=1) have much smaller values

For better visualization of the structure, the logarithm reveals more detail:

plt.figure(figsize=(8, 8), dpi=150)
plt.imshow(np.log(cm_bicyclobutane), cmap='coolwarm')
plt.colorbar(label='log(Magnitude)')
plt.title('Log Coulomb Matrix for Bicyclobutane')
plt.show()

Eigenvalue Analysis

The eigenvalues of the Coulomb matrix provide another perspective on molecular structure:

These eigenvalues are often used as features themselves, providing a more compact representation than the full matrix.

Practical Limitations

The Coulomb matrix has significant limitations that explain why it’s been largely superseded by modern methods. Understanding these constraints is crucial for knowing when and how to use this descriptor.

The Size Problem

Every molecule must be represented by the same size matrix, which creates several issues:

• Padding overhead: Small molecules get padded with zeros up to the maximum size
• Quadratic scaling: An $N$-atom molecule requires $N^2$ features
• Fixed maximum size: You can’t represent molecules larger than your preset limit
• Inefficient storage: Most elements are zero for small molecules in large matrices

For a dataset ranging from 5-atom to 50-atom molecules, every molecule needs a 50x50 matrix. That’s 2,500 features, most of which are zero for smaller molecules.

Permutation Sensitivity

Despite being called “invariant,” the Coulomb matrix can actually change if you reorder the atoms in your input file. The standard solution is to sort atoms by the L2 norm of their matrix rows, but this introduces its own problems:

• Symmetry breaking: Equivalent atoms might be ordered differently
• Numerical instability: Small coordinate changes can flip the ordering
• Loss of chemical intuition: The sorted order doesn’t reflect meaningful chemistry

Interestingly, some studies suggest that adding controlled noise to create multiple permutations can actually improve machine learning performance [5].

Limited Scope

The Coulomb matrix works well only for specific types of systems:

• Small molecules: Performance degrades for large systems due to size scaling
• Gas-phase: Not suitable for periodic systems like crystals or surfaces
• Single conformations: Each 3D structure gets its own matrix
• Non-reactive: Doesn’t capture bond-breaking or formation

For periodic systems, you’d need specialized variants like the Ewald sum matrix [6].

Why Learn It Anyway?

Given these limitations, why spend time understanding the Coulomb matrix? Several reasons:

Educational value: It’s conceptually straightforward and provides excellent intuition for how molecular descriptors work. The mathematical formulation is simple enough to implement from scratch.

Historical importance: Many subsequent methods build on ideas first explored with Coulomb matrices. Understanding this foundation helps you appreciate why newer methods were developed.

Benchmarking: It remains useful as a baseline method for comparing new descriptors on small molecular datasets.

Proof of concept: For exploratory work on small, well-defined datasets, the Coulomb matrix can still provide quick insights.

If you’re working on practical problems with larger datasets or diverse molecular sizes, consider modern alternatives like graph neural networks, descriptors from DScribe’s extended library, or learned representations from transformer models.

Putting It in Context

To see the Coulomb matrix applied to real problems, I’ve written a detailed guide using it for molecular classification:

• Coulomb Matrix Eigenvalues: Can You Hear the Shape of a Molecule?: A comprehensive analysis of alkane isomers, from unsupervised clustering limits to supervised classification successes.

For comparison with modern approaches, check out my post on 3D conformer generation with the GEOM dataset, which showcases more sophisticated molecular representations. For technical specifications and benchmarks, see the GEOM dataset card.

The Coulomb matrix may be dated, but it remains an excellent entry point into the world of molecular machine learning. Once you understand its strengths and limitations, you’ll be better equipped to appreciate why the field has moved toward more sophisticated approaches.

Have questions about molecular descriptors or want to discuss other approaches to molecular machine learning? I’d be happy to explore these topics further.

References

• [1]: M. Rupp, A. Tkatchenko, K.-R. Müller, and O. A. von Lilienfeld, “Fast and Accurate Modeling of Molecular Atomization Energies with Machine Learning,” Physical Review Letters, 108(5), 058301 (2012). https://doi.org/10.1103/PhysRevLett.108.058301 arXiv:1109.2618
• [2] L. Himanen, M. O. J. Jäger, E. V. Morooka, F. F. Canova, Y. S. Ranawat, D. Z. Gao, P. Rinke, and A. S. Foster, “DScribe: Library of descriptors for machine learning in materials science,” Computer Physics Communications, 247, 106949 (2020). https://doi.org/10.1016/j.cpc.2019.106949 arXiv:1904.08875
• [3] J. Schrier, “Can one hear the shape of a molecule (from its Coulomb matrix eigenvalues)?,” Journal of Chemical Information and Modeling, 60(8), 3804-3811 (2020). https://doi.org/10.1021/acs.jcim.0c00631
• [4] A. H. Larsen, J. J. Mortensen, J. Blomqvist, I. E. Castelli, R. Christensen, M. Dułak, J. Friis, M. N. Groves, B. Hammer, C. Hargus, E. D. Hermes, P. C. Jennings, P. B. Jensen, J. Kermode, J. R. Kitchin, E. L. Kolsbjerg, J. Kubal, K. Kaasbjerg, S. Lysgaard, J. B. Maronsson, T. Maxson, T. Olsen, L. Pastewka, A. Peterson, C. Rostgaard, J. Schiøtz, O. Schütt, M. Strange, K. S. Thygesen, T. Vegge, L. Vilhelmsen, M. Walter, Z. Zeng, and K. W. Jacobsen, “The Atomic Simulation Environment - A Python library for working with atoms,” J. Phys.: Condens. Matter, 29, 273002 (2017). https://doi.org/10.1088/1361-648X/aa680e documentation
• [5] G. Montavon, K. Hansen, S. Fazli, M. Rupp, F. Biegler, A. Ziehe, A. Tkatchenko, A. Lilienfeld, and K.-R. Müller, “Learning invariant representations of molecules for atomization energy prediction,” Advances in Neural Information Processing Systems, 25 (2012). Available online: https://proceedings.neurips.cc/paper_files/paper/2012/file/115f89503138416a242f40fb7d7f338e-Paper.pdf
• [6] F. Faber, A. Lindmaa, O. A. von Lilienfeld, and R. Armiento, “Crystal structure representations for machine learning models of formation energies,” International Journal of Quantum Chemistry, 115(16), 1094-1101 (2015). https://doi.org/10.1002/qua.24917

Analyzing congressional data reveals the underlying mechanics of the legislative process. Legislative text is a large, structured corpus well suited to text classification and other NLP tasks. I scraped data from Congress.gov to analyze what actually happens to the thousands of bills introduced each session and to build a foundational dataset for downstream machine learning tasks.

This analysis focuses on the 117th Congress (2021-2023), examining 15,000+ bills to understand basic patterns: Which bills get introduced? How many receive votes? What factors influence success?

This post covers the foundational exploratory analysis and data collection process, setting the stage for predictive modeling and policy area classification.

Data Collection

My primary source is Congress.gov, maintained by the Library of Congress. I focused on the 117th Congress (2021-2023), collecting data on bills and joint resolutions, omitting simple resolutions, concurrent resolutions, and amendments.

Data collected:

Technical Implementation

Building a usable NLP dataset requires careful handling of the source. Congress.gov loads content dynamically and presents nested DOM structures, so the scraper combines static HTML parsing with a headless browser to render JavaScript before parsing.

Implementation details:

• Python for core orchestration and data schema management
• Selenium for executing JavaScript and loading dynamic page elements
• BeautifulSoup for structured HTML parsing
• Regex for text normalization and extracting clean legislative text for language models

The crawler used 5-second delays between requests to respect server limits, a roughly 3-day collection run. It handles edge cases in congressional text formatting and writes one JSON record per bill on a fixed schema. The crawler and processed data are available on GitHub.

For each bill, I queried two pages:

• All info page: https://www.congress.gov/bill/117th-congress/{bill_type}/{bill_id}/all-info
• Text page: https://www.congress.gov/bill/117th-congress/{bill_type}/{bill_id}/text?format=txt

The parsing process involved targeting specific HTML elements and implementing basic caching to avoid redundant requests.

Key Findings

The analysis reveals clear patterns in congressional activity. Most bills never receive votes, and success rates vary significantly by party and policy area.

Legislative Outcomes

The fundamental question: what happens to bills after introduction?

Each bill has a tracker status indicating its position in the legislative process. The eight possible statuses can be grouped into three meaningful categories:

• Introduced: Bills introduced but never voted on
• Stalled: Bills that saw votes but didn’t become law (since the 117th Congress ended, these effectively died)
• Law: Bills signed by the President

Key insights:

• Only 7% of introduced bills ever receive a vote
• Of bills that receive votes, 36% become law
• Overall, just 2% of introduced bills become law

Sponsor Analysis

The bill sponsor (the primary member who introduces legislation) provides insights into party and geographic patterns.

Party Breakdown

Party comparison:

• Democrats: 7.5% of bills moved beyond introduction; 2.6% became law
• Republicans: 5.5% of bills moved beyond introduction; 2.1% became law
• When bills do advance, Republicans have a slightly higher success rate (38% vs 35%)

Geographic Distribution

Top 10 states by bills introduced:

Per-representative normalization reveals different patterns:

Top Individual Sponsors

Most prolific legislators by bills introduced:

Effectiveness score (laws enacted / total bills):

$$ \text{effectiveness} = \frac{\text{bills that became law}}{\text{total bills introduced}} $$

Policy Focus Areas

Each bill is assigned a primary policy area. Here are the most active areas by legislative outcome:

Notable patterns: Health dominates introductions but has lower success rates, while government operations and armed forces bills are more likely to become law.

Next Steps

This analysis establishes baseline patterns: most bills fail, party affiliation affects success rates, and certain policy areas perform better than others.

Future work could explore:

• Committee dynamics and voting patterns
• Geographic analysis of state-level interests
• Bill text analysis using NLP techniques
• Predictive modeling for bill outcomes

Update: I’ve since applied machine learning to this type of data in Congressional Bill Policy Area Classification, using 48K+ bills from three Congresses to automatically categorize bills by policy area.

The complete dataset and code are publicly available to support further research into legislative transparency.

In computer vision or scientific computing, a common problem frequently arises: given two sets of points, what is the optimal rigid body transformation for their alignment? The Kabsch algorithm provides a nice solution.

What are some concrete situations where this crops up?

• Molecular Dynamics: Your points are a set of atoms (with physically relevant types), and you want to compare two molecular conformations. Are they the same structure with minor noise or rotation? Or are they different conformations, like a different folding of a protein? This is especially helpful when applying generative models to chemical structures. For example, if you are building a 3D Molecular VAE in PyTorch or working with Flow Matching models, Kabsch alignment ensures your generative loss function remains rotationally invariant.
• Computer Vision: You have two point clouds from 3D scans of an object taken from different angles. You want to align them to reconstruct the full shape. Or perhaps you’re generating 3D shapes from 2D images and need to compare the generated shape to a ground truth scan. Anytime a 3D system is represented as a point cloud, the Kabsch algorithm can help with alignment.

Of course, existing libraries implement this algorithm. However, often I find it beneficial to implement algorithms from scratch to build intuition. Furthermore, modern machine learning applications require automatic differentiation, so we will implement the algorithm in PyTorch, TensorFlow, and JAX.

Below, we’ll cover the math behind the Kabsch algorithm (and its scaling variant, the Kabsch-Umeyama algorithm) and provide complete, differentiable implementations in NumPy, PyTorch, TensorFlow, and JAX, demonstrating both single-pair and batched computations for ML applications.

The Math

Let’s say we have two sets of paired points, $P={\mathbf{p}_i} \in \mathbb{R}^{N \times D}$ and $Q={\mathbf{q}_i} \in \mathbb{R}^{N \times D}$, for $i = 1, \dots, N$ (where $D$ is the dimensionality and $N$ is the number of points). We want to find a translation vector $\mathbf{t}$ and rotation matrix $R$ to transform $P$ to align with $Q$.

The optimization problem is:

$$ \min_{\mathbf{t}, \ R} \mathcal{L}(\mathbf{t}, R) = \frac{1}{2} \sum_{i=1}^N | \mathbf{q}_i - (R\mathbf{p}_i + \mathbf{t}) |^2 $$

where $\mathbf{t}^\ast \in \mathbb{R}^D$ and $R^\ast \in \mathbb{R}^{D \times D}$ are the optimal translation and rotation.

Often we use a weighted version with weights $w_i$ (e.g., atomic masses in molecular dynamics):

$$ \min_{\mathbf{t}, \ R} \mathcal{L}(\mathbf{t}, R) = \frac{1}{2} \sum_{i=1}^N w_i | \mathbf{q}_i - (R\mathbf{p}_i + \mathbf{t}) |^2 $$

The Translation

The translation and rotation are coupled, but they separate cleanly once we work in centroid-centered coordinates. Compute the centroids (averages) of both point sets:

$$ \bar{\mathbf{p}} = \frac{1}{N} \sum_{i=1}^N \mathbf{p}_i \quad \text{and} \quad \bar{\mathbf{q}} = \frac{1}{N} \sum_{i=1}^N \mathbf{q}_i $$

For any fixed rotation $R$, the translation that minimizes $\mathcal{L}$ is found by setting $\partial \mathcal{L} / \partial \mathbf{t} = 0$. It maps the rotated source centroid onto the target centroid:

$$ \mathbf{t} = \bar{\mathbf{q}} - R\bar{\mathbf{p}} $$

A tempting shortcut is to write $\mathbf{t} = \bar{\mathbf{q}} - \bar{\mathbf{p}}$, but that is only correct when $R = I$. In general the translation depends on the rotation, so we compute it after solving for $R$. Substituting this optimal $\mathbf{t}$ back into the objective cancels the centroids and leaves a rotation-only problem in the centered coordinates $\mathbf{p}_i^\prime = \mathbf{p}_i - \bar{\mathbf{p}}$ and $\mathbf{q}_i^\prime = \mathbf{q}_i - \bar{\mathbf{q}}$:

$$ \mathcal{L}(R) = \frac{1}{2} \sum_{i=1}^N | \mathbf{q}_i^\prime - R\mathbf{p}_i^\prime |^2 $$

which is what the next section solves.

The Rotation Matrix

We now minimize $\mathcal{L}(R)$ over rotations, using the centered points $\mathbf{p}_i^\prime$ and $\mathbf{q}_i^\prime$ from above. Compute the cross-covariance matrix between the centered sets:

$$ C = P^{\prime T} Q^\prime = \sum_{i=1}^N \mathbf{p}_i^{\prime T} \mathbf{q}_i^{\prime} \in \mathbb{R}^{D \times D} $$

This is a fairly lightweight operation since $D$ is typically small (e.g., 3 for 3D points), even if $N$ is large.

With $C$ in hand, we want to compute its Singular Value Decomposition (SVD):

$$ C = U \Sigma V^T $$

This operation is computationally expensive. It scales cubically with $D$ (i.e., $O(D^3)$). However, since we’re often interested in cases where $D$ is small (e.g., 2D or 3D points), this is manageable.

Next, we check for improper rotations (i.e., reflections) and correct for them where necessary:

$$ d = \text{sign}(\det(V U^T)) $$

If $d = -1$, we need to flip the last column of $V$ in the final rotation matrix.

Let $B = \text{diag}(1, 1, d)$. The optimal rotation matrix comes out:

$$ R^\ast = V B U^T $$

Summary

In a nutshell, the Kabsch algorithm boils down to:

1. Compute centroids of $P$ and $Q$ ($\bar{\mathbf{p}}$ and $\bar{\mathbf{q}}$)
2. Center both point sets by subtracting centroids: $P^\prime$ and $Q^\prime$
3. Compute cross-covariance matrix $C = P^{\prime T} Q^\prime$
4. Compute SVD: $C = U \Sigma V^T$ (expensive step)
5. Compute $d = \text{sign}(\det(V U^T))$ and $B = \text{diag}(1, 1, d)$
6. Optimal rotation: $R^\ast = V B U^T$
7. Optimal translation (using the rotation from step 6): $\mathbf{t}^\ast = \bar{\mathbf{q}} - R^\ast\bar{\mathbf{p}}$

The resulting root-mean-square deviation (RMSD) between aligned point sets is

$$ \text{RMSD} = \sqrt{\frac{1}{N} \sum_{i=1}^N | \mathbf{q}_i - (R^\ast\mathbf{p}_i + \mathbf{t}^\ast) |^2} $$

which is frequently used as a measure of similarity between molecular structures or as a metric in loss functions for ML applications.

The Kabsch-Umeyama Algorithm (Scaling)

While the standard Kabsch algorithm solves for optimal rotation and translation, the Kabsch-Umeyama algorithm extends this by also finding an optimal scaling factor $c$. This is essential when aligning structures of different scales, such as a 3D scan versus a ground truth model.

(Note: This is sometimes searched for as the “Absch-Umeyama algorithm” due to typos, but the correct attribution is to Shinji Umeyama based on Wolfgang Kabsch’s work.)

The method estimates the transformation $\mathbf{q}_i \approx c R \mathbf{p}_i + \mathbf{t}$. The optimal scale is the trace of the (reflection-corrected) singular values of the cross-covariance divided by the variance of the source points about their centroid. See the Umeyama paper notes for the full derivation.

A Note on SVD and Automatic Differentiation

While modern frameworks allow us to backpropagate through the Singular Value Decomposition (SVD), it comes with a known stability issue: if the cross-covariance matrix has identical (degenerate) singular values (which can occur if the point clouds are perfectly aligned or have certain symmetries), the gradient of the SVD approaches infinity, causing NaN values during backpropagation. If you plan to use this algorithm as a loss function for a neural network, it is often necessary to add a tiny epsilon to the matrix before computing the SVD, or to utilize an SVD gradient patch. The Kabsch-Horn Cookbook library provides a SafeSVD primitive that floors the singular-value-gap denominator at machine epsilon in the backward pass, producing finite gradients at degenerate inputs across PyTorch, JAX, TensorFlow, and MLX.

Implementation

Let’s implement the algorithm in different frameworks. Note that for simplicity, the following implementations cover the unweighted Kabsch algorithm. If your application (like molecular dynamics) requires weights (e.g., atomic masses), the Kabsch-Horn Cookbook library provides per-point weighted alignment out of the box.

NumPy

import numpy as np


def kabsch_numpy(P, Q):
    """
    Computes the optimal rotation and translation to align two sets of points (P -> Q),
    and their RMSD.

    :param P: A Nx3 matrix of points
    :param Q: A Nx3 matrix of points
    :return: A tuple containing the optimal rotation matrix, the optimal
             translation vector, and the RMSD.
    """
    assert P.shape == Q.shape, "Matrix dimensions must match"

    # Compute centroids
    centroid_P = np.mean(P, axis=0)
    centroid_Q = np.mean(Q, axis=0)

    # Center the points
    p = P - centroid_P
    q = Q - centroid_Q

    # Compute the covariance matrix
    H = np.dot(p.T, q)

    # SVD
    U, S, Vt = np.linalg.svd(H)

    # Validate right-handed coordinate system
    if np.linalg.det(np.dot(Vt.T, U.T)) < 0.0:
        Vt[-1, :] *= -1.0

    # Optimal rotation
    R = np.dot(Vt.T, U.T)

    # Optimal translation (depends on R, so computed after it)
    t = centroid_Q - np.dot(R, centroid_P)

    # RMSD
    rmsd = np.sqrt(np.sum(np.square(np.dot(p, R.T) - q)) / P.shape[0])

    return R, t, rmsd

Here’s a quick test to verify correctness:

def test_numpy():
    np.random.seed(12345)

    P = np.random.randn(100, 3)

    alpha = np.random.rand() * 2 * np.pi
    R = np.array([[np.cos(alpha), -np.sin(alpha), 0],
                    [np.sin(alpha), np.cos(alpha), 0],
                    [0, 0, 1]])
    t = np.random.randn(3) * 10

    Q = np.dot(P, R.T) + t

    R_opt, t_opt, rmsd = kabsch_numpy(P, Q)

    print('RMSD: {}'.format(rmsd))
    print('R:\n{}'.format(R))
    print('R_opt:\n{}'.format(R_opt))
    print('t:\n{}'.format(t))
    print('t_opt:\n{}'.format(t_opt))

    l2_t = np.linalg.norm(t - t_opt)
    l2_R = np.linalg.norm(R - R_opt)
    print('l2_t: {}'.format(l2_t))
    print('l2_R: {}'.format(l2_R))

Running this test shows the algorithm correctly recovers the rotation and translation:

RMSD: 3.2111501877699246e-15
R:
[[-0.8475392 -0.5307328  0.       ]
 [ 0.5307328 -0.8475392  0.       ]
 [ 0.         0.         1.       ]]
R_opt:
[[-8.47539198e-01 -5.30732803e-01 -2.95434260e-16]
 [ 5.30732803e-01 -8.47539198e-01  2.92859649e-16]
 [ 0.00000000e+00 -2.77555756e-16  1.00000000e+00]]
t:
[ 5.99726796  1.50078468 -3.34633977]
t_opt:
[ 5.99726796  1.50078468 -3.34633977]
l2_t: 2.7012892057857038e-15
l2_R: 8.028174304721057e-16

Both the rotation and the translation are recovered to within floating-point precision (the residuals l2_t and l2_R are on the order of 1e-15).

For batch processing:

def kabsch_numpy_batched(P, Q):
    """
    Computes the optimal rotation and translation to align two sets of points (P -> Q),
    and their RMSD.

    :param P: A BxNx3 matrix of points
    :param Q: A BxNx3 matrix of points
    :return: A tuple containing the optimal rotation matrix, the optimal
             translation vector, and the RMSD.
    """
    assert P.shape == Q.shape, "Matrix dimensions must match"

    # Compute centroids
    centroid_P = np.mean(P, axis=1, keepdims=True)  # Bx1x3
    centroid_Q = np.mean(Q, axis=1, keepdims=True)  # Bx1x3

    # Center the points
    p = P - centroid_P  # BxNx3
    q = Q - centroid_Q  # BxNx3

    # Compute the covariance matrix
    H = np.matmul(p.transpose(0, 2, 1), q)  # Bx3x3

    # SVD
    U, S, Vt = np.linalg.svd(H)  # Bx3x3

    # Validate right-handed coordinate system
    d = np.linalg.det(np.matmul(Vt.transpose(0, 2, 1), U.transpose(0, 2, 1)))
    flip = d < 0.0
    if flip.any():
        Vt[flip, -1, :] *= -1.0

    # Optimal rotation
    R = np.matmul(Vt.transpose(0, 2, 1), U.transpose(0, 2, 1))  # Bx3x3

    # Optimal translation (depends on R, so computed after it)
    t = centroid_Q.squeeze(1) - np.matmul(centroid_P, R.transpose(0, 2, 1)).squeeze(1)  # Bx3

    # RMSD
    rmsd = np.sqrt(np.sum(np.square(np.matmul(p, R.transpose(0, 2, 1)) - q), axis=(1, 2)) / P.shape[1])

    return R, t, rmsd

PyTorch

The PyTorch implementation now uses broadcasting to ensure differentiability:

import torch


def kabsch_torch(P, Q):
    """
    Computes the optimal rotation and translation to align two sets of points (P -> Q),
    and their RMSD.
    :param P: A Nx3 matrix of points
    :param Q: A Nx3 matrix of points
    :return: A tuple containing the optimal rotation matrix, the optimal
             translation vector, and the RMSD.
    """
    assert P.shape == Q.shape, "Matrix dimensions must match"

    # Compute centroids
    centroid_P = torch.mean(P, dim=0)
    centroid_Q = torch.mean(Q, dim=0)

    # Center the points
    p = P - centroid_P
    q = Q - centroid_Q

    # Compute the covariance matrix
    H = torch.matmul(p.transpose(0, 1), q)

    # SVD
    U, S, Vt = torch.linalg.svd(H)

    # 1. Calculate determinant
    d = torch.det(torch.matmul(Vt.transpose(0, 1), U.transpose(0, 1)))

    # 2. Build diagonal B tensor without in-place mutation
    # We use stack to preserve gradients and graph connections
    B_diag = torch.stack([torch.tensor(1.0, device=d.device, dtype=d.dtype),
                          torch.tensor(1.0, device=d.device, dtype=d.dtype),
                          torch.sign(d)])

    # 3. Scale columns of Vt.T via broadcasting, then multiply by U^T
    # Vt.T: (3, 3). B_diag: (3) -> B_diag[None, :]: (1, 3)
    R = torch.matmul(Vt.transpose(0, 1) * B_diag[None, :], U.transpose(0, 1))

    # Optimal translation (depends on R, so computed after it)
    t = centroid_Q - centroid_P @ R.transpose(0, 1)

    # RMSD
    rmsd = torch.sqrt(torch.sum(torch.square(torch.matmul(p, R.transpose(0, 1)) - q)) / P.shape[0])

    return R, t, rmsd

And our batched version:

def kabsch_torch_batched(P, Q):
    """
    Computes the optimal rotation and translation to align two sets of points (P -> Q),
    and their RMSD, in a batched manner.
    :param P: A BxNx3 matrix of points
    :param Q: A BxNx3 matrix of points
    :return: A tuple containing the optimal rotation matrix, the optimal
             translation vector, and the RMSD.
    """
    assert P.shape == Q.shape, "Matrix dimensions must match"

    # Compute centroids
    centroid_P = torch.mean(P, dim=1, keepdims=True)  # Bx1x3
    centroid_Q = torch.mean(Q, dim=1, keepdims=True)  # Bx1x3

    # Center the points
    p = P - centroid_P  # BxNx3
    q = Q - centroid_Q  # BxNx3

    # Compute the covariance matrix
    H = torch.matmul(p.transpose(1, 2), q)  # Bx3x3

    # SVD
    U, S, Vt = torch.linalg.svd(H)  # Bx3x3

    # 1. Calculate batched determinant
    d = torch.det(torch.matmul(Vt.transpose(1, 2), U.transpose(1, 2)))  # B

    # 2. Build batched B_diag without in-place mutation or control flow
    ones = torch.ones_like(d)
    B_diag = torch.stack([ones, ones, torch.sign(d)], dim=-1) # Bx3

    # 3. Scale columns of Vt.T and multiply
    # Vt.T: (B, 3, 3). B_diag: (B, 3). B_diag[:, None, :]: (B, 1, 3).
    R = torch.matmul(Vt.transpose(1, 2) * B_diag[:, None, :], U.transpose(1, 2))

    # Optimal translation (depends on R, so computed after it)
    t = centroid_Q.squeeze(1) - torch.matmul(centroid_P, R.transpose(1, 2)).squeeze(1)  # Bx3

    # RMSD
    rmsd = torch.sqrt(torch.sum(torch.square(torch.matmul(p, R.transpose(1, 2)) - q), dim=(1, 2)) / P.shape[1])

    return R, t, rmsd

TensorFlow

The TensorFlow implementation returns S, U, and V directly. To handle immutability and potential compilation (e.g., via @tf.function), we avoid explicit conditional branching by constructing a correction matrix $B$ and broadcasting it.

import tensorflow as tf


def kabsch_tensorflow(P, Q):
    """
    Computes the optimal rotation and translation to align two sets of points (P -> Q),
    and their RMSD.

    :param P: A Nx3 matrix of points
    :param Q: A Nx3 matrix of points
    :return: A tuple containing the optimal rotation matrix, the optimal
             translation vector, and the RMSD.
    """
    P = tf.convert_to_tensor(P, dtype=tf.float32)
    Q = tf.convert_to_tensor(Q, dtype=tf.float32)

    assert P.shape == Q.shape, "Matrix dimensions must match"

    # Compute centroids
    centroid_P = tf.reduce_mean(P, axis=0)
    centroid_Q = tf.reduce_mean(Q, axis=0)

    # Center the points
    p = P - centroid_P
    q = Q - centroid_Q

    # Compute the covariance matrix
    H = tf.matmul(tf.transpose(p), q)

    # SVD
    S, U, V = tf.linalg.svd(H)

    # 1. Calculate determinant
    # Note: V in TF SVD is V, not V^T.
    # R = V * U^T. Det(R) = Det(V * U^T)
    d = tf.linalg.det(tf.matmul(V, tf.transpose(U)))

    # 2. Build diagonal B tensor: [1.0, 1.0, sign(d)]
    # Use static shape 3 if possible, or infer from D. Assuming D=3 here.
    B_diag = tf.stack([1.0, 1.0, tf.sign(d)])

    # 3. Scale columns of V via broadcasting (V * B_diag), then multiply by U^T
    # V is DxD, B_diag is D. V * B_diag[None, :] multiplies each column j by B_diag[j]
    R = tf.matmul(V * B_diag[None, :], tf.transpose(U))

    # Optimal translation (depends on R, so computed after it)
    t = centroid_Q - tf.linalg.matvec(R, centroid_P)

    # RMSD
    rmsd = tf.sqrt(tf.reduce_sum(tf.square(tf.matmul(p, tf.transpose(R)) - q)) / P.shape[0])

    return R, t, rmsd

and a batched version:

def kabsch_tensorflow_batched(P, Q):
    """
    Computes the optimal rotation and translation to align two sets of points (P -> Q),
    and their RMSD.

    :param P: A Nx3 matrix of points
    :param Q: A Nx3 matrix of points
    :return: A tuple containing the optimal rotation matrix, the optimal
             translation vector, and the RMSD.
    """
    P = tf.convert_to_tensor(P, dtype=tf.float32)
    Q = tf.convert_to_tensor(Q, dtype=tf.float32)

    assert P.shape == Q.shape, "Matrix dimensions must match"

    # Compute centroids
    centroid_P = tf.reduce_mean(P, axis=1, keepdims=True)
    centroid_Q = tf.reduce_mean(Q, axis=1, keepdims=True)

    # Center the points
    p = P - centroid_P
    q = Q - centroid_Q

    # Compute the covariance matrix
    H = tf.matmul(tf.transpose(p, perm=[0, 2, 1]), q)

    # SVD
    S, U, V = tf.linalg.svd(H)

    # 1. Calculate batched determinant
    d = tf.linalg.det(tf.matmul(V, tf.transpose(U, perm=[0, 2, 1])))

    # 2. Build batched B_diag: shape (B, 3)
    ones = tf.ones_like(d)
    B_diag = tf.stack([ones, ones, tf.sign(d)], axis=-1)

    # 3. Scale columns of V (Broadcasting adds the middle dimension)
    # V: (B, 3, 3), B_diag: (B, 3) -> B_diag[:, None, :]: (B, 1, 3)
    R = tf.matmul(V * B_diag[:, None, :], tf.transpose(U, perm=[0, 2, 1]))

    # Optimal translation (depends on R, so computed after it)
    t = tf.squeeze(centroid_Q, axis=1) - tf.linalg.matvec(R, tf.squeeze(centroid_P, axis=1))  # Bx3

    # RMSD
    rmsd = tf.sqrt(tf.reduce_sum(tf.square(tf.matmul(p, tf.transpose(R, perm=[0, 2, 1])) - q), axis=(1, 2)) / P.shape[1])

    return R, t, rmsd

JAX

The JAX implementation closely mirrors NumPy, replacing np with jnp. However, we again avoid if statements and in-place assignment (which JAX disallows) by using the broadcasting B-matrix approach.

import jax.numpy as jnp


def kabsch_jax(P, Q):
    """
    Computes the optimal rotation and translation to align two sets of points (P -> Q),
    and their RMSD.

    :param P: A Nx3 matrix of points
    :param Q: A Nx3 matrix of points
    :return: A tuple containing the optimal rotation matrix, the optimal
             translation vector, and the RMSD.
    """
    P = jnp.array(P)
    Q = jnp.array(Q)
    assert P.shape == Q.shape, "Matrix dimensions must match"

    # Compute centroids
    centroid_P = jnp.mean(P, axis=0)
    centroid_Q = jnp.mean(Q, axis=0)

    # Center the points
    p = P - centroid_P
    q = Q - centroid_Q

    # Compute the covariance matrix
    H = jnp.dot(p.T, q)

    # SVD
    U, S, Vt = jnp.linalg.svd(H)

    # 1. Calculate determinant
    d = jnp.linalg.det(jnp.dot(Vt.T, U.T))

    # 2. Build diagonal B array
    B_diag = jnp.array([1.0, 1.0, jnp.sign(d)])

    # 3. Scale columns of Vt.T and multiply by U.T
    # Vt.T is V.
    R = jnp.dot(Vt.T * B_diag[None, :], U.T)

    # Optimal translation (depends on R, so computed after it)
    t = centroid_Q - jnp.dot(R, centroid_P)

    # RMSD
    rmsd = jnp.sqrt(jnp.sum(jnp.square(jnp.dot(p, R.T) - q)) / P.shape[0])

    return R, t, rmsd

and batched:

def kabsch_jax_batched(P, Q):
    """
    Computes the optimal rotation and translation to align two sets of points (P -> Q),
    and their RMSD.

    :param P: A BxNx3 matrix of points
    :param Q: A BxNx3 matrix of points
    :return: A tuple containing the optimal rotation matrix, the optimal
             translation vector, and the RMSD.
    """
    P = jnp.array(P)
    Q = jnp.array(Q)
    assert P.shape == Q.shape, "Matrix dimensions must match"

    # Compute centroids
    centroid_P = jnp.mean(P, axis=1, keepdims=True)  # Bx1x3
    centroid_Q = jnp.mean(Q, axis=1, keepdims=True)  # Bx1x3

    # Center the points
    p = P - centroid_P  # BxNx3
    q = Q - centroid_Q  # BxNx3

    # Compute the covariance matrix
    H = jnp.matmul(p.transpose(0, 2, 1), q)  # Bx3x3

    # SVD
    U, S, Vt = jnp.linalg.svd(H)  # Bx3x3

    # 1. Calculate batched determinant
    d = jnp.linalg.det(jnp.matmul(Vt.transpose(0, 2, 1), U.transpose(0, 2, 1)))

    # 2. Build batched B_diag
    ones = jnp.ones_like(d)
    B_diag = jnp.stack([ones, ones, jnp.sign(d)], axis=-1)

    # 3. Scale columns of Vt.T and multiply by U.T
    # Vt.T: (B, 3, 3). B_diag: (B, 3).
    R = jnp.matmul(Vt.transpose(0, 2, 1) * B_diag[:, None, :], U.transpose(0, 2, 1))

    # Optimal translation (depends on R, so computed after it)
    t = centroid_Q.squeeze(1) - jnp.matmul(centroid_P, R.transpose(0, 2, 1)).squeeze(1)  # Bx3

    # RMSD
    rmsd = jnp.sqrt(jnp.sum(jnp.square(jnp.matmul(p, R.transpose(0, 2, 1)) - q), axis=(1, 2)) / P.shape[1])

    return R, t, rmsd

Extensions

The Kabsch algorithm has several important extensions that go beyond the formulation dealt with here:

• Quaternion Form: The algorithm can be reformulated using quaternions for better numerical stability, particularly useful in applications requiring high precision.
• Iterative Versions: More robust variants that handle noise better and have improved scaling properties for large point sets. This also can be advantageous for setups with limited computational resources.
• Weighted Kabsch: Extensions that incorporate point weights (e.g., atomic masses in molecular dynamics). While SciPy provides a weighted version, it lacks batch processing capabilities.
• The Umeyama Algorithm: If your point sets are rotated, translated, and scaled differently, the Umeyama algorithm is the direct extension of Kabsch. It solves the same optimization problem but introduces a scaling factor $c$, finding the optimal alignment for $Q \approx c R P + t$.

Several of these extensions are implemented in the Kabsch-Horn Cookbook library, which provides differentiable Kabsch, Horn, and Umeyama alignment across NumPy, PyTorch, JAX, TensorFlow, and MLX.

Further Reading

• Wikipedia, Kabsch Algorithm
• Zalo on Kabsch: An interactive shape matching demo.

Original Papers

• [Kabsch 1976] Kabsch, W. (1976). “A solution for the best rotation to relate two sets of vectors.” Acta Crystallographica Section A, 32(5), 922-923. DOI: 10.1107/S0567739476001873 The original paper: a closed-form, non-iterative optimal-rotation solution derived via Lagrange multipliers and eigendecomposition of $\tilde{R}R$ (the SVD reformulation came later; see Arun et al. 1987). See also: paper notes.
• [Kabsch 1978] Kabsch, W. (1978). “A discussion of the solution for the best rotation to relate two sets of vectors.” Acta Crystallographica Section A, 34(5), 827-828. DOI: 10.1107/S0567739478001680 The follow-up paper correcting for improper rotations (reflections).
• [Arun et al. 1987] Arun, K. S., Huang, T. S., & Blostein, S. D. (1987). “Least-Squares Fitting of Two 3-D Point Sets.” IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-9(5), 698-700. DOI: 10.1109/TPAMI.1987.4767965 The first SVD-based formulation for 3D point set alignment. See also: paper notes.
• [Horn et al. 1988] Horn, B. K. P., Hilden, H. M., & Negahdaripour, S. (1988). “Closed-form solution of absolute orientation using orthonormal matrices.” Journal of the Optical Society of America A, 5(7), 1127-1135. DOI: 10.1364/JOSAA.5.001127 The matrix square root (polar decomposition) approach to the same problem. See also: paper notes.
• [Horn 1987] Horn, B. K. P. (1987). “Closed-form solution of absolute orientation using unit quaternions.” Journal of the Optical Society of America A, 4(4), 629-642. DOI: 10.1364/JOSAA.4.000629 An alternative quaternion-based closed-form solution that also handles scale. See also: paper notes.
• [Umeyama 1991] Umeyama, S. (1991). “Least-squares estimation of transformation parameters between two point patterns.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(4), 376-380. DOI: 10.1109/34.88573 The extension of the algorithm to include optimal scaling in addition to rotation and translation. See also: paper notes.

This project provides an “input-to-analysis” workflow for simulating adatom diffusion on FCC metal surfaces. It demonstrates how to set up surface diffusion simulations in LAMMPS, manage EAM potentials, and parse trajectory data into energy and trajectory plots using Python. The LAMMPS input scripts are adapted from Eric N. Hahn’s adatom tutorial; the Python analysis layer (plot_energy.py, plot_xy.py) is my own, written while in CSElab (Harvard, 2023).

The workflow covers two material systems (Copper (Cu) and Platinum (Pt)) providing comparative datasets that highlight how atomic mass and bonding strength affect surface dynamics.

Features

Simulation Architecture

The project separates simulation logic from analysis code:

Each directory contains:

• LAMMPS input scripts (.in files) defining the physics
• EAM potential files for metallic bonding (the Cu potential is committed; the Pt potential must be downloaded separately from the NIST Interatomic Potentials Repository, so the Pt system does not run as-checked-out)
• Python analysis scripts for trajectory and energy parsing

Key Features

• EAM Potentials: Uses Embedded Atom Method alloy potentials to accurately model metallic bonding and surface energies, providing accuracy beyond simple Lennard-Jones potentials
• Automated Analysis: Python pipeline (plot_energy.py, plot_xy.py) that parses raw thermodynamic logs and trajectory dumps to generate “health check” dashboards
• Workflow Orchestration: Demonstrates the “Input → Simulation → Analysis” loop, automating the transition from raw .lammpstrj files to publication-ready plots
• Kokkos Support: Includes Kokkos execution commands for GPU/multi-threaded runs

Simulation Parameters

Usage

The repository includes LAMMPS input scripts and Python analysis scripts. Run the LAMMPS scripts to generate trajectory data, then use the Python scripts to visualize the results.

Results

This workflow is documented in detail in companion blog posts:

• LAMMPS Tutorial: Copper and Platinum Adatom Diffusion - Complete setup walkthrough with line-by-line script explanation and comparison of how heavier atoms behave differently on surfaces

I developed an automated GROMACS pipeline to generate molecular dynamics (MD) datasets for machine learning applications. The workflow automates the simulation of capped dipeptides across nine distinct residue types, creating a diverse training set suitable for Neural Network Potentials (NNPs). The pipeline is built off Luca Tubiana’s GROMACS tutorial (University of Trento); the Python analysis layer and the curated dipeptide dataset are my own.

Features

Automated Simulation Pipeline

• End-to-End Scripting: Bash-automated workflow handling topology generation (pdb2gmx), solvation, ionization, and equilibration
• Langevin Dynamics: Implemented Stochastic Dynamics (SD) integration to ensure proper canonical (NVT) ensemble sampling
• High-Resolution Output: Configured to capture 0.1 ps (100 fs) resolution trajectories, critical for capturing fast bond vibrations
• Force Extraction: Optimized output to .trr format preserving uncompressed atomic forces, a key requirement for force-matching in ML potentials

; md_langevin.mdp
integrator  = sd        ; Stochastic dynamics for proper sampling
dt          = 0.001     ; 1 fs timestep
nstxout     = 100       ; Output every 100 steps = 0.1 ps resolution
tc-grps     = Protein Non-Protein
tau_t       = 0.1  0.1  ; Friction constant (ps)

Chemical Diversity Suite

Designed to stress-test ML models against varied kinematic constraints:

Usage

The pipeline is executed via bash scripts, requiring GROMACS to be installed.

Results

• Data Volume vs. Fidelity: Balanced high-frequency force outputs (every 100 steps) against storage constraints by automating post-processing extraction of forces into lightweight .xvg formats
• Force Field Consistency: Standardized the Amber03 force field and TIP3P water model across all residues to ensure consistent potential energy surfaces for downstream model training

Note: This pipeline uses Amber03 for consistency across residue types. For production ML potentials, consider swapping to Charmm36m or similar modern force fields.

Retrospective

• Demonstrative, not production-scale: the 1 ns trajectories exercise the pipeline and capture fast bond vibrations, but proper conformational sampling needs 100 ns to 1 µs runs. This is a working reference, not a finished dataset.
• Dated force field: Amber03 / TIP3P keeps the potential energy surface consistent across residues, but it is not state-of-the-art for ML-potential training; CHARMM36m or Amber ff19SB would be the upgrade path.
• Paused, not abandoned: a candidate to revive and extend (more residues, longer trajectories, Ramachandran analysis) for future force-matching work.

Related Work

• Mini-Protein Dynamics - Detailed blog post on the simulation methodology

This is my undergraduate senior thesis, completed at Drexel University in 2021. The scope (308 million posts across four platforms, structural topology analysis, and domain adaptation experiments with Transformer models) was unusually broad for a senior thesis, spanning large-scale data engineering, graph-structural analysis, and representation-learning experiments.

Social media research is often siloed by platform, with tools built specifically for Twitter’s flat structure or Reddit’s tree structure. This fragmentation makes cross-platform analysis difficult. In this work, I introduce PyConversations, an open-source Python package that normalizes data from Twitter, Facebook, Reddit, and 4chan into a single, platform-agnostic data model. (Note: the repository is archived and no longer actively maintained.)

Leveraging this tool, I processed over 308 million posts to analyze the structural “shape” of online conversations. I then evaluated the efficacy of domain-adaptive pre-training (DAPT) for Transformer-based language models, finding that training on a toxic domain (4chan) boosts hate-speech detection by over 5 F1.

The Engineering Problem: Data Normalization

Social media platforms impose different structural constraints on discourse, making it difficult to feed heterogeneous data into a single ML pipeline:

• Twitter: Technically allows infinite depth, but functionally operates as a flat stream or shallow tree.
• Facebook: Enforces a hard limit of two depth levels (comments and replies), resulting in “short and fat” conversation trees.
• Reddit & 4chan: Allow for deep, branching tree structures.

To solve this, I designed a Universal Message Schema and the PyConversations library. This system ingests raw dumps from these disparate sources and maps them to a unified Directed Acyclic Graph (DAG) format, preserving the parent-child relationships regardless of the source platform’s constraints.

Key Contributions

• PyConversations Library: An open-source package for robust conversational analysis, featuring graph-based traversing and filtering.
• Massive Dataset Analysis: Processed a collection of 308 million posts and 15.8 million conversations, creating one of the largest comparative cross-platform analyses at the time of thesis submission.
• Structural Insights: Quantified how UI constraints shape human behavior. For instance, Facebook’s depth limit forces users to “bunch” comments, creating uniquely wide conversation trees compared to Reddit’s deep, narrow threads.
• Domain Adaptation Experiments: Continued-pretrained RoBERTa on platform-specific slices (e.g., the 4chan-adapted RoBERTa-4chan), demonstrating that exposing models to toxic domains improved hate-speech detection F1 by over 5 points.

Structural Analysis Findings

By treating conversations as graphs, we uncovered distinct topological signatures for each platform:

The “Shape” of Discourse

We measured the width (max posts at any depth) and depth (max distance from root) of conversation trees.

• Facebook exhibited a “short and fat” topology due to its 2-level nesting limit.
• 4chan threads were surprisingly shallow despite having no depth limits. This suggests that the platform’s ephemerality (threads are deleted quickly) and the “bump limit” mechanic discourage long-term dialogue, though data scraping limitations on this transient platform also contribute to this topology.
• Reddit maintained the most robust tree structures, with “good faith” communities like r/ChangeMyView showing distinct patterns of sustained engagement.

Information Density

We analyzed Innovation Rate, a measure of how quickly a text introduces new vocabulary. We found that Twitter threads have negative innovation rates (indicating high novelty per token) likely forced by the strict character limits. In contrast, Reddit posts showed higher redundancy, typical of longer-form essay writing.

Representation Learning & Domain Adaptation

We experimented with “Warm-Start” tuning: taking a standard RoBERTa model and pre-training it further on platform-specific data before fine-tuning on downstream tasks (TweetEval).

• Limited gains on most general tasks: Domain-adaptive pre-training added little on sentiment and emotion (from well under 1 up to a few F1 points), with irony detection the exception (+5.6 to +5.9 F1). Base RoBERTa already covers most of the signal for general NLP tasks.
• The Toxic Exception: The notable exception was Hate Speech Detection. The 4chan-adapted model (RoBERTa-4chan) was the strongest here, outperforming the baseline by over 5 F1. This highlights that for specialized, out-of-distribution language (like toxic slang), domain adaptation remains valuable.

Significance

This work bridges the gap between Computational Social Science and ML Engineering. It provides the community with a reusable tool (PyConversations) to handle the messy reality of social data and offers empirical evidence on the limits and benefits of domain-adaptive pre-training for LLMs.

Citation

@thesis{heidenreich2021look,
  title={Look, Don't Tweet: Representation Learning and Social Media},
  author={Hunter Heidenreich},
  year={2021},
  school={Drexel University},
  type={Undergraduate Senior Thesis}
}

Related Work

For related work on how social media content surfaces in digital journalism, including a dataset of embedded tweets across 273,899 news articles, see NewsTweet Dataset: Social Media in Digital Journalism.

Undergraduate thesis exploring representation learning for social media text and developing tools for cross-platform conversational analysis. Built PyConversations, a Python module for analyzing social media conversations, and found that domain-specific approaches often outperform large pre-trained models.

Features

PyConversations Module

• Graph-based modeling: Models conversations as Directed Acyclic Graphs (DAGs) to quantify topological structure (depth, width, density)
• Unified interface: Polymorphic design normalizing heterogeneous data from Twitter, Reddit, 4chan, and Facebook into a single analysis schema
• Linguistic dynamics: Implements information-theoretic feature extraction, including harmonic mixing laws and entropy measures
• Stream processing: Memory-efficient generators to ingest and traverse multi-gigabyte JSON dumps (e.g., 135M+ Reddit posts) without loading the full corpus into RAM

Research Contributions

• Representation learning: Investigated domain-specific vs. general-purpose Transformers (BERT vs. specialized variants) on social media text
• Topological analysis: Demonstrated that conversational structure (context) is as critical as content for classification tasks
• Cross-platform study: Comparative analysis of communication dynamics across moderated (Reddit/Twitter) and unmoderated (4chan) spaces

Usage

The PyConversations module can be imported into Python scripts to parse and analyze social media datasets.

Results

• Model performance: Smaller, domain-specific approaches frequently outperformed standard pre-trained models
• Context importance: Conversational context and dialogue structure proved crucial for understanding social media interactions
• Domain adaptation: Social media text benefits from specialized handling over generic approaches
• Cross-platform challenges: Different platforms require adapted approaches despite seeming similarities

Team & Recognition

• Hunter Heidenreich - Lead Researcher and Developer
• Jake Williams - Faculty Advisor
• First Place - Research Undergraduate Senior Thesis at Drexel University

Impact

This library served as the engineering backbone for my thesis, Look, Don’t Tweet, enabling the processing of 308 million posts to evaluate Transformer performance on toxic data.

The findings about model performance suggested that specialized domains require tailored model architectures, a perspective that has become more relevant as the field continues to evolve.

A word embedding maps words to real-valued vectors:

$$ \text{word} \rightarrow \mathbb{R}^n $$

where $n$ represents the dimensionality of the embedding space.

The goal is simple: position semantically similar words close together in vector space. This dense representation typically uses hundreds of dimensions, a massive reduction from the millions required by one-hot encoding.

Word embeddings are grounded in Zellig Harris’ distributional hypothesis: words appearing in similar contexts tend to have similar meanings. This forms the foundation of distributional semantics.

Different embedding algorithms capture various aspects of this distributional principle. This post explores the main methods for creating word embeddings and their applications in natural language processing.

While modern foundation models and large Vision-Language Models rely on subword tokenizers (like BPE) and Transformer embedding layers, the goal is the same: mapping discrete text to a continuous vector space where math can capture meaning. These foundational techniques build the intuition for the embedding layers in today’s models.

Why Word Embeddings Matter in NLP

Computers require numerical representations to apply machine learning algorithms to text. Word embeddings bridge this gap by converting text into dense vectors that preserve semantic and syntactic relationships.

Key advantages:

1. Dense representation: Hundreds of dimensions provide a compact alternative to vocabulary-sized sparse vectors.
2. Semantic preservation: Similar words cluster together in vector space.
3. Mathematical operations: Enable analogical reasoning ($\text{king} - \text{man} + \text{woman} \approx \text{queen}$).
4. Transfer learning: Pre-trained embeddings work across multiple tasks and domains.

Modern deep learning architectures leverage these properties extensively. The development of universal, pre-trained embeddings was a significant step forward. We can use versatile embeddings that generalize across applications, eliminating the need to train task-specific representations from scratch.

Word Embedding Approaches

One-Hot Encoding and Count Vectorization

One-hot encoding represents the simplest approach to word vectorization. Each word gets a unique dimension in a vocabulary-sized vector, marked with 1 for presence and 0 elsewhere. Count vectorization extends this by counting the occurrences of each word in a document.

Characteristics:

• High dimensionality: Vector length equals vocabulary size.
• Extreme sparsity: Most dimensions contain zeros.
• No relationships: Treats all words as equally distant.
• Computational efficiency: Simple to implement and understand.

While lacking semantic information, count vectorization serves as a foundation for more complex methods. Let’s look at a practical implementation using scikit-learn’s CountVectorizer.

from sklearn.feature_extraction.text import CountVectorizer

# Initialize the vectorizer
vectorizer = CountVectorizer()

# Sample text for demonstration
sample_text = ["One of the most basic ways we can numerically represent words "
               "is through the one-hot encoding method (also sometimes called "
               "count vectorizing)."]

# Fit the vectorizer to our text data
vectorizer.fit(sample_text)

# Examine the vocabulary and word indices
print('Vocabulary:')
print(vectorizer.vocabulary_)

# Transform text to vectors
vector = vectorizer.transform(sample_text)
print('Full vector:')
print(vector.toarray())

At scale, count vectorization introduces engineering challenges. With millions of documents, the vocabulary grows large, and the sparse matrices become expensive to store and compute on. In these scaling scenarios, practitioners often turn to the Hashing Trick (via HashingVectorizer) to bound the dimensionality, or they move entirely to the dense embeddings discussed later in this post.

We can see count vectorization in action with a real dataset, building a simple text classifier for the 20 Newsgroups dataset:

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn import metrics

# Load train and test splits, removing metadata for a cleaner signal
newsgroups_train = fetch_20newsgroups(subset='train',
                                      remove=('headers', 'footers', 'quotes'))
newsgroups_test = fetch_20newsgroups(subset='test',
                                     remove=('headers', 'footers', 'quotes'))

# Initialize and fit vectorizer on training data
vectorizer = CountVectorizer()
X_train = vectorizer.fit_transform(newsgroups_train.data)

# Build and train classifier
classifier = MultinomialNB(alpha=0.01)
classifier.fit(X_train, newsgroups_train.target)

# Transform test data and make predictions
X_test = vectorizer.transform(newsgroups_test.data)
y_pred = classifier.predict(X_test)

# Evaluate performance
accuracy = metrics.accuracy_score(newsgroups_test.target, y_pred)
print(f'Accuracy: {accuracy:.3f}')

This provides a solid baseline. To capture actual semantic meaning and reduce dimensionality, we must move beyond simple counting.

TF-IDF (Term Frequency-Inverse Document Frequency)

TF-IDF extends one-hot encoding by weighting terms based on their importance across a document collection. TF-IDF combines:

• Term Frequency (TF): How often a word appears in a document
• Inverse Document Frequency (IDF): How rare a word is across all documents

This weighting scheme reduces the impact of common words (like “the” or “and”) while emphasizing distinctive terms that appear frequently in specific documents but rarely elsewhere.

Advantages:

• Captures document-level importance
• Reduces impact of stop words
• Effective for information retrieval tasks

Limitations:

• Still high-dimensional and sparse
• No semantic relationships between terms
• Context-independent representation

Co-Occurrence Matrices

Co-occurrence matrices capture word relationships by recording which terms appear together within defined contexts (sentences, paragraphs, or fixed windows). The resulting matrix has dimensions equal to vocabulary size squared, with entries showing co-occurrence frequency.

Key properties:

• Global statistics: Captures corpus-wide word relationships
• Symmetric relationships: Mutual co-occurrence patterns
• Extreme dimensionality: Vocabulary size squared creates storage challenges
• Sparse representation: Most word pairs never co-occur

While computationally expensive to store and process, co-occurrence matrices form the foundation for advanced methods like GloVe that compress this information into dense representations.

Neural Network-Based Embeddings

Neural Probabilistic Language Models

Neural probabilistic models pioneered the use of neural networks for learning word embeddings. These models learn dense representations as a byproduct of language modeling, predicting the next word in a sequence.

Training process:

1. Initialize random dense embeddings for each vocabulary word
2. Use embeddings as inputs to predict language modeling objectives
3. Update embeddings through backpropagation based on prediction errors
4. Resulting embeddings capture patterns useful for the training task

This approach demonstrated that task-specific embeddings could be learned jointly with model objectives, establishing the foundation for modern embedding methods.

Word2Vec

Word2Vec made word embeddings practical at scale by introducing efficient training algorithms for massive corpora. It popularized compelling vector arithmetic properties, enabling analogical reasoning like the famous “$\text{king} - \text{man} + \text{woman} \approx \text{queen}$” example (a vector-offset regularity first reported by Mikolov, Yih & Zweig (2013) on recurrent-network language-model embeddings).

Two training architectures:

Continuous Bag-of-Words (CBOW)

Predicts target words from surrounding context words. Given a window of context words, the model learns to predict the central word.

Skip-Gram

Predicts context words from target words. Given a central word, the model learns to predict surrounding words within a defined window.

Key advantages:

• Computational efficiency: Much faster than neural probabilistic models
• Scalable training: Can process billion-word corpora effectively
• Quality embeddings: Captures semantic and syntactic relationships
• Flexible context: Window size controls topical vs. functional similarity

The choice of window size significantly impacts learned relationships. Larger windows capture topical associations, while smaller windows focus on syntactic and functional similarities.

GloVe (Global Vectors)

GloVe combines the best aspects of matrix factorization methods (which capture global corpus statistics) and local context window approaches like Word2Vec. Matrix factorization methods excel at global patterns but struggle with analogical reasoning, while Word2Vec captures local relationships but may miss global structure.

Key innovation: GloVe trains on a global word-context co-occurrence matrix, incorporating corpus-wide statistical information while maintaining the analogical reasoning capabilities that made Word2Vec successful.

Advantages over Word2Vec:

• Global optimization: Leverages entire corpus statistics
• Better performance: Often outperforms Word2Vec on word similarity and analogy tasks
• Stable training: More consistent convergence due to global objective function

The result is embeddings that capture both local syntactic patterns and global semantic relationships more effectively.

Contextual Embedding Methods

FastText

FastText addresses a critical limitation of previous methods: handling out-of-vocabulary (OOV) words. By incorporating subword information, FastText can generate meaningful representations for previously unseen words.

Subword approach:

• Decomposes words into character n-grams (typically 3-6 characters)
• Represents words as sums of their component n-grams
• Trains using skip-gram objective with negative sampling

Key advantages:

• OOV handling: Can embed unseen words using known subword components
• Morphological awareness: Captures relationships between related word forms
• Multilingual support: Facebook released pre-trained embeddings for 294 languages
• Robust performance: Particularly effective for morphologically rich languages

For example, if the model knows “navigate,” it can provide meaningful representation for “circumnavigate” by leveraging shared subword components, even if “circumnavigate” wasn’t in the training data.

Poincaré Embeddings

Poincaré embeddings introduce a novel approach by learning representations in hyperbolic space. This geometric innovation specifically targets hierarchical relationships in data.

Hyperbolic geometry advantages:

• Natural hierarchy encoding: Distance represents similarity, while norm encodes hierarchical level
• Efficient representation: Requires fewer dimensions for hierarchical data
• Mathematical elegance: Leverages properties of hyperbolic space for embedding optimization

Applications: Particularly effective for data with inherent hierarchical structure, such as:

• WordNet taxonomies
• Organizational charts
• Computer network topologies
• Knowledge graphs

The original paper demonstrates good efficiency in reproducing WordNet relationships with significantly lower dimensionality compared to traditional embedding methods.

Contextual Embeddings

ELMo (Embeddings from Language Models)

ELMo represents a paradigm shift toward contextual word representations. ELMo generates dynamic representations based on sentence context, adapting to word usage patterns.

Architecture:

• Bidirectional LSTM: Processes text in both forward and backward directions
• Character-level input: Handles OOV words and captures morphological patterns
• Multi-layer representations: Combines different abstraction levels

Layer specialization:

• Lower layers: Excel at syntactic tasks (POS tagging, parsing)
• Higher layers: Capture semantic relationships (word sense disambiguation)
• Combined layers: Weighted combination achieves good performance

Key innovation: ELMo embeddings vary by context. The word “bank” receives different representations in “river bank” versus “financial bank,” addressing polysemy directly through contextual awareness.

This approach achieved strong performance across numerous NLP tasks by providing context-sensitive representations that adapt to word usage patterns.

Probabilistic FastText

Probabilistic FastText addresses polysemy (words with multiple meanings) through probabilistic modeling. Traditional embeddings conflate different word senses into single representations, limiting their precision.

The polysemy problem: Consider “rock” which can mean:

• Rock music (genre)
• A stone (geological object)
• Rocking motion (verb)

Standard embeddings average these meanings, producing representations that may not capture any sense precisely.

Probabilistic approach: Probabilistic FastText represents words as Gaussian mixture models: probability distributions that can capture multiple distinct meanings as separate components.

Advantages:

• Multi-sense representation: Each word sense gets its own distribution
• Context sensitivity: Can select appropriate sense based on usage context
• Uncertainty quantification: Probabilistic framework captures embedding confidence

This approach provides a more nuanced treatment of lexical ambiguity, particularly valuable for words with distinct, context-dependent meanings.

Summary and Future Directions

Word embeddings have evolved from simple one-hot encodings to contextual representations that capture nuanced linguistic relationships. Each approach offers distinct advantages:

Static embeddings (Word2Vec, GloVe, FastText) provide:

• Computational efficiency for large-scale applications
• Pre-trained models available for numerous languages
• Clear analogical reasoning capabilities
• Good performance on many downstream tasks

Contextual embeddings (ELMo, BERT, GPT) offer:

• Dynamic representations based on sentence context
• Better handling of polysemy and word sense disambiguation
• Strong performance on complex NLP tasks
• Ability to capture subtle contextual nuances

Choosing the right approach depends on:

• Task requirements: Static embeddings for efficiency, contextual for accuracy
• Data availability: Pre-trained models vs. domain-specific training
• Computational constraints: Static embeddings require less processing power
• Language coverage: Consider availability of pre-trained models for target languages

The field continues advancing toward more efficient contextual models, better multilingual representations, and embeddings that capture increasingly complex linguistic phenomena.

For a from-scratch Word2Vec implementation in PyTorch (Skip-gram and CBOW, with hierarchical softmax and negative sampling) that takes these concepts further, see the PyTorch Word2Vec project.

Overview

Manually creating a university schedule involves solving a Constraint Satisfaction Problem (CSP) with multiple variables:

• Hard Constraints: No time overlaps between classes.
• Soft Constraints: Preferences for “no 8 AMs,” specific lunch breaks, or maximizing free days.

The naive approach (manually checking every possible combination) becomes intractable as the number of courses and sections grows.

Features

I built a script that:

1. Scraped Data: Parsed the Drexel WebTMS (Term Master Schedule) using lxml to build a localized dataset of course availability.
2. Solved for X: Implemented a recursive backtracking algorithm to generate every valid schedule permutation that satisfied user-defined constraints.

The Algorithm

The core of this project is a recursive_generator function that implements a valid CSP solver using backtracking. It performs a recursive depth-first search that:

1. Takes a set of variables (courses).
2. Checks constraints (time overlaps, lunch hours, max classes per day).
3. Backtracks when a branch fails.

It is the same backtracking pattern used in everything from Sudoku solvers to compiler register allocation.

Usage/Gameplay

The tool is run via the command line, taking a list of desired courses and outputting valid schedule combinations.

Results

This tool saved me (and several friends) hours of planning time each quarter. While the scraping logic was fragile (dependent on 2017 HTML structures), the core logic (a depth-first search through the state space of possible schedules) remains a fundamental algorithmic pattern.

Built in under 24 hours at the Drexel 2017 Music Hackathon, this project attempts to answer a question: What does order sound like?

The system uses a webcam to scan a Rubik’s cube face and algorithmically generates audio based on the color configuration. A scrambled cube generates dissonant, complex waveforms; a solved cube resolves into a pure, harmonious chord.

Features

This freshman-year project was built on first principles:

• Manual Waveform Synthesis: The audio engine generates raw 8-bit PCM audio byte-by-byte using sine functions (math.sin), played at a 16 kHz sample rate.
• Algorithmic Harmony: Colors are mapped to musical intervals. The “center” color establishes the root note (Tonic), while the surrounding “cubies” determine the chord structure and melody using equal temperament frequency calculations ($f = f_0 \cdot 2^{n/12}$).

Usage/Gameplay

The application runs via a Python script, requiring a webcam to scan the Rubik’s cube.

Results

Looking back at this code 8 years later, it serves as a “time capsule” of my early engineering mindset.

• The “Hack”: The computer vision relied on hardcoded pixel coordinates and raw OS shell calls, classic “glue code” behavior typical of hackathons.
• The Lesson: While brittle, the project successfully demonstrated how to bridge the gap between physical entropy and digital signal processing using fundamental programming concepts.

Related Content

• Video Demonstration

This is a project I worked on with Emmanuel Espino and Jason Zogheb for the Drexel 2017 Music Hackathon at the ExCITe Center. The Rubik’s Cube Player uses a webcam to capture images of a Rubik’s cube face and converts the color patterns into musical pitches.

The interesting part is that the more solved a cube face is, the more harmonious the generated melody sounds. We built this in less than 24 hours using Python, computer vision libraries, and audio synthesis techniques.

Technical Details

• Computer Vision: Uses webcam capture and image processing to identify cube colors
• Audio Generation: Converts color patterns to musical pitches and waveforms
• Real-time Processing: Live analysis and audio playback
• Harmony Algorithm: More solved faces produce more consonant musical intervals

Links

• GitHub Repository: Rubik’s Player
• Project Page: Rubik’s Cube Player
• Contributors: Emmanuel Espino, Jason Zogheb, and myself