Hand-Drawn Structure Recognition

Hand-drawn chemical structure recognition addresses the challenge of interpreting molecules sketched by hand, whether photographed from paper, scanned from notebooks, or captured from whiteboard images. These inputs differ substantially from the clean, typeset diagrams that most OCSR systems target: strokes vary in thickness and regularity, characters are ambiguous, and bond angles are imprecise. Early work like Ramel (1999) and Ouyang & Davis (2007) applied structural analysis and domain-constrained sketch recognition. More recent approaches use deep learning, including DECIMER’s hand-drawn variant, ChemReco’s EfficientNet-Transformer pipeline, and data augmentation strategies like OCSAug’s diffusion-based pipeline to bridge the domain gap between synthetic training data and real hand-drawn inputs.

Year	Paper	Key Idea
1999	Structural Analysis of Handwritten Chemical Formulas	Structural graph representation with recursive specialists
2007	Hand-Drawn Chemical Diagram Recognition (AAAI 2007)	Sketch recognition using chemical valence to correct errors
2008	Handwritten Chemical Ring Recognition with Neural Networks	Two-phase neural network pipeline for 23 types of heterocyclic rings
2021	ChemPix: Hand-Drawn Hydrocarbon Structure Recognition	CNN-LSTM image captioning for hand-drawn hydrocarbons to SMILES
2023	Handwritten Chemical Structure Recognition with RCGD	End-to-end framework with guided graph traversal and SSML markup
2024	AtomLenz: Atom-Level OCSR with Limited Supervision	Weakly supervised object detection and graph construction for OCSR
2024	ChemReco: Hand-Drawn Chemical Structure Recognition	EfficientNet and Transformer achieving 96.9% accuracy
2024	Enhanced DECIMER for Hand-Drawn Structure Recognition	EfficientNetV2 + Transformer for hand-drawn structures to SMILES
2025	OCSAug: Diffusion-Based Augmentation for Hand-Drawn OCSR	DDPM and RePaint augmentation to bridge synthetic-to-real domain gap

Computational Chemistry

Overview of the OCSAug pipeline showing DDPM training, masked RePaint augmentation, and OCSR fine-tuning phases.

OCSAug: Diffusion-Based Augmentation for Hand-Drawn OCSR

OCSAug uses Denoising Diffusion Probabilistic Models (DDPM) and the RePaint algorithm with custom masking to generate synthetic hand-drawn chemical structure images, improving OCSR performance by 1.918-3.820x on the DECIMER benchmark.

Computational Chemistry

AtomLenz learns atom-level detection from hand-drawn molecular images with weak supervision

AtomLenz: Atom-Level OCSR with Limited Supervision

Introduces AtomLenz, an OCSR tool that combines object detection with a molecular graph constructor. Features a novel weakly supervised training scheme (ProbKT*) to learn atom-level localization from SMILES-only data, achieving state-of-the-art results on hand-drawn images.

Computational Chemistry

Overview of the ChemReco pipeline showing synthetic data generation and EfficientNet+Transformer architecture for hand-drawn chemical structure recognition

ChemReco: Hand-Drawn Chemical Structure Recognition

ChemReco automates the recognition of hand-drawn chemical structures using a synthetic data pipeline and an EfficientNet+Transformer architecture, achieving 96.90% accuracy on C-H-O molecules.

Computational Chemistry

Diagram showing the DECIMER hand-drawn OCSR pipeline from hand-drawn chemical structure image through EfficientNetV2 encoder and Transformer decoder to predicted SMILES output

Enhanced DECIMER for Hand-Drawn Structure Recognition

This paper presents an enhanced deep learning architecture for Optical Chemical Structure Recognition (OCSR) specifically optimized for hand-drawn inputs. By pairing an EfficientNetV2 encoder with a Transformer decoder and training on over 150 million synthetic images, the model achieves 73.25% exact match accuracy on a real-world hand-drawn benchmark of 5,088 images.

Computational Chemistry

Overview of the ChemPix CNN-LSTM pipeline converting a hand-drawn hydrocarbon sketch to a SMILES string

ChemPix: Hand-Drawn Hydrocarbon Structure Recognition

Proposes a CNN-LSTM architecture that treats chemical structure recognition as an image captioning task. Introduces a synthetic data generation pipeline with augmentation, degradation, and background addition to train models that generalize to hand-drawn inputs without seeing real data during training.

Computational Chemistry

Handwritten Chemical Structure Recognition with RCGD

Proposes a Random Conditional Guided Decoder (RCGD) and a Structure-Specific Markup Language (SSML) to handle the ambiguity and complexity of handwritten chemical structure recognition, validated on a new benchmark dataset (EDU-CHEMC) with 50,000 handwritten images.

Computational Chemistry

Handwritten chemical ring recognition neural network architecture

Handwritten Chemical Ring Recognition with Neural Networks

Proposes a specialized Classifier-Recognizer architecture that first categorizes rings by heteroatom (S, N, O) and then identifies the specific ring using optimized grid inputs.

Computational Chemistry

Structural Analysis of Handwritten Chemical Formulas

This paper proposes a strategy for interpreting handwritten chemical formulas by converting bitmap images into a dynamic structural graph of quadrilaterals. It achieves ~97% recognition on graphical elements by using recursive ‘specialists’ to identify chemical bonds and rings.

Computational Chemistry

Hand-Drawn Chemical Diagram Recognition (AAAI 2007)

An early method paper (AAAI ‘07) proposing a multi-stage sketch recognition pipeline. It introduces a domain verification step that uses chemical rules to refine ink parsing, achieving a 27% error reduction over geometric-only baselines.