MaCBench is a Resource contribution that provides a comprehensive benchmark for evaluating vision language models (VLLMs) on real-world chemistry and materials science tasks. Rather than testing general-purpose visual reasoning or text-only scientific knowledge, MaCBench specifically targets the interplay between visual and textual modalities across the scientific workflow. The benchmark contains 779 multiple-choice questions and 374 numeric-answer questions organized into 11 topics across three pillars: data extraction, experimental execution, and data interpretation. Through systematic ablation studies, the authors identify fundamental limitations in spatial reasoning, cross-modal synthesis, and multi-step inference that current VLLMs exhibit.

Why Multimodal Evaluation Matters for Chemistry

Scientific research inherently requires integrating multiple information modalities: reading plots, interpreting spectra, evaluating laboratory setups, and connecting visual observations with domain knowledge. While text-only benchmarks like ChemBench have evaluated LLM capabilities in chemistry, and general multimodal benchmarks have tested visual reasoning, no prior work had systematically assessed how VLLMs handle the specific multimodal demands of the chemistry and materials science workflow.

Existing evaluations treated either the scientific reasoning dimension or the multimodal dimension in isolation. This left a critical gap: can VLLMs reliably assist with tasks that require both visual perception and scientific reasoning simultaneously? For example, identifying laboratory equipment is a perception task, but evaluating whether a laboratory setup is safe requires integrating visual understanding with domain-specific knowledge about hazards.

The authors designed MaCBench to fill this gap by constructing tasks that mirror actual scientific workflows and by including ablation studies that isolate specific failure modes.

Benchmark Design: Three Pillars of Scientific Work

The benchmark is structured around three pillars reflecting the scientific process:

Data Extraction covers parsing scientific literature, including extracting values from tables and plots, interpreting chemical structure diagrams, and identifying reaction components. Tasks range from simple value extraction to complex spatial reasoning about molecular relationships (e.g., identifying isomeric relationships between compounds).

Experimental Execution evaluates understanding of laboratory operations and crystallographic analysis. This includes equipment identification, safety assessment of laboratory setups, and interpretation of crystal structure renderings (space group assignment, atomic species counting, density calculations).

Data Interpretation tests analysis of experimental outputs: spectral analysis (XRD, NMR, mass spectrometry), electronic structure interpretation, adsorption isotherm analysis, and AFM image interpretation.

Each task uses a single prompt template containing multiple questions. All questions pair images with text-based prompts. The dataset was curated manually, with questions reviewed by multiple scientists before inclusion. A BigBench canary string is embedded in each file to prevent data contamination during future model training.

Evaluation of Frontier VLLMs and Ablation Studies

The authors evaluated four frontier VLLMs: Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro, and Llama 3.2 90B Vision. Performance is reported relative to random baselines to account for the varying number of answer choices across MCQ tasks:

$$ \text{acc}_{\text{rel}} = \text{acc} - \text{acc}_{\text{baseline}} $$

Each benchmark run was repeated five times to capture variability, with standard deviations reported as error bars.

Overall Performance Landscape

Claude 3.5 Sonnet was the leading model across all three task families, though no model dominated across all individual tasks. Key findings:

• Equipment identification: average accuracy of 0.77 (strong perception performance)
• Hand-drawn molecule to SMILES matching: average accuracy of 0.80
• Table composition extraction: average accuracy of 0.53 (Llama 3.2 indistinguishable from random guessing)
• Isomer relationship identification: average accuracy of 0.24 (barely above the 0.14 baseline)
• Laboratory safety assessment: average accuracy of 0.46
• AFM image interpretation: average accuracy of 0.24
• NMR and mass spectrometry analysis: average accuracy of 0.35

Ablation Studies: Four Dimensions of Failure

The authors designed ablations isolating four specific dimensions:

1. Modality (Image vs. Text): When identical information was presented as text instead of images, performance improved consistently across all tasks. For XRD peak identification, models showed a roughly 35% performance increase when peaks were provided as text rather than displayed visually. Even crystal structure volume calculations differed by four percentage points between visual and textual input of unit cell parameters.

2. Multi-Step Reasoning: Performance degraded consistently as tasks required more reasoning steps. For XRD analysis, identifying the highest peak achieved 0.74 average accuracy, while ranking relative peak intensities dropped to 0.28. Isotherm analysis showed the same pattern: finding the maximum value was easier than ordering multiple values.

3. Scientific Terminology: Removing domain-specific terminology (e.g., using IUPAC names instead of SMILES notation) improved performance on several tasks, suggesting models are sensitive to specific vocabularies rather than understanding underlying concepts. Gemini 1.5 Pro showed particular sensitivity to exact prompt wording, with large performance variations from minor changes like replacing “image” with “diagram” or “plot.”

4. Guidance: Adding step-by-step instructions improved performance for most models on spectral analysis and XRD pattern matching, with the notable exception of Claude 3.5 Sonnet, whose performance did not improve with guidance.

Internet Frequency Correlation

The authors measured the correlation between model performance and the number of Google search results for various crystal structures (as a proxy for training data frequency). For all tested cases, structures with correct model responses had higher Internet presence. This effect held even for pure perception tasks like counting atomic species, suggesting models may rely on memorized patterns rather than genuine visual reasoning.

Limitations of Current VLLMs for Scientific Assistance

The results reveal three fundamental limitations of current VLLMs:

Spatial reasoning failure: Models perform well on perception tasks (identifying equipment, matching hand-drawn molecules) but fail when spatial understanding is required (stereochemistry assignment at 0.24 accuracy, space group identification at 0.45). This limitation undermines one of the most intuitive potential use cases of vision models.

Incomplete cross-modal integration: The consistent performance gap between text and image presentations of identical information demonstrates that current models have not developed robust strategies for visual information processing. The models process text and images through fundamentally different pathways, with text consistently yielding better results.

Multi-step reasoning brittleness: The systematic degradation across reasoning steps indicates that chaining logical operations, a core requirement for scientific reasoning, remains a fundamental weakness.

The authors note that compared to text-only benchmarks (e.g., ChemBench), multimodal systems show much higher performance variability across tasks, suggesting greater fragility. They propose that advances in synthetic training data generation (particularly for spatial reasoning) and modality transformation training tasks could help address these limitations. They also acknowledge that future workflows with machine-actionable data formats may reduce the need for some multimodal parsing capabilities.

The benchmark does not encompass the full scope of scientific reasoning, and the evaluated models are not exhaustive of all available architectures. The authors call for continued research across wider task and model sets, along with interpretability studies to distinguish genuine reasoning from pattern matching.

Reproducibility Details

Data

Both datasets are available on HuggingFace. Questions are stored in extended BigBench format with base-64-encoded images and BigBench canary strings.

Algorithms

The evaluation pipeline builds on the ChemBench framework (v0.3.0). Answer extraction uses regex-based parsing backed by an LLM extractor (Claude 3.5 Sonnet) for fallback cases. Refusal detection combines LLM Guard regex patterns with a fine-tuned DistilRoBERTa model, with up to five retries for refused responses.

Scoring:

• MCQs: correct if Hamming loss is zero (exact match)
• Numeric: correct if mean absolute error falls within specified tolerance (default 1%, up to 5% for specific tasks)
• Random baseline: random option selection for MCQs; mean of all target values in a topic for numeric questions

Models

Four frontier VLLMs evaluated:

• Claude 3.5 Sonnet (Anthropic)
• GPT-4o (OpenAI)
• Gemini 1.5 Pro (Google)
• Llama 3.2 90B Vision (Meta)

Default quality/resolution settings were used for each provider.

Evaluation

Hardware

The paper does not specify hardware requirements. All evaluations were run through commercial API endpoints.

Artifacts

Paper Information

Citation: Alampara, N., Schilling-Wilhelmi, M., Ríos-García, M., Mandal, I., Khetarpal, P., Grover, H. S., Krishnan, N. M. A., & Jablonka, K. M. (2025). Probing the limitations of multimodal language models for chemistry and materials research. Nature Computational Science, 5(10), 952-961. https://doi.org/10.1038/s43588-025-00836-3

@article{alampara2025macbench,
  title={Probing the limitations of multimodal language models for chemistry and materials research},
  author={Alampara, Nawaf and Schilling-Wilhelmi, Mara and R{\'\i}os-Garc{\'\i}a, Marti{\~n}o and Mandal, Indrajeet and Khetarpal, Pranav and Grover, Hargun Singh and Krishnan, N. M. Anoop and Jablonka, Kevin Maik},
  journal={Nature Computational Science},
  volume={5},
  number={10},
  pages={952--961},
  year={2025},
  publisher={Nature Publishing Group},
  doi={10.1038/s43588-025-00836-3}
}

Abstract

Traditional OCR pipelines are often brittle, while modern “OCR-free” Vision-Language Models (VLMs) frequently lack precise token-to-pixel alignment. To address this, we introduce GutenOCR, a family of VLMs designed specifically as a “grounded OCR front-end.” By fine-tuning Qwen2.5-VL on a curriculum of synthetic and real-world documents, GutenOCR provides both high-quality text transcription and explicit geometric grounding (bounding boxes) through a unified, prompt-based interface. This approach allows downstream systems to request exactly the data format they need, from plain text to complex JSON structures.

Key Contributions & Results

• Unified Interface: Transforms Qwen2.5-VL models into specialized OCR systems supporting full-page reading, detection, localized reading, and conditional detection via prompting.
• In-Domain Improvements: GutenOCR-7B more than doubles the composite grounded OCR score of its base model (0.40 to 0.82) on 10.5K held-out pages, with the gains concentrated in localized reading and detection.
• Fox Benchmark: GutenOCR-3B achieves a region-level Character Error Rate (CER) of 0.053, surpassing even the dedicated Fox model (0.059), and sharply improves line-level OCR over its Qwen2.5-VL backbones (0.817 to 0.240), though the specialized Fox model still leads at line level.
• Curriculum Learning: Training uses a three-stage curriculum across synthetic data, real-world business documents, and long-context scientific articles to progressively build layout and grounding competency.
• Trade-offs: While GutenOCR reads content accurately (high Page F1), it orders text based on 2D layout columns. It also experiences catastrophic forgetting of color-based prompts and clear negative transfer on math formula recognition (notably the 3B model), to the point the paper recommends the un-fine-tuned backbone for equation-heavy use.

Methodology

• Data: The training mixture combines large-scale real-world documents (business forms, scientific articles) with synthetic data designed to teach precise grounding (e.g., “Grounded LaTeX” and “SynthDoG Grounding”).
• Curriculum Learning: Training progresses through three stages, starting with short contexts and synthetic data, moving to real-world business documents, and finishing with long-context scientific articles (up to 16k tokens).
• Unified Interface: The model treats “pipeline” stages (detection, reading, grounding) as different input-output schemas of a single model, allowing downstream systems to request exactly the data format they need (e.g., plain text vs. JSON boxes).

Models

We release 3B and 7B parameter models on HuggingFace:

• GutenOCR-3B (HuggingFace)
• GutenOCR-7B (HuggingFace)

You can try GutenOCR directly at ocr.bevaya.ai, where you can upload a document image and see the model’s parsed text output alongside bounding-box highlights on the original image.

Why This Matters

GutenOCR is proposed as a foundational layer for systems where every extracted answer must be explicitly linked to supporting pixels. By providing stable, grounded outputs, it enables human-in-the-loop workflows where reviewers can easily verify hallucinations or missing text by checking the predicted bounding boxes. This work pairs closely with our release of PubMed-OCR, which provides the large-scale, high-density annotations necessary to train such layout-aware models.

Resources

• Live Demo: Try GutenOCR on your own documents.
• Paper (arXiv): Full technical report.
• Code (GitHub): Training code and model release.
• GutenOCR-3B (HuggingFace): 3B parameter model weights.
• GutenOCR-7B (HuggingFace): 7B parameter model weights.

Citation

@misc{heidenreich2026gutenocrgroundedvisionlanguagefrontend,
      title={GutenOCR: A Grounded Vision-Language Front-End for Documents},
      author={Hunter Heidenreich and Ben Elliott and Olivia Dinica and Yosheb Getachew},
      year={2026},
      eprint={2601.14490},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2601.14490},
}

Related Work

• PubMed-OCR: The large-scale annotation dataset used to train GutenOCR’s layout-aware grounding capabilities.
• LLMs for Page Stream Segmentation: Complementary work on document understanding at the page-stream level.
• The Evolution of Page Stream Segmentation: Rules to LLMs: Background on the history and evolution of document processing pipelines.
• The Reliability Trap: When 99% Accuracy Isn’t Enough: Explores calibration challenges in deployed PSS systems, directly relevant to GutenOCR’s deployment context as an OCR front-end.

This paper represents a $\Psi_{\text{Method}}$ projection that proposes a novel architectural pipeline for indexing and searching chemical literature. The framework unifies text, molecular diagrams, and structured reaction records. It also contains a secondary $\Psi_{\text{Resource}}$ projection, providing a functional demonstration tool and curating a specific benchmark dataset for Suzuki coupling reactions.

The Gap in Passage-Level Chemical Retrieval

Scientific literature documents chemical reactions through a combination of text and visual diagrams. Textual descriptions detail parameters like yield and operational temperature, whereas diagrams graphically model these structural transformations. Existing tools such as SciFinder or Reaxys perform document-level or individual compound retrieval. They fail to explicitly link molecular figures to localized textual descriptions. This structure prevents researchers from directly extracting a corresponding reaction diagram alongside the exact textual protocol. Researchers require passage-level retrieval of synthesis protocols to efficiently access complete reaction conditions.

Core Innovation: Unified Multimodal Indexing

The core methodological innovation is a multimodal passage-level indexing and linking pipeline.

• Unified Indexing: The framework processes text and diagrams in parallel and directly links them into a single index structure. This architecture supports search queries utilizing raw text, discrete SMILES strings, or multimodal combinations.
• Compound-Passage Linking: The mechanism applies conflict-resolution logic linking chemical diagrams to specific text citations using two parallel heuristics:
  1. Token-based Alignment: Matching parsed diagram labels against documented text strings (e.g., “compound 5”) using normalized Levenshtein distance.
  2. Fingerprint-based Alignment: Matching chemical structures against generated SMILES strings via structural Tanimoto Similarity.
• ReactionMiner Integration: The pipeline parses and incorporates formatted reaction records (reactants, products, catalysts, quantitative yields) directly derived from segmented text passages.

Methodology & Expert Evaluation

The authors evaluated the system utilizing a chemical case study targeting specific synthesis domains alongside qualitative expert assessment.

• Dataset: Evaluators processed a corpus of 7 research manuscripts and 6 supplementary data documents detailing Suzuki coupling reactions.
• Volume: The resulting index processed 1,282 extracted passages (indexing 538), extracted 383 unique SMILES, and logged 219 parsed reactions.
• Qualitative Evaluation: Practicing structural chemists developed real-world queries (such as cross-referencing the conceptual “Burke group” alongside an explicit structural SMARTS pattern) to gauge retrieval capability.

Key Findings & System Limitations

• Diagram-to-Text Linking: The pipeline accurately paired visual molecular diagrams with structurally derived text details, permitting testers to navigate directly from a molecule query card to the exact origin passage within the source PDF.
• Contextual Insight Extraction: Specialized chemists found the parsed reaction representations (yield metrics, isolated catalysts) functionally pragmatic as high-level extractive summaries.
• Extrapolative Retrieval: The architecture permitted the effective retrieval of targeted chemical derivatives (such as benzo[b]thiophen-2-ylboronic acid) via structurally related input queries (dibenzothiophene).

The system evaluation highlights several architectural restrictions:

• Domain-Restricted Validation: The initial validation is entirely qualitative and bounded to the specific subclass of Suzuki coupling reactions. The evaluation omits standardized quantitative retrieval baselines (e.g., MAP, NDCG) and lacks systematic ablation data for the fusion scoring mechanism.
• Algorithmic Transparency: The multimodal query routing mechanism does not clearly indicate the dominant retrieval feature. This hides whether keyword text or structural similarity actually drove the final result placement. This ambiguity limits operator control.
• Optical Processing Brittleness: The embedded vision inference and primitive parsing pipelines display inherent fragility, producing intermittent failures when associating text passages with correctly parsed molecular diagrams.
• Metadata Logging Incompleteness: Practicing chemists requested additional structured metadata targets (such as specific molar equivalents and parameterized mol% values) to successfully bridge the extracted data stream directly into digital electronic lab notebooks.

Reproducibility

Artifacts

Data

• Source: The corpus features 7 primary research papers and 6 auxiliary supplementary information documents focusing on Suzuki coupling reactions, sourced from practicing chemists at UIUC. This evaluation dataset is strictly internal and not publicly available.
• Preprocessing:
  • Engineers convert source PDFs to full-page raster images.
  • The system extracts localized graphical layout and raw text via PyTesseract.
  • The pipeline segments valid passage chunks emphasizing reaction-related sentences utilizing product-indicative lexicons and topic modeling.

Algorithms

• Diagram Extraction: A YOLOv8 model identifies and segments molecular regions within structured PDF pages.
• Diagram Parsing: The architecture relies on ChemScraper to infer structural semantics from raw diagrams:
  • Born-digital PDFs: SymbolScraper extracts vector lines and polygons directly from bounding box definitions.
  • Raster images: The system employs the Line Segment Detector (LSD) and watershed bounding algorithms to isolate native geometric primitives.
• Text Entity Extraction: The framework deploys ChemDataExtractor 2.0 to extract explicit molecular aliases. A translation layer maps these entities to string representations via OPSIN.
• Linking Logic (Fusion Score):
  • Text Link: The algorithm calculates a normalized Levenshtein ratio connecting visual diagram labels against proximal text mentions based on calculated edit distance.
  • Structure Link: The algorithm computes the discrete Tanimoto Similarity between generated 2048-bit Morgan fingerprints extracted from localized visual diagram features and baseline text SMILES queries: $$ T(A, B) = \frac{A \cdot B}{|A|^{2} + |B|^{2} - A \cdot B} $$ where $A$ and $B$ represent the boolean bit vectors of the respective fingerprint pairs.
  • Conflict Resolution Protocol: The system fuses structural geometry bounds and discrete textual tokenization metrics, prioritizing the ranking sequence that yields a higher terminal similarity score. During final retrieval, the candidate subset is systematically re-ranked leveraging the hybrid calculation of the BM25 explicit metric and the localized count of exact SMILES pattern hits.

Models

• Reaction Extraction Parameters: The engineers configure a LLaMA-3.1-8b model fine-tuned entirely via LoRA targeting custom tokens representing reaction entities (compounds, reagents, thermal inputs) directly pulled from text sub-chunks. Exact prompt constraints, the fine-tuning dataset, and specific LoRA hyperparameters are omitted from the source text.
• Diagram Processing Bounds: The codebase incorporates a segmentation-aware multi-task neural network topology built into ChemScraper to execute low-level raster image parsing tasks.

Evaluation

• Search Engine Base: The authors implemented their indexing framework scaling atop PyTerrier.
• Text Feature Ranking: The metric utilizes standalone BM25 bounds mapping keyword-similarity.
• Structure Feature Operations: The topology operates RDKit bindings powering substructure coordinate mapping logic and exact molecular similarity searches.
• Multimodal Fusion Processing:
  • The algorithm filters out terminal candidates mapping initial structural properties (SMILES queries) against the document-wide lexical properties (BM25 scores).
  • The final fusion routing assigns the strongest positive weight to retrieved passages that accumulate dense local clusters of structurally exact verified SMILES patterns.

Hardware

• Compute Infrastructure: The hardware and parameter requirements to host the multi-stage vision extractors (YOLOv8, ChemScraper) alongside a local 8B LLM are entirely unspecified in the paper.

Paper Information

Citation: Shah, A. K., et al. (2025). Multimodal Search in Chemical Documents and Reactions. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘25). ACM. https://doi.org/10.48550/arXiv.2502.16865

Publication: SIGIR ‘25 (Demo Track), 2025

@misc{shahMultimodalSearchChemical2025,
  title = {Multimodal {{Search}} in {{Chemical Documents}} and {{Reactions}}},
  author = {Shah, Ayush Kumar and Dey, Abhisek and Luo, Leo and Amador, Bryan and Philippy, Patrick and Zhong, Ming and Ouyang, Siru and Friday, David Mark and Bianchi, David and Jackson, Nick and Zanibbi, Richard and Han, Jiawei},
  year = 2025,
  month = feb,
  number = {arXiv:2502.16865},
  eprint = {2502.16865},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2502.16865},
  archiveprefix = {arXiv}
}

Additional Resources:

• Online Demo (Note: While the landing page advertises the system as open-source, the exact repository URL and installation prerequisites are omitted from the official manuscript.)

This is primarily a Methodological paper ($\Psi_{\text{Method}}$) that introduces a novel pipeline (MERMaid) for extracting structured chemical data from unstructured PDF documents. It proposes a specific architecture combining fine-tuned vision models (VisualHeist) with vision-language models (DataRaider) and a retrieval-augmented generation system (KGWizard) to solve the problem of multimodal data ingestion.

Secondarily, it is a Resource paper ($\Psi_{\text{Resource}}$) as it releases the source code, prompts, and a new benchmark dataset (MERMaid-100) consisting of annotated reaction data across three chemical domains.

The Inaccessibility of Diagrammatic Reaction Data

• Data Inaccessibility: A significant volume of chemical knowledge currently resides in “print-optimized” PDF formats, specifically within graphical elements like figures, schemes, and tables, which resist standard text mining.
• Limitations of Prior Work: Existing tools (e.g., ChemDataExtractor, OpenChemIE) focus primarily on text, struggle with multimodal parsing, or lack the “contextual awareness” needed to interpret implicit information (e.g., “standard conditions” with modifications in optimization tables).
• Need for Structured Data: To enable self-driving laboratories and data-driven discovery, this unstructured literature must be converted into machine-actionable formats like knowledge graphs.

The MERMaid Pipeline: Vision Models and LLM RAG

• VisualHeist (Fine-tuned Segmentation): A custom fine-tuned model based on Microsoft’s Florence-2 that accurately segments figures, captions, and footnotes, even in messy supplementary materials.
• DataRaider (Context-Aware Extraction): A VLM-powered module (using GPT-4o) with a two-step prompt framework that performs “self-directed context completion.” It can infer missing reaction parameters from context and resolve footnote labels (e.g., linking “condition a” in a table to its footnote description).
• KGWizard (Schema-Adaptive Graph Construction): A text-to-graph engine that uses LLMs as higher-order functions to synthesize parsers dynamically. It employs Retrieval-Augmented Generation (RAG) to check for existing nodes during creation, implicitly resolving coreferences (e.g., unifying “MeCN” and “Acetonitrile”).
• Topic-Agnostic Design: MERMaid features a flexible design that works across three distinct domains: organic electrosynthesis, photocatalysis, and organic synthesis.

Benchmarking Segmentation and Extraction Accuracy

• Segmentation Benchmarking: The authors compared VisualHeist against OpenChemIE (LayoutParser) and PDFigCapX using a dataset of 121 PDFs from 5 publishers.
• End-to-End Extraction: Evaluated the full pipeline on MERMaid-100, a curated dataset of 100 articles across three domains (organic electrosynthesis, photocatalysis, organic synthesis).
  • Validating extraction of specific parameters (e.g., catalysts, solvents, yields) using “hard-match” accuracy.
• Knowledge Graph Construction: Automatically generated knowledge graphs for the three domains and assessed the structural integrity and coreference resolution accuracy.

End-to-End Extraction Performance

• Segmentation Results: VisualHeist achieved >93% F1 score across all document types (including pre-2000 papers and supplementary materials), outperforming OpenChemIE by 15-75% and PDFigCapX by 28-75% across all metrics.
• Extraction Accuracy: DataRaider achieved >92% accuracy for VLM-based parameter extraction and near-unity accuracy for domain-specific reaction parameters (e.g., anode, cathode, photocatalyst).
• Graph Building: KGWizard achieved 96% accuracy in node creation and coreference resolution.
• Overall Performance: The pipeline demonstrated an 87% end-to-end overall accuracy.
• Limitations: The architecture relies heavily on closed-weight models (GPT-4o) for reasoning and graph construction, which risks future reproducibility if API snapshots are deprecated. Additionally, the system remains vulnerable to cumulative error propagation from upstream OCR/OCSR tools like RxnScribe.
• Availability: The authors provide a modular, extensible framework that can be adapted to other scientific domains.

Reproducibility Details

Data

• Training Data (VisualHeist):
  • Dataset of 3,435 figures and 1,716 tables annotated from 3,518 PDF pages.
  • Includes main text, supplementary materials, and unformatted archive papers.
• Evaluation Data (MERMaid-100):
  • 100 PDF articles curated from three domains: organic electrosynthesis, photocatalysis, and organic synthesis.
  • Includes 104 image-caption/table-heading pairs relevant to reaction optimization.
  • Available for download at Zenodo (DOI: 10.5281/zenodo.14917752).

Algorithms

• Two-Step Prompt Framework (DataRaider):
  • Step 1: Generic base prompt + domain keys to extract “reaction dictionaries” and “footnote dictionaries”. Uses “fill-in-the-blank” inference for missing details.
  • Step 2: Safety check prompt where the VLM updates the reaction dictionary using the footnote dictionary to resolve entry-specific modifications.
• LLM-Synthesized Parsers (KGWizard):
  • Uses LLM as a function $g_{A,B}: A \times B \rightarrow (X \rightarrow Y)$ to generate Python code (parsers) dynamically based on input schema instructions.
• RAG for Coreference:
  • During graph construction, the system queries the existing database for matching values (e.g., “MeCN”) before creating new nodes to prevent duplication.
• Batching:
  • Articles processed in dynamic batch sizes (starting at 1, increasing to 30) to balance speed and redundancy checks.

Models

• VisualHeist: Fine-tuned Florence-2-large (Microsoft vision foundation model).
  • Hyperparameters: 12 epochs, learning rate $5 \times 10^{-6}$, batch size 4.
• DataRaider & KGWizard: GPT-4o (version gpt-4o-2024-08-06). Note: Requires an active OpenAI API key. The pipeline’s long-term reproducibility is currently tied to the continued availability of this specific closed-source endpoint.
• RxnScribe: Used for Optical Chemical Structure Recognition (OCSR) to convert reactant/product images to SMILES.

Evaluation

• Metrics:
  • Segmentation: Precision, Recall, F1, Accuracy.
  • Caption Extraction: Evaluated via Jaccard similarity, mapping predicted token sets $A$ and true token sets $B$ to a threshold condition: $$J(A, B) = \frac{|A \cap B|}{|A \cup B|} \ge 0.70$$
  • Data Extraction: Evaluated via Hard-Match accuracy, requiring exact correspondence between predicted sets ($\hat{Y}$) and ground-truth parameters ($Y$) for specific roles (e.g., anode vs. cathode): $$\text{HMA} = \frac{1}{|N|} \sum_{i=1}^{N} \mathbb{1}[y_i = \hat{y}_i]$$
• Baselines: OpenChemIE (LayoutParser + EasyOCR) and PDFigCapX.

Hardware

• Training (VisualHeist): 2x NVLINK Nvidia RTX A6000 GPUs (48GB VRAM) + Intel Xeon w7-2495X CPU (48 cores).
• DataRaider Evaluation: 13th Gen Intel Core i7-1360P CPU (12 cores).
• Inference Costs:
  • DataRaider: ~$0.051 per image.
  • KGWizard: ~$0.40 per JSON.
• Timing:
  • VisualHeist inference: ~4.5 seconds/image.
  • DataRaider inference: ~41.3 seconds/image.
  • KGWizard processing: ~110.6 seconds/file.

Paper Information

Citation: Leong, S. X., Pablo-García, S., Wong, B., & Aspuru-Guzik, A. (2025). MERMaid: Universal multimodal mining of chemical reactions from PDFs using vision-language models. Matter, 8(12), 102331. https://doi.org/10.1016/j.matt.2025.102331

Publication: Matter, 2025

Artifacts:

@article{leong2025mermaid,
  title={MERMaid: Universal multimodal mining of chemical reactions from PDFs using vision-language models},
  author={Leong, Shi Xuan and Pablo-Garc{\'i}a, Sergio and Wong, Brandon and Aspuru-Guzik, Al{\'a}n},
  journal={Matter},
  volume={8},
  number={12},
  pages={102331},
  year={2025},
  doi={10.1016/j.matt.2025.102331}
}

Methodological Paper ($\Psi_{\text{Method}}$)

This work proposes InstructMol, a novel multi-modal architecture and training paradigm. It focuses on engineering a system that aligns a pre-trained molecular graph encoder with a general-purpose Large Language Model (LLM). The paper’s primary contribution is the Two-Stage Instruction Tuning strategy (Alignment Pre-training + Task-Specific Tuning) designed to bridge the modality gap between 2D molecular graphs and natural language.

Bridging Specialist and Generalist Models

Current AI approaches in drug discovery typically fall into two categories. Specialist models deliver high accuracy on specific tasks (such as property prediction) but require extensive labeled datasets and lack conversational adaptability. Conversely, generalist LLMs offer strong reasoning and dialogue capabilities but struggle to natively interpret complex structural data, often relying on brittle 1D text representations of molecules like SMILES.

There is a practical need for a unified “Molecular Assistant” capable of visually interpreting molecular graphs, reasoning about structure in natural language, and adapting across tasks like synthesis planning and property analysis without training from scratch.

Two-Stage Modality Alignment

The core novelty lies in the architecture and the two-stage training pipeline designed to align differing modalities efficiently:

1. MoleculeSTM Integration: InstructMol initializes its graph encoder with MoleculeSTM, which is already pre-aligned with text via contrastive learning, facilitating easier downstream alignment.
2. Two-Stage Alignment Strategy:
   • Stage 1 (Alignment Pre-training): Freezes both the LLM and Graph Encoder; trains only a linear projector using a massive dataset of molecule-description pairs to map graph features into the LLM’s token space.
   • Stage 2 (Task-Specific Instruction Tuning): Freezes the Graph Encoder; fine-tunes the Projector and the LLM (using LoRA) on specific downstream tasks. This allows the model to adapt its reasoning capabilities while preserving the structural understanding gained in Stage 1.

Task Evaluation in Drug Discovery

The authors evaluated InstructMol across three distinct categories of drug discovery tasks, comparing it against generalist LLMs (Vicuna, LLaMA, Galactica) and specialist models (ChemBERTa, MolT5):

1. Property Prediction:
   • Regression: Predicting quantum mechanical properties (HOMO, LUMO, Gap) using the QM9 dataset.
   • Classification: Predicting biological activity (BACE, BBBP, HIV) using MoleculeNet.
2. Molecule Description Generation: Generating natural language descriptions of molecules using the ChEBI-20 dataset.
3. Chemical Reaction Analysis:
   • Forward Reaction Prediction: Predicting products from reactants.
   • Reagent Prediction: Identifying necessary reagents.
   • Retrosynthesis: Suggesting reactants for a given product.

Ablation Studies tested the impact of the projector type (Linear vs. MLP), LLM scale (7B vs 13B), and the necessity of the two-stage training approach.

Core Findings and Limitations

• Improvement Over Baseline Generalists: InstructMol significantly outperformed generalist LLMs (like LLaMA and Galactica) on all tasks, demonstrating the value of incorporating explicit graph modalities.
• Reducing the Gap with Specialists: While InstructMol brings versatile reasoning capabilities, it still trails highly optimized specialist models (such as Uni-Mol and MolT5) on tasks like molecule description generation. This remaining gap likely stems from its reliance on a relatively small alignment pre-training dataset (~264K PubChem pairs) and the information bottleneck of using a simple linear projector, compared to the millions of structures used to train expert foundational models.
• Importance of Alignment: Ablation studies confirmed that skipping Stage 1 (Alignment Pre-training) degraded performance, proving that a dedicated phase for projecting graph features into text space is crucial.
• Limitation: The model struggles with highly imbalanced datasets (e.g., HIV) and complex reaction mixtures where mapping multiple graph tokens to text becomes ambiguous.

Reproducibility Details

Data

The training pipeline utilizes distinct datasets for the two stages. Note: As of the latest repository update, the finely-processed instruction-tuning datasets (e.g., the filtered ~264K PubChem pairs and instruction-formatted subset pairs) are listed as “coming soon”, requiring manual recreation for full reproduction.

Algorithms

• Two-Stage Training:
  1. Alignment Pre-training: Updates only the Projector. The objective maximizes the probability of generating the target description token sequence $\mathbf{X}_A$ given the molecule input $\mathbf{X}_M$ and instruction $\mathbf{X}_I$: $$p(\mathbf{X}_A | \mathbf{X}_M, \mathbf{X}_I) = \prod_{i=1}^L p_\theta(x_i | \mathbf{X}_G \parallel \mathbf{X}_S, \mathbf{X}_I, \mathbf{X}_{A,<i})$$
  2. Instruction Tuning: Updates Projector + LLM (via LoRA) using standard autoregressive language modeling on task-specific instructions. The objective minimizes the negative log-likelihood of generating the target response $R$ of length $L$: $$\mathcal{L}(\theta) = -\sum_{i=1}^L \log p(R_i | I, M, R_{<i}; \theta)$$ where $I$ represents the instruction and $M$ is the multi-modal molecular input.
• LoRA (Low-Rank Adaptation): Applied to the LLM in Stage 2. Rank $r=64$, Scaling $\alpha=16$.
• Optimization: AdamW optimizer. Learning rate starts at 2e-3 (Stage 1) and 8e-5 (Stage 2) with cosine decay. Warm-up ratio 0.03.

Models

Note: The official repository currently lists the final fine-tuned InstructMol weights as “coming soon.” Consequently, one must fine-tune the components using the provided scripts. Base model weights (Vicuna-7B and MoleculeSTM) are publicly available via Hugging Face.

• Graph Encoder ($f_g$):
  • Architecture: Graph Isomorphism Network (GIN) with 5 layers.
  • Hidden Dimension: 300.
  • Initialization: MoleculeSTM checkpoint (pre-trained via contrastive learning).
  • Status: Frozen during Stage 2.
• LLM:
  • Base: Vicuna-v1.3-7B.
  • Status: Frozen in Stage 1; LoRA fine-tuned in Stage 2.
• Projector:
  • Architecture: Linear Layer.
  • Function: Maps node-level graph representation $Z_G \in \mathbb{R}^{N \times d}$ to the LLM’s word embedding space dimensions.

Evaluation

• Metric Libraries: RDKit for validity/fingerprints, standard NLP libraries for BLEU/ROUGE.
• Reaction Metrics: Fingerprint Tanimoto Similarity (FTS), Exact Match, Levenshtein distance, and validity (via RDKit).
• Description Metrics: BLEU-2, BLEU-4, ROUGE-1, ROUGE-2, ROUGE-L, METEOR.

Hardware

• Compute: 4 x NVIDIA RTX A6000 (48GB VRAM).
• Training Time:
  • Stage 1: 5 epochs.
  • Stage 2: 20-50 epochs (Description Generation), 10 epochs (Properties/Reactions).
• Batch Size: 128 for both stages.

Artifacts

Paper Information

Citation: Cao, H., Liu, Z., Lu, X., Yao, Y., & Li, Y. (2025). InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery. Proceedings of the 31st International Conference on Computational Linguistics, 354-379.

Publication: COLING 2025

@inproceedings{caoInstructMolMultiModalIntegration2025,
  title = {{{InstructMol}}: {{Multi-Modal Integration}} for {{Building}} a {{Versatile}} and {{Reliable Molecular Assistant}} in {{Drug Discovery}}},
  shorttitle = {{{InstructMol}}},
  booktitle = {Proceedings of the 31st {{International Conference}} on {{Computational Linguistics}}},
  author = {Cao, He and Liu, Zijing and Lu, Xingyu and Yao, Yuan and Li, Yu},
  editor = {Rambow, Owen and Wanner, Leo and Apidianaki, Marianna and {Al-Khalifa}, Hend and Eugenio, Barbara Di and Schockaert, Steven},
  year = 2025,
  month = jan,
  pages = {354--379},
  url = {https://aclanthology.org/2025.coling-main.25/},
  publisher = {Association for Computational Linguistics},
  address = {Abu Dhabi, UAE},
  abstract = {The rapid evolution of artificial intelligence in drug discovery encounters challenges with generalization and extensive training, yet Large Language Models (LLMs) offer promise in reshaping interactions with complex molecular data. Our novel contribution, InstructMol, a multi-modal LLM, effectively aligns molecular structures with natural language via an instruction-tuning approach, utilizing a two-stage training strategy that adeptly combines limited domain-specific data with molecular and textual information. InstructMol showcases substantial performance improvements in drug discovery-related molecular tasks, surpassing leading LLMs and significantly reducing the gap with specialists, thereby establishing a robust foundation for a versatile and dependable drug discovery assistant.}
}

Additional Resources:

• Official Repository

This is primarily a Method paper with a significant Resource contribution.

Method: The paper proposes a novel “Cross-modal Dialogue Foundation Model” architecture that aligns five distinct chemical modalities (2D graphs, 3D conformations, images, MS2 spectra, IR spectra) to a single LLM decoder using separate encoders and projection modules. It establishes strong baseline performance across multiple modalities compared against current generalist models.

Resource: The paper addresses the scarcity of multimodal chemical data by constructing a 7.6M instruction-tuning dataset. This dataset is largely synthesized from seed SMILES strings using approximate calculations (MMFF94, CFM-ID, Chemprop-IR) and specialist model predictions.

Bridging Experimental Data and LLMs

Existing chemical AI models generally fall into two distinct categories. Task-specific specialist models achieve high accuracy on singular objectives, such as property prediction or molecular generation, but require strict formatting and lack conversational flexibility. Conversely, early chemical large language models provide natural language interaction but are restricted to text and SMILES strings. ChemDFM-X addresses this gap by enabling large multimodal models to process the experimental characterization data (MS2 spectra and IR spectra) and visual data routinely used in practical chemistry workflows.

Synthetic Data Scaling for Modality Alignment

The core novelty lies in the “Any-to-Text” alignment strategy via synthetic data scaling:

1. Comprehensive Modality Support: ChemDFM-X incorporates experimental characterization data (MS2 and IR spectra) alongside 2D graphs, 3D conformations, and images. The data representations are formally defined mathematically rather than as raw pixels:

   • Molecular Graph: An undirected graph $G = (\textbf{V}, \textbf{E})$ with atom set $\textbf{V}$ and bond set $\textbf{E}$.
   • Molecular Conformation: An undirected graph $G = (\textbf{V}’, \textbf{E})$ storing spatial coordinates: $\textbf{v}_i = (x_i, y_i, z_i, a_i)$.
   • MS2 Spectrum: Treated as a point sequence of discrete mass-to-charge ratios and intensities, tokenized via a discrete codebook: $\textbf{M} = ((r_1, I_1), (r_2, I_2), \dots, (r_n, I_n))$.
   • IR Spectrum: Treated as a dense sequence of continuous wave lengths and absorption intensities, directly reshaped for feature extraction: $\textbf{R} = ((w_1, t_1), (w_2, t_2), \dots, (w_l, t_l))$.

   The authors trained new Sequence Transformer encoders from scratch for the MS2 and IR modalities since suitable pre-trained models did not exist.

2. Synthetic Data Generation Pipeline: The authors generated a 7.6M sample dataset by starting with 1.3M seed SMILES and using “approximate calculations” to generate missing modalities:

   • 3D conformations via MMFF94 force field optimization
   • MS2 spectra via CFM-ID 4.0 (Competitive Fragmentation Modeling)
   • IR spectra via Chemprop-IR (Message Passing Neural Network)

3. Cross-Modal Synergy: The model demonstrates that training on reaction images improves recognition performance by leveraging semantic chemical knowledge (reaction rules) to correct visual recognition errors, an emergent capability from multimodal training.

Multimodal Benchmarking with ChemLLMBench

The model was evaluated using a customized version of ChemLLMBench and MoleculeNet across three modality categories:

1. Structural Modalities (2D Graphs & 3D Conformations):

   • Molecule recognition and captioning
   • Property prediction (MoleculeNet: BACE, BBBP, ClinTox, HIV, Tox21)
   • Compared against specialist models (Mole-BERT, Uni-Mol, MolXPT, MolCA) and generalist models (3D-MoLM, ChemDFM, ChemLLM)

2. Visual Modalities (Images):

   • Single molecule image recognition
   • Reaction image recognition
   • Compared against GPT-4O, Gemini 1.5 Pro, Qwen-VL, LLaVA, and specialist models MolNextr and MolScribe

3. Characterization Modalities (MS2 & IR Spectra):

   • Spectral analysis tasks (identifying molecules from spectra)
   • Contextualized spectral interpretation (combining spectra with reaction context)
   • Novel evaluation requiring integration of spectroscopic data with reaction knowledge

Cross-Modal Synergy and Generalist Performance

Key Findings:

1. Leading Generalist Performance: ChemDFM-X establishes a new benchmark among existing generalist models (such as 3D-MOLM and ChemLLM), achieving performance metrics that match dedicated specialist models across several multimodal tasks.

2. Failure of General LMMs: General vision models (GPT-4O, Gemini 1.5 Pro, Qwen-VL, LLaVA, InternLM-XComposer2, DocOwl) failed significantly on chemical image recognition tasks (0% accuracy for most models on molecule and reaction recognition, Table 9), demonstrating that chemical domain knowledge cannot be assumed from general pre-training.

3. Cross-Modal Error Correction: In reaction image recognition, ChemDFM-X achieved higher accuracy (53.0%) than on single molecules (46.0%) (Table 9). The authors conclude the model uses its internal knowledge of chemical reaction rules to correct recognition errors in the visual modality, an emergent capability from multimodal training.

4. Reliance on Reaction Context for Spectra: In zero-shot scenarios, ChemDFM-X essentially fails at pure spectral recognition (achieving 0% and 1% top-1 accuracy on MS2 and IR spectra alone, Table 11). However, when SMILES-based reaction context is included, performance rises to 45% (MS2) and 64% (IR) on the reaction prediction task, and 29% (MS2) and 60% (IR) on retrosynthesis (Table 11). This indicates the model uses spectral data as a soft prior to constrain textual deductions. Furthermore, the paper compares ChemDFM-X’s spectral identification performance exclusively against text-only LLMs that cannot process spectra, omitting comparisons against established specialist tools.

5. Surrogate Distillation Trade-offs: Because the spectral training data relies entirely on outputs from CFM-ID 4.0 and Chemprop-IR, ChemDFM-X effectively distills these surrogate models. Any inherent predictive biases or inaccuracies from these underlying tools are permanently embedded in the new ChemDFM-X encoders.

Main Conclusion: The “separate encoders + unified decoder” architecture with synthetic data generation enables effective multimodal chemical understanding, bridging the gap between specialist and generalist AI systems for chemistry.

Reproducibility Details

Data

The authors constructed a 7.6M sample instruction-tuning dataset derived from 1.3M seed SMILES (sourced from PubChem and USPTO). Note: The final 7.6M multimodal tuning dataset itself isn’t publicly available.

Generation Pipeline:

Data Augmentation:

• Molecule images augmented with “handwritten” style using the ChemPix pipeline
• Multiple rendering styles (RDKit default, Indigo clean)
• Spectra generated at multiple energy levels (10eV, 20eV, 40eV for MS2)

Algorithms

Architecture: “Separate Encoders + Unified Decoder”

Code Availability: The authors have only released inference logic. The cross-modal projection training and synthetic data-generation scripts are closed.

Modality Alignment:

• Each modality has a dedicated encoder (frozen pre-trained models where available)
• For graph, conformation, MS2, and IR modalities: 2-layer MLP projector (Linear, GELU, Linear) maps encoder features to LLM input space
• For images: H-Reducer module compresses image tokens by factor of $n=8$ to handle high-resolution chemical images, then projects to LLM input space
• All projected features are concatenated and fed to the unified LLM decoder

Models

Base LLM:

• ChemDFM (13B): LLaMA-based model pre-trained on chemical text and SMILES

Modality Encoders:

Design Rationale: MS2 and IR encoders trained from scratch as Sequence Transformers treating spectral peaks as token sequences, since no suitable pre-trained models exist for chemical spectra.

Evaluation

Metrics:

• Accuracy (Acc) for recognition tasks
• BLEU-2/4 and METEOR for captioning tasks
• AUC-ROC for property prediction (classification)

Code Availability: The adapted code for evaluating on ChemLLMBench and their custom spectral recognition tasks is closed-source.

Benchmarks:

• ChemLLMBench: Adapted for multimodal inputs across molecule captioning, property prediction, and reaction understanding
• MoleculeNet: Standard molecular property prediction tasks (BACE, BBBP, ClinTox, HIV, Tox21)
• USPTO: Reaction prediction and retrosynthesis tasks
• Custom Spectral Tasks: Novel evaluations requiring spectral interpretation

Hardware

Note: The type and quantity of GPUs used, along with the total training wall-time, were not published.

Training Configuration:

• Total Batch Size: 256
• Epochs: 3
• Optimizer: AdamW

Modality-Specific Learning Rates (Peak):

Note: Different learning rates reflect the varying degrees of domain adaptation required. Images (general CLIP) need more adaptation than graphs (chemical Mole-BERT).

Artifacts

Paper Information

Citation: Zhao, Z., Chen, B., Li, J., Chen, L., Wen, L., Wang, P., Zhu, Z., Zhang, D., Wan, Z., Li, Y., Dai, Z., Chen, X., & Yu, K. (2024). ChemDFM-X: Towards Large Multimodal Model for Chemistry. Science China Information Sciences, 67(12), 220109. https://doi.org/10.1007/s11432-024-4243-0

Publication: Science China Information Sciences, December 2024

Additional Resources:

• arXiv Version
• Code Repository

@article{zhaoChemDFMXLargeMultimodal2024,
  title = {{{ChemDFM-X}}: {{Towards Large Multimodal Model}} for {{Chemistry}}},
  author = {Zhao, Zihan and Chen, Bo and Li, Jingpiao and Chen, Lu and Wen, Liyang and Wang, Pengyu and Zhu, Zichen and Zhang, Danyang and Wan, Ziping and Li, Yansi and Dai, Zhongyang and Chen, Xin and Yu, Kai},
  year = {2024},
  month = dec,
  journal = {Science China Information Sciences},
  volume = {67},
  number = {12},
  pages = {220109},
  doi = {10.1007/s11432-024-4243-0},
  archiveprefix = {arXiv},
  eprint = {2409.13194},
  primaryclass = {cs.LG}
}

This paper is a combination of Method (primary) and Resource (secondary).

It is primarily a Method paper because it proposes ChemVLM, a novel multimodal architecture specifically tailored for the chemical domain, utilizing a “ViT-MLP-LLM” framework. The authors introduce a specific two-stage training strategy to align visual features with chemical text representations.

Secondarily, it is a Resource paper as it introduces a comprehensive suite of three new datasets: ChemOCR, MMCR-Bench, and MMChemBench, developed to rigorously evaluate multimodal capabilities in chemistry, covering OCR, reasoning, and property prediction.

Bridging the Visual Gap in Chemical LLMs

The primary motivation is the limitation of existing models in handling the multimodal nature of chemistry.

• Visual Data Gap: Chemical tasks heavily rely on visual information (molecular structures, reactions) which purely text-based chemical LLMs cannot process.
• Limitations of Generalist Models: General multimodal models (like GPT-4V or LLaVA) lack specialized chemical domain knowledge, leading to hallucinations or misinterpretations.
• Inadequacy of OCR Tools: Traditional chemical OCR tools (like MolScribe) excel at modality conversion (Image-to-SMILES) but fail at complex reasoning tasks.

Domain-Specific Data Curation and Benchmarking

• Data-Driven Alignment: The underlying “ViT-MLP-LLM” framework is standard in multimodal modeling, paralleling architectures like LLaVA. The core innovation here is the rigorous creation of a bilingual multimodal dataset spanning hand-drawn molecules, reactions, and exam questions augmented with style transfers. The training data pipeline heavily relies on generating synthetic variance using tools like RanDepict and RDKit to introduce distortions, rotations, and handwritten styles, alongside GPT-4 generated prompts to ensure linguistic diversity.
• Model Integration: ChemVLM merges InternViT-6B (a large-scale vision transformer) with ChemLLM-20B (a chemical language model). Visual features $X_v$ are mapped into the linguistic embedding space via an MLP projector, producing aligned token sequences alongside text instructions $X_q$. The joint multimodal sequence is trained using standard autoregressive next-token prediction: $$ \mathcal{L} = -\sum_{i} \log P(y_i \mid X_v, X_q, y_{<i}) $$
• Three Custom Benchmarks: The authors introduce tailored benchmarks to assess distinct competencies:
  • ChemOCR: For image-to-SMILES conversion.
  • MMCR-Bench: College entrance exam questions testing complex logical reasoning.
  • MMChemBench: For molecule captioning and zero-shot property prediction.

Evaluating Chemical OCR and Reasoning

The authors benchmarked ChemVLM against both open-source (LLaVA, Qwen-VL, InternVL) and proprietary (GPT-4V) models across three primary domains:

1. Chemical OCR: Evaluated on 1,000 image-text pairs from ChemOCR. The primary metric is the Tanimoto similarity between the Morgan fingerprints of the generated structure ($A$) and the ground-truth SMILES ($B$): $$ T(A, B) = \frac{|A \cap B|}{|A| + |B| - |A \cap B|} $$ They report both the average Tanimoto similarity and the strict exact-match rate (Tanimoto@1.0).
2. Multimodal Chemical Reasoning (MMCR): Tested on MMCR-Bench (1,000 exam questions), ScienceQA, and CMMU. Performance was scored based on accuracy for multiple-choice and fill-in-the-blank questions.
3. Multimodal Molecule Understanding: Evaluated on MMChemBench for molecule captioning and property prediction.
4. Text-Only Reasoning: Tested on SciBench, a text-only benchmark for university-level science, to ensure the model retains fundamental linguistic reasoning.
5. Generalization: Tested on non-chemistry subjects within the CMMU framework (Biology, Physics, Math) to assess cross-domain competence.

Performance Gains and Existing Limitations

• Multimodal Reasoning Leadership: ChemVLM achieved state-of-the-art results on MMCR-Bench (41.7%), surpassing generalist models like GPT-4V (40.1%). However, scoring for portions of these benchmarks relied heavily on an LLM-as-a-judge (the Qwen-max API), which can introduce bias as LLM evaluators often favor structural characteristics and verbosity produced by similar autoregressive models. Furthermore, the model was fine-tuned on 200,000 exam questions and tested on MMCR-Bench (also derived from Chinese college entrance exams). While the authors state the data was deduplicated, the potential for data leakage remains a significant unaddressed confounder.
• Superior Understanding: In molecule captioning and prediction, ChemVLM showed significant improvements over general baseline models, scoring 80.9% on prediction compared to GPT-4V’s 38.6%. This is a natural consequence of testing a custom-trained model on domain-specific benchmarks.
• OCR Capabilities vs. Dedicated Tools: ChemVLM outperformed generalist MLLMs in chemical structure recognition, achieving an average Tanimoto similarity of 71.0% (vs. GPT-4V’s 15.0%). However, it remains significantly inferior to pure structural OCR tools like MolScribe in strict modality conversion tasks, only achieving an exact structural match (Tanimoto@1.0) of 42.9% compared to MolScribe’s 89.1%.
• Textual Retention and Generalization Claims: The authors claim the diverse training strategy imparts broad scientific reasoning, pointing to performance retention on non-chemistry subjects (Biology, Physics, Math) and strong results on the purely textual SciBench benchmark. However, this cross-domain generalization highly likely stems from the underlying base model (ChemLLM-20B/InternLM2) or the inclusion of 1.3 million “General” visual QA pairs in their training blend, rather than emergent general scientific skills originating purely from learning chemistry representations.

Reproducibility Details

Data

The training and evaluation data relied on a mix of open-source repositories and custom curation. Many of the curated datasets have been formally released by the authors on Hugging Face (di-zhang-fdu/chemvlm-sft-datasets).

Preprocessing: Images were augmented using RanDepict for style variation. Text data (SMILES) was validated and cleaned. Prompts were diversified using GPT-4 to generate different linguistic styles.

Algorithms

• Architecture: “ViT-MLP-LLM” structure.
  • Vision Encoder: InternViT-6B, processing images at $448 \times 448$ resolution. Images are segmented into tiles (max 12).
  • Projector: Multi-Layer Perceptron (MLP) initialized randomly to map visual features to text embedding space.
  • LLM: ChemLLM-20B, a domain-specific model.
• Training Strategy: Two-stage supervised fine-tuning.
  1. Modal Alignment: Freeze LLM and base Vision Encoder weights. Train only the randomly initialized MLP projector and LoRA layers (rank 32) of the Vision Encoder. Uses diverse multimodal data.
  2. Supervised Fine-Tuning (SFT): Keep LLM and Vision Encoder base weights frozen, but add LoRA (rank 16) to the LLM and retain LoRA (rank 32) on the Vision Encoder. The MLP projector is fully trained. Data includes specialized chemistry and general corpora.
• Optimization:
  • Optimizer: AdamW
  • Context Length: 2048 tokens
  • Chat Template: InternLM2 dialogue schema

Models

• ChemVLM-26B: The primary model released. It combines the 6B parameter vision encoder and the 20B parameter language model. Weights are fully available at AI4Chem/ChemVLM-26B-1-2. An 8B version is also available.
• Baselines: Comparisons were made against GPT-4V, Qwen-VL-Chat, LLaVA-v1.5-13B, InternVL-v1.5, and Yi-VL-Plus.

Evaluation

Performance was measured across three distinct task types. Exact evaluation scripts have been released in the official repository.

Hardware

• Training Compute: Training utilized 16 NVIDIA A100 (80GB) GPUs.
• Configuration:
  • Batch size: 4 (per GPU, resulting in an effective global batch size of 256)
  • Gradient Accumulation: 4 iterations
  • Precision: Deepspeed bfloat16 (bf16) with ZeRO-3 offloading strategy
  • Framework: Training runs on the InternVL-v1.5 codebase rather than standalone scripts.
• Inference Compute: Evaluating the 26B model requires at least one 80GB A100 GPU (with Flash Attention + bfloat16). The 8B variant requires a GPU with at least 48GB of VRAM.

Artifacts

Paper Information

Citation: Li, J., et al. (2025). ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area. Proceedings of the AAAI Conference on Artificial Intelligence, 39(1), 415-423. https://doi.org/10.1609/aaai.v39i1.32020

Publication: AAAI 2025

@inproceedings{li2025chemvlm,
  title={ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area},
  author={Li, Junxian and Zhang, Di and Wang, Xunzhi and Hao, Zeying and Lei, Jingdi and Tan, Qian and Zhou, Cai and Liu, Wei and Yang, Yaotian and Xiong, Xinrui and Wang, Weiyun and Chen, Zhe and Wang, Wenhai and Li, Wei and Su, Mao and Zhang, Shufei and Ouyang, Wanli and Li, Yuqiang and Zhou, Dongzhan},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={39},
  number={1},
  pages={415--423},
  year={2025},
  url={https://doi.org/10.1609/aaai.v39i1.32020},
  doi={10.1609/aaai.v39i1.32020}
}

Additional Resources:

• Official Repository