Molecular Encoders

This group covers models designed primarily to learn fixed-dimensional molecular representations from chemical string notations (SMILES, SELFIES, InChI). These encoders serve as feature extractors for downstream tasks like property prediction, virtual screening, and molecular similarity search. For models that fuse molecular strings or graphs with additional modalities (text, property vectors, knowledge graphs), see Multimodal Molecular Models.

Pre-trained Encoder Models

Paper	Year	Architecture	Key Idea
Seq2seq Fingerprint	2017	GRU encoder-decoder	Unsupervised fingerprints learned from SMILES translation
Mol2vec	2018	Word2vec	Substructure embeddings inspired by NLP word vectors
CDDD	2019	Encoder-decoder	Descriptors learned by translating between SMILES and InChI
SMILES-BERT	2019	BERT	Masked pretraining on 18M+ SMILES from ZINC
SMILES Transformer	2019	Transformer	Unsupervised transformer fingerprints for low-data tasks
ChemBERTa	2020	RoBERTa	RoBERTa evaluation on 77M PubChem SMILES
MolBERT	2020	BERT	Domain-relevant auxiliary task pretraining
X-MOL	2020	Transformer	Large-scale pretraining on 1.1 billion molecules
ChemBERTa-2	2022	RoBERTa	Scaling to 77M molecules, comparing MLM vs MTR objectives
MoLFormer	2022	Linear-attention Transformer	Linear attention at 1.1B molecule scale
SELFormer	2023	RoBERTa	SELFIES-based pretraining on 2M ChEMBL molecules
BARTSmiles	2024	BART	Denoising pretraining on 1.7B SMILES from ZINC20
SMI-TED	2025	Encoder-decoder Transformer	Foundation model trained on 91M PubChem molecules
ChemBERTa-3	2026	RoBERTa	Open-source scalable chemical foundation models

Scaling, Evaluation & Surveys

Paper	Year	Key Idea
Neural Scaling of Deep Chemical Models	2023	Power-law scaling relations for chemical LMs and GNNs
Systematic Review of Deep Learning CLMs (2020-2024)	2024	PRISMA review of 72 papers on chemical language models
Transformer CLMs for SMILES: Literature Review 2024	2024	Review of transformer-based CLMs operating on SMILES
Survey of Transformer Architectures in Molecular Science	2024	Survey of 12 transformer architecture families in molecular science
AMORE	2025	Testing ChemLLM robustness to SMILES variants

Machine Learning Fundamentals

Log-log plots showing power-law scaling of ChemGPT validation loss versus model size and GNN force field loss versus dataset size

Neural Scaling of Deep Chemical Models

Frey et al. discover empirical power-law scaling relations for both chemical language models (ChemGPT, up to 1B parameters) and equivariant GNN interatomic potentials, finding that neither domain has saturated with respect to model size, data, or compute.

Computational Chemistry

BARTSmiles ablation study summary showing impact of pre-training strategies on downstream task performance

BARTSmiles: BART Pre-Training for Molecular SMILES

BARTSmiles pre-trains a BART-large model on 1.7 billion SMILES strings from ZINC20 and achieves the best reported results on 11 classification, regression, and generation benchmarks.

Computational Chemistry

MoLFormer-XL architecture diagram showing SMILES tokens flowing through a linear attention transformer to MoleculeNet benchmark results and attention-structure correlation

MoLFormer: Large-Scale Chemical Language Representations

MoLFormer is a transformer encoder with linear attention and rotary positional embeddings, pretrained via masked language modeling on 1.1 billion molecules from PubChem and ZINC. MoLFormer-XL outperforms GNN baselines on most MoleculeNet classification and regression tasks, and attention analysis reveals that the model learns interatomic spatial relationships directly from SMILES strings.

Computational Chemistry

SELFormer architecture diagram showing SELFIES token input flowing through a RoBERTa transformer encoder to molecular property predictions

SELFormer: A SELFIES-Based Molecular Language Model

SELFormer is a transformer-based chemical language model that uses SELFIES instead of SMILES as input. Pretrained on 2M ChEMBL compounds via masked language modeling, it achieves strong classification performance on MoleculeNet tasks, outperforming ChemBERTa-2 by ~12% on average across BACE, BBBP, and HIV.

Computational Chemistry

ChemBERTa-3 visualization showing muscular arms lifting a stack of building blocks representing molecular data with SMILES notation, symbolizing the power and scalability of the open-source training framework

ChemBERTa-3: Open Source Chemical Foundation Models

ChemBERTa-3 provides a unified, scalable infrastructure for pretraining and benchmarking chemical foundation models. It addresses reproducibility gaps in previous studies like MoLFormer through standardized scaffold splitting and open-source tooling.

Computational Chemistry

ChemBERTa-2 visualization showing flowing SMILES strings in blue tones representing molecular data streams

ChemBERTa-2: Scaling Molecular Transformers to 77M

This work investigates the scaling hypothesis for molecular transformers, training RoBERTa models on 77M SMILES from PubChem. It compares Masked Language Modeling (MLM) against Multi-Task Regression (MTR) pretraining, finding that MTR yields better downstream performance but is computationally heavier.

Computational Chemistry

ChemBERTa masked language modeling visualization showing SMILES string CC(=O)O with masked tokens

ChemBERTa: Molecular Property Prediction via Transformers

This paper introduces ChemBERTa, a RoBERTa-based model pretrained on 77M SMILES strings. It systematically evaluates the impact of pretraining dataset size, tokenization strategies, and input representations (SMILES vs. SELFIES) on downstream MoleculeNet tasks, finding that performance scales positively with data size.

Pre-trained Encoder Models#

Scaling, Evaluation & Surveys#

Pre-trained Encoder Models

Scaling, Evaluation & Surveys