Where to cut, how deep is a controlled comparison of the two dominant subword tokenizers, byte-pair encoding (BPE) and Unigram-LM, over chemistry SMILES. Holding the corpus, the OpenSMILES glyph base, and the target vocabulary size fixed, the study asks whether the two algorithms learn the same vocabulary. They do not: across every matched condition they build near-disjoint sets of multi-glyph pieces, the same atoms carved into almost none of the same subwords.

Abstract

Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE’s principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open: the complete glyph base already covers every conformant molecule, so the learned pieces add compression rather than coverage, and a tiny alphabet under hard valence constraints could drive two frequency-based algorithms to converge. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces above the shared base never exceeds $0.161$, and at most $0.05$ once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into $29$–$41%$ more tokens; the arms largely agree on where to cut but not how deeply, so BPE’s segmentation is a strict coarsening of Unigram-LM’s on $80$–$99%$ of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale, past where embeddings remain learnable; only token-frequency imbalance attenuates in magnitude, shrinking with vocabulary size and most on the natural-products corpus, without closing. The subword algorithm is therefore a modeling decision, not a free default. We release all trained tokenizers and per-condition measurements.

Key Contributions & Results

  • Near-disjoint vocabularies. Across all 22 matched conditions, cross-algorithm Jaccard overlap on the learned multi-glyph pieces never exceeds $0.161$, and drops to at most $0.05$ once weighted toward the high-frequency pieces a model updates most. The two algorithms share the glyph base and almost none of the subwords built on top of it.
  • Same cuts, different depth. Unigram-LM segments held-out molecules into $29$–$41%$ more tokens. The two arms largely agree on where to cut but disagree on how deeply, so BPE’s segmentation is a strict coarsening of Unigram-LM’s on $80$–$99%$ of molecules. The disagreement is nesting, not conflict.
  • The separation is structural. It holds across corpus typology, boundary policy, and vocabulary size, persisting even at $8\times$ the headline vocabulary, past the regime where token embeddings remain learnable. Only token-frequency imbalance attenuates in magnitude (shrinking with vocabulary size, most on the natural-products corpus) without closing.
  • No language models trained. Every result is a property of the tokenizers and the corpora, so the comparison is exactly controlled. The claim is about vocabularies, not downstream task scores.
  • Released artifacts. All trained tokenizers and per-condition measurements are archived for re-analysis on Zenodo.

Study Design

  • Controlled grid. The comparison fixes a 165-token OpenSMILES glyph base, then trains BPE and Unigram-LM to matched target vocabulary sizes at the small scales where token embeddings stay learnable. Corpus, glyph base, and target size are held constant so any divergence is attributable to the algorithm.
  • Three corpus typologies. Diverse, drug-like, and natural-products corpora span the chemistry a SMILES tokenizer is likely to meet, testing whether the divergence is a property of one chemical distribution or of the algorithms themselves.
  • Both boundary policies. The grid runs under both pre-tokenization boundary policies, separating the effect of the subword algorithm from the effect of where pre-tokenization allows pieces to form.
  • Mechanism diagnostics. The study reports chemistry-side measurements, previously unreported for SMILES, of three mechanism diagnostics carried from the natural-language literature (dead-zone surplus, scaffold fraction, and whole-pretoken absorption), alongside a structural-character and non-canonicity battery that localizes where the two arms diverge.

Interactive Demo

The live demo takes any molecule, aligns both algorithms on the shared glyph stream, and surfaces the paper’s three contrasts directly: membership (near-disjoint vocabularies), granularity (Unigram-LM stays near-atomic and emits more tokens), and compatibility (the disagreement is nesting, so BPE’s parse is usually a coarsening of Unigram-LM’s). It runs on free CPU: the tokenizers are re-implemented in pure Python and validated byte-faithful against the reference Smirk tokenizer.

Nicotine and serotonin under the two algorithms' V=1024 vocabularies, with BPE building larger pieces and Unigram-LM emitting more, smaller ones
Nicotine and serotonin under the two algorithms’ V=1024 vocabularies. Darker shading marks a larger piece. BPE (blue) builds a few large pieces spanning whole rings and chains; Unigram-LM (orange) stays near-atomic and emits more tokens per molecule (+5 and +7 here).

Why This Matters

Chemical language models inherit BPE from natural language as an unexamined default. This study shows that the choice of subword algorithm changes the learned vocabulary itself, not merely the compression ratio: BPE and Unigram-LM hand a model different pieces to embed and different segment counts to attend over. That makes the tokenizer a modeling decision to be reported and controlled, the same way an architecture or an objective is, rather than a free default carried over without scrutiny. The released tokenizers and per-condition measurements let others re-analyze the comparison or extend it to downstream training.

Resources

  • Paper (arXiv): Full technical report (ChemRxiv identifier to follow).
  • Code (GitHub): The measurement and pipeline package, reproducible from SHA-pinned corpora.
  • Interactive Demo: See both tokenizers carve up any molecule, in-browser.
  • Artifacts (Zenodo): Trained tokenizers and full measurement deposits.

Citation

@misc{heidenreich2026neardisjoint,
  title        = {{Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES}},
  author       = {Heidenreich, Hunter},
  year         = {2026},
  eprint       = {2607.05691},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  doi          = {10.5281/zenodo.21228245},
  url          = {https://arxiv.org/abs/2607.05691}
}