Measuring it requires finding bills that share text. Political scientists do this to estimate how policy diffuses between states; journalists have done it to identify model bills moving between statehouses. A method everyone in this lineage uses is the one biologists use to align gene sequences (Smith-Waterman local alignment): take two bills, find the longest run of matching words, score it for similarity.

Because this is unsupervised, you’ll need a cutoff. Score everything against everything and you have hundreds of millions of bill pairs with a number attached; somewhere on that scale is the line between coincidence and copying. Where you put that line is where every downstream finding comes from, because every count of how much two states share is computed over the pairs above it.

Here is the scale, and here is a line:

The line sits on the steep slope rather than out in the quiet part of the distribution, so moving it a little moves the population a lot: 53 million pairs sit in the bucket immediately below it.

Data and a Threshold

This data is from a CC0 corpus of state bills. The accompanying paper’s conclusion advises future scholars to work in the upper quantiles, and says that for searching out bill pairs “this can be done through thresholding on high alignment scores (e.g., 50+) to indicate the presence of highly similar policy proposals in legislation.” That said, they acknowledge that such thresholding “may discard informative variation,” and assert that high-score thresholding “would focus the analysis on bill pairs that reliably contain very similar policy enactments, and pose little risk of a false positive.”

That is a balanced statement about high-score thresholding in general, not necessarily a recommendation of 50. Read strictly, that advice is about searching out bill pairs: producing candidates worth a human’s attention. Anybody who opens a candidate and finds a PostScript block will throw it away, which is a fair defence of the sentence as written.

Unfortunately, it stops working at scale. The cutoff selects 2,968,117 pairs, which is not a set anybody reads, and the point of a scored corpus is that the score stands in for the reading. So the question is what the cutoff hands you when nobody looks.

I classified the aligned text of every one of those 2,968,117 pairs. 785,330 of them, 26.46%, are not legislative text at all.

What Is Actually Up There

They are page furniture: the line numbers printed down a page margin, the rule that separates a bill’s summary from its body, and a block of PDF printer settings that was never meant to be read by anyone. Text extraction turned them into words. Two bills produced by the same software then “share” those words, and the aligner scores it like any other match.

That is the part I did not expect. The artifact is not sprinkled through the corpus and then swept up incidentally by the cutoff. It is concentrated almost entirely in the region the cutoff selects. A block of printer settings shared between two documents is the most perfect match in the whole corpus.

It Manufactures Edges

If the artifact were spread evenly it would inflate a denominator and leave the ranking of state pairs alone. It is not spread evenly. A quirk of drafting or scanning software belongs to a jurisdiction, so the damage lands on the pairs that jurisdiction belongs to.

Ranked by how many above-threshold pairs they share, the most connected pair of states in this data is Arizona and Nevada, by a factor of nine over the runner-up. 99.9% of those pairs are two PDFs agreeing about their printer settings.

Strip the artifacts and Arizona-Nevada falls from rank 1 to rank 754 of 1,247. Nine of the top twenty dyads leave the top 100.

Two caveats on that table:

• Ranking by raw pair count rewards states with many bills and good text coverage, and coverage in this corpus varies by 100 percentage points across states. This means the ranking is volume-sensitive before any artifact enters it.
• The removal column only subtracts, so it cannot show a dyad crossing into the top 100 from underneath.

And one thing I am not claiming. This ranking is mine. The paper’s own dyadic validation regresses an aggregate alignment score on diffusion ties across 1,225 state pairs; it does not rank dyads by above-threshold count, and I have not re-estimated any published result. I would expect the contamination to survive a change of statistic, since 460,447 pairs at roughly 440 apiece is a great deal of mass in any sum over az-nv, but I have not run that version and it is not what the census shows.

What the census shows is narrower and needs no model: threshold this deposit at a number its own paper offers, and the highest-degree edge in the resulting network is an extraction artifact.

Where It Comes From

The blocks are not spread across the corpus. Scanning all 571,554 bills finds them in 3,551 Arizona bills, 3,178 Nevada bills, one Delaware bill, and nowhere else. So I fetched the original PDFs from both legislatures and re-extracted them.

These PDFs carry an embedded file attachment holding the Acrobat Distiller job options used to produce them, Flate-compressed inside a /Type /EmbeddedFile stream. For one Arizona bill, decompressing that attachment gives 14,581 bytes against the 14,580 characters sitting in the corpus for that same bill. Opening <<, closing >> setpagedevice, indentation and all 220 newlines included. The two are identical once the attachment’s trailing newline comes off.

The corpus’s text extraction emitted PDF file attachments as document text. Not a font problem, not an encoding problem, and not a property of the source documents. pdftotext walks page content and ignores attachments, which is why it reproduces the block 0 times in 92 from documents that demonstrably contain it.

Two details make this worse than a scoring problem. Smith-Waterman is not a cheap algorithm to run. So to reduce the number of bills one runs this alignment scoring for, resesarchers first perform a coarse retrieval stage to pre-filter comparisons to those with the highest similarity as determined by bag-of-words methods (like BM-25). The block also wins the retrieval: a bill carrying one spends 15.00% of its candidate slots on other bills carrying one, against a 1.36% base rate, and a candidate that retrieval never returned was never scored at all.

The paper anticipated the general worry. Bills are compared against the 500 nearest neighbours Elasticsearch returns, and the authors checked that cap by re-running at 1,000, concluding it introduces no selection bias “because the set of bills that result in modest or large text reuse scores is always smaller than 500.” That reasoning holds when the top 500 is filled by candidates competing on legislative language. What the crowding number shows is a population where the premise is doubtful: across the 6,700 queries carrying a distiller block, an elevenfold enrichment means a large share of those slots is won on a string that is not language at all. I have not re-run the authors’ check, so this is a reason to think it deserves re-running on the artifact-carrying subset, not a finding that it fails.

And the bias cannot be signed. A distiller block is appended after the page text, so excising it can only lower a score. A line-number run sits between two passages of prose, so excising it joins them and the score rises, by up to 63 points. The artifact invents reuse in the pairs it creates and conceals reuse in the pairs it interrupts.

The Problem No Cleaning Fixes

Everything above is a defect. A better extractor and a screening pass fix all of it, and anything built on stripped text is recoverable.

The pairs that survive every screen have a separate problem, and this one is a property of the measure rather than of this corpus. It survives perfect text.

Two states implementing the same federal statute share text word for word, and neither one copied the other. The alignment is exact. The reuse is real. It just does not run between the two states in the edge.

Who Did the “Reading”

I tried a language model first. MiniMax M3 labelled every prose pair in my sample, three times over, and on the 477 where all three passes parsed, its verdicts flipped on 33.5%.

Where it broke is the useful part. The instability was worst exactly at the bar, around 57% stable in the 50 to 99 band against 100% above 500, which left two readings the run could not separate: either the model was unreliable on hard cases, or pairs near the bar are genuinely undecidable from the text.

So I read 80 of the pairs again on a blinded page, stratified toward the 50 to 99 band where the instability lived, with score, bill ids and jurisdictions hidden. And so did Claude Opus, on the identical page. Whereas M3 had been the arms-length instrument, prompted and run in batch, Claude was the model I had been working the audit with in Claude Code. It knew my hypotheses and had already seen some of these spans.

The agreement number below is not an inter-rater statistic. It is a person and a model that shared a hypothesis, labelling the same page. I am reporting it because the structure of the disagreement turned out to matter more than its size, and because the answer to the question the M3 run could not settle came out of it: agreement held at 77.4% in the 50 to 99 band, roughly flat where M3’s own stability had collapsed. Pairs near the bar are decidable. The unreliable instrument was the model that kept changing its mind.

No reading makes this class mostly boilerplate. The stricter reading still puts two thirds of it on unambiguous shared source. The 11% both call generic is what you would guess: enacting clauses, appropriation formulae, quorum boilerplate, bill titles on unrelated subjects. The pairs that survive the artifact screen are, in the main, real reuse.

The third row is the finding and the weakest row in the table. Every pair I called shared source, Claude did too. The split is not about where a line was drawn. Instead, the model could name an upstream source and I could not. Text you cannot place reasonably looks like it might be generic.

The Names Are the Result, Not the Labels

A name is checkable in a way a label is not.

Claude named 37 upstream sources across the 80 pairs. I checked eight of them against published law, choosing the ones the finding leans on hardest:

Seven verbatim, one partial, none refuted, and all eight sit in the disputed column. That is the result. Text that a careful reader calls generic formula is sometimes a named federal act reproduced word for word, and unlike every similarity score in this corpus, that is a claim a stranger can destroy with a search engine.

The twenty-nine unchecked names are not evidence, and the reason is on the record. On a control pair reading this act shall be known and may be cited as the school protection act, M3 confidently named an ALEC model act, the Liberty Preservation Act, which has nothing to do with it. A model asked to name a source will produce a plausible one whether or not it exists. The eight are worth something precisely because they were checked; the rest are leads.

The Part That Survives a Clean Corpus

A threshold on text similarity is selecting two different phenomena and cannot tell them apart. States adopting a private model act is policy diffusion, which is what the measure was built to find. States independently implementing the same federal statute is mandated uniformity, which is a different causal story with the same textual signature. The alignment is equally exact in both cases, which is precisely why no score separates them. A higher threshold does not help. A better aligner does not help. Cleaning the text does not help, because there is nothing wrong with the text.

And the error has structure, which is what makes it bite. States differ in how much federal law they implement and how fast, so a diffusion network built on this measure counts federal conformity as interstate influence, and it does so unevenly across exactly the dyads people draw conclusions from.

None of which is news to the diffusion literature. Vertical influence, federal mandates and common exposure are long-standing confounds in that field, and there are methods built to handle them. The narrow claim is the one worth making, and it is about the instrument: a text-reuse score offers no purchase on the distinction at all, because the two cases have an identical textual signature. Whatever a researcher does about common exposure, they have to do it from outside this measure, and a network drawn straight from alignment scores carries the confound with no way to see it.

The fix is a labelled source: to know whether copying ran between the two jurisdictions in your edge, you have to know what was copied.

How Far That Goes, and How Far It Does Not

The sample is not the population. The 80 pairs were drawn to over-represent the 50 to 99 band, so the 23% is a rate within a deliberately skewed draw and I would not quote it as a corpus rate. Read it as context for the eight verified names rather than as the finding.

There is one human in this design. Me, not blind, holding the hypothesis while labelling. Everything else in the reading pipeline was a model. What that design can establish is bounded, and what it did establish is the checkable half: eight names, seven exact.

What the 80 pairs support is that the distinction exists, that it is not rare, and that where it has been tested it holds. How large it is corpus-wide is unmeasured, and a second human annotator who had never read this audit is the thing that would change it. That is not a run I can do, so the blinded page and both sets of labels ship with the code, and somebody else can.

So, Would a Language Model Have Helped?

I came at this wanting to know whether a language model was useful for legislative text reuse. Having spent the time, the answer is yes, and not where I expected.

The contamination needed no model at all. It is a defect: a few dozen lines of pattern matching find every class, removal is lossless because each one is self-delimiting, and anything built on stripped text is recoverable.

The uniformity confound is the opposite. It is not removable at any threshold and a better similarity score will not touch it. What would separate it is knowing what was copied, which is a provenance question, and provenance is the kind of question a language model is actually good at.

It was the one thing both models did well here. Asked to name upstream sources, M3 produced the Uniform Controlled Substances Act, the Streamlined Sales and Use Tax Agreement, the Uniform Vehicle Code, 18 U.S.C. 2510, and those names were more durable than its own verdicts, agreeing across passes 77% of the time against 66.5% for the labels. None of that batch was checked against published law. The eight that were checked came from the blinded run and seven were exact.

With the condition from the section above attached, and it is not a small one: the behaviour that produces a correct name also produces the ALEC misfire, and nothing in the output tells you which one you are holding. Only a check does.

That is the whole thing in miniature:

A similarity label is an opinion. A named source is a claim. Only the second kind can be refuted.

A wrong name announces itself the moment somebody checks it. A wrong similarity score is invisible forever. The second failure mode is the one you want.

So the honest next artifact here is not a better detector. It is a labelled source corpus, built by something that can propose a name and verified by somebody who checks it.

If You Use This Deposit

Strip all three classes before thresholding. Each is self-delimiting, so removal is exact rather than heuristic.

Three things that are easy to get wrong. Do not just exclude Arizona and Nevada, because the other two classes cover more dyads than the distiller block does. Treat Hawaii separately, because its text is OCR-damaged underneath the artifact, that damage is what its source documents actually say, and neither removal nor re-extraction fixes it. And before removing a run, check that it ascends by one, because a run that ascends on every second position is a statutory rate schedule with a counter threaded through it, and excising it deletes enacted numbers.

Do not trust a high threshold to be safe either. The most reassuring number here, that the band above 500 is 99.1% clean, is an accident: the distiller blocks stop scoring at 442 only because Arizona ran Distiller 8.2.5 and Nevada ran 6.0, so the two job-options files agree only through their shared prefix. Two jurisdictions on the same version would share a byte-identical 14,500-character block, which aligns into the thousands.

The Full Audit

The writeup behind all of this is a technical report: the four independent instruments, the removal experiment and its label-blind control, the scrape-era census that explains why the damage partitions by legislative session rather than by state, the limitations including a cluster bootstrap that widens my own headline interval by a factor of five, and an appendix of the four things earlier drafts got wrong. It is at REPORT.md in the page-furniture repo, alongside the screen itself and the dated results file behind every number here.

The 80-pair adjudication is in there too. The per-pair record has every pair, both labels, the upstream act Claude named where it named one, and which eight I checked against published law. The 29 unchecked names are published as well, because a name nobody has checked is a lead, and I would rather ship it as one than quietly drop it.

Attribution

The corpus was collected by Open States and assembled by Burgess et al. It is CC0; US legislative text is uncopyrightable at origin under Georgia v. Public.Resource.Org (2020).

Burgess, Matthew, Eugenia Giraudy, Julian Katz-Samuels, Joe Walsh, Derek Willis, Lauren Haynes, and Rayid Ghani. “The Legislative Influence Detector: Finding Text Reuse in State Legislation.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 57-66, 2016, doi:10.1145/2939672.2939697.

The deposit measured here is Linder, Fridolin J., “Replication Data for: Measuring Policy Similarity Through Bill Text Reuse”, Harvard Dataverse, doi:10.7910/DVN/CZ25GF, accompanying Linder, Desmarais, Burgess and Giraudy, “Text as Policy: Measuring Policy Similarity through Bill Text Reuse”, Policy Studies Journal, doi:10.1111/psj.12257. The prior art is Citron, Daniel T. and Paul Ginsparg, “Patterns of text reuse in a scientific corpus”, PNAS 112(1):25-30, 2015, doi:10.1073/pnas.1415135111.

This is a Systematization paper ($\Psi_{\text{Systematization}}$). It organizes existing literature into two distinct evolutionary phases: Rule-based systems (1990s-2010s) and Machine Learning-based systems (2015-present). It synthesizes performance metrics across these paradigms to highlight the shift from simple classification to “image captioning” (sequence generation).

Justification: The paper focuses on “organizing and synthesizing existing literature” and answers the core question: “What do we know?” The dominant contribution is systematization based on several key indicators:

1. Survey Structure: The paper explicitly structures content by categorizing the field into two distinct historical and methodological groups: “Rule-based systems” and “ML-based systems”. It traces the “evolution of approaches from rule-based structure analyses to complex statistical models”, moving chronologically from early tools like OROCS and OSRA (1990s-2000s) to modern Deep Learning approaches like DECIMER and Vision Transformers.

2. Synthesis of Knowledge: The paper aggregates performance metrics from various distinct studies into unified comparison tables (Table 1 for rule-based and Table 2 for ML-based). It synthesizes technical details of different models, explaining how specific architectures (CNNs, LSTMs, Attention mechanisms) are applied to the specific problem of Optical Chemical Structure Recognition (OCSR).

3. Identification of Gaps: The authors dedicate specific sections to “Gaps of rule-based systems” and “Gaps of ML-based systems”. It concludes with recommendations for future development, such as the need for “standardized datasets” and specific improvements in image augmentation and evaluation metrics.

Motivation for Digitization in Cheminformatics

The primary motivation is the need to digitize vast amounts of chemical knowledge locked in non-digital formats (e.g., scanned PDFs, older textbooks). This is challenging because:

1. Representational Variety: A single chemical formula can be drawn in many visually distinct ways (e.g., different orientations, bond styles, fonts).
2. Legacy Data: Older documents contain noise, low resolution, and disconnected strokes that confuse standard computer vision models.
3. Lack of Standardization: There is no centralized database or standardized benchmark for evaluating OCSR performance, making comparison difficult.

Key Insights and the Paradigm Shift

The paper provides a structured comparison of the “evolution” of OCSR, specifically identifying the pivot point where the field moved from object detection to NLP-inspired sequence generation.

Key insights include:

• The Paradigm Shift: Identifying that OCSR has effectively become an “image captioning” problem where the “caption” is a SMILES or InChI string.
• Metric Critique: It critically analyzes the flaws in current evaluation metrics, noting that Levenshtein Distance (LD) is better than simple accuracy but still fails to capture semantic chemical severity (e.g., mistaking “F” for “S” is worse than a wrong digit).
• Hybrid Potential: Despite the dominance of ML, the authors argue that rule-based heuristics are still valuable for post-processing validation (e.g., checking element order, sequence structure, and formula correspondence).

Comparative Analysis of Rule-Based vs. ML Systems

As a review paper, it aggregates experimental results from primary sources. It compares:

• Rule-based systems: OSRA, chemoCR, Imago, Markov Logic OCSR, and various heuristic approaches.
• ML-based systems: DECIMER (multiple versions), MSE-DUDL, ICMDT (Image Captioning Model based on Deep Transformer-in-Transformer), and other BMS Kaggle competition solutions.

It contrasts these systems using:

• Datasets: BMS (synthetic, 4M images), PubChem (synthetic), U.S. Patents (real-world scanned).
• Metrics: Tanimoto similarity (structural overlap) and Levenshtein distance (string edit distance).

Outcomes, Critical Gaps, and Recommendations

1. Transformers are SOTA: Attention-based encoder-decoder models outperform CNN-RNN hybrids. DECIMER 1.0 achieved 96.47% Tanimoto $= 1.0$ on its test set using an EfficientNet-B3 encoder and Transformer decoder.
2. Data Hungry: Modern approaches require massive datasets (millions of images) and significant compute. DECIMER 1.0 trained on 39M images for 14 days on TPU, while the original DECIMER took 27 days on a single GPU. Rule-based systems required neither large data nor heavy compute but hit a performance ceiling.
3. Critical Gaps:
   • Super-atoms: Current models struggle with abbreviated super-atoms (e.g., “Ph”, “COOH”).
   • Stereochemistry: 3D information (wedges/dashes) is often lost or misinterpreted.
   • Resolution: Models are brittle to resolution changes; some require high-res, others fail if images aren’t downscaled.
4. Recommendation: Future systems should integrate “smart” pre-processing (denoising without cropping) and use domain-specific distance metrics. The authors also note that post-processing formula validation (checking element order, sequence structure, and formula correspondence) increases accuracy by around 5-6% on average. They suggest exploring Capsule Networks as an alternative to CNNs, since capsules add position invariance through routing-by-agreement rather than max-pooling.

Reproducibility

As a review paper, this work does not introduce original code, models, or datasets. The paper itself is open access via the Journal of Cheminformatics. This section summarizes the technical details of the systems reviewed.

Data

The review identifies the following key datasets used for training OCSR models:

Algorithms

The review highlights the progression of algorithms:

• Rule-Based: Hough transforms for bond detection, vectorization/skeletonization, and OCR for atom labels.
• Sequence Modeling:
  • Image Captioning: Encoder (CNN/ViT) → Decoder (RNN/Transformer).
  • Tokenization: Parsing InChI/SMILES into discrete tokens (e.g., splitting C13 into C, 13).
  • Beam Search: Used in inference (typical $k=15-20$) to find the most likely chemical string.

Models

Key architectures reviewed:

• DECIMER 1.0: Uses EfficientNet-B3 (Encoder) and Transformer (Decoder). Predicts SELFIES strings (more robust than SMILES).
• Swin Transformer: Often used in Kaggle ensembles as the visual encoder due to better handling of variable image sizes.
• Grid LSTM: Used in older deep learning approaches (MSE-DUDL) to capture spatial dependencies.

Evaluation

Metrics standard in the field:

• Levenshtein Distance (LD): Edit distance between predicted and ground truth strings. Lower is better. Formally, for two sequences $a$ and $b$ (e.g. SMILES strings) of lengths $|a|$ and $|b|$, the recursive distance $LD(a, b)$ is bounded from $0$ to $\max(|a|, |b|)$.
• Tanimoto Similarity: Measures overlap of molecular fingerprints ($0.0 - 1.0$). Higher is better. DECIMER 1.0 achieved a Tanimoto of 0.99 on PubChem data (Table 2). Calculated as: $$ \begin{aligned} T(A, B) = \frac{N_c}{N_a + N_b - N_c} \end{aligned} $$ where $N_a$ and $N_b$ are the number of bits set to 1 in fingerprints $A$ and $B$, and $N_c$ is the number of common bits set to 1.
• 1-1 Match Rate: Exact string matching (accuracy). For DECIMER 1.0, 96.47% of results achieved Tanimoto $= 1.0$.

Hardware

• Training Cost: High for SOTA. DECIMER 1.0 required ~14 days on TPU. The original DECIMER took ~27 days on a single NVIDIA GPU.
• Inference: Transformer models are heavy; rule-based systems run on standard CPUs but with lower accuracy.

Paper Information

Citation: Musazade, F., Jamalova, N., & Hasanov, J. (2022). Review of techniques and models used in optical chemical structure recognition in images and scanned documents. Journal of Cheminformatics, 14(1), 61. https://doi.org/10.1186/s13321-022-00642-3

Publication: Journal of Cheminformatics 2022

@article{musazadeReviewTechniquesModels2022,
  title = {Review of Techniques and Models Used in Optical Chemical Structure Recognition in Images and Scanned Documents},
  author = {Musazade, Fidan and Jamalova, Narmin and Hasanov, Jamaladdin},
  year = 2022,
  month = sep,
  journal = {Journal of Cheminformatics},
  volume = {14},
  number = {1},
  pages = {61},
  doi = {10.1186/s13321-022-00642-3}
}

This is a personal working taxonomy, a lens I use to orient myself when reading papers, not a formal classification scheme. The categories are fuzzy in practice, and reasonable people will assign the same paper differently. The goal is clarity of thought, not consensus.

The taxonomy uses a superposition model where each paper is viewed as a linear combination of seven fundamental contribution types (basis vectors). Examples throughout draw primarily from chemistry and materials science, though the framework applies across physical sciences.

The framework helps answer: “What is this paper’s primary contribution?” by identifying rhetorical patterns and structural elements that signal different research paradigms.

Core Principle: The Superposition Model

All papers in this domain can be viewed as a superposition of fundamental contribution vectors. This is an analogy, not a formal mathematical claim: the seven types below are not linearly independent in the strict sense, and most papers project onto more than one.

Most papers exhibit a profile across multiple basis vectors, blending contribution types (e.g., Method + Theory). One vector usually provides the primary narrative thrust; secondary vectors supply the supporting evidence.

I tend to classify a paper by identifying its Primary Projection (the dominant contribution) and Secondary Projections (supporting work). When a paper seems to split evenly across two types, I look at which type’s indicators dominate the abstract and section headings, and I treat the secondary projection as context that supports the primary narrative, not a co-equal claim.

The Seven Basis Vectors ($\Psi$)

Assessment Guide: Rhetorical Indicators

These rhetorical patterns tend to signal which vector dominates. They are heuristics, not rules; the goal is to notice what the paper is primarily arguing for, not to tick boxes.

1. $\Psi_{\text{Method}}$: The Methodological Paper

Focuses on proposing a novel mechanism, architecture, or approximation (e.g., a new Transformer variant, a GNN with symmetry, a new DFT functional).

Rhetorical Indicators:

• Ablation Study: Authors systematically remove components of their system to prove their specific innovation drives the performance gain. In physics venues, this often appears as parameter sensitivity analysis or direct comparison to prior potentials and functionals rather than a classic ablation table.
• Baseline Comparison: A prominent table comparing the new method against the State-of-the-Art (SOTA)
• Pseudo-code: An explicit block detailing the algorithmic steps (e.g., for training, sampling, or inference)

Examples in the notes: Flow Matching for Generative Modeling (introduces a new continuous normalizing flow mechanism with ablations), MOFFlow (novel SE(3) flow matching architecture for metal-organic frameworks with SOTA comparisons).

2. $\Psi_{\text{Theory}}$: The Theoretical Paper

Focuses on mathematical guarantees, proofs, or derivations from first principles.

Rhetorical Indicators:

• Mathematical Proof Sections: Sections titled “Theorem 1,” “Proof of Equivariance,” or “Formal Bounds”
• Analysis of Limits/Capacity: Investigates the expressivity (e.g., comparing a GNN to the Weisfeiler-Lehman Test) or analyzes the geometry of the optimization landscape
• Generalization/OOD: Derives generalization bounds on test error or formally defines “chemical space coverage” for out-of-distribution (OOD) behavior
• Exact Constraints: Derives exact conditions that true physical functions (like the universal Density Functional) must satisfy

Examples in the notes: The Convexity Principle and Interacting Gases (formal theoretical analysis connecting flow-based generative models to thermodynamic principles).

3. $\Psi_{\text{Resource}}$: The Infrastructure Paper

Focuses on creating and sharing foundational tools for the community.

Rhetorical Indicators:

• Curation Description: Detailed steps on how data was generated, filtered, or curated (e.g., describing millions of CPU-hours of DFT calculations for a dataset like QM9). In physics contexts, this often takes the form of DFT protocol descriptions and computational settings rather than a formal “datasheet.”
• “Datasheets” and “Data Cards”: Inclusion of formal documentation detailing provenance, copyright, and potential biases in the data. In physics contexts, this often appears as detailed computational settings, DFT functional choices, and convergence criteria rather than a named datasheet document.
• Benchmark Definition: Argues that “Metric X on Dataset Y” is the correct proxy for progress in a specific scientific task

Examples in the notes: Molecular Sets (MOSES) (releases a benchmark dataset and evaluation suite for molecular generation), SELFIES 2023 (releases an updated molecular string representation standard with tooling and validation).

4. $\Psi_{\text{Systematization}}$: The Review Paper

Focuses on organizing and synthesizing existing literature.

Rhetorical Indicators:

• Survey Structure: Follows a linear, often chronological, progression or is grouped by architecture (e.g., Variational Autoencoders (VAEs), GANs, Diffusion models)
• Systematization of Knowledge (SoK): A higher-order contribution that proposes a new taxonomy or a unified framework to connect disparate concepts
• Citation Density: References a large fraction of the relevant prior literature; often includes a comparison table spanning many prior works across years or venues

Examples in the notes: Embedded Atom Method Review (1993) (organizes and contextualizes the body of work on EAM potentials into a coherent reference).

5. $\Psi_{\text{Position}}$: The Sociological Paper

Focuses on meta-science, arguing for a change in community norms, or critiquing systemic issues.

I see two subtypes come up often. The first is the roadmap/perspective paper: a constructive argument for where the field should focus, often written by senior researchers after a period of reflection. The second is the critique paper: a more adversarial argument that something the community is currently doing is wrong or counterproductive. Both share the same core signal (the argument itself is the contribution), but they have different tones and are often targeted at different audiences.

Rhetorical Indicators:

• Venue/Track: Often found in “Position Tracks” or called “Blue Sky” or “Forward Looking” papers
• Argumentative Tone: Uses qualitative or quantitative analysis (meta-analysis) to argue for a shift in how research is conducted or funded (e.g., a paper arguing that AI contracts the focus of science)
• Argument as Contribution: The paper presents no new experimental findings; the primary contribution is the argument itself

Examples in the notes: Fold Graciously (position paper arguing for a shift in how the protein folding community evaluates progress), SELFIES 2022 (makes a normative argument for adopting SELFIES over SMILES for robust molecular generation; the paper also has Method and Resource characteristics, but the argument-first framing and advocacy tone push it toward Position for me).

6. $\Psi_{\text{Discovery}}$: The Empirical Discovery Paper

Focuses on the discovery of novel scientific artifacts using AI/ML tools. The key criterion is experimental or higher-fidelity confirmation of the AI’s prediction (wet-lab synthesis, physical characterization, or a higher-fidelity simulation), rather than performance on a held-out test set.

Rhetorical Indicators:

• Structure: Follows a workflow: (1) Computational Screening (AI selects candidates), (2) Validation (wet-lab synthesis, physical characterization, or independent computational confirmation)
• Core Claim: The primary contribution is a new material, molecule, or physical finding, with the AI/ML part serving as the necessary first step
• Key Question: Does the AI’s prediction hold true against an independent ground truth (e.g., a physical experiment or a higher-fidelity simulation)?

Examples in the notes: Oxidation-Reduction Oscillations on Pt/SiO2 (1994) (validates a computationally-predicted dynamic phenomenon against physical experiment).

7. $\Psi_{\text{Application}}$: The Application Paper

Applies an existing method, architecture, or technique to a new scientific domain or task without introducing a new mechanism, dataset, or experimentally confirmed finding. The primary contribution is demonstrating feasibility or utility of transfer.

Rhetorical Indicators:

• “We Apply X to Y” framing: The abstract explicitly names an existing method and a new domain; the contribution is the connection, not the method itself.
• Benchmark-style Results: Performance reported on domain-specific tasks using existing metrics; baselines are often simple (classical methods, prior domain tools) rather than ML SOTA.
• Absence of novelty claims: The paper does not argue for a new architecture, does not release a new dataset, and does not report a validated experimental discovery.

Examples in the notes: Benchmarking LLMs for Molecular Property Prediction (evaluates GPT-3.5, GPT-4, and Llama-2 on six existing OGB molecular tasks without modifying the models), Maxsmi: SMILES Augmentation for Property Prediction (systematic evaluation of five augmentation strategies with existing CNN/RNN architectures), RNNs vs Transformers for Molecular Generation Tasks (empirical comparison of existing architectures on SMILES and SELFIES generation).

Edge Cases and Disambiguation

These are the pairs I find myself second-guessing most often. I’ve written down how I tend to resolve them, not as rules, but as a record of my reasoning.

Method vs. Discovery: The key question I ask is whether there is independent validation beyond a held-out test set. If the paper reports a wet-lab confirmation or a higher-fidelity simulation verifying the AI’s output, I lean toward Discovery: the finding is real and the AI was the tool. If the architecture is the thing being argued for, and validation is limited to benchmarks, I lean toward Method. One edge case I see often in computational chemistry is ML-predicts + DFT-validates: no wet lab, but DFT is genuinely independent of the ML model. I treat this as Discovery if the DFT result is the claimed contribution, and Method if the ML model’s performance against DFT is what’s being argued for.

Resource vs. Systematization: I think of this as artifact-first vs. interpretation-first. If the paper’s main output is something you can download and use (a benchmark, a curated dataset, a software library), it’s Resource. If the paper’s main output is a new way of thinking about a body of literature (a taxonomy, a conceptual unification), it’s Systematization. The presence of a clear benchmark definition with metrics tends to push toward Resource even when there is substantial review content.

Application vs. Method: The cleanest distinguisher here is whether anything new was designed. If an existing method is taken off the shelf and pointed at a new domain, that’s Application. If the domain’s requirements motivated a modification to the method (even a small one), then there may be a legitimate Method component. Application papers often read as feasibility demonstrations; Method papers read as capability expansions.

When I’m still stuck: I look at three things in roughly this order: the abstract’s final sentence (in my experience, especially in ML venues, authors tend to signal what they most want credit for in that last sentence), section heading vocabulary (presence of “theorem,” “algorithm,” “dataset,” “survey,” or “we find” is usually diagnostic), and venue track (position/perspective tracks, SoK tracks, and empirical tracks each carry strong prior signal). I treat the result as a best guess, not a verdict.

Applications

This framework has been useful to me in a few concrete ways:

• Organizing literature reviews: Knowing a paper’s vector profile tells me where it belongs in a review’s structure: whether it fits in the “methods” or “findings” or “community context” section, rather than forcing everything into chronological order.
• Understanding conference and journal acceptance criteria: Different venues weight vectors differently. Physics journals favor Discovery; ML venues favor Method and Theory; interdisciplinary venues like Nature or Science often weight Discovery heavily but also publish high-impact Position papers. Knowing this helps calibrate expectations.
• Identifying gaps in research portfolios: A collection that is heavy on Method but light on Resource or Discovery points to work that may lack empirical grounding or shared infrastructure. The vector breakdown makes these gaps visible.
• Recognizing different types of scientific contribution: Not everything that looks like a methods paper is one, and not everything that looks like a survey is one. The rhetorical patterns here help me notice when my first impression was wrong and reconsider what is actually being claimed.

On-Site Enforcement

Every note_type: paper entry on this site declares its Primary Projection through a single-valued paper_types frontmatter list. A schema validator rejects notes that omit this field or use values outside the seven canonical vectors, and the classification is surfaced as a Hugo taxonomy at /paper-types/. Browsing a vector landing page such as /paper-types/method/ or /paper-types/systematization/ is the fastest way to pull up every note that shares a contribution type.

Enforcement is deliberately single-select: it captures the Primary Projection only. Secondary vectors still exist in the text of each note, but the tag axis is chosen to discriminate between classes rather than to describe a paper exhaustively.