Measuring it requires finding bills that share text. Political scientists do this to estimate how policy diffuses between states; journalists have done it to identify model bills moving between statehouses. A method everyone in this lineage uses is the one biologists use to align gene sequences (Smith-Waterman local alignment): take two bills, find the longest run of matching words, score it for similarity.

Because this is unsupervised, you’ll need a cutoff. Score everything against everything and you have hundreds of millions of bill pairs with a number attached; somewhere on that scale is the line between coincidence and copying. Where you put that line is where every downstream finding comes from, because every count of how much two states share is computed over the pairs above it.

Here is the scale, and here is a line:

The line sits on the steep slope rather than out in the quiet part of the distribution, so moving it a little moves the population a lot: 53 million pairs sit in the bucket immediately below it.

Data and a Threshold

This data is from a CC0 corpus of state bills. The accompanying paper’s conclusion advises future scholars to work in the upper quantiles, and says that for searching out bill pairs “this can be done through thresholding on high alignment scores (e.g., 50+) to indicate the presence of highly similar policy proposals in legislation.” That said, they acknowledge that such thresholding “may discard informative variation,” and assert that high-score thresholding “would focus the analysis on bill pairs that reliably contain very similar policy enactments, and pose little risk of a false positive.”

That is a balanced statement about high-score thresholding in general, not necessarily a recommendation of 50. Read strictly, that advice is about searching out bill pairs: producing candidates worth a human’s attention. Anybody who opens a candidate and finds a PostScript block will throw it away, which is a fair defence of the sentence as written.

Unfortunately, it stops working at scale. The cutoff selects 2,968,117 pairs, which is not a set anybody reads, and the point of a scored corpus is that the score stands in for the reading. So the question is what the cutoff hands you when nobody looks.

I classified the aligned text of every one of those 2,968,117 pairs. 785,330 of them, 26.46%, are not legislative text at all.

What Is Actually Up There

They are page furniture: the line numbers printed down a page margin, the rule that separates a bill’s summary from its body, and a block of PDF printer settings that was never meant to be read by anyone. Text extraction turned them into words. Two bills produced by the same software then “share” those words, and the aligner scores it like any other match.

That is the part I did not expect. The artifact is not sprinkled through the corpus and then swept up incidentally by the cutoff. It is concentrated almost entirely in the region the cutoff selects. A block of printer settings shared between two documents is the most perfect match in the whole corpus.

It Manufactures Edges

If the artifact were spread evenly it would inflate a denominator and leave the ranking of state pairs alone. It is not spread evenly. A quirk of drafting or scanning software belongs to a jurisdiction, so the damage lands on the pairs that jurisdiction belongs to.

Ranked by how many above-threshold pairs they share, the most connected pair of states in this data is Arizona and Nevada, by a factor of nine over the runner-up. 99.9% of those pairs are two PDFs agreeing about their printer settings.

Strip the artifacts and Arizona-Nevada falls from rank 1 to rank 754 of 1,247. Nine of the top twenty dyads leave the top 100.

Two caveats on that table:

• Ranking by raw pair count rewards states with many bills and good text coverage, and coverage in this corpus varies by 100 percentage points across states. This means the ranking is volume-sensitive before any artifact enters it.
• The removal column only subtracts, so it cannot show a dyad crossing into the top 100 from underneath.

And one thing I am not claiming. This ranking is mine. The paper’s own dyadic validation regresses an aggregate alignment score on diffusion ties across 1,225 state pairs; it does not rank dyads by above-threshold count, and I have not re-estimated any published result. I would expect the contamination to survive a change of statistic, since 460,447 pairs at roughly 440 apiece is a great deal of mass in any sum over az-nv, but I have not run that version and it is not what the census shows.

What the census shows is narrower and needs no model: threshold this deposit at a number its own paper offers, and the highest-degree edge in the resulting network is an extraction artifact.

Where It Comes From

The blocks are not spread across the corpus. Scanning all 571,554 bills finds them in 3,551 Arizona bills, 3,178 Nevada bills, one Delaware bill, and nowhere else. So I fetched the original PDFs from both legislatures and re-extracted them.

These PDFs carry an embedded file attachment holding the Acrobat Distiller job options used to produce them, Flate-compressed inside a /Type /EmbeddedFile stream. For one Arizona bill, decompressing that attachment gives 14,581 bytes against the 14,580 characters sitting in the corpus for that same bill. Opening <<, closing >> setpagedevice, indentation and all 220 newlines included. The two are identical once the attachment’s trailing newline comes off.

The corpus’s text extraction emitted PDF file attachments as document text. Not a font problem, not an encoding problem, and not a property of the source documents. pdftotext walks page content and ignores attachments, which is why it reproduces the block 0 times in 92 from documents that demonstrably contain it.

Two details make this worse than a scoring problem. Smith-Waterman is not a cheap algorithm to run. So to reduce the number of bills one runs this alignment scoring for, resesarchers first perform a coarse retrieval stage to pre-filter comparisons to those with the highest similarity as determined by bag-of-words methods (like BM-25). The block also wins the retrieval: a bill carrying one spends 15.00% of its candidate slots on other bills carrying one, against a 1.36% base rate, and a candidate that retrieval never returned was never scored at all.

The paper anticipated the general worry. Bills are compared against the 500 nearest neighbours Elasticsearch returns, and the authors checked that cap by re-running at 1,000, concluding it introduces no selection bias “because the set of bills that result in modest or large text reuse scores is always smaller than 500.” That reasoning holds when the top 500 is filled by candidates competing on legislative language. What the crowding number shows is a population where the premise is doubtful: across the 6,700 queries carrying a distiller block, an elevenfold enrichment means a large share of those slots is won on a string that is not language at all. I have not re-run the authors’ check, so this is a reason to think it deserves re-running on the artifact-carrying subset, not a finding that it fails.

And the bias cannot be signed. A distiller block is appended after the page text, so excising it can only lower a score. A line-number run sits between two passages of prose, so excising it joins them and the score rises, by up to 63 points. The artifact invents reuse in the pairs it creates and conceals reuse in the pairs it interrupts.

The Problem No Cleaning Fixes

Everything above is a defect. A better extractor and a screening pass fix all of it, and anything built on stripped text is recoverable.

The pairs that survive every screen have a separate problem, and this one is a property of the measure rather than of this corpus. It survives perfect text.

Two states implementing the same federal statute share text word for word, and neither one copied the other. The alignment is exact. The reuse is real. It just does not run between the two states in the edge.

Who Did the “Reading”

I tried a language model first. MiniMax M3 labelled every prose pair in my sample, three times over, and on the 477 where all three passes parsed, its verdicts flipped on 33.5%.

Where it broke is the useful part. The instability was worst exactly at the bar, around 57% stable in the 50 to 99 band against 100% above 500, which left two readings the run could not separate: either the model was unreliable on hard cases, or pairs near the bar are genuinely undecidable from the text.

So I read 80 of the pairs again on a blinded page, stratified toward the 50 to 99 band where the instability lived, with score, bill ids and jurisdictions hidden. And so did Claude Opus, on the identical page. Whereas M3 had been the arms-length instrument, prompted and run in batch, Claude was the model I had been working the audit with in Claude Code. It knew my hypotheses and had already seen some of these spans.

The agreement number below is not an inter-rater statistic. It is a person and a model that shared a hypothesis, labelling the same page. I am reporting it because the structure of the disagreement turned out to matter more than its size, and because the answer to the question the M3 run could not settle came out of it: agreement held at 77.4% in the 50 to 99 band, roughly flat where M3’s own stability had collapsed. Pairs near the bar are decidable. The unreliable instrument was the model that kept changing its mind.

No reading makes this class mostly boilerplate. The stricter reading still puts two thirds of it on unambiguous shared source. The 11% both call generic is what you would guess: enacting clauses, appropriation formulae, quorum boilerplate, bill titles on unrelated subjects. The pairs that survive the artifact screen are, in the main, real reuse.

The third row is the finding and the weakest row in the table. Every pair I called shared source, Claude did too. The split is not about where a line was drawn. Instead, the model could name an upstream source and I could not. Text you cannot place reasonably looks like it might be generic.

The Names Are the Result, Not the Labels

A name is checkable in a way a label is not.

Claude named 37 upstream sources across the 80 pairs. I checked eight of them against published law, choosing the ones the finding leans on hardest:

Seven verbatim, one partial, none refuted, and all eight sit in the disputed column. That is the result. Text that a careful reader calls generic formula is sometimes a named federal act reproduced word for word, and unlike every similarity score in this corpus, that is a claim a stranger can destroy with a search engine.

The twenty-nine unchecked names are not evidence, and the reason is on the record. On a control pair reading this act shall be known and may be cited as the school protection act, M3 confidently named an ALEC model act, the Liberty Preservation Act, which has nothing to do with it. A model asked to name a source will produce a plausible one whether or not it exists. The eight are worth something precisely because they were checked; the rest are leads.

The Part That Survives a Clean Corpus

A threshold on text similarity is selecting two different phenomena and cannot tell them apart. States adopting a private model act is policy diffusion, which is what the measure was built to find. States independently implementing the same federal statute is mandated uniformity, which is a different causal story with the same textual signature. The alignment is equally exact in both cases, which is precisely why no score separates them. A higher threshold does not help. A better aligner does not help. Cleaning the text does not help, because there is nothing wrong with the text.

And the error has structure, which is what makes it bite. States differ in how much federal law they implement and how fast, so a diffusion network built on this measure counts federal conformity as interstate influence, and it does so unevenly across exactly the dyads people draw conclusions from.

None of which is news to the diffusion literature. Vertical influence, federal mandates and common exposure are long-standing confounds in that field, and there are methods built to handle them. The narrow claim is the one worth making, and it is about the instrument: a text-reuse score offers no purchase on the distinction at all, because the two cases have an identical textual signature. Whatever a researcher does about common exposure, they have to do it from outside this measure, and a network drawn straight from alignment scores carries the confound with no way to see it.

The fix is a labelled source: to know whether copying ran between the two jurisdictions in your edge, you have to know what was copied.

How Far That Goes, and How Far It Does Not

The sample is not the population. The 80 pairs were drawn to over-represent the 50 to 99 band, so the 23% is a rate within a deliberately skewed draw and I would not quote it as a corpus rate. Read it as context for the eight verified names rather than as the finding.

There is one human in this design. Me, not blind, holding the hypothesis while labelling. Everything else in the reading pipeline was a model. What that design can establish is bounded, and what it did establish is the checkable half: eight names, seven exact.

What the 80 pairs support is that the distinction exists, that it is not rare, and that where it has been tested it holds. How large it is corpus-wide is unmeasured, and a second human annotator who had never read this audit is the thing that would change it. That is not a run I can do, so the blinded page and both sets of labels ship with the code, and somebody else can.

So, Would a Language Model Have Helped?

I came at this wanting to know whether a language model was useful for legislative text reuse. Having spent the time, the answer is yes, and not where I expected.

The contamination needed no model at all. It is a defect: a few dozen lines of pattern matching find every class, removal is lossless because each one is self-delimiting, and anything built on stripped text is recoverable.

The uniformity confound is the opposite. It is not removable at any threshold and a better similarity score will not touch it. What would separate it is knowing what was copied, which is a provenance question, and provenance is the kind of question a language model is actually good at.

It was the one thing both models did well here. Asked to name upstream sources, M3 produced the Uniform Controlled Substances Act, the Streamlined Sales and Use Tax Agreement, the Uniform Vehicle Code, 18 U.S.C. 2510, and those names were more durable than its own verdicts, agreeing across passes 77% of the time against 66.5% for the labels. None of that batch was checked against published law. The eight that were checked came from the blinded run and seven were exact.

With the condition from the section above attached, and it is not a small one: the behaviour that produces a correct name also produces the ALEC misfire, and nothing in the output tells you which one you are holding. Only a check does.

That is the whole thing in miniature:

A similarity label is an opinion. A named source is a claim. Only the second kind can be refuted.

A wrong name announces itself the moment somebody checks it. A wrong similarity score is invisible forever. The second failure mode is the one you want.

So the honest next artifact here is not a better detector. It is a labelled source corpus, built by something that can propose a name and verified by somebody who checks it.

If You Use This Deposit

Strip all three classes before thresholding. Each is self-delimiting, so removal is exact rather than heuristic.

Three things that are easy to get wrong. Do not just exclude Arizona and Nevada, because the other two classes cover more dyads than the distiller block does. Treat Hawaii separately, because its text is OCR-damaged underneath the artifact, that damage is what its source documents actually say, and neither removal nor re-extraction fixes it. And before removing a run, check that it ascends by one, because a run that ascends on every second position is a statutory rate schedule with a counter threaded through it, and excising it deletes enacted numbers.

Do not trust a high threshold to be safe either. The most reassuring number here, that the band above 500 is 99.1% clean, is an accident: the distiller blocks stop scoring at 442 only because Arizona ran Distiller 8.2.5 and Nevada ran 6.0, so the two job-options files agree only through their shared prefix. Two jurisdictions on the same version would share a byte-identical 14,500-character block, which aligns into the thousands.

The Full Audit

The writeup behind all of this is a technical report: the four independent instruments, the removal experiment and its label-blind control, the scrape-era census that explains why the damage partitions by legislative session rather than by state, the limitations including a cluster bootstrap that widens my own headline interval by a factor of five, and an appendix of the four things earlier drafts got wrong. It is at REPORT.md in the page-furniture repo, alongside the screen itself and the dated results file behind every number here.

The 80-pair adjudication is in there too. The per-pair record has every pair, both labels, the upstream act Claude named where it named one, and which eight I checked against published law. The 29 unchecked names are published as well, because a name nobody has checked is a lead, and I would rather ship it as one than quietly drop it.

Attribution

The corpus was collected by Open States and assembled by Burgess et al. It is CC0; US legislative text is uncopyrightable at origin under Georgia v. Public.Resource.Org (2020).

Burgess, Matthew, Eugenia Giraudy, Julian Katz-Samuels, Joe Walsh, Derek Willis, Lauren Haynes, and Rayid Ghani. “The Legislative Influence Detector: Finding Text Reuse in State Legislation.” Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 57-66, 2016, doi:10.1145/2939672.2939697.

The deposit measured here is Linder, Fridolin J., “Replication Data for: Measuring Policy Similarity Through Bill Text Reuse”, Harvard Dataverse, doi:10.7910/DVN/CZ25GF, accompanying Linder, Desmarais, Burgess and Giraudy, “Text as Policy: Measuring Policy Similarity through Bill Text Reuse”, Policy Studies Journal, doi:10.1111/psj.12257. The prior art is Citron, Daniel T. and Paul Ginsparg, “Patterns of text reuse in a scientific corpus”, PNAS 112(1):25-30, 2015, doi:10.1073/pnas.1415135111.

This post explores machine learning approaches for classifying congressional bills by policy area, using data from the 115th to 117th Congresses (2017-2023). We’ll examine:

• The fundamentals of bill classification
• Traditional machine learning models as baselines
• Performance analysis across different time periods and policy domains

This work establishes baselines for future deep learning approaches to legislative text classification.

This post builds on the data foundation established in Exploring the 117th U.S. Congress.

Motivation

Automatically classifying congressional bills by policy area has practical value for researchers, journalists, and citizens who need to navigate thousands of bills each Congress. Machine learning can help identify patterns in legislative priorities and track policy trends over time.

Data

The data comes from scraping Congress.gov for all bills from the 115th through 117th Congresses. Each bill includes:

• Bill ID and title
• Summary (when available): the earliest summary provided
• Full text (when available): the earliest text version
• Policy area classification

Our task is to predict policy area from text features:

$$ f(X) = \hat{y}, \quad \text{where} \quad X = { \text{title}, \text{summary}, \text{text} }, \quad \hat{y} \in { \text{policy areas} } $$

The complete dataset is available at Hugging Face: hheiden/us-congress-bill-policy-115_117.

Bills by Congress

Our dataset contains the following distribution:

Policy Areas

Each bill receives a policy area label from Congress.gov (see glossary). The dataset includes 33 policy areas, though these classes are highly imbalanced.

The following table shows the number of bills in each policy area across the three Congresses:

The class imbalance is severe: Social Sciences and History has only 15 bills across all three Congresses, while Health has 5,911 bills. This imbalance presents modeling challenges, as minority classes may lack sufficient representative samples.

Text Statistics

We analyzed token counts using spaCy to understand the computational requirements for each text field.

Title Token Statistics:

Summary Token Statistics:

Full Text Token Statistics:

These statistics reveal computational trade-offs:

• Titles average ~12 tokens: computationally efficient but limited information.
• Summaries average ~100 tokens: good balance of information and efficiency.
• Full text averages ~2,700 tokens with 129M total tokens: detailed but computationally expensive. Processing this volume of text introduces real-world engineering challenges, such as memory constraints and a higher noise-to-signal ratio typical of long legal documents.

We’ll prototype with titles and summaries before considering full text, given the computational costs involved.

Evaluation Framework

Experimental Design

We train models on one Congress and test on others, creating a 3x3 evaluation grid. This setup evaluates both within-Congress performance (same session) and cross-Congress generalization (different sessions). We expect temporal drift between Congress sessions to impact performance.

Metrics and Hyperparameter Tuning

We use weighted average F1 score to handle class imbalance, ensuring fair evaluation across all policy areas regardless of frequency.

For within-Congress evaluation, we report cross-validated scores. For cross-Congress evaluation, we test on the entire target Congress dataset.

Hyperparameter tuning uses Cross-Validation Grid Search with folds set to min(3, n_samples) to ensure all classes are represented. We apply the best parameters from training to test generalization across different Congresses.

Baseline Models

We evaluate three traditional machine learning approaches using TF-IDF vectorization:

Text Preprocessing

We convert text to numerical features using TF-IDF (term frequency-inverse document frequency), which weighs word importance by frequency within documents relative to the entire corpus. This creates normalized feature vectors suitable for machine learning classification.

Multinomial Naive Bayes

We start with Multinomial Naive Bayes as our simplest baseline. Despite its “naive” independence assumption between features, this model often performs surprisingly well for text classification tasks and serves as an important benchmark. If more complex models can’t beat Naive Bayes, it signals potential issues with the approach or data.

The model’s feature_log_prob_ attribute reveals the most influential words for each policy area, providing interpretable insights into classification patterns.

You can see the code for training the Naive Bayes model below:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB

# Create a pipeline with TF-IDF vectorizer and Multinomial Naive Bayes classifier
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(lowercase=True, dtype=np.float32)),
    ('clf', MultinomialNB()),
])

# Define the parameters for grid search
parameters = {  
    'tfidf__ngram_range': [(1, 1), (1, 2), (1, 3)],
    'tfidf__max_df': (0.05, 0.1, 0.25, 0.5),
    'tfidf__min_df': (1, 2, 5, 10),
    'clf__alpha': (1, 0.1, 0.01, 0.001),
}

# Perform grid search with cross-validation
grid_search = GridSearchCV(
    pipeline,
    parameters,
    scoring='f1_weighted',
    n_jobs=-1,
    refit=True,
    cv=3,
)
grid_search.fit(X_train, y_train)

Logistic Regression

Logistic regression provides a natural step up in complexity from Naive Bayes. It uses the logistic function to convert linear combinations of features into probabilities, making it an excellent baseline for comparison with more sophisticated models while remaining interpretable.

You can see the code for training the Logistic Regression model below:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

# Create a pipeline with TF-IDF vectorizer and Logistic Regression classifier
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(lowercase=True, dtype=np.float32)),
    ('clf', LogisticRegression(max_iter=1000, random_state=42, class_weight='balanced')),
])

# Define the parameters for grid search
parameters = {  
    'tfidf__ngram_range': [(1, 1), (1, 2)],
    'tfidf__max_df': (0.05, 0.1, 0.25),
    'clf__C': [0.1, 1, 10],
}

# Perform grid search with cross-validation
grid_search = GridSearchCV(
    pipeline,
    parameters,
    scoring='f1_weighted',
    n_jobs=-1,
    refit=True,
    cv=3,
)
grid_search.fit(X_train, y_train)

XGBoost

We include XGBoost as our tree-based ensemble method. While XGBoost typically excels on structured tabular data, we test whether its gradient boosting approach can effectively handle TF-IDF features for text classification.

You can see the code for training the XGBoost model below:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from xgboost import XGBClassifier

# Create a pipeline with TF-IDF vectorizer and XGBoost classifier
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(lowercase=True, dtype=np.float32)),
    ('clf', XGBClassifier(use_label_encoder=False, eval_metric='mlogloss', objective='multi:softmax', seed=42, n_jobs=-1)),
])

# Define the parameters for grid search
parameters = {  
    'tfidf__max_df': (0.05, 0.1, 0.25),
    'clf__max_depth': (3, 6, 9),
    'clf__n_estimators': (100, 200, 300),
}

# Perform grid search with cross-validation
grid_search = GridSearchCV(
    pipeline,
    parameters,
    scoring='f1_weighted',
    refit=True,
    cv=3,
    verbose=3,
)
grid_search.fit(X_train, y_train, clf__sample_weight=sample_weight)

Results

We evaluate models on three input types:

• Title-only: Quick prototyping with limited context
• Summary-only: Balanced information content and computational efficiency
• Full text: Maximum context with computational constraints (limited hyperparameter tuning)

Title-Only Inputs

Naive Bayes

Title-only Naive Bayes experiments are run with the following settings:

sweep_nb(
    data,
    X_key='title',
    y_key='policy_area',
    tfidf_params={
        'lowercase': True,
        'dtype': np.float32,
    },
    tfidf_grid={
        'ngram_range': [(1, 1), (1, 2)],
        'max_df': (0.05, 0.1, 0.25, 0.5),
        'min_df': (1, 2, 5),
    },
    nb_params={},
    nb_grid={
        'alpha': (1, 0.1, 0.01, 0.001),
    },
)

and the results:

Training on Congress 115
Best score: 0.661
Refit Time: 0.570
Best parameters set:
	clf__alpha: 0.01
	tfidf__max_df: 0.05
	tfidf__min_df: 1
	tfidf__ngram_range: (1, 2)
Testing on Congress 116 F1: 0.6369760774921475
Testing on Congress 117 F1: 0.5488274400521962

Training on Congress 116
Best score: 0.677
Refit Time: 0.499
Best parameters set:
	clf__alpha: 0.01
	tfidf__max_df: 0.05
	tfidf__min_df: 1
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.691175262953872
Testing on Congress 117 F1: 0.6798043069585031

Training on Congress 117
Best score: 0.670
Refit Time: 0.565
Best parameters set:
	clf__alpha: 0.01
	tfidf__max_df: 0.25
	tfidf__min_df: 1
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.6168474701996426
Testing on Congress 116 F1: 0.6981574942116808

Mean fit time: 0.54 ± 0.03s

Results Summary

The results demonstrate several key findings:

• Fast training: Sub-second training times make this highly practical
• Solid baseline performance: F1 scores around 0.65-0.70 provide a reasonable starting point
• Consistent hyperparameters: Similar optimal settings across Congresses suggest stable patterns
• Temporal effects: Performance generally decreases when training and testing on Congresses further apart in time

Training on the 116th Congress yields the best cross-Congress performance, likely due to its temporal proximity to both adjacent sessions.

The model learns interpretable features for each policy area. For example, Agriculture bills are strongly associated with terms like “farm,” “crop,” and “livestock,” while Armed Forces bills correlate with “military,” “defense,” and “veterans.”

Logistic Regression

Title-only Logistic Regression experiments are run with the following settings:

sweep_logreg(
    data,
    X_key='title',
    y_key='policy_area',
    tfidf_params={
        'lowercase': True,
        'dtype': np.float32,
    },
    tfidf_grid={
        'ngram_range': [(1, 1), (1, 2)],
        'max_df': (0.05, 0.1, 0.25),
    },
    logreg_params={
        'max_iter': 1000,
        'random_state': 42,
        'class_weight': 'balanced',
    },
    logreg_grid={
        'C': [0.1, 1, 10],
    },
)

and the results:

Training on Congress 115
Best score: 0.704
Refit Time: 32.063
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.05
	tfidf__ngram_range: (1, 2)
Testing on Congress 116 F1: 0.6809188275881766
Testing on Congress 117 F1: 0.601917336933838

Training on Congress 116
Best score: 0.714
Refit Time: 31.227
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.05
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.7408989977276476
Testing on Congress 117 F1: 0.7200639105208106

Training on Congress 117
Best score: 0.711
Refit Time: 34.083
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.05
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.674418393892329
Testing on Congress 116 F1: 0.7405934743144291

Mean fit time: 32.46 ± 1.20s

Results Summary

Logistic regression improves upon Naive Bayes performance:

• Higher F1 scores: Generally 5-7 percentage points better than Naive Bayes
• Consistent hyperparameters: Optimal settings remain stable across Congresses
• Reasonable training time: 30-35 seconds per model remains manageable
• Strong cross-Congress generalization: F1 scores consistently above 0.70

XGBoost

Title-only XGBoost experiments are run with the following settings:

sweep_xgb(
    data,
    X_key='title',
    y_key='policy_area',
    tfidf_grid={
        'max_df': (0.05,),
    },
    xgb_grid={
        'max_depth': (6,),
        'eta': (0.3,),
    },
)

and the results:

Training on Congress 115
Best score: 0.591
Refit Time: 198.063
Best parameters set:
	clf__eta: 0.3
	clf__max_depth: 6
	clf__num_class: 33
	tfidf__max_df: 0.05
Testing on Congress 116 F1: 0.5649530686141018
Testing on Congress 117 F1: 0.5215939580735101

Training on Congress 116
Best score: 0.600
Refit Time: 264.824
Best parameters set:
	clf__eta: 0.3
	clf__max_depth: 6
	clf__num_class: 33
	tfidf__max_df: 0.05
Testing on Congress 115 F1: 0.6037922738570368
Testing on Congress 117 F1: 0.5965027418245722

Training on Congress 117
Best score: 0.595
Refit Time: 249.799
Best parameters set:
	clf__eta: 0.3
	clf__max_depth: 6
	clf__num_class: 33
	tfidf__max_df: 0.05
Testing on Congress 115 F1: 0.5600491477899472
Testing on Congress 116 F1: 0.60815381664894

Mean fit time: 237.56 ± 28.60s

Results Summary

XGBoost underperforms relative to expectations:

• Poor performance: F1 scores significantly below linear models (0.55-0.60 range)
• Long training times: 4+ minutes per model with limited hyperparameter exploration
• Questionable value: The computational cost doesn’t justify the poor performance

Given these results, we focus on the more promising linear models for subsequent experiments with longer text inputs.

Training Efficiency

The computational costs vary dramatically:

XGBoost’s poor performance despite high computational cost suggests that tree-based methods may not be well-suited for sparse TF-IDF features. This is a classic example of the “curse of dimensionality”: tree-based models struggle to make effective splits in highly sparse, high-dimensional bag-of-words spaces compared to linear models that simply assign weights to all features simultaneously. We’ll focus on linear models for the remaining experiments.

Summary-Only Results

Using bill summaries provides substantially more context than titles alone, leading to significant performance improvements.

Naive Bayes Performance

The summary-based models show dramatic improvement over title-only versions:

• F1 scores: 0.85+ within-Congress, 0.77-0.86 cross-Congress
• Training time: Still fast at ~3.4 seconds
• Strong generalization: Consistent performance across time periods

Logistic Regression Performance

Logistic regression slightly outperforms Naive Bayes on summaries:

• F1 scores: 0.86+ within-Congress, 0.79-0.87 cross-Congress
• Training time: Reasonable at ~12 seconds
• Stable hyperparameters: Consistent optimal settings across Congresses

The performance difference between models suggests they rely on similar feature patterns, with logistic regression better capturing feature interactions.

Logistic Regression

Summary-only Logistic Regression experiments are run with the following settings:

sweep_logreg(
    data,
    X_key='summary',
    y_key='policy_area',
    tfidf_params={
        'lowercase': True,
        'dtype': np.float32,
    },
    tfidf_grid={
        # 'ngram_range': [(1, 1), (1, 2)],
        'max_df': (0.05, 0.1, 0.25),
    },
    logreg_params={
        'max_iter': 1000,
        'random_state': 42,
        'class_weight': 'balanced',
    },
    logreg_grid={
        'C': [0.1, 1, 10],
    },
)

And the results:

Training on Congress 115
Best score: 0.862
Refit Time: 9.007
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.25
Testing on Congress 116 F1: 0.8284864693401133
Testing on Congress 117 F1: 0.7934161507811646

Training on Congress 116
Best score: 0.865
Refit Time: 13.897
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.25
Testing on Congress 115 F1: 0.8637852557418315
Testing on Congress 117 F1: 0.8594775615031977

Training on Congress 117
Best score: 0.862
Refit Time: 12.167
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.25
Testing on Congress 115 F1: 0.8355736563084967
Testing on Congress 116 F1: 0.8696403838390832

Mean fit time: 11.69 ± 2.02s

Full Text Results

We test whether complete bill text improves performance over summaries, using optimal hyperparameters from summary experiments.

Naive Bayes on Full Text

Surprisingly, full text yields slightly lower performance than summaries:

• F1 scores: 0.84-0.85 within-Congress, 0.77-0.86 cross-Congress
• Training time: ~50 seconds (10x slower than summaries)
• Performance drop: Likely due to increased noise in lengthy documents

Logistic Regression on Full Text

Logistic regression shows the strongest performance on full text:

• F1 scores: 0.87-0.88 within-Congress, 0.83-0.89 cross-Congress
• Training time: ~70 seconds
• Best overall performance: up to 0.89 F1 on the strongest single cross-Congress pair (best within-Congress score 0.877)

The logistic regression model benefits from having access to complete legislative language while effectively regularizing against noise.

Key Findings

This baseline study establishes several important results:

Best performing model: Logistic regression trained on full bill text reaches up to 0.89 F1 on the strongest single cross-Congress pair (best within-Congress score 0.877), providing a strong benchmark for future deep learning approaches.

Text input comparison:

• Titles: Limited but fast (F1 ~0.65-0.70)
• Summaries: Good balance of performance and efficiency (F1 ~0.85)
• Full text: Best performance but computationally expensive (certified weighted-F1 0.871-0.877; up to ~0.89 on the strongest single cross-Congress pair)

Cross-Congress generalization: Models trained on one Congress generalize reasonably well to others, though performance decreases with temporal distance between sessions.

Model performance ranking: Logistic Regression > Naive Bayes » XGBoost for this text classification task.

Next Steps

The strong baseline performance sets the stage for several research directions:

1. Deep learning models: Transformer-based approaches using pre-trained language models
2. Dataset expansion: Including additional Congresses and more detailed bill metadata
3. Error analysis: Understanding failure cases and class-specific performance patterns
4. Feature engineering: Exploring domain-specific text preprocessing and feature extraction

The complete dataset and experimental code are available for researchers interested in building upon these baselines.

Resources:

• Dataset: Hugging Face: hheiden/us-congress-bill-policy-115_117
• Leaderboard: Policy Area Classification Leaderboard
• Project: Congressional Knowledge Graph & Policy Classification

Analyzing congressional data reveals the underlying mechanics of the legislative process. Legislative text is a large, structured corpus well suited to text classification and other NLP tasks. I scraped data from Congress.gov to analyze what actually happens to the thousands of bills introduced each session and to build a foundational dataset for downstream machine learning tasks.

This analysis focuses on the 117th Congress (2021-2023), examining 15,000+ bills to understand basic patterns: Which bills get introduced? How many receive votes? What factors influence success?

This post covers the foundational exploratory analysis and data collection process, setting the stage for predictive modeling and policy area classification.

Data Collection

My primary source is Congress.gov, maintained by the Library of Congress. I focused on the 117th Congress (2021-2023), collecting data on bills and joint resolutions, omitting simple resolutions, concurrent resolutions, and amendments.

Data collected:

Technical Implementation

Building a usable NLP dataset requires careful handling of the source. Congress.gov loads content dynamically and presents nested DOM structures, so the scraper combines static HTML parsing with a headless browser to render JavaScript before parsing.

Implementation details:

• Python for core orchestration and data schema management
• Selenium for executing JavaScript and loading dynamic page elements
• BeautifulSoup for structured HTML parsing
• Regex for text normalization and extracting clean legislative text for language models

The crawler used 5-second delays between requests to respect server limits, a roughly 3-day collection run. It handles edge cases in congressional text formatting and writes one JSON record per bill on a fixed schema. The crawler and processed data are available on GitHub.

For each bill, I queried two pages:

• All info page: https://www.congress.gov/bill/117th-congress/{bill_type}/{bill_id}/all-info
• Text page: https://www.congress.gov/bill/117th-congress/{bill_type}/{bill_id}/text?format=txt

The parsing process involved targeting specific HTML elements and implementing basic caching to avoid redundant requests.

Key Findings

The analysis reveals clear patterns in congressional activity. Most bills never receive votes, and success rates vary significantly by party and policy area.

Legislative Outcomes

The fundamental question: what happens to bills after introduction?

Each bill has a tracker status indicating its position in the legislative process. The eight possible statuses can be grouped into three meaningful categories:

• Introduced: Bills introduced but never voted on
• Stalled: Bills that saw votes but didn’t become law (since the 117th Congress ended, these effectively died)
• Law: Bills signed by the President

Key insights:

• Only 7% of introduced bills ever receive a vote
• Of bills that receive votes, 36% become law
• Overall, just 2% of introduced bills become law

Sponsor Analysis

The bill sponsor (the primary member who introduces legislation) provides insights into party and geographic patterns.

Party Breakdown

Party comparison:

• Democrats: 7.5% of bills moved beyond introduction; 2.6% became law
• Republicans: 5.5% of bills moved beyond introduction; 2.1% became law
• When bills do advance, Republicans have a slightly higher success rate (38% vs 35%)

Geographic Distribution

Top 10 states by bills introduced:

Per-representative normalization reveals different patterns:

Top Individual Sponsors

Most prolific legislators by bills introduced:

Effectiveness score (laws enacted / total bills):

$$ \text{effectiveness} = \frac{\text{bills that became law}}{\text{total bills introduced}} $$

Policy Focus Areas

Each bill is assigned a primary policy area. Here are the most active areas by legislative outcome:

Notable patterns: Health dominates introductions but has lower success rates, while government operations and armed forces bills are more likely to become law.

Next Steps

This analysis establishes baseline patterns: most bills fail, party affiliation affects success rates, and certain policy areas perform better than others.

Future work could explore:

• Committee dynamics and voting patterns
• Geographic analysis of state-level interests
• Bill text analysis using NLP techniques
• Predictive modeling for bill outcomes

Update: I’ve since applied machine learning to this type of data in Congressional Bill Policy Area Classification, using 48K+ bills from three Congresses to automatically categorize bills by policy area.

The complete dataset and code are publicly available to support further research into legislative transparency.

A computational social science project that constructed a dataset of 47,000+ US congressional bills by extracting legislative text and metadata from the 115th-117th Congresses. The project creates a “legislative graph” (linking sponsors, committees, and bill text) and establishes TF-IDF baseline models for policy area classification across 33 (highly imbalanced) policy classes, now hosted on Hugging Face to support reproducible political science research.

Features

Intelligent Data Acquisition

Standard APIs impose strict rate limits. I built a Selenium-based extraction engine to handle Congress.gov’s complex DOM structures.

• Optimization: Targeted aggregate endpoints (e.g., /all-info) to pull each bill’s text and metadata in fewer requests.
• Resilience: Implemented a local caching layer to store raw HTML, separating the fetch step from the parse step. This made the parse step re-runnable without re-fetching, and minimized server load during iterative development.
• Graph construction: Beyond simple text, the script extracts relational data including co-sponsorship networks, committee assignments, and related bill lineage.

Natural Language Processing

• Corpus construction: Cleaned and normalized legislative text, removing procedural artifacts (e.g., “A BILL TO…”) to isolate semantic policy content.
• Feature engineering: Utilized TF-IDF vectorization with N-gram analysis to capture legislative jargon.
• Modeling: Benchmarked Naive Bayes, Logistic Regression, and gradient-boosted trees (XGBoost), reaching ~0.86 weighted F1 on bill summaries and ~0.88 on full text within-Congress (0.877; up to 0.89 on the strongest cross-Congress pair). Weighted F1, not raw accuracy, is the honest metric here: the 33 policy classes are severely imbalanced (Health has 5,911 bills; Social Sciences and History has 15).

Usage

The dataset is available on Hugging Face and can be loaded directly via the datasets library. The scraper can be run locally to fetch new bills.

Results

• Temporal drift: Policy priorities and terminology showed measurable shifts across congressional sessions (115th vs 117th).
• Classification success: Simple linear models (Logistic Regression and Naive Bayes) proved effective at distinguishing policy domains, outperforming gradient-boosted trees on these sparse TF-IDF features and suggesting legislative language is highly structured.

Impact & Deliverables

• Hugging Face dataset: Released a machine-readable, ML-ready dataset of modern bills (115th-117th Congresses) on Hugging Face for reproducible research.
• Open source tooling: Published the scraper and parsing logic to allow others to extend the dataset to future congresses.
• Academic benchmark: Establishing a clear baseline for “Government NLP” tasks, aiding in the automated transparency and monitoring of new legislation.

Related Work

• 117th Congress Data Exploration
• Congressional Bill Policy Area Classification