Method.

This paper is primarily a Methodological contribution. It proposes a novel probabilistic architecture, the Hierarchical Ideal Point Topic Model (HIPTM), designed to solve the specific limitations of existing political science models that typically rely on either voting data or text data in isolation. The paper validates this method by demonstrating its superior performance in predicting “Tea Party” membership compared to text-only baselines and its ability to provide interpretable “framing” analysis.

What is the motivation?

The primary motivation is to better understand political polarization, specifically the “Tea Party” phenomenon within the Republican party during the 112th Congress.

An ideal point is a scalar score representing a legislator’s ideological position, estimated from voting patterns. Standard “Ideal Point” models (like DW-NOMINATE) typically project legislators onto a single liberal-conservative dimension using only binary voting data. This is insufficient for capturing complex, multi-dimensional intra-party conflicts where legislators might agree on votes but differ on policy “framing” or specific sub-issues. Furthermore, existing multi-dimensional models often produce dimensions that are difficult for humans to interpret.

What is the novelty here?

The core novelty is the Hierarchical Ideal Point Topic Model (HIPTM). It distinguishes itself from prior work through three main technical innovations:

1. Joint Modeling of Three Data Sources: It integrates roll call votes, the text of bills, and the floor speeches of legislators into a single probabilistic framework.
2. Hierarchical Topic Structure: It models “frames” as a second level of the topic hierarchy. “Issues” (level 1) are fixed and non-polarized, while “Frames” (level 2) are discovered dynamically and carry polarity (ideal point weights). For example, Health Care is an issue; “government overreach” vs. “patient protection” are frames legislators use when debating it.
3. Text-Based Ideal Point Prediction: HIPTM regresses ideal points on speech text, allowing it to predict the political alignment of legislators based solely on their writing or speeches without requiring voting records for inference.

What experiments were performed?

The authors validated the model using data from the 112th U.S. Congress (Republican legislators only).

• Prediction Task: Classifying legislators as members of the “Tea Party Caucus”.
• Baselines: The model was compared against Support Vector Machines (SVM) trained on:
  • TF-IDF vectors (Text only)
  • Normalized TF-IDF vectors (Text only)
  • Binary Vote vectors (Vote only)
• Metric: Area Under the Receiver Operating Characteristic Curve (AUC-ROC) via 5-fold cross-validation.
• Qualitative Analysis: The authors examined the “span” of ideal points within specific topics (e.g., Macroeconomics, Health) to identify which issues were most polarized between Tea Party and Establishment Republicans.

What were the outcomes and conclusions drawn?

• Quantitative Performance: HIPTM features combined with voting data (HIPTM-VOTE) achieved the highest classification performance (AUC-ROC in the ~0.70-0.75 range, approximate, read from Figure 2). Vote-only features slightly trail HIPTM-VOTE, while text-only baselines (TF-IDF, normalized TF-IDF) fall considerably lower. The one-dimensional Tea Party ideal points correlate with DW-NOMINATE ($\rho = 0.91$). When voting data was withheld (simulating a candidate without a record), HIPTM’s text-based features outperformed standard text baselines TF-IDF and normalized TF-IDF (approximate, read from Figure 3).
• Political Insight: The model identified “Government Operations,” “Macroeconomics,” and “Transportation” as the three most polarized topics between Tea Party and establishment Republicans.
• Framing Analysis: The hierarchical topic structure reveals how legislators frame issues differently. For Macroeconomics, frame M3 (most Tea Party-oriented) focuses on criticizing government overspending, while frame M1 (least Tea Party-oriented) focuses on the downsides of a government shutdown. For Health, frame H3 captures Tea Party framing of the Affordable Care Act as an unconstitutional government takeover, while frame H1 frames opposition in terms of implementation costs and health care exchanges.
• Framing vs. Voting Taxonomy: The authors construct a 2x2 taxonomy of disagreement across issues, crossing whether ideal points are polarized with whether issue frames are polarized. Issues like Civil Rights fall in the “neither polarized” quadrant, where cooperation is expected. Banking/Finance and Transportation fall in the “ideal points polarized, frames not” quadrant, where Republicans frame the issue similarly but have underlying policy disagreements. Issues like Health and Public Lands fall in the “frames polarized, ideal points not” quadrant: Republicans voted similarly but framed the issue very differently. Issues like Macroeconomics and Government Operations fall in the “both polarized” quadrant, posing the greatest challenge for Republican leadership.
• Sub-group Identification: The model identifies legislators whose language marks them as ideologically aligned with the Tea Party even without formal caucus membership. For example, Jeff Flake (R-AZ) received the second-highest ideal point, disagreeing with Freedom Works on only one of 60 key votes, despite not being a Tea Party Caucus member. Justin Amash (R-MI), founder and chairman of the Liberty Caucus, agreed with Freedom Works on every key vote since 2011. Conversely, some self-identified Tea Partiers like Rodney Alexander (R-LA) only agreed with Freedom Works 48% of the time. Alexander and Ander Crenshaw (R-FL, 50% agreement) are categorized as “Green Tea” by Gervais and Morris (2014): Republican legislators who associate with the Tea Party on their own initiative but lack support from Tea Party organizations.

Limitations

• HIPTM does not formally distinguish frames from other kinds of subtopics. For example, the model discovered a strongly Tea Party-oriented frame under “Labor, Employment and Immigration” that reflected a Boeing labor dispute specific to South Carolina legislators, capturing geographic rather than ideological framing.
• The model is validated only on Republican legislators in the 112th Congress. Generalization to other parties, chambers, or time periods is untested.

Reproducibility Details

Data

The study focuses on the 112th U.S. Congress (Jan 2011 - Jan 2013).

Algorithms

The model uses a Stochastic EM approach for inference.

• Generative Process:
  • Speeches: Modeled as a mixture of $K$ Hierarchical Dirichlet Processes (HDPs). A legislator chooses an issue $z$, then a frame $t$ from a Dirichlet Process, then a word $w$.
  • Bills: Modeled using Latent Dirichlet Allocation (LDA). Each bill is a mixture over $K$ issues.
  • Votes: Modeled via a probabilistic ideal point function (logistic/inverse-logit). The probability of a “Yes” vote depends on the bill’s polarity $x_b$, popularity $y_b$, and the legislator’s issue-specific ideal point $u_{a,k}$.
• Optimization Steps:
  1. Sampling: Issue assignments $z$ and frame assignments $t$ are sampled for tokens in speeches and bills.
  2. Regression: Frame-specific regression weights $\eta_{k,j}$ are optimized using L-BFGS.
  3. Ideal Points: Legislator ideal points $u_{a,k}$ and bill parameters ($x_b, y_b$) are updated using Gradient Ascent.

Models

• Ideal Point Definition: A legislator’s ideal point on issue $k$ ($u_{a,k}$) is defined as a linear combination of the ideal points of the frames they use ($\eta_{k,j}$), weighted by their usage frequency ($\hat{\psi}_{a,k,j}$).
• Topic Hierarchy:
  • Level 1 (Issues): Fixed at $K=19$ (based on Policy Agendas Project major headings). These nodes use informed Dirichlet priors.
  • Level 2 (Frames): Unbounded number of frames per issue, discovered non-parametrically via Dirichlet Process.
• Prediction Features: The model runs for 1,000 iterations total with a 500-iteration burn-in. After burn-in, the sampled state is kept every 50 iterations, and feature values are averaged over the 10 stored models.

Evaluation

• Primary Metric: AUC-ROC (Area Under the Receiver Operating Characteristic Curve).
• Classifier: $\text{SVM}^{\text{light}}$ (Joachims, 1999).
• Cross-Validation: 5-fold stratified sampling.

Artifacts

No official code release accompanies this paper. The inference algorithm (Stochastic EM with Gibbs sampling, L-BFGS, and gradient ascent) is described in detail in Section 4 of the paper, but a full reimplementation would be required.

Paper Information

Citation: Nguyen, V., Boyd-Graber, J., Resnik, P., & Miler, K. (2015). Tea Party in the House: A Hierarchical Ideal Point Topic Model and Its Application to Republican Legislators in the 112th Congress. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, 1438-1448. https://doi.org/10.3115/v1/P15-1139

Publication: ACL 2015

@inproceedings{nguyenTeaPartyHouse2015,
  title = {Tea {{Party}} in the {{House}}: {{A Hierarchical Ideal Point Topic Model}} and {{Its Application}} to {{Republican Legislators}} in the 112th {{Congress}}},
  shorttitle = {Tea {{Party}} in the {{House}}},
  booktitle = {Proceedings of the 53rd {{Annual Meeting}} of the {{Association}} for {{Computational Linguistics}} and the 7th {{International Joint Conference}} on {{Natural Language Processing}} ({{Volume}} 1: {{Long Papers}})},
  author = {Nguyen, Viet-An and {Boyd-Graber}, Jordan and Resnik, Philip and Miler, Kristina},
  year = {2015},
  pages = {1438--1448},
  publisher = {Association for Computational Linguistics},
  address = {Beijing, China},
  doi = {10.3115/v1/P15-1139},
  urldate = {2023-11-02},
  abstract = {We introduce the Hierarchical Ideal Point Topic Model, which provides a rich picture of policy issues, framing, and voting behavior using a joint model of votes, bill text, and the language that legislators use when debating bills. We use this model to look at the relationship between Tea Party Republicans and ``establishment'' Republicans in the U.S. House of Representatives during the 112th Congress.},
  langid = {english}
}

Additional Resources:

• ACL Anthology: Tea Party in the House
• Gervais, B. T., & Morris, I. L. (2012). Reading the tea leaves: Understanding Tea Party Caucus membership in the US House of Representatives. PS: Political Science & Politics, 45(2), 245-250.
• Gervais, B. T., & Morris, I. L. (2014). Black Tea, Green Tea, White Tea, and Coffee: Understanding the variation in attachment to the Tea Party among members of Congress. In Annual Meeting of the American Political Science Association. (Source of the “Green Tea” Republican taxonomy cited in the paper)

This is a Method paper. It proposes a novel neural architecture that modifies how bill embeddings are constructed by explicitly incorporating sponsor metadata alongside text. The authors validate this method by comparing it against text-only baselines (MWE and CNN) and demonstrating superior performance in a newly defined “out-of-session” evaluation setting.

What is the motivation?

Existing models for predicting legislative roll-call votes rely heavily on text or voting history within a single session. However, these models fail to generalize across sessions because the underlying data generation process changes. Specifically, the ideological position of bills on similar topics shifts depending on which party is in power. A model trained on a single session learns an implicit ideological prior that becomes inaccurate when the political context changes in subsequent sessions.

What is the novelty here?

The core novelty is a neural architecture that augments bill text representations with sponsor ideology, specifically the percentage of Republican vs. Democrat sponsors.

• Sponsor-Weighted Embeddings: They compute a composite embedding where the text representation is weighted by party sponsorship percentages ($p_{r}, p_{d}$) and party-specific influence vectors ($a_{r}, a_{d}$).
• Out-of-Session Evaluation: They introduce a rigorous evaluation setting where models trained on past sessions (e.g., 2005-2012) are tested on future sessions (e.g., 2013-2014) to test generalization, which previous work had ignored.

What experiments were performed?

The authors evaluated their models using a dataset of U.S. Congressional bills from 2005 to 2016.

• Models Tested: They compared text-only models (MWE (Mean Word Embedding), CNN) against metadata-augmented versions (MWE+Meta, CNN+Meta) and a “Meta-Only” baseline (using dummy text).
• Settings:
  • In-Session: 5-fold cross-validation on 2005-2012 data.
  • Out-of-Session: Training on 2005-2012 and testing on 2013-2014 and 2015-2016.
• Baselines: Comparisons included a “Guess Yes” baseline and an SVM trained on bag-of-words summaries with sponsor indicators.

What outcomes/conclusions?

• Metadata is Critical: Augmenting text with sponsor metadata consistently outperformed text-only models. The CNN+Meta model achieved the highest accuracy in-session (86.21% vs. 83.24% for CNN) and on 2013-2014 out-of-session (83.59%), while MWE+Meta achieved the best 2015-2016 accuracy (71.90%).
• Generalization: Text-only models degraded significantly in out-of-session testing. For example, CNN dropped from 83.24% in-session to 77.49% on 2013-2014 and 69.63% on 2015-2016, confirming that text alone fails to capture shifting ideological contexts.
• Sponsor Signal: The Meta-Only model (using no text) outperformed text-only models in the 2013-2014 out-of-session test (82.28% vs. 77.57% for MWE), suggesting that in some contexts, the author’s identity provides a stronger predictive signal than the bill’s content.
• 2015-2016 Difficulty: All models performed worse on the 2015-2016 session, where intra-party divisions within the House Republican caucus disrupted typical voting dynamics.

Reproducibility Details

Data

• Source: Collected from GovTrack. The paper text references the “106th to 111th” Congressional sessions, but the data tables show coverage from 2005 to 2016, which corresponds to the 109th through 114th sessions.
• Content: Non-unanimous roll call votes, full text of bills/resolutions, and Congressional Research Service (CRS) summaries.
• Filtering: Bills with unanimous votes were excluded.
• Preprocessing:
  • Text lowercased and stop-words removed.
  • Summaries truncated to $N=400$ words; full text truncated to $N=2000$ words (80th percentile lengths).
• Splits:
  • Training: Sessions 2005-2012 (1718 bills).
  • Testing: Sessions 2013-2014 (360 bills) and 2015-2016 (382 bills).

Algorithms

• Bill Representation ($v_{B}$): $$v_{B}=((a_{r}p_{r})\cdot T_{r})+((a_{d}p_{d})\cdot T_{d})$$ Where $T$ is the text embedding (CNN or MWE), $p$ is the percentage of sponsors from a party, and $a$ is a learnable party influence vector. $T_{r}$ and $T_{d}$ are Republican and Democratic copies of the same bill’s text representation, each weighted by the corresponding party’s sponsorship proportion.
• Vote Prediction:
  • Project bill embedding to legislator space: $v_{BL}=W_{B}v_{B}+b_{B}$.
  • Alignment score: $W_{v}(v_{BL}\odot v_{L})+b_{v}$ (using element-wise multiplication).
  • Output: Sigmoid activation.
• Optimization: AdaMax algorithm with binary cross-entropy loss.

Models

• Text Encoders:
  • CNN: 4-grams with 400 filter maps.
  • MWE: Mean Word Embedding.
• Embeddings:
  • Initialized with 50-dimensional GloVe vectors.
  • Embeddings are non-static (updated during training).
  • Legislator embedding size ($v_{L}$): 25 dimensions.
• Initialization: Weights initialized with Glorot uniform distribution.

Evaluation

• Metrics: Accuracy.
• Comparison:
  • In-session: 5-fold cross-validation.
  • Out-of-session: Train on past sessions, predict future sessions.

Hardware

• Training Config: Models trained for 50 epochs with mini-batches of size 50. No specific GPU or compute requirements are reported.

Artifacts

No official code repository or pretrained models were released with this paper.

Paper Information

Citation: Kornilova, A., Argyle, D., & Eidelman, V. (2018). Party Matters: Enhancing Legislative Embeddings with Author Attributes for Vote Prediction. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 510-515. https://doi.org/10.18653/v1/p18-2081

Publication: ACL 2018

@inproceedings{kornilovaPartyMattersEnhancing2018,
  title = {Party {{Matters}}: {{Enhancing Legislative Embeddings}} with {{Author Attributes}} for {{Vote Prediction}}},
  shorttitle = {Party {{Matters}}},
  author = {Kornilova, Anastassia and Argyle, Daniel and Eidelman, Vlad},
  year = {2018},
  booktitle = {Proceedings of the 56th {{Annual Meeting}} of the {{Association}} for {{Computational Linguistics}} ({{Volume}} 2: {{Short Papers}})},
  pages = {510--515},
  publisher = {Association for Computational Linguistics},
  address = {Melbourne, Australia},
  doi = {10.18653/v1/p18-2081},
  eprint = {1805.08182},
  archiveprefix = {arXiv},
}

This is a Methodological ($\Psi_{\text{Method}}$) paper. It introduces a “general nonlinear logit model” and a specific estimation algorithm (NOMINATE) to analyze political choice data. The paper focuses on deriving a metric spatial map from nominal data (yea/nay votes). It validates this method by comparing it against existing techniques like Guttman scaling and factor analysis, demonstrating that the new method recovers geometric structures that previous methods obscured.

What is the motivation?

Prior research relied on “black box” statistical methods (like factor analysis or nonmetric scaling) or Guttman scaling to analyze legislative behavior. These methods had significant limitations:

• Metric Recovery: They struggled to accurately recover the underlying Euclidean coordinates of legislators and choices from nominal data.
• Dimensionality: They tended to exaggerate the number of dimensions (issues) because they did not account for probabilistic error in voting.
• Identification: Pure Guttman scaling (assuming perfect voting) identifies only the order of legislators, leaving the location of policy alternatives unknown.

The authors sought to bridge the “crucial gap” between spatial theory and data by developing a model-driven procedure that simultaneously estimates the locations of choosers and choices while accounting for error.

What is the novelty here?

The core contribution is the NOMINATE (Nominal Three-step Estimation) procedure. Key innovations include:

• Simultaneous Estimation: This method estimates coordinates for both the legislators ($x_i$) and the roll call outcomes ($z_{jl}$) in a common space simultaneously.
• Probabilistic Utility: It employs a specific bell-shaped utility function with a stochastic error term (log of the inverse exponential), allowing for a tractable probabilistic voting model.
• Metric Unfolding: It successfully performs “unfolding methodology for nominal level data,” recovering metric distances solely from binary choices.

What experiments were performed?

The authors validated the model through both historical data analysis and synthetic testing:

• US House Analysis (1957-58): Analyzed 172 roll calls from the 85th Congress to compare NOMINATE results against Miller and Stokes’ influential Guttman scales.
• US Senate Analysis (1979-1982): Performed separate estimations for four years of Senate voting to assess stability and validity.
• Monte Carlo Simulations: Generated synthetic data (98 legislators and 291 roll calls in most runs, 50 legislators in one run) for different values of $\beta$ to test the robustness of parameter recovery under known “truth” conditions.
• Robustness Checks: Tested sensitivity to “perfect” legislators (who never vote against their side) and outliers (like Senator Proxmire).

What outcomes/conclusions?

• Unidimensionality: A single liberal-conservative dimension correctly classified ~80% of individual choices in the US House and Senate.
• Dimensionality Reduction: The model demonstrated that distinct “issue scales” found in previous research (e.g., social welfare vs. foreign policy) could largely be mapped onto a single dimension when error is accounted for.
• Strategic Behavior: The analysis revealed that majority leadership tends to place roll call midpoints slightly away from the median legislator to increase the probability of passage.
• Geometric Mean Probability: The authors introduced the geometric mean probability as a more robust metric than simple classification error for evaluating probabilistic models.
• Limitations: The authors acknowledge that the model is restricted to one dimension with a common utility function, and that civil rights voting represents a genuinely separate dimension not captured by the liberal-conservative axis. Standard errors computed from the alternating procedure are theoretically approximate (computed from separate information matrices rather than the full joint matrix), though Monte Carlo tests showed them to be reasonably reliable in practice. Extensions to multidimensional models and variable utility functions are deferred to later work.

Reproducibility Details

Data

The paper analyzes roll call voting matrices (a roll call is a procedure in which each legislator’s name is called and their individual vote is recorded, producing a complete public record of who voted which way) where rows are legislators and columns are roll calls.

Algorithms

The NOMINATE algorithm maximizes the log-likelihood of observed choices using a constrained nonlinear maximum likelihood procedure.

Utility Function: The utility of legislator $i$ for outcome $j$ on roll call $l$ is: $$U_{ijl}=\beta~\exp\left[\frac{-\omega^{2}d_{ijl}^{2}}{2}\right]+\epsilon_{ijl}$$ Where $d_{ijl}$ is the Euclidean distance between legislator $i$ and outcome $j$.

Optimization Strategy (Global Iteration): Because estimating ~800 parameters simultaneously is impractical, the algorithm uses an alternating three-step method:

1. Utility Parameters: Estimate $\beta$ and $\omega$ while holding legislator ($x$) and roll call ($z$) coordinates fixed.
2. Legislator Coordinates: Estimate $x_i$ for each legislator (independent of others) holding $\beta, \omega, z$ fixed.
3. Roll Call Coordinates: Estimate $z_{yl}, z_{nl}$ for each roll call holding $\beta, \omega, x$ fixed.

This cycle repeats until parameters correlate at the 0.99 level between iterations.

Models

The model estimates the following parameters for a one-dimensional space:

• Legislator Coordinates ($x_i$): The ideal point of each legislator, normalized to the range $[-1, +1]$.
• Outcome Coordinates ($z_{yl}, z_{nl}$): The spatial location of the “Yea” and “Nay” policy outcomes for each vote.
• Signal-to-Noise ($\beta$): Represents the weight of the spatial component versus the error term.
• Weighting ($\omega$): A shape parameter for the utility function (often fixed to $0.5$ in practice due to collinearity with $\beta$).

Evaluation

Performance is evaluated primarily via classification accuracy and probabilistic fit.

Hardware

• Development: DEC-2060
• Production: VAX-11/780

Reproducibility Status

This paper predates modern open-source conventions. No original source code was released, and the NOMINATE algorithm was described at an overview level rather than with full pseudocode. However, the underlying roll call voting data for the U.S. Congress is now freely available through the Voteview project, which Poole and Rosenthal later maintained. Modern open-source reimplementations exist, including the R packages wnominate and pscl. Reproducibility status: Partially Reproducible (data available, modern reimplementations exist, but original code not released).

Paper Information

Citation: Poole, K. T., & Rosenthal, H. (1985). A Spatial Model for Legislative Roll Call Analysis. American Journal of Political Science, 29(2), 357-384. https://doi.org/10.2307/2111172

Publication: American Journal of Political Science 1985

@article{pooleSpatialModelLegislative1985,
  title = {A {{Spatial Model}} for {{Legislative Roll Call Analysis}}},
  author = {Poole, Keith T. and Rosenthal, Howard},
  year = 1985,
  journal = {American Journal of Political Science},
  volume = {29},
  number = {2},
  pages = {357--384},
  doi = {10.2307/2111172}
}

Additional Resources:

• Wikipedia: NOMINATE
• Voteview (Modern Repository)

This post explores machine learning approaches for classifying congressional bills by policy area, using data from the 115th to 117th Congresses (2017-2023). We’ll examine:

• The fundamentals of bill classification
• Traditional machine learning models as baselines
• Performance analysis across different time periods and policy domains

This work establishes baselines for future deep learning approaches to legislative text classification.

This post builds on the data foundation established in Exploring the 117th U.S. Congress.

Motivation

Automatically classifying congressional bills by policy area has practical value for researchers, journalists, and citizens who need to navigate thousands of bills each Congress. Machine learning can help identify patterns in legislative priorities and track policy trends over time.

Data

The data comes from scraping Congress.gov for all bills from the 115th through 117th Congresses. Each bill includes:

• Bill ID and title
• Summary (when available): the earliest summary provided
• Full text (when available): the earliest text version
• Policy area classification

Our task is to predict policy area from text features:

$$ f(X) = \hat{y}, \quad \text{where} \quad X = { \text{title}, \text{summary}, \text{text} }, \quad \hat{y} \in { \text{policy areas} } $$

The complete dataset is available at Hugging Face: hheiden/us-congress-bill-policy-115_117.

Bills by Congress

Our dataset contains the following distribution:

Policy Areas

Each bill receives a policy area label from Congress.gov (see glossary). The dataset includes 33 policy areas, though these classes are highly imbalanced.

The following table shows the number of bills in each policy area across the three Congresses:

The class imbalance is severe: Social Sciences and History has only 15 bills across all three Congresses, while Health has 5,911 bills. This imbalance presents modeling challenges, as minority classes may lack sufficient representative samples.

Text Statistics

We analyzed token counts using spaCy to understand the computational requirements for each text field.

Title Token Statistics:

Summary Token Statistics:

Full Text Token Statistics:

These statistics reveal computational trade-offs:

• Titles average ~12 tokens: computationally efficient but limited information.
• Summaries average ~100 tokens: good balance of information and efficiency.
• Full text averages ~2,700 tokens with 129M total tokens: detailed but computationally expensive. Processing this volume of text introduces real-world engineering challenges, such as memory constraints and a higher noise-to-signal ratio typical of long legal documents.

We’ll prototype with titles and summaries before considering full text, given the computational costs involved.

Evaluation Framework

Experimental Design

We train models on one Congress and test on others, creating a 3x3 evaluation grid. This setup evaluates both within-Congress performance (same session) and cross-Congress generalization (different sessions). We expect temporal drift between Congress sessions to impact performance.

Metrics and Hyperparameter Tuning

We use weighted average F1 score to handle class imbalance, ensuring fair evaluation across all policy areas regardless of frequency.

For within-Congress evaluation, we report cross-validated scores. For cross-Congress evaluation, we test on the entire target Congress dataset.

Hyperparameter tuning uses Cross-Validation Grid Search with folds set to min(3, n_samples) to ensure all classes are represented. We apply the best parameters from training to test generalization across different Congresses.

Baseline Models

We evaluate three traditional machine learning approaches using TF-IDF vectorization:

Text Preprocessing

We convert text to numerical features using TF-IDF (term frequency-inverse document frequency), which weighs word importance by frequency within documents relative to the entire corpus. This creates normalized feature vectors suitable for machine learning classification.

Multinomial Naive Bayes

We start with Multinomial Naive Bayes as our simplest baseline. Despite its “naive” independence assumption between features, this model often performs surprisingly well for text classification tasks and serves as an important benchmark. If more complex models can’t beat Naive Bayes, it signals potential issues with the approach or data.

The model’s feature_log_prob_ attribute reveals the most influential words for each policy area, providing interpretable insights into classification patterns.

You can see the code for training the Naive Bayes model below:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB

# Create a pipeline with TF-IDF vectorizer and Multinomial Naive Bayes classifier
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(lowercase=True, dtype=np.float32)),
    ('clf', MultinomialNB()),
])

# Define the parameters for grid search
parameters = {  
    'tfidf__ngram_range': [(1, 1), (1, 2), (1, 3)],
    'tfidf__max_df': (0.05, 0.1, 0.25, 0.5),
    'tfidf__min_df': (1, 2, 5, 10),
    'clf__alpha': (1, 0.1, 0.01, 0.001),
}

# Perform grid search with cross-validation
grid_search = GridSearchCV(
    pipeline,
    parameters,
    scoring='f1_weighted',
    n_jobs=-1,
    refit=True,
    cv=3,
)
grid_search.fit(X_train, y_train)

Logistic Regression

Logistic regression provides a natural step up in complexity from Naive Bayes. It uses the logistic function to convert linear combinations of features into probabilities, making it an excellent baseline for comparison with more sophisticated models while remaining interpretable.

You can see the code for training the Logistic Regression model below:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

# Create a pipeline with TF-IDF vectorizer and Logistic Regression classifier
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(lowercase=True, dtype=np.float32)),
    ('clf', LogisticRegression(max_iter=1000, random_state=42, class_weight='balanced')),
])

# Define the parameters for grid search
parameters = {  
    'tfidf__ngram_range': [(1, 1), (1, 2)],
    'tfidf__max_df': (0.05, 0.1, 0.25),
    'clf__C': [0.1, 1, 10],
}

# Perform grid search with cross-validation
grid_search = GridSearchCV(
    pipeline,
    parameters,
    scoring='f1_weighted',
    n_jobs=-1,
    refit=True,
    cv=3,
)
grid_search.fit(X_train, y_train)

XGBoost

We include XGBoost as our tree-based ensemble method. While XGBoost typically excels on structured tabular data, we test whether its gradient boosting approach can effectively handle TF-IDF features for text classification.

You can see the code for training the XGBoost model below:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from xgboost import XGBClassifier

# Create a pipeline with TF-IDF vectorizer and XGBoost classifier
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(lowercase=True, dtype=np.float32)),
    ('clf', XGBClassifier(use_label_encoder=False, eval_metric='mlogloss', objective='multi:softmax', seed=42, n_jobs=-1)),
])

# Define the parameters for grid search
parameters = {  
    'tfidf__max_df': (0.05, 0.1, 0.25),
    'clf__max_depth': (3, 6, 9),
    'clf__n_estimators': (100, 200, 300),
}

# Perform grid search with cross-validation
grid_search = GridSearchCV(
    pipeline,
    parameters,
    scoring='f1_weighted',
    refit=True,
    cv=3,
    verbose=3,
)
grid_search.fit(X_train, y_train, clf__sample_weight=sample_weight)

Results

We evaluate models on three input types:

• Title-only: Quick prototyping with limited context
• Summary-only: Balanced information content and computational efficiency
• Full text: Maximum context with computational constraints (limited hyperparameter tuning)

Title-Only Inputs

Naive Bayes

Title-only Naive Bayes experiments are run with the following settings:

sweep_nb(
    data,
    X_key='title',
    y_key='policy_area',
    tfidf_params={
        'lowercase': True,
        'dtype': np.float32,
    },
    tfidf_grid={
        'ngram_range': [(1, 1), (1, 2)],
        'max_df': (0.05, 0.1, 0.25, 0.5),
        'min_df': (1, 2, 5),
    },
    nb_params={},
    nb_grid={
        'alpha': (1, 0.1, 0.01, 0.001),
    },
)

and the results:

Training on Congress 115
Best score: 0.661
Refit Time: 0.570
Best parameters set:
	clf__alpha: 0.01
	tfidf__max_df: 0.05
	tfidf__min_df: 1
	tfidf__ngram_range: (1, 2)
Testing on Congress 116 F1: 0.6369760774921475
Testing on Congress 117 F1: 0.5488274400521962

Training on Congress 116
Best score: 0.677
Refit Time: 0.499
Best parameters set:
	clf__alpha: 0.01
	tfidf__max_df: 0.05
	tfidf__min_df: 1
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.691175262953872
Testing on Congress 117 F1: 0.6798043069585031

Training on Congress 117
Best score: 0.670
Refit Time: 0.565
Best parameters set:
	clf__alpha: 0.01
	tfidf__max_df: 0.25
	tfidf__min_df: 1
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.6168474701996426
Testing on Congress 116 F1: 0.6981574942116808

Mean fit time: 0.54 ± 0.03s

Results Summary

The results demonstrate several key findings:

• Fast training: Sub-second training times make this highly practical
• Solid baseline performance: F1 scores around 0.65-0.70 provide a reasonable starting point
• Consistent hyperparameters: Similar optimal settings across Congresses suggest stable patterns
• Temporal effects: Performance generally decreases when training and testing on Congresses further apart in time

Training on the 116th Congress yields the best cross-Congress performance, likely due to its temporal proximity to both adjacent sessions.

The model learns interpretable features for each policy area. For example, Agriculture bills are strongly associated with terms like “farm,” “crop,” and “livestock,” while Armed Forces bills correlate with “military,” “defense,” and “veterans.”

Logistic Regression

Title-only Logistic Regression experiments are run with the following settings:

sweep_logreg(
    data,
    X_key='title',
    y_key='policy_area',
    tfidf_params={
        'lowercase': True,
        'dtype': np.float32,
    },
    tfidf_grid={
        'ngram_range': [(1, 1), (1, 2)],
        'max_df': (0.05, 0.1, 0.25),
    },
    logreg_params={
        'max_iter': 1000,
        'random_state': 42,
        'class_weight': 'balanced',
    },
    logreg_grid={
        'C': [0.1, 1, 10],
    },
)

and the results:

Training on Congress 115
Best score: 0.704
Refit Time: 32.063
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.05
	tfidf__ngram_range: (1, 2)
Testing on Congress 116 F1: 0.6809188275881766
Testing on Congress 117 F1: 0.601917336933838

Training on Congress 116
Best score: 0.714
Refit Time: 31.227
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.05
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.7408989977276476
Testing on Congress 117 F1: 0.7200639105208106

Training on Congress 117
Best score: 0.711
Refit Time: 34.083
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.05
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.674418393892329
Testing on Congress 116 F1: 0.7405934743144291

Mean fit time: 32.46 ± 1.20s

Results Summary

Logistic regression improves upon Naive Bayes performance:

• Higher F1 scores: Generally 5-7 percentage points better than Naive Bayes
• Consistent hyperparameters: Optimal settings remain stable across Congresses
• Reasonable training time: 30-35 seconds per model remains manageable
• Strong cross-Congress generalization: F1 scores consistently above 0.70

XGBoost

Title-only XGBoost experiments are run with the following settings:

sweep_xgb(
    data,
    X_key='title',
    y_key='policy_area',
    tfidf_grid={
        'max_df': (0.05,),
    },
    xgb_grid={
        'max_depth': (6,),
        'eta': (0.3,),
    },
)

and the results:

Training on Congress 115
Best score: 0.591
Refit Time: 198.063
Best parameters set:
	clf__eta: 0.3
	clf__max_depth: 6
	clf__num_class: 33
	tfidf__max_df: 0.05
Testing on Congress 116 F1: 0.5649530686141018
Testing on Congress 117 F1: 0.5215939580735101

Training on Congress 116
Best score: 0.600
Refit Time: 264.824
Best parameters set:
	clf__eta: 0.3
	clf__max_depth: 6
	clf__num_class: 33
	tfidf__max_df: 0.05
Testing on Congress 115 F1: 0.6037922738570368
Testing on Congress 117 F1: 0.5965027418245722

Training on Congress 117
Best score: 0.595
Refit Time: 249.799
Best parameters set:
	clf__eta: 0.3
	clf__max_depth: 6
	clf__num_class: 33
	tfidf__max_df: 0.05
Testing on Congress 115 F1: 0.5600491477899472
Testing on Congress 116 F1: 0.60815381664894

Mean fit time: 237.56 ± 28.60s

Results Summary

XGBoost underperforms relative to expectations:

• Poor performance: F1 scores significantly below linear models (0.55-0.60 range)
• Long training times: 4+ minutes per model with limited hyperparameter exploration
• Questionable value: The computational cost doesn’t justify the poor performance

Given these results, we focus on the more promising linear models for subsequent experiments with longer text inputs.

Training Efficiency

The computational costs vary dramatically:

XGBoost’s poor performance despite high computational cost suggests that tree-based methods may not be well-suited for sparse TF-IDF features. This is a classic example of the “curse of dimensionality”: tree-based models struggle to make effective splits in highly sparse, high-dimensional bag-of-words spaces compared to linear models that simply assign weights to all features simultaneously. We’ll focus on linear models for the remaining experiments.

Summary-Only Results

Using bill summaries provides substantially more context than titles alone, leading to significant performance improvements.

Naive Bayes Performance

The summary-based models show dramatic improvement over title-only versions:

• F1 scores: 0.85+ within-Congress, 0.77-0.86 cross-Congress
• Training time: Still fast at ~3.4 seconds
• Strong generalization: Consistent performance across time periods

Logistic Regression Performance

Logistic regression slightly outperforms Naive Bayes on summaries:

• F1 scores: 0.86+ within-Congress, 0.79-0.87 cross-Congress
• Training time: Reasonable at ~12 seconds
• Stable hyperparameters: Consistent optimal settings across Congresses

The performance difference between models suggests they rely on similar feature patterns, with logistic regression better capturing feature interactions.

Logistic Regression

Summary-only Logistic Regression experiments are run with the following settings:

sweep_logreg(
    data,
    X_key='summary',
    y_key='policy_area',
    tfidf_params={
        'lowercase': True,
        'dtype': np.float32,
    },
    tfidf_grid={
        # 'ngram_range': [(1, 1), (1, 2)],
        'max_df': (0.05, 0.1, 0.25),
    },
    logreg_params={
        'max_iter': 1000,
        'random_state': 42,
        'class_weight': 'balanced',
    },
    logreg_grid={
        'C': [0.1, 1, 10],
    },
)

And the results:

Training on Congress 115
Best score: 0.862
Refit Time: 9.007
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.25
Testing on Congress 116 F1: 0.8284864693401133
Testing on Congress 117 F1: 0.7934161507811646

Training on Congress 116
Best score: 0.865
Refit Time: 13.897
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.25
Testing on Congress 115 F1: 0.8637852557418315
Testing on Congress 117 F1: 0.8594775615031977

Training on Congress 117
Best score: 0.862
Refit Time: 12.167
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.25
Testing on Congress 115 F1: 0.8355736563084967
Testing on Congress 116 F1: 0.8696403838390832

Mean fit time: 11.69 ± 2.02s

Full Text Results

We test whether complete bill text improves performance over summaries, using optimal hyperparameters from summary experiments.

Naive Bayes on Full Text

Surprisingly, full text yields slightly lower performance than summaries:

• F1 scores: 0.84-0.85 within-Congress, 0.77-0.86 cross-Congress
• Training time: ~50 seconds (10x slower than summaries)
• Performance drop: Likely due to increased noise in lengthy documents

Logistic Regression on Full Text

Logistic regression shows the strongest performance on full text:

• F1 scores: 0.87-0.88 within-Congress, 0.83-0.89 cross-Congress
• Training time: ~70 seconds
• Best overall performance: up to 0.89 F1 on the strongest single cross-Congress pair (best within-Congress score 0.877)

The logistic regression model benefits from having access to complete legislative language while effectively regularizing against noise.

Key Findings

This baseline study establishes several important results:

Best performing model: Logistic regression trained on full bill text reaches up to 0.89 F1 on the strongest single cross-Congress pair (best within-Congress score 0.877), providing a strong benchmark for future deep learning approaches.

Text input comparison:

• Titles: Limited but fast (F1 ~0.65-0.70)
• Summaries: Good balance of performance and efficiency (F1 ~0.85)
• Full text: Best performance but computationally expensive (certified weighted-F1 0.871-0.877; up to ~0.89 on the strongest single cross-Congress pair)

Cross-Congress generalization: Models trained on one Congress generalize reasonably well to others, though performance decreases with temporal distance between sessions.

Model performance ranking: Logistic Regression > Naive Bayes » XGBoost for this text classification task.

Next Steps

The strong baseline performance sets the stage for several research directions:

1. Deep learning models: Transformer-based approaches using pre-trained language models
2. Dataset expansion: Including additional Congresses and more detailed bill metadata
3. Error analysis: Understanding failure cases and class-specific performance patterns
4. Feature engineering: Exploring domain-specific text preprocessing and feature extraction

The complete dataset and experimental code are available for researchers interested in building upon these baselines.

Resources:

• Dataset: Hugging Face: hheiden/us-congress-bill-policy-115_117
• Leaderboard: Policy Area Classification Leaderboard
• Project: Congressional Knowledge Graph & Policy Classification

Analyzing congressional data reveals the underlying mechanics of the legislative process. Legislative text is a large, structured corpus well suited to text classification and other NLP tasks. I scraped data from Congress.gov to analyze what actually happens to the thousands of bills introduced each session and to build a foundational dataset for downstream machine learning tasks.

This analysis focuses on the 117th Congress (2021-2023), examining 15,000+ bills to understand basic patterns: Which bills get introduced? How many receive votes? What factors influence success?

This post covers the foundational exploratory analysis and data collection process, setting the stage for predictive modeling and policy area classification.

Data Collection

My primary source is Congress.gov, maintained by the Library of Congress. I focused on the 117th Congress (2021-2023), collecting data on bills and joint resolutions, omitting simple resolutions, concurrent resolutions, and amendments.

Data collected:

Technical Implementation

Building a usable NLP dataset requires careful handling of the source. Congress.gov loads content dynamically and presents nested DOM structures, so the scraper combines static HTML parsing with a headless browser to render JavaScript before parsing.

Implementation details:

• Python for core orchestration and data schema management
• Selenium for executing JavaScript and loading dynamic page elements
• BeautifulSoup for structured HTML parsing
• Regex for text normalization and extracting clean legislative text for language models

The crawler used 5-second delays between requests to respect server limits, a roughly 3-day collection run. It handles edge cases in congressional text formatting and writes one JSON record per bill on a fixed schema. The crawler and processed data are available on GitHub.

For each bill, I queried two pages:

• All info page: https://www.congress.gov/bill/117th-congress/{bill_type}/{bill_id}/all-info
• Text page: https://www.congress.gov/bill/117th-congress/{bill_type}/{bill_id}/text?format=txt

The parsing process involved targeting specific HTML elements and implementing basic caching to avoid redundant requests.

Key Findings

The analysis reveals clear patterns in congressional activity. Most bills never receive votes, and success rates vary significantly by party and policy area.

Legislative Outcomes

The fundamental question: what happens to bills after introduction?

Each bill has a tracker status indicating its position in the legislative process. The eight possible statuses can be grouped into three meaningful categories:

• Introduced: Bills introduced but never voted on
• Stalled: Bills that saw votes but didn’t become law (since the 117th Congress ended, these effectively died)
• Law: Bills signed by the President

Key insights:

• Only 7% of introduced bills ever receive a vote
• Of bills that receive votes, 36% become law
• Overall, just 2% of introduced bills become law

Sponsor Analysis

The bill sponsor (the primary member who introduces legislation) provides insights into party and geographic patterns.

Party Breakdown

Party comparison:

• Democrats: 7.5% of bills moved beyond introduction; 2.6% became law
• Republicans: 5.5% of bills moved beyond introduction; 2.1% became law
• When bills do advance, Republicans have a slightly higher success rate (38% vs 35%)

Geographic Distribution

Top 10 states by bills introduced:

Per-representative normalization reveals different patterns:

Top Individual Sponsors

Most prolific legislators by bills introduced:

Effectiveness score (laws enacted / total bills):

$$ \text{effectiveness} = \frac{\text{bills that became law}}{\text{total bills introduced}} $$

Policy Focus Areas

Each bill is assigned a primary policy area. Here are the most active areas by legislative outcome:

Notable patterns: Health dominates introductions but has lower success rates, while government operations and armed forces bills are more likely to become law.

Next Steps

This analysis establishes baseline patterns: most bills fail, party affiliation affects success rates, and certain policy areas perform better than others.

Future work could explore:

• Committee dynamics and voting patterns
• Geographic analysis of state-level interests
• Bill text analysis using NLP techniques
• Predictive modeling for bill outcomes

Update: I’ve since applied machine learning to this type of data in Congressional Bill Policy Area Classification, using 48K+ bills from three Congresses to automatically categorize bills by policy area.

The complete dataset and code are publicly available to support further research into legislative transparency.

A computational social science project that constructed a dataset of 47,000+ US congressional bills by extracting legislative text and metadata from the 115th-117th Congresses. The project creates a “legislative graph” (linking sponsors, committees, and bill text) and establishes TF-IDF baseline models for policy area classification across 33 (highly imbalanced) policy classes, now hosted on Hugging Face to support reproducible political science research.

Features

Intelligent Data Acquisition

Standard APIs impose strict rate limits. I built a Selenium-based extraction engine to handle Congress.gov’s complex DOM structures.

• Optimization: Targeted aggregate endpoints (e.g., /all-info) to pull each bill’s text and metadata in fewer requests.
• Resilience: Implemented a local caching layer to store raw HTML, separating the fetch step from the parse step. This made the parse step re-runnable without re-fetching, and minimized server load during iterative development.
• Graph construction: Beyond simple text, the script extracts relational data including co-sponsorship networks, committee assignments, and related bill lineage.

Natural Language Processing

• Corpus construction: Cleaned and normalized legislative text, removing procedural artifacts (e.g., “A BILL TO…”) to isolate semantic policy content.
• Feature engineering: Utilized TF-IDF vectorization with N-gram analysis to capture legislative jargon.
• Modeling: Benchmarked Naive Bayes, Logistic Regression, and gradient-boosted trees (XGBoost), reaching ~0.86 weighted F1 on bill summaries and up to ~0.89 on full text (cross-validated). Weighted F1, not raw accuracy, is the honest metric here: the 33 policy classes are severely imbalanced (Health has 5,911 bills; Social Sciences and History has 15).

Usage

The dataset is available on Hugging Face and can be loaded directly via the datasets library. The scraper can be run locally to fetch new bills.

Results

• The “partisan vocabulary”: Feature importance analysis revealed distinct linguistic markers separating Democratic and Republican legislation, identifiable even without metadata.
• Temporal drift: Policy priorities and terminology showed measurable shifts across congressional sessions (115th vs 117th).
• Classification success: Simple linear models (Logistic Regression and Naive Bayes) proved effective at distinguishing policy domains, outperforming gradient-boosted trees on these sparse TF-IDF features and suggesting legislative language is highly structured.

Impact & Deliverables

• Hugging Face dataset: Released a machine-readable, ML-ready dataset of modern bills (115th-117th Congresses) on Hugging Face for reproducible research.
• Open source tooling: Published the scraper and parsing logic to allow others to extend the dataset to future congresses.
• Academic benchmark: Establishing a clear baseline for “Government NLP” tasks, aiding in the automated transparency and monitoring of new legislation.

Related Work

• 117th Congress Data Exploration
• Congressional Bill Policy Area Classification