Method.

This paper is primarily a Methodological contribution. It proposes a novel probabilistic architecture, the Hierarchical Ideal Point Topic Model (HIPTM), designed to solve the specific limitations of existing political science models that typically rely on either voting data or text data in isolation. The paper validates this method by demonstrating its superior performance in predicting “Tea Party” membership compared to text-only baselines and its ability to provide interpretable “framing” analysis.

What is the motivation?

The primary motivation is to better understand political polarization, specifically the “Tea Party” phenomenon within the Republican party during the 112th Congress.

An ideal point is a scalar score representing a legislator’s ideological position, estimated from voting patterns. Standard “Ideal Point” models (like DW-NOMINATE) typically project legislators onto a single liberal-conservative dimension using only binary voting data. This is insufficient for capturing complex, multi-dimensional intra-party conflicts where legislators might agree on votes but differ on policy “framing” or specific sub-issues. Furthermore, existing multi-dimensional models often produce dimensions that are difficult for humans to interpret.

What is the novelty here?

The core novelty is the Hierarchical Ideal Point Topic Model (HIPTM). It distinguishes itself from prior work through three main technical innovations:

1. Joint Modeling of Three Data Sources: It integrates roll call votes, the text of bills, and the floor speeches of legislators into a single probabilistic framework.
2. Hierarchical Topic Structure: It models “frames” as a second level of the topic hierarchy. “Issues” (level 1) are fixed and non-polarized, while “Frames” (level 2) are discovered dynamically and carry polarity (ideal point weights). For example, Health Care is an issue; “government overreach” vs. “patient protection” are frames legislators use when debating it.
3. Text-Based Ideal Point Prediction: HIPTM regresses ideal points on speech text, allowing it to predict the political alignment of legislators based solely on their writing or speeches without requiring voting records for inference.

What experiments were performed?

The authors validated the model using data from the 112th U.S. Congress (Republican legislators only).

• Prediction Task: Classifying legislators as members of the “Tea Party Caucus”.
• Baselines: The model was compared against Support Vector Machines (SVM) trained on:
  • TF-IDF vectors (Text only)
  • Normalized TF-IDF vectors (Text only)
  • Binary Vote vectors (Vote only)
• Metric: Area Under the Receiver Operating Characteristic Curve (AUC-ROC) via 5-fold cross-validation.
• Qualitative Analysis: The authors examined the “span” of ideal points within specific topics (e.g., Macroeconomics, Health) to identify which issues were most polarized between Tea Party and Establishment Republicans.

What were the outcomes and conclusions drawn?

• Quantitative Performance: HIPTM features combined with voting data (HIPTM-VOTE) achieved the highest classification performance (AUC-ROC in the ~0.70-0.75 range, approximate, read from Figure 2). Vote-only features slightly trail HIPTM-VOTE, while text-only baselines (TF-IDF, normalized TF-IDF) fall considerably lower. The one-dimensional Tea Party ideal points correlate with DW-NOMINATE ($\rho = 0.91$). When voting data was withheld (simulating a candidate without a record), HIPTM’s text-based features outperformed standard text baselines TF-IDF and normalized TF-IDF (approximate, read from Figure 3).
• Political Insight: The model identified “Government Operations,” “Macroeconomics,” and “Transportation” as the three most polarized topics between Tea Party and establishment Republicans.
• Framing Analysis: The hierarchical topic structure reveals how legislators frame issues differently. For Macroeconomics, frame M3 (most Tea Party-oriented) focuses on criticizing government overspending, while frame M1 (least Tea Party-oriented) focuses on the downsides of a government shutdown. For Health, frame H3 captures Tea Party framing of the Affordable Care Act as an unconstitutional government takeover, while frame H1 frames opposition in terms of implementation costs and health care exchanges.
• Framing vs. Voting Taxonomy: The authors construct a 2x2 taxonomy of disagreement across issues, crossing whether ideal points are polarized with whether issue frames are polarized. Issues like Civil Rights fall in the “neither polarized” quadrant, where cooperation is expected. Banking/Finance and Transportation fall in the “ideal points polarized, frames not” quadrant, where Republicans frame the issue similarly but have underlying policy disagreements. Issues like Health and Public Lands fall in the “frames polarized, ideal points not” quadrant: Republicans voted similarly but framed the issue very differently. Issues like Macroeconomics and Government Operations fall in the “both polarized” quadrant, posing the greatest challenge for Republican leadership.
• Sub-group Identification: The model identifies legislators whose language marks them as ideologically aligned with the Tea Party even without formal caucus membership. For example, Jeff Flake (R-AZ) received the second-highest ideal point, disagreeing with Freedom Works on only one of 60 key votes, despite not being a Tea Party Caucus member. Justin Amash (R-MI), founder and chairman of the Liberty Caucus, agreed with Freedom Works on every key vote since 2011. Conversely, some self-identified Tea Partiers like Rodney Alexander (R-LA) only agreed with Freedom Works 48% of the time. Alexander and Ander Crenshaw (R-FL, 50% agreement) are categorized as “Green Tea” by Gervais and Morris (2014): Republican legislators who associate with the Tea Party on their own initiative but lack support from Tea Party organizations.

Limitations

• HIPTM does not formally distinguish frames from other kinds of subtopics. For example, the model discovered a strongly Tea Party-oriented frame under “Labor, Employment and Immigration” that reflected a Boeing labor dispute specific to South Carolina legislators, capturing geographic rather than ideological framing.
• The model is validated only on Republican legislators in the 112th Congress. Generalization to other parties, chambers, or time periods is untested.

Reproducibility Details

Data

The study focuses on the 112th U.S. Congress (Jan 2011 - Jan 2013).

Algorithms

The model uses a Stochastic EM approach for inference.

• Generative Process:
  • Speeches: Modeled as a mixture of $K$ Hierarchical Dirichlet Processes (HDPs). A legislator chooses an issue $z$, then a frame $t$ from a Dirichlet Process, then a word $w$.
  • Bills: Modeled using Latent Dirichlet Allocation (LDA). Each bill is a mixture over $K$ issues.
  • Votes: Modeled via a probabilistic ideal point function (logistic/inverse-logit). The probability of a “Yes” vote depends on the bill’s polarity $x_b$, popularity $y_b$, and the legislator’s issue-specific ideal point $u_{a,k}$.
• Optimization Steps:
  1. Sampling: Issue assignments $z$ and frame assignments $t$ are sampled for tokens in speeches and bills.
  2. Regression: Frame-specific regression weights $\eta_{k,j}$ are optimized using L-BFGS.
  3. Ideal Points: Legislator ideal points $u_{a,k}$ and bill parameters ($x_b, y_b$) are updated using Gradient Ascent.

Models

• Ideal Point Definition: A legislator’s ideal point on issue $k$ ($u_{a,k}$) is defined as a linear combination of the ideal points of the frames they use ($\eta_{k,j}$), weighted by their usage frequency ($\hat{\psi}_{a,k,j}$).
• Topic Hierarchy:
  • Level 1 (Issues): Fixed at $K=19$ (based on Policy Agendas Project major headings). These nodes use informed Dirichlet priors.
  • Level 2 (Frames): Unbounded number of frames per issue, discovered non-parametrically via Dirichlet Process.
• Prediction Features: The model runs for 1,000 iterations total with a 500-iteration burn-in. After burn-in, the sampled state is kept every 50 iterations, and feature values are averaged over the 10 stored models.

Evaluation

• Primary Metric: AUC-ROC (Area Under the Receiver Operating Characteristic Curve).
• Classifier: $\text{SVM}^{\text{light}}$ (Joachims, 1999).
• Cross-Validation: 5-fold stratified sampling.

Artifacts

No official code release accompanies this paper. The inference algorithm (Stochastic EM with Gibbs sampling, L-BFGS, and gradient ascent) is described in detail in Section 4 of the paper, but a full reimplementation would be required.

Paper Information

Citation: Nguyen, V., Boyd-Graber, J., Resnik, P., & Miler, K. (2015). Tea Party in the House: A Hierarchical Ideal Point Topic Model and Its Application to Republican Legislators in the 112th Congress. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, 1438-1448. https://doi.org/10.3115/v1/P15-1139

Publication: ACL 2015

@inproceedings{nguyenTeaPartyHouse2015,
  title = {Tea {{Party}} in the {{House}}: {{A Hierarchical Ideal Point Topic Model}} and {{Its Application}} to {{Republican Legislators}} in the 112th {{Congress}}},
  shorttitle = {Tea {{Party}} in the {{House}}},
  booktitle = {Proceedings of the 53rd {{Annual Meeting}} of the {{Association}} for {{Computational Linguistics}} and the 7th {{International Joint Conference}} on {{Natural Language Processing}} ({{Volume}} 1: {{Long Papers}})},
  author = {Nguyen, Viet-An and {Boyd-Graber}, Jordan and Resnik, Philip and Miler, Kristina},
  year = {2015},
  pages = {1438--1448},
  publisher = {Association for Computational Linguistics},
  address = {Beijing, China},
  doi = {10.3115/v1/P15-1139},
  urldate = {2023-11-02},
  abstract = {We introduce the Hierarchical Ideal Point Topic Model, which provides a rich picture of policy issues, framing, and voting behavior using a joint model of votes, bill text, and the language that legislators use when debating bills. We use this model to look at the relationship between Tea Party Republicans and ``establishment'' Republicans in the U.S. House of Representatives during the 112th Congress.},
  langid = {english}
}

Additional Resources:

• ACL Anthology: Tea Party in the House
• Gervais, B. T., & Morris, I. L. (2012). Reading the tea leaves: Understanding Tea Party Caucus membership in the US House of Representatives. PS: Political Science & Politics, 45(2), 245-250.
• Gervais, B. T., & Morris, I. L. (2014). Black Tea, Green Tea, White Tea, and Coffee: Understanding the variation in attachment to the Tea Party among members of Congress. In Annual Meeting of the American Political Science Association. (Source of the “Green Tea” Republican taxonomy cited in the paper)

This is a Method paper. It proposes a novel neural architecture that modifies how bill embeddings are constructed by explicitly incorporating sponsor metadata alongside text. The authors validate this method by comparing it against text-only baselines (MWE and CNN) and demonstrating superior performance in a newly defined “out-of-session” evaluation setting.

What is the motivation?

Existing models for predicting legislative roll-call votes rely heavily on text or voting history within a single session. However, these models fail to generalize across sessions because the underlying data generation process changes. Specifically, the ideological position of bills on similar topics shifts depending on which party is in power. A model trained on a single session learns an implicit ideological prior that becomes inaccurate when the political context changes in subsequent sessions.

What is the novelty here?

The core novelty is a neural architecture that augments bill text representations with sponsor ideology, specifically the percentage of Republican vs. Democrat sponsors.

• Sponsor-Weighted Embeddings: They compute a composite embedding where the text representation is weighted by party sponsorship percentages ($p_{r}, p_{d}$) and party-specific influence vectors ($a_{r}, a_{d}$).
• Out-of-Session Evaluation: They introduce a rigorous evaluation setting where models trained on past sessions (e.g., 2005-2012) are tested on future sessions (e.g., 2013-2014) to test generalization, which previous work had ignored.

What experiments were performed?

The authors evaluated their models using a dataset of U.S. Congressional bills from 2005 to 2016.

• Models Tested: They compared text-only models (MWE (Mean Word Embedding), CNN) against metadata-augmented versions (MWE+Meta, CNN+Meta) and a “Meta-Only” baseline (using dummy text).
• Settings:
  • In-Session: 5-fold cross-validation on 2005-2012 data.
  • Out-of-Session: Training on 2005-2012 and testing on 2013-2014 and 2015-2016.
• Baselines: Comparisons included a “Guess Yes” baseline and an SVM trained on bag-of-words summaries with sponsor indicators.

What outcomes/conclusions?

• Metadata is Critical: Augmenting text with sponsor metadata consistently outperformed text-only models. The CNN+Meta model achieved the highest accuracy in-session (86.21% vs. 83.24% for CNN) and on 2013-2014 out-of-session (83.59%), while MWE+Meta achieved the best 2015-2016 accuracy (71.90%).
• Generalization: Text-only models degraded significantly in out-of-session testing. For example, CNN dropped from 83.24% in-session to 77.49% on 2013-2014 and 69.63% on 2015-2016, confirming that text alone fails to capture shifting ideological contexts.
• Sponsor Signal: The Meta-Only model (using no text) outperformed text-only models in the 2013-2014 out-of-session test (82.28% vs. 77.57% for MWE), suggesting that in some contexts, the author’s identity provides a stronger predictive signal than the bill’s content.
• 2015-2016 Difficulty: All models performed worse on the 2015-2016 session, where intra-party divisions within the House Republican caucus disrupted typical voting dynamics.

Reproducibility Details

Data

• Source: Collected from GovTrack. The paper text references the “106th to 111th” Congressional sessions, but the data tables show coverage from 2005 to 2016, which corresponds to the 109th through 114th sessions.
• Content: Non-unanimous roll call votes, full text of bills/resolutions, and Congressional Research Service (CRS) summaries.
• Filtering: Bills with unanimous votes were excluded.
• Preprocessing:
  • Text lowercased and stop-words removed.
  • Summaries truncated to $N=400$ words; full text truncated to $N=2000$ words (80th percentile lengths).
• Splits:
  • Training: Sessions 2005-2012 (1718 bills).
  • Testing: Sessions 2013-2014 (360 bills) and 2015-2016 (382 bills).

Algorithms

• Bill Representation ($v_{B}$): $$v_{B}=((a_{r}p_{r})\cdot T_{r})+((a_{d}p_{d})\cdot T_{d})$$ Where $T$ is the text embedding (CNN or MWE), $p$ is the percentage of sponsors from a party, and $a$ is a learnable party influence vector. $T_{r}$ and $T_{d}$ are Republican and Democratic copies of the same bill’s text representation, each weighted by the corresponding party’s sponsorship proportion.
• Vote Prediction:
  • Project bill embedding to legislator space: $v_{BL}=W_{B}v_{B}+b_{B}$.
  • Alignment score: $W_{v}(v_{BL}\odot v_{L})+b_{v}$ (using element-wise multiplication).
  • Output: Sigmoid activation.
• Optimization: AdaMax algorithm with binary cross-entropy loss.

Models

• Text Encoders:
  • CNN: 4-grams with 400 filter maps.
  • MWE: Mean Word Embedding.
• Embeddings:
  • Initialized with 50-dimensional GloVe vectors.
  • Embeddings are non-static (updated during training).
  • Legislator embedding size ($v_{L}$): 25 dimensions.
• Initialization: Weights initialized with Glorot uniform distribution.

Evaluation

• Metrics: Accuracy.
• Comparison:
  • In-session: 5-fold cross-validation.
  • Out-of-session: Train on past sessions, predict future sessions.

Hardware

• Training Config: Models trained for 50 epochs with mini-batches of size 50. No specific GPU or compute requirements are reported.

Artifacts

No official code repository or pretrained models were released with this paper.

Paper Information

Citation: Kornilova, A., Argyle, D., & Eidelman, V. (2018). Party Matters: Enhancing Legislative Embeddings with Author Attributes for Vote Prediction. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 510-515. https://doi.org/10.18653/v1/p18-2081

Publication: ACL 2018

@inproceedings{kornilovaPartyMattersEnhancing2018,
  title = {Party {{Matters}}: {{Enhancing Legislative Embeddings}} with {{Author Attributes}} for {{Vote Prediction}}},
  shorttitle = {Party {{Matters}}},
  author = {Kornilova, Anastassia and Argyle, Daniel and Eidelman, Vlad},
  year = {2018},
  booktitle = {Proceedings of the 56th {{Annual Meeting}} of the {{Association}} for {{Computational Linguistics}} ({{Volume}} 2: {{Short Papers}})},
  pages = {510--515},
  publisher = {Association for Computational Linguistics},
  address = {Melbourne, Australia},
  doi = {10.18653/v1/p18-2081},
  eprint = {1805.08182},
  archiveprefix = {arXiv},
}

Sarcasm detection represents one of the most challenging problems in NLP. The difficulties include:

Context dependence: Sarcasm relies on situational knowledge and shared understanding that extends beyond the text itself.

Subtlety: Even humans struggle with sarcastic interpretation, especially in written text without vocal cues.

Cultural variability: Sarcastic expressions vary significantly across cultures and regions.

Annotation disagreement: Human annotators often disagree on what constitutes sarcasm.

These challenges raise a fundamental question: can sarcasm detection be well-defined as a computational problem? This case study explores what happens when we try (and reveals a common pitfall in dataset construction).

The Dataset: A Hidden Flaw

I used the Sarcasm News Headlines dataset, which combines headlines from The Onion (satirical) and The Huffington Post (traditional news). The dataset contains ~50,000 examples.

from datasets import load_dataset

dataset = load_dataset("raquiba/Sarcasm_News_Headline")
print(dataset["train"][0])
print(dataset["train"][1])

{'headline': 'thirtysomething scientists unveil doomsday clock of hair loss',
 'is_sarcastic': 1}
{'headline': 'dem rep. totally nails why congress is falling short on gender, racial equality',
 'is_sarcastic': 0}

The critical flaw: This dataset uses binary classification based on source domain. The Onion headlines are labeled sarcastic, HuffPost headlines are not. This creates a dangerous shortcut where models learn to detect the publication source.

After preprocessing to standardize column names:

dataset = dataset.map(
    lambda example: {"text": example["headline"], "label": example["is_sarcastic"]},
    remove_columns=["headline", "article_link", "is_sarcastic"]
)

Fine-Tuning RoBERTa

I fine-tuned a pre-trained RoBERTa model using standard practices:

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments

model_name = "FacebookAI/roberta-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)

# Tokenize the data
def tokenize_function(examples):
    return tokenizer(examples["text"], truncation=True, max_length=512)

tokenized_datasets = dataset.map(tokenize_function, batched=True)

# Training configuration
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=5,
    per_device_train_batch_size=32,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["test"],
    tokenizer=tokenizer,
)

trainer.train()

Results: Too Good to Be True

The model achieved high accuracy:

This should immediately raise red flags. Sarcasm detection is notoriously difficult, even for humans. Such high accuracy suggests the model learned a proxy task.

My hypothesis: The model bypassed sarcasm detection entirely, learning only to distinguish between The Onion and HuffPost writing styles.

Interacting with the Model

Let’s test our hypothesis by interacting with the model.

First, let’s load the model and tokenizer:

from transformers import pipeline

model = AutoModelForSequenceClassification.from_pretrained('results/2024-02-25_20-24-51/checkpoint-4475')

clf = pipeline('text-classification', model=model, tokenizer=tokenizer)

Now, let’s test the model with some examples.

First, let’s try an Onion article from this week, something I know to be sarcastic and not in the training data. Let’s use “Alabama Supreme Court Justice Invokes ‘VeggieTales’ In Ruling”:

clf("Alabama Supreme Court Justice Invokes ‘VeggieTales' In Ruling")

[{'label': 'LABEL_0', 'score': 0.99916672706604}]

The model is extremely confident that this is not sarcastic.

Let’s try a different Onion article, possibly even more difficult: Breaking News Trump Booed, Frozen Burritos, And More: This Week In Breaking News February 24, 2024:

clf("Breaking News Trump Booed, Frozen Burritos, And More: This Week In Breaking News February 24, 2024")

[{'label': 'LABEL_0', 'score': 0.9993497729301453}]

Again, very confident that this is not sarcastic. Hmm. It could be the temporal accuracy of our model just cannot capture the sarcasm of the Onion in 2024.

Let’s try one more Onion article, this one that is still recent but a bit more of a low-hanging fruit: Mom Only Likes The Other Outback Steakhouse:

clf("Mom Only Likes The Other Outback Steakhouse")

[{'label': 'LABEL_1', 'score': 0.9997231364250183}]

Finally, a correct prediction! The model is confident that this is sarcastic. Our model detects only very specific types of sarcasm. It fails to generalize to new, unseen data within the same domain.

Let’s also try some headlines from the Huffington Post, which the model should predict as not sarcastic. Let’s try the five most recent headlines from the Huffington Post:

• Donald Trump Won South Carolina - But There’s 1 Big Caveat
• Man Sets Himself On Fire In Front Of Israeli Embassy In Washington
• Israeli Media Report Progress On Reaching A Temporary Truce In Gaza And A Hostage-Prisoner Exchange
• A White Liberal Is Trying To Oust A Progressive Black Congressman. His Comments Could Make That Job Harder.
• Climate Change-Fueled Winter Extremes Put 90% Of This Country At ‘High Risk’

clf([
    "Donald Trump Won South Carolina - But There's 1 Big Caveat",
    "Man Sets Himself On Fire In Front Of Israeli Embassy In Washington",
    "Israeli Media Report Progress On Reaching A Temporary Truce In Gaza And A Hostage-Prisoner Exchange",
    "A White Liberal Is Trying To Oust A Progressive Black Congressman. His Comments Could Make That Job Harder.",
    "Climate Change-Fueled Winter Extremes Put 90% Of This Country At 'High Risk'"
])

[{'label': 'LABEL_0', 'score': 0.9993808269500732},
 {'label': 'LABEL_0', 'score': 0.9993786811828613},
 {'label': 'LABEL_0', 'score': 0.9985186457633972},
 {'label': 'LABEL_0', 'score': 0.9993883371353149},
 {'label': 'LABEL_0', 'score': 0.9993487000465393}]

The model is extremely confident that these are not sarcastic.

The model detects sarcasm in limited cases. It fails to generalize to new, unseen data within the same domain. This is a common problem in machine learning. Training a model that performs well on a specific dataset is straightforward. Training a model that generalizes to new, unseen data remains a significant challenge. Furthermore, our sarcasm detection project resulted in a domain classifier. For fuzzier concepts like sarcasm, it’s important to be clear about what we’re actually detecting, and to collect the necessary scale of data to capture the full range of the concept.

Key Takeaways

This case study reveals a fundamental problem in ML: high accuracy guarantees only performance on the training distribution. Here’s what actually happened:

1. Dataset bias: Using publication source as a proxy for sarcasm created a shortcut for the model
2. Domain classification: The model exclusively learned to distinguish writing styles
3. Poor generalization: New examples from the same sources often failed

This is a common pitfall when building datasets for subjective concepts. The lesson: high accuracy must be accompanied by validation of the model’s actual learned behavior.

For better sarcasm detection, we’d need:

• Diverse sources beyond two publications
• Human annotation across multiple contexts
• Careful evaluation on out-of-domain examples

Instructive failures in ML projects provide valuable lessons about our assumptions and the limitations of our approaches.

This post explores machine learning approaches for classifying congressional bills by policy area, using data from the 115th to 117th Congresses (2017-2023). We’ll examine:

• The fundamentals of bill classification
• Traditional machine learning models as baselines
• Performance analysis across different time periods and policy domains

This work establishes baselines for future deep learning approaches to legislative text classification.

This post builds on the data foundation established in Exploring the 117th U.S. Congress.

Why This Matters

Automatically classifying congressional bills by policy area has practical value for researchers, journalists, and citizens who need to navigate thousands of bills each Congress. Machine learning can help identify patterns in legislative priorities and track policy trends over time.

This work establishes baseline performance for text classification on legislative data, providing a foundation for more sophisticated approaches.

Data

The data comes from scraping Congress.gov for all bills from the 115th through 117th Congresses. Each bill includes:

• Bill ID and title
• Summary (when available): the earliest summary provided
• Full text (when available): the earliest text version
• Policy area classification

Our task is to predict policy area from text features:

$$ f(X) = \hat{y}, \quad \text{where} \quad X = { \text{title}, \text{summary}, \text{text} }, \quad \hat{y} \in { \text{policy areas} } $$

The complete dataset is available at Hugging Face: hheiden/us-congress-bill-policy-115_117.

Bills by Congress

Our dataset contains the following distribution:

Policy Areas

Each bill receives a policy area label from Congress.gov (see glossary). The dataset includes 33 policy areas, though these classes are highly imbalanced.

The following table shows the number of bills in each policy area across the three Congresses:

The class imbalance is severe: Social Sciences and History has only 15 bills across all three Congresses, while Health has 5,911 bills. This imbalance presents modeling challenges, as minority classes may lack sufficient representative samples.

Text Statistics

We analyzed token counts using spaCy to understand the computational requirements for each text field.

Title Token Statistics:

Summary Token Statistics:

Full Text Token Statistics:

These statistics reveal computational trade-offs:

• Titles average ~12 tokens: computationally efficient but limited information.
• Summaries average ~100 tokens: good balance of information and efficiency.
• Full text averages ~2,700 tokens with 129M total tokens: detailed but computationally expensive. Processing this volume of text introduces real-world engineering challenges, such as memory constraints and a higher noise-to-signal ratio typical of long legal documents.

We’ll prototype with titles and summaries before considering full text, given the computational costs involved.

Evaluation Framework

Experimental Design

We train models on one Congress and test on others, creating a 3x3 evaluation grid. This setup evaluates both within-Congress performance (same session) and cross-Congress generalization (different sessions). We expect temporal drift between Congress sessions to impact performance.

Metrics and Hyperparameter Tuning

We use weighted average F1 score to handle class imbalance, ensuring fair evaluation across all policy areas regardless of frequency.

For within-Congress evaluation, we report cross-validated scores. For cross-Congress evaluation, we test on the entire target Congress dataset.

Hyperparameter tuning uses Cross-Validation Grid Search with folds set to min(3, n_samples) to ensure all classes are represented. We apply the best parameters from training to test generalization across different Congresses.

Baseline Models

We evaluate three traditional machine learning approaches using TF-IDF vectorization:

Text Preprocessing

We convert text to numerical features using TF-IDF (term frequency-inverse document frequency), which weighs word importance by frequency within documents relative to the entire corpus. This creates normalized feature vectors suitable for machine learning classification.

Multinomial Naive Bayes

We start with Multinomial Naive Bayes as our simplest baseline. Despite its “naive” independence assumption between features, this model often performs surprisingly well for text classification tasks and serves as an important benchmark. If more complex models can’t beat Naive Bayes, it signals potential issues with the approach or data.

The model’s feature_log_prob_ attribute reveals the most influential words for each policy area, providing interpretable insights into classification patterns.

You can see the code for training the Naive Bayes model below:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB

# Create a pipeline with TF-IDF vectorizer and Multinomial Naive Bayes classifier
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(lowercase=True, dtype=np.float32)),
    ('clf', MultinomialNB()),
])

# Define the parameters for grid search
parameters = {  
    'tfidf__ngram_range': [(1, 1), (1, 2), (1, 3)],
    'tfidf__max_df': (0.05, 0.1, 0.25, 0.5),
    'tfidf__min_df': (1, 2, 5, 10),
    'clf__alpha': (1, 0.1, 0.01, 0.001),
}

# Perform grid search with cross-validation
grid_search = GridSearchCV(
    pipeline,
    parameters,
    scoring='f1_weighted',
    n_jobs=-1,
    refit=True,
    cv=3,
)
grid_search.fit(X_train, y_train)

Logistic Regression

Logistic regression provides a natural step up in complexity from Naive Bayes. It uses the logistic function to convert linear combinations of features into probabilities, making it an excellent baseline for comparison with more sophisticated models while remaining interpretable.

You can see the code for training the Logistic Regression model below:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

# Create a pipeline with TF-IDF vectorizer and Logistic Regression classifier
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(lowercase=True, dtype=np.float32)),
    ('clf', LogisticRegression(max_iter=1000, random_state=42, class_weight='balanced')),
])

# Define the parameters for grid search
parameters = {  
    'tfidf__ngram_range': [(1, 1), (1, 2)],
    'tfidf__max_df': (0.05, 0.1, 0.25),
    'clf__C': [0.1, 1, 10],
}

# Perform grid search with cross-validation
grid_search = GridSearchCV(
    pipeline,
    parameters,
    scoring='f1_weighted',
    n_jobs=-1,
    refit=True,
    cv=3,
)
grid_search.fit(X_train, y_train)

XGBoost

We include XGBoost as our tree-based ensemble method. While XGBoost typically excels on structured tabular data, we test whether its gradient boosting approach can effectively handle TF-IDF features for text classification.

You can see the code for training the XGBoost model below:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline
from xgboost import XGBClassifier

# Create a pipeline with TF-IDF vectorizer and XGBoost classifier
pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(lowercase=True, dtype=np.float32)),
    ('clf', XGBClassifier(use_label_encoder=False, eval_metric='mlogloss', objective='multi:softmax', seed=42, n_jobs=-1)),
])

# Define the parameters for grid search
parameters = {  
    'tfidf__max_df': (0.05, 0.1, 0.25),
    'clf__max_depth': (3, 6, 9),
    'clf__n_estimators': (100, 200, 300),
}

# Perform grid search with cross-validation
grid_search = GridSearchCV(
    pipeline,
    parameters,
    scoring='f1_weighted',
    refit=True,
    cv=3,
    verbose=3,
)
grid_search.fit(X_train, y_train, clf__sample_weight=sample_weight)

Results

We evaluate models on three input types:

• Title-only: Quick prototyping with limited context
• Summary-only: Balanced information content and computational efficiency
• Full text: Maximum context with computational constraints (limited hyperparameter tuning)

Title-Only Inputs

Naive Bayes

Title-only Naive Bayes experiments are run with the following settings:

sweep_nb(
    data,
    X_key='title',
    y_key='policy_area',
    tfidf_params={
        'lowercase': True,
        'dtype': np.float32,
    },
    tfidf_grid={
        'ngram_range': [(1, 1), (1, 2)],
        'max_df': (0.05, 0.1, 0.25, 0.5),
        'min_df': (1, 2, 5),
    },
    nb_params={},
    nb_grid={
        'alpha': (1, 0.1, 0.01, 0.001),
    },
)

and the results:

Training on Congress 115
Best score: 0.661
Refit Time: 0.570
Best parameters set:
	clf__alpha: 0.01
	tfidf__max_df: 0.05
	tfidf__min_df: 1
	tfidf__ngram_range: (1, 2)
Testing on Congress 116 F1: 0.6369760774921475
Testing on Congress 117 F1: 0.5488274400521962

Training on Congress 116
Best score: 0.677
Refit Time: 0.499
Best parameters set:
	clf__alpha: 0.01
	tfidf__max_df: 0.05
	tfidf__min_df: 1
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.691175262953872
Testing on Congress 117 F1: 0.6798043069585031

Training on Congress 117
Best score: 0.670
Refit Time: 0.565
Best parameters set:
	clf__alpha: 0.01
	tfidf__max_df: 0.25
	tfidf__min_df: 1
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.6168474701996426
Testing on Congress 116 F1: 0.6981574942116808

Mean fit time: 0.54 ± 0.03s

Results Summary

The results demonstrate several key findings:

• Fast training: Sub-second training times make this highly practical
• Solid baseline performance: F1 scores around 0.65-0.70 provide a reasonable starting point
• Consistent hyperparameters: Similar optimal settings across Congresses suggest stable patterns
• Temporal effects: Performance generally decreases when training and testing on Congresses further apart in time

Training on the 116th Congress yields the best cross-Congress performance, likely due to its temporal proximity to both adjacent sessions.

The model learns interpretable features for each policy area. For example, Agriculture bills are strongly associated with terms like “farm,” “crop,” and “livestock,” while Armed Forces bills correlate with “military,” “defense,” and “veterans.”

Logistic Regression

Title-only Logistic Regression experiments are run with the following settings:

sweep_logreg(
    data,
    X_key='title',
    y_key='policy_area',
    tfidf_params={
        'lowercase': True,
        'dtype': np.float32,
    },
    tfidf_grid={
        'ngram_range': [(1, 1), (1, 2)],
        'max_df': (0.05, 0.1, 0.25),
    },
    logreg_params={
        'max_iter': 1000,
        'random_state': 42,
        'class_weight': 'balanced',
    },
    logreg_grid={
        'C': [0.1, 1, 10],
    },
)

and the results:

Training on Congress 115
Best score: 0.704
Refit Time: 32.063
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.05
	tfidf__ngram_range: (1, 2)
Testing on Congress 116 F1: 0.6809188275881766
Testing on Congress 117 F1: 0.601917336933838

Training on Congress 116
Best score: 0.714
Refit Time: 31.227
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.05
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.7408989977276476
Testing on Congress 117 F1: 0.7200639105208106

Training on Congress 117
Best score: 0.711
Refit Time: 34.083
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.05
	tfidf__ngram_range: (1, 2)
Testing on Congress 115 F1: 0.674418393892329
Testing on Congress 116 F1: 0.7405934743144291

Mean fit time: 32.46 ± 1.20s

Results Summary

Logistic regression improves upon Naive Bayes performance:

• Higher F1 scores: Generally 5-7 percentage points better than Naive Bayes
• Consistent hyperparameters: Optimal settings remain stable across Congresses
• Reasonable training time: 30-35 seconds per model remains manageable
• Strong cross-Congress generalization: F1 scores consistently above 0.70

XGBoost

Title-only XGBoost experiments are run with the following settings:

sweep_xgb(
    data,
    X_key='title',
    y_key='policy_area',
    tfidf_grid={
        'max_df': (0.05,),
    },
    xgb_grid={
        'max_depth': (6,),
        'eta': (0.3,),
    },
)

and the results:

Training on Congress 115
Best score: 0.591
Refit Time: 198.063
Best parameters set:
	clf__eta: 0.3
	clf__max_depth: 6
	clf__num_class: 33
	tfidf__max_df: 0.05
Testing on Congress 116 F1: 0.5649530686141018
Testing on Congress 117 F1: 0.5215939580735101

Training on Congress 116
Best score: 0.600
Refit Time: 264.824
Best parameters set:
	clf__eta: 0.3
	clf__max_depth: 6
	clf__num_class: 33
	tfidf__max_df: 0.05
Testing on Congress 115 F1: 0.6037922738570368
Testing on Congress 117 F1: 0.5965027418245722

Training on Congress 117
Best score: 0.595
Refit Time: 249.799
Best parameters set:
	clf__eta: 0.3
	clf__max_depth: 6
	clf__num_class: 33
	tfidf__max_df: 0.05
Testing on Congress 115 F1: 0.5600491477899472
Testing on Congress 116 F1: 0.60815381664894

Mean fit time: 237.56 ± 28.60s

Results Summary

XGBoost underperforms relative to expectations:

• Poor performance: F1 scores significantly below linear models (0.55-0.60 range)
• Long training times: 4+ minutes per model with limited hyperparameter exploration
• Questionable value: The computational cost doesn’t justify the poor performance

Given these results, we focus on the more promising linear models for subsequent experiments with longer text inputs.

Training Efficiency

The computational costs vary dramatically:

XGBoost’s poor performance despite high computational cost suggests that tree-based methods may not be well-suited for sparse TF-IDF features. This is a classic example of the “curse of dimensionality”: tree-based models struggle to make effective splits in highly sparse, high-dimensional bag-of-words spaces compared to linear models that simply assign weights to all features simultaneously. We’ll focus on linear models for the remaining experiments.

Summary-Only Results

Using bill summaries provides substantially more context than titles alone, leading to significant performance improvements.

Naive Bayes Performance

The summary-based models show dramatic improvement over title-only versions:

• F1 scores: 0.85+ within-Congress, 0.77-0.86 cross-Congress
• Training time: Still fast at ~3.4 seconds
• Strong generalization: Consistent performance across time periods

Logistic Regression Performance

Logistic regression slightly outperforms Naive Bayes on summaries:

• F1 scores: 0.86+ within-Congress, 0.79-0.87 cross-Congress
• Training time: Reasonable at ~12 seconds
• Stable hyperparameters: Consistent optimal settings across Congresses

The performance difference between models suggests they rely on similar feature patterns, with logistic regression better capturing feature interactions.

Logistic Regression

Summary-only Logistic Regression experiments are run with the following settings:

sweep_logreg(
    data,
    X_key='summary',
    y_key='policy_area',
    tfidf_params={
        'lowercase': True,
        'dtype': np.float32,
    },
    tfidf_grid={
        # 'ngram_range': [(1, 1), (1, 2)],
        'max_df': (0.05, 0.1, 0.25),
    },
    logreg_params={
        'max_iter': 1000,
        'random_state': 42,
        'class_weight': 'balanced',
    },
    logreg_grid={
        'C': [0.1, 1, 10],
    },
)

And the results:

Training on Congress 115
Best score: 0.862
Refit Time: 9.007
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.25
Testing on Congress 116 F1: 0.8284864693401133
Testing on Congress 117 F1: 0.7934161507811646

Training on Congress 116
Best score: 0.865
Refit Time: 13.897
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.25
Testing on Congress 115 F1: 0.8637852557418315
Testing on Congress 117 F1: 0.8594775615031977

Training on Congress 117
Best score: 0.862
Refit Time: 12.167
Best parameters set:
	clf__C: 10
	tfidf__max_df: 0.25
Testing on Congress 115 F1: 0.8355736563084967
Testing on Congress 116 F1: 0.8696403838390832

Mean fit time: 11.69 ± 2.02s

Full Text Results

We test whether complete bill text improves performance over summaries, using optimal hyperparameters from summary experiments.

Naive Bayes on Full Text

Surprisingly, full text yields slightly lower performance than summaries:

• F1 scores: 0.84-0.85 within-Congress, 0.77-0.86 cross-Congress
• Training time: ~50 seconds (10x slower than summaries)
• Performance drop: Likely due to increased noise in lengthy documents

Logistic Regression on Full Text

Logistic regression shows the strongest performance on full text:

• F1 scores: 0.87-0.88 within-Congress, 0.83-0.89 cross-Congress
• Training time: ~70 seconds
• Best overall performance: Approaches 90% F1 on some Congress pairs

The logistic regression model benefits from having access to complete legislative language while effectively regularizing against noise.

Key Findings

This baseline study establishes several important results:

Best performing model: Logistic regression trained on full bill text achieves up to 89% F1 score, providing a strong benchmark for future deep learning approaches.

Text input comparison:

• Titles: Limited but fast (F1 ~0.65-0.70)
• Summaries: Good balance of performance and efficiency (F1 ~0.85)
• Full text: Best performance but computationally expensive (F1 ~0.87-0.89)

Cross-Congress generalization: Models trained on one Congress generalize reasonably well to others, though performance decreases with temporal distance between sessions.

Model performance ranking: Logistic Regression > Naive Bayes » XGBoost for this text classification task.

Next Steps

The strong baseline performance sets the stage for several research directions:

1. Deep learning models: Transformer-based approaches using pre-trained language models
2. Dataset expansion: Including additional Congresses and more detailed bill metadata
3. Error analysis: Understanding failure cases and class-specific performance patterns
4. Feature engineering: Exploring domain-specific text preprocessing and feature extraction

The complete dataset and experimental code are available for researchers interested in building upon these baselines.

Resources:

• Dataset: Hugging Face: hheiden/us-congress-bill-policy-115_117
• Leaderboard: Policy Area Classification Leaderboard
• Project: Congressional Knowledge Graph & Policy Classification

A computational social science project that constructed a dataset of 47,000+ US congressional bills by extracting legislative text and metadata from the 115th-117th Congresses. The project creates a novel “legislative graph” (linking sponsors, committees, and bill text) and establishes a machine learning benchmark for policy area classification (87% accuracy), now hosted on Hugging Face to support reproducible political science research.

Features

Intelligent Data Acquisition

Standard APIs impose strict rate limits. I engineered a custom Selenium-based extraction engine to handle Congress.gov’s complex DOM structures.

• Optimization: Targeted aggregate endpoints (e.g., /all-info) to reduce HTTP request volume by ~90% per bill.
• Resilience: Implemented a local caching layer to store raw HTML, separating the fetch step from the parse step. This ensured 100% reproducibility and minimized server load during iterative development.
• Graph construction: Beyond simple text, the script extracts relational data including co-sponsorship networks, committee assignments, and related bill lineage.

Natural Language Processing

• Corpus construction: Cleaned and normalized legislative text, removing procedural artifacts (e.g., “A BILL TO…”) to isolate semantic policy content.
• Feature engineering: Utilized TF-IDF vectorization with N-gram analysis to capture legislative jargon.
• Modeling: Benchmarked Naive Bayes, Logistic Regression, and SVMs, achieving 87.3% accuracy on policy area prediction (cross-validated).

Usage

The dataset is available on Hugging Face and can be loaded directly via the datasets library. The scraper can be run locally to fetch new bills.

Results

• The “partisan vocabulary”: Feature importance analysis revealed distinct linguistic markers separating Democratic and Republican legislation, identifiable even without metadata.
• Temporal drift: Policy priorities and terminology showed measurable shifts across congressional sessions (115th vs 117th).
• Classification success: Simple linear models (SVM/LogReg) proved remarkably effective at distinguishing policy domains, suggesting legislative language is highly structured.

Impact & Deliverables

• Hugging Face dataset: Released the first machine-readable, ML-ready dataset of modern bills, democratizing access for researchers.
• Open source tooling: Published the scraper and parsing logic to allow others to extend the dataset to future congresses.
• Academic benchmark: Establishing a clear baseline for “Government NLP” tasks, aiding in the automated transparency and monitoring of new legislation.

Related Work

• 117th Congress Data Exploration
• Congressional Bill Policy Area Classification

A word embedding maps words to real-valued vectors:

$$ \text{word} \rightarrow \mathbb{R}^n $$

where $n$ represents the dimensionality of the embedding space.

The goal is simple: position semantically similar words close together in vector space. This dense representation typically uses hundreds of dimensions, a massive reduction from the millions required by one-hot encoding.

Word embeddings are grounded in Zellig Harris’ distributional hypothesis: words appearing in similar contexts tend to have similar meanings. This forms the foundation of distributional semantics.

Different embedding algorithms capture various aspects of this distributional principle. This post explores the main methods for creating word embeddings and their applications in natural language processing.

While modern foundation models and terabyte-scale Vision-Language Models (VLMs) rely on advanced subword tokenizers (like BPE) and massive Transformer embedding layers, the fundamental goal remains exactly the same: mapping discrete text to a continuous vector space where math can capture meaning. Understanding these foundational techniques provides the necessary intuition for debugging and scaling today’s production ML systems.

Why Word Embeddings Matter in NLP

Computers require numerical representations to apply machine learning algorithms to text. Word embeddings bridge this gap by converting text into dense vectors that preserve semantic and syntactic relationships.

Key advantages:

1. Dense representation: Hundreds of dimensions provide a compact alternative to vocabulary-sized sparse vectors.
2. Semantic preservation: Similar words cluster together in vector space.
3. Mathematical operations: Enable analogical reasoning ($\text{king} - \text{man} + \text{woman} \approx \text{queen}$).
4. Transfer learning: Pre-trained embeddings work across multiple tasks and domains.

Modern deep learning architectures leverage these properties extensively. The development of universal, pre-trained embeddings was a significant step forward. We can use versatile embeddings that generalize across applications, eliminating the need to train task-specific representations from scratch.

Word Embedding Approaches

One-Hot Encoding and Count Vectorization

One-hot encoding represents the simplest approach to word vectorization. Each word gets a unique dimension in a vocabulary-sized vector, marked with 1 for presence and 0 elsewhere. Count vectorization extends this by counting the occurrences of each word in a document.

Characteristics:

• High dimensionality: Vector length equals vocabulary size.
• Extreme sparsity: Most dimensions contain zeros.
• No relationships: Treats all words as equally distant.
• Computational efficiency: Simple to implement and understand.

While lacking semantic information, count vectorization serves as a foundation for more complex methods. Let’s look at a practical implementation using scikit-learn’s CountVectorizer.

from sklearn.feature_extraction.text import CountVectorizer

# Initialize the vectorizer
vectorizer = CountVectorizer()

# Sample text for demonstration
sample_text = ["One of the most basic ways we can numerically represent words "
               "is through the one-hot encoding method (also sometimes called "
               "count vectorizing)."]

# Fit the vectorizer to our text data
vectorizer.fit(sample_text)

# Examine the vocabulary and word indices
print('Vocabulary:')
print(vectorizer.vocabulary_)

# Transform text to vectors
vector = vectorizer.transform(sample_text)
print('Full vector:')
print(vector.toarray())

In a production environment, count vectorization introduces significant engineering challenges. When processing millions of documents, the vocabulary size explodes. Storing and computing on these massive sparse matrices quickly leads to memory exhaustion. In these scaling scenarios, practitioners often turn to the Hashing Trick (via HashingVectorizer) to bound the dimensionality, or they move entirely to the dense embeddings discussed later in this post.

We can see count vectorization in action with a real dataset, building a simple text classifier for the 20 Newsgroups dataset:

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn import metrics

# Load train and test splits, removing metadata for a cleaner signal
newsgroups_train = fetch_20newsgroups(subset='train',
                                      remove=('headers', 'footers', 'quotes'))
newsgroups_test = fetch_20newsgroups(subset='test',
                                     remove=('headers', 'footers', 'quotes'))

# Initialize and fit vectorizer on training data
vectorizer = CountVectorizer()
X_train = vectorizer.fit_transform(newsgroups_train.data)

# Build and train classifier
classifier = MultinomialNB(alpha=0.01)
classifier.fit(X_train, newsgroups_train.target)

# Transform test data and make predictions
X_test = vectorizer.transform(newsgroups_test.data)
y_pred = classifier.predict(X_test)

# Evaluate performance
accuracy = metrics.accuracy_score(newsgroups_test.target, y_pred)
print(f'Accuracy: {accuracy:.3f}')

This provides a solid baseline. To capture actual semantic meaning and reduce dimensionality, we must move beyond simple counting.

TF-IDF (Term Frequency-Inverse Document Frequency)

TF-IDF extends one-hot encoding by weighting terms based on their importance across a document collection. TF-IDF combines:

• Term Frequency (TF): How often a word appears in a document
• Inverse Document Frequency (IDF): How rare a word is across all documents

This weighting scheme reduces the impact of common words (like “the” or “and”) while emphasizing distinctive terms that appear frequently in specific documents but rarely elsewhere.

Advantages:

• Captures document-level importance
• Reduces impact of stop words
• Effective for information retrieval tasks

Limitations:

• Still high-dimensional and sparse
• No semantic relationships between terms
• Context-independent representation

Co-Occurrence Matrices

Co-occurrence matrices capture word relationships by recording which terms appear together within defined contexts (sentences, paragraphs, or fixed windows). The resulting matrix has dimensions equal to vocabulary size squared, with entries showing co-occurrence frequency.

Key properties:

• Global statistics: Captures corpus-wide word relationships
• Symmetric relationships: Mutual co-occurrence patterns
• Extreme dimensionality: Vocabulary size squared creates storage challenges
• Sparse representation: Most word pairs never co-occur

While computationally expensive to store and process, co-occurrence matrices form the foundation for advanced methods like GloVe that compress this information into dense representations.

Neural Network-Based Embeddings

Neural Probabilistic Language Models

Neural probabilistic models pioneered the use of neural networks for learning word embeddings. These models learn dense representations as a byproduct of language modeling, predicting the next word in a sequence.

Training process:

1. Initialize random dense embeddings for each vocabulary word
2. Use embeddings as inputs to predict language modeling objectives
3. Update embeddings through backpropagation based on prediction errors
4. Resulting embeddings capture patterns useful for the training task

This approach demonstrated that task-specific embeddings could be learned jointly with model objectives, establishing the foundation for modern embedding methods.

Word2Vec

Word2Vec revolutionized word embeddings by introducing efficient training algorithms for massive corpora. It became the first method to demonstrate compelling vector arithmetic properties, enabling analogical reasoning like the famous “$\text{king} - \text{man} + \text{woman} \approx \text{queen}$” example.

Two training architectures:

Continuous Bag-of-Words (CBOW)

Predicts target words from surrounding context words. Given a window of context words, the model learns to predict the central word.

Skip-Gram

Predicts context words from target words. Given a central word, the model learns to predict surrounding words within a defined window.

Key advantages:

• Computational efficiency: Much faster than neural probabilistic models
• Scalable training: Can process billion-word corpora effectively
• Quality embeddings: Captures semantic and syntactic relationships
• Flexible context: Window size controls topical vs. functional similarity

The choice of window size significantly impacts learned relationships. Larger windows capture topical associations, while smaller windows focus on syntactic and functional similarities.

GloVe (Global Vectors)

GloVe combines the best aspects of matrix factorization methods (which capture global corpus statistics) and local context window approaches like Word2Vec. Matrix factorization methods excel at global patterns but struggle with analogical reasoning, while Word2Vec captures local relationships but may miss global structure.

Key innovation: GloVe trains on a global word-context co-occurrence matrix, incorporating corpus-wide statistical information while maintaining the analogical reasoning capabilities that made Word2Vec successful.

Advantages over Word2Vec:

• Global optimization: Leverages entire corpus statistics
• Better performance: Often outperforms Word2Vec on word similarity and analogy tasks
• Stable training: More consistent convergence due to global objective function

The result is embeddings that capture both local syntactic patterns and global semantic relationships more effectively.

Contextual Embedding Methods

FastText

FastText addresses a critical limitation of previous methods: handling out-of-vocabulary (OOV) words. By incorporating subword information, FastText can generate meaningful representations for previously unseen words.

Subword approach:

• Decomposes words into character n-grams (typically 3-6 characters)
• Represents words as sums of their component n-grams
• Trains using skip-gram objective with negative sampling

Key advantages:

• OOV handling: Can embed unseen words using known subword components
• Morphological awareness: Captures relationships between related word forms
• Multilingual support: Facebook released pre-trained embeddings for 294 languages
• Robust performance: Particularly effective for morphologically rich languages

For example, if the model knows “navigate,” it can provide meaningful representation for “circumnavigate” by leveraging shared subword components, even if “circumnavigate” wasn’t in the training data.

Poincaré Embeddings

Poincaré embeddings introduce a novel approach by learning representations in hyperbolic space. This geometric innovation specifically targets hierarchical relationships in data.

Hyperbolic geometry advantages:

• Natural hierarchy encoding: Distance represents similarity, while norm encodes hierarchical level
• Efficient representation: Requires fewer dimensions for hierarchical data
• Mathematical elegance: Leverages properties of hyperbolic space for embedding optimization

Applications: Particularly effective for data with inherent hierarchical structure, such as:

• WordNet taxonomies
• Organizational charts
• Computer network topologies
• Knowledge graphs

The original paper demonstrates good efficiency in reproducing WordNet relationships with significantly lower dimensionality compared to traditional embedding methods.

Contextual Embeddings

ELMo (Embeddings from Language Models)

ELMo represents a paradigm shift toward contextual word representations. ELMo generates dynamic representations based on sentence context, adapting to word usage patterns.

Architecture:

• Bidirectional LSTM: Processes text in both forward and backward directions
• Character-level input: Handles OOV words and captures morphological patterns
• Multi-layer representations: Combines different abstraction levels

Layer specialization:

• Lower layers: Excel at syntactic tasks (POS tagging, parsing)
• Higher layers: Capture semantic relationships (word sense disambiguation)
• Combined layers: Weighted combination achieves good performance

Key innovation: ELMo embeddings vary by context. The word “bank” receives different representations in “river bank” versus “financial bank,” addressing polysemy directly through contextual awareness.

This approach achieved strong performance across numerous NLP tasks by providing context-sensitive representations that adapt to word usage patterns.

Probabilistic FastText

Probabilistic FastText addresses polysemy (words with multiple meanings) through probabilistic modeling. Traditional embeddings conflate different word senses into single representations, limiting their precision.

The polysemy problem: Consider “rock” which can mean:

• Rock music (genre)
• A stone (geological object)
• Rocking motion (verb)

Standard embeddings average these meanings, producing representations that may not capture any sense precisely.

Probabilistic approach: Probabilistic FastText represents words as Gaussian mixture models: probability distributions that can capture multiple distinct meanings as separate components.

Advantages:

• Multi-sense representation: Each word sense gets its own distribution
• Context sensitivity: Can select appropriate sense based on usage context
• Uncertainty quantification: Probabilistic framework captures embedding confidence

This approach provides a more nuanced treatment of lexical ambiguity, particularly valuable for words with distinct, context-dependent meanings.

Summary and Future Directions

Word embeddings have evolved from simple one-hot encodings to contextual representations that capture nuanced linguistic relationships. Each approach offers distinct advantages:

Static embeddings (Word2Vec, GloVe, FastText) provide:

• Computational efficiency for large-scale applications
• Pre-trained models available for numerous languages
• Clear analogical reasoning capabilities
• Good performance on many downstream tasks

Contextual embeddings (ELMo, BERT, GPT) offer:

• Dynamic representations based on sentence context
• Better handling of polysemy and word sense disambiguation
• Strong performance on complex NLP tasks
• Ability to capture subtle contextual nuances

Choosing the right approach depends on:

• Task requirements: Static embeddings for efficiency, contextual for accuracy
• Data availability: Pre-trained models vs. domain-specific training
• Computational constraints: Static embeddings require less processing power
• Language coverage: Consider availability of pre-trained models for target languages

The field continues advancing toward more efficient contextual models, better multilingual representations, and embeddings that capture increasingly complex linguistic phenomena.

For a production-grade Word2Vec implementation in PyTorch that takes these concepts further, see the High-Performance Word2Vec project.