Sarcasm detection represents one of the most challenging problems in NLP. The difficulties include:

Context dependence: Sarcasm relies on situational knowledge and shared understanding that extends beyond the text itself.

Subtlety: Even humans struggle with sarcastic interpretation, especially in written text without vocal cues.

Cultural variability: Sarcastic expressions vary significantly across cultures and regions.

Annotation disagreement: Human annotators often disagree on what constitutes sarcasm.

These challenges raise a fundamental question: can sarcasm detection be well-defined as a computational problem? This case study explores what happens when we try (and reveals a common pitfall in dataset construction).

The Dataset: A Hidden Flaw

I used the Sarcasm News Headlines dataset, which combines headlines from The Onion (satirical) and The Huffington Post (traditional news). The dataset contains ~50,000 examples.

from datasets import load_dataset

dataset = load_dataset("raquiba/Sarcasm_News_Headline")
print(dataset["train"][0])
print(dataset["train"][1])

{'headline': 'thirtysomething scientists unveil doomsday clock of hair loss',
 'is_sarcastic': 1}
{'headline': 'dem rep. totally nails why congress is falling short on gender, racial equality',
 'is_sarcastic': 0}

The critical flaw: This dataset uses binary classification based on source domain. The Onion headlines are labeled sarcastic, HuffPost headlines are not. This creates a dangerous shortcut where models learn to detect the publication source.

After preprocessing to standardize column names:

dataset = dataset.map(
    lambda example: {"text": example["headline"], "label": example["is_sarcastic"]},
    remove_columns=["headline", "article_link", "is_sarcastic"]
)

Fine-Tuning RoBERTa

I fine-tuned a pre-trained RoBERTa model using standard practices:

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments

model_name = "FacebookAI/roberta-base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=2)

# Tokenize the data
def tokenize_function(examples):
    return tokenizer(examples["text"], truncation=True, max_length=512)

tokenized_datasets = dataset.map(tokenize_function, batched=True)

# Training configuration
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=5,
    per_device_train_batch_size=32,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    eval_dataset=tokenized_datasets["test"],
    tokenizer=tokenizer,
)

trainer.train()

Results: Too Good to Be True

The model achieved high accuracy:

This should immediately raise red flags. Sarcasm detection is notoriously difficult, even for humans. Such high accuracy suggests the model learned a proxy task.

My hypothesis: The model bypassed sarcasm detection entirely, learning only to distinguish between The Onion and HuffPost writing styles.

Interacting with the Model

Let’s test our hypothesis by interacting with the model.

First, let’s load the model and tokenizer:

from transformers import pipeline

model = AutoModelForSequenceClassification.from_pretrained('results/2024-02-25_20-24-51/checkpoint-4475')

clf = pipeline('text-classification', model=model, tokenizer=tokenizer)

Now, let’s test the model with some examples.

First, let’s try an Onion article from this week, something I know to be sarcastic and not in the training data. Let’s use “Alabama Supreme Court Justice Invokes ‘VeggieTales’ In Ruling”:

clf("Alabama Supreme Court Justice Invokes ‘VeggieTales' In Ruling")

[{'label': 'LABEL_0', 'score': 0.99916672706604}]

The model is extremely confident that this is not sarcastic.

Let’s try a different Onion article, possibly even more difficult: Breaking News Trump Booed, Frozen Burritos, And More: This Week In Breaking News February 24, 2024:

clf("Breaking News Trump Booed, Frozen Burritos, And More: This Week In Breaking News February 24, 2024")

[{'label': 'LABEL_0', 'score': 0.9993497729301453}]

Again, very confident that this is not sarcastic. Hmm. It could be the temporal accuracy of our model just cannot capture the sarcasm of the Onion in 2024.

Let’s try one more Onion article, this one that is still recent but a bit more of a low-hanging fruit: Mom Only Likes The Other Outback Steakhouse:

clf("Mom Only Likes The Other Outback Steakhouse")

[{'label': 'LABEL_1', 'score': 0.9997231364250183}]

Finally, a correct prediction! The model is confident that this is sarcastic. Our model detects only very specific types of sarcasm. It fails to generalize to new, unseen data within the same domain.

Let’s also try some headlines from the Huffington Post, which the model should predict as not sarcastic. Let’s try the five most recent headlines from the Huffington Post:

• Donald Trump Won South Carolina - But There’s 1 Big Caveat
• Man Sets Himself On Fire In Front Of Israeli Embassy In Washington
• Israeli Media Report Progress On Reaching A Temporary Truce In Gaza And A Hostage-Prisoner Exchange
• A White Liberal Is Trying To Oust A Progressive Black Congressman. His Comments Could Make That Job Harder.
• Climate Change-Fueled Winter Extremes Put 90% Of This Country At ‘High Risk’

clf([
    "Donald Trump Won South Carolina - But There's 1 Big Caveat",
    "Man Sets Himself On Fire In Front Of Israeli Embassy In Washington",
    "Israeli Media Report Progress On Reaching A Temporary Truce In Gaza And A Hostage-Prisoner Exchange",
    "A White Liberal Is Trying To Oust A Progressive Black Congressman. His Comments Could Make That Job Harder.",
    "Climate Change-Fueled Winter Extremes Put 90% Of This Country At 'High Risk'"
])

[{'label': 'LABEL_0', 'score': 0.9993808269500732},
 {'label': 'LABEL_0', 'score': 0.9993786811828613},
 {'label': 'LABEL_0', 'score': 0.9985186457633972},
 {'label': 'LABEL_0', 'score': 0.9993883371353149},
 {'label': 'LABEL_0', 'score': 0.9993487000465393}]

The model is extremely confident that these are not sarcastic.

The model detects sarcasm in limited cases. It fails to generalize to new, unseen data within the same domain. This is a common problem in machine learning. Training a model that performs well on a specific dataset is straightforward. Training a model that generalizes to new, unseen data remains a significant challenge. Furthermore, our sarcasm detection project resulted in a domain classifier. For fuzzier concepts like sarcasm, it’s important to be clear about what we’re actually detecting, and to collect the necessary scale of data to capture the full range of the concept.

Key Takeaways

This case study reveals a fundamental problem in ML: high accuracy guarantees only performance on the training distribution. Here’s what actually happened:

1. Dataset bias: Using publication source as a proxy for sarcasm created a shortcut for the model
2. Domain classification: The model exclusively learned to distinguish writing styles
3. Poor generalization: New examples from the same sources often failed

This is a common pitfall when building datasets for subjective concepts. The lesson: high accuracy must be accompanied by validation of the model’s actual learned behavior.

For better sarcasm detection, we’d need:

• Diverse sources beyond two publications
• Human annotation across multiple contexts
• Careful evaluation on out-of-domain examples

Instructive failures in ML projects provide valuable lessons about our assumptions and the limitations of our approaches.

Social media research is often siloed by platform, with tools built specifically for Twitter’s flat structure or Reddit’s tree structure. This fragmentation makes cross-platform analysis difficult. In this work, I introduce PyConversations, an open-source Python package that normalizes data from Twitter, Facebook, Reddit, and 4chan into a single, platform-agnostic data model. (Note: the repository is archived and no longer actively maintained.)

Leveraging this tool, I processed over 308 million posts to analyze the structural “shape” of online conversations. I then evaluated the efficacy of domain-adaptive pre-training (DAPT) for Transformer-based language models, finding that training on a toxic domain (4chan) boosts hate-speech detection by over 5 F1.

The Engineering Problem: Data Normalization

Social media platforms impose different structural constraints on discourse, making it difficult to feed heterogeneous data into a single ML pipeline:

• Twitter: Technically allows infinite depth, but functionally operates as a flat stream or shallow tree.
• Facebook: Enforces a hard limit of two depth levels (comments and replies), resulting in “short and fat” conversation trees.
• Reddit & 4chan: Allow for deep, branching tree structures.

To solve this, I designed a Universal Message Schema and the PyConversations library. This system ingests raw dumps from these disparate sources and maps them to a unified Directed Acyclic Graph (DAG) format, preserving the parent-child relationships regardless of the source platform’s constraints.

Key Contributions

• PyConversations Library: An open-source package for robust conversational analysis, featuring graph-based traversing and filtering.
• Massive Dataset Analysis: Processed a collection of 308 million posts and 15.8 million conversations, creating one of the largest comparative cross-platform analyses at the time of thesis submission.
• Structural Insights: Quantified how UI constraints shape human behavior. For instance, Facebook’s depth limit forces users to “bunch” comments, creating uniquely wide conversation trees compared to Reddit’s deep, narrow threads.
• Domain Adaptation Experiments: Continued-pretrained RoBERTa on platform-specific slices (e.g., the 4chan-adapted RoBERTa-4chan), demonstrating that exposing models to toxic domains improved hate-speech detection F1 by over 5 points.

Structural Analysis Findings

By treating conversations as graphs, I uncovered distinct topological signatures for each platform:

The “Shape” of Discourse

I measured the width (max posts at any depth) and depth (max distance from root) of conversation trees.

• Facebook exhibited a “short and fat” topology due to its 2-level nesting limit.
• 4chan threads were surprisingly shallow despite having no depth limits. This suggests that the platform’s ephemerality (threads are deleted quickly) and the “bump limit” mechanic discourage long-term dialogue, though data scraping limitations on this transient platform also contribute to this topology.
• Reddit maintained the most robust tree structures, with “good faith” communities like r/ChangeMyView showing distinct patterns of sustained engagement.

Information Density

I analyzed Innovation Rate, a measure of how quickly a text introduces new vocabulary. Twitter and Facebook posts regress to negative rates; Reddit and 4chan posts sit well above zero.

Representation Learning & Domain Adaptation

I experimented with “Warm-Start” tuning: taking a standard RoBERTa model and pre-training it further on platform-specific data before fine-tuning on downstream tasks (TweetEval).

• Limited gains on most general tasks: Domain-adaptive pre-training added little on sentiment and emotion (from well under 1 up to a few F1 points), with irony detection the exception (+5.6 to +5.9 F1). Base RoBERTa already covers most of the signal for general NLP tasks.
• The Toxic Exception: The notable exception was Hate Speech Detection. The 4chan-adapted model (RoBERTa-4chan) was the strongest here, outperforming the baseline by over 5 F1. This highlights that for specialized, out-of-distribution language (like toxic slang), domain adaptation remains valuable.

Citation

@thesis{heidenreich2021look,
  title={Look, Don't Tweet: Representation Learning and Social Media},
  author={Hunter Heidenreich},
  year={2021},
  school={Drexel University},
  type={Undergraduate Senior Thesis}
}

Related Work

For related work on how social media content surfaces in digital journalism, including a dataset of embedded tweets across 273,899 news articles, see NewsTweet Dataset: Social Media in Digital Journalism.

We introduce NewsTweet, a dataset and data collection pipeline designed to study the embedding of social media in digital journalism. Our descriptive analysis of articles collected from Google News (chosen for its significant role in shaping attention) reveals that 13% of stories include embedded tweets. The dataset provides a foundation for exploring how social media content is sourced and which users become newsworthy. (Note: this is an arXiv preprint from 2020 and was not published at a peer-reviewed venue.)

Key Contributions

• Large-Scale Dataset: A dataset of 273,899 news articles, with 35,218 containing embedded tweets, collected from Google News RSS feeds over a four-month period.
• Data Collection Pipeline: Details an automated pipeline for acquiring news articles, extracting embedded tweets, and collecting the corresponding user timelines from Twitter’s API.
• Descriptive Statistics: Presents statistics on the prevalence of tweet embedding across different news categories, outlets, and users, highlighting key patterns.

Data Availability

The NewsTweet dataset is not publicly available for direct download. Due to Twitter/X’s Terms of Service restrictions on redistributing tweet content, the full dataset cannot be shared openly. Researchers interested in accessing the data or the collection pipeline are encouraged to contact the authors via the arXiv paper (arXiv:2008.02870).

Dataset Characteristics

Scale and Coverage

• News Sources: 5,961 unique news domains aggregated through Google News RSS feeds.
• Time Period: Data collection initiated on May 15th, 2019, with the paper describing the first four months of data.
• Collection Velocity: The pipeline averaged 2,302 articles per day, with approximately 296 containing embedded tweets.
• Content Types: Focuses specifically on embedded tweets from Twitter, the most frequently embedded platform.
• Metadata: Includes article source, Google News category (e.g., Sports, Health), and full tweet and user objects from the Twitter API.

Technical Implementation

• RSS-to-API Pipeline: Automatically crawls Google News RSS feeds to extract article HTML, identifying embedded tweet IDs to fetch full objects via the Twitter API.
• Artifact Filtering: Implements cleaning protocols to handle artifacts, such as detecting and excluding YouTube pages that appear as articles in Google News feeds.
• Longitudinal Tracking: Features a “top-off” mechanism that continuously tracks discovered users, updating their timelines to capture historical context.
• Rate Limit Management: Utilizes a random sampling queue to maintain continuous data collection across thousands of users without exceeding Twitter API limits.

Key Findings

Embedding Prevalence

• 13% of news articles in our Google News-sourced collection contained embedded tweets.
• Significant variation across categories: Sports (24% of articles) and Entertainment (14%) had the highest rates of embedding, while Health (2%) had the lowest.
• News outlets that publish the most articles are well-known mass media organizations, while outlets with the highest average number of embeds per article are often focused on Sports and Entertainment.

User and Content Patterns

• Public figures dominate: Well-known figures like politicians and celebrities, alongside organizations, are embedded far more often than ordinary users.
• Some users have a small number of their tweets embedded many times, while others gain newsworthiness from a wider range of their content.
• The Health category, despite having few embedded tweets, had the highest proportion of unique tweets (93%), suggesting that when tweets are embedded, they are less likely to be reused across multiple stories.
• “Catch-up” Phenomenon: Data reveals a class of users with high “embedding effectiveness”: those embedded more frequently than they tweet. This suggests journalists often use embeddings to “catch readers up” on backstories for previously unknown individuals.

Significance

The dataset is a foundation for studying how social media surfaces in journalism: how sourcing routines are evolving in the digital age, how traditional outlets and social platforms interact, and how previously-unknown users become newsworthy, grounded in the per-category and per-outlet embedding rates and the user-newsworthiness patterns the dataset captures.

My Contribution

I am the second of six authors on this paper. My contributions focused on the descriptive analysis: writing code to process the collected data, generating summary tables and statistics, and helping write and review the manuscript.

Citation

@misc{mujib2020newstweetdatasetsocialmedia,
  title={NewsTweet: A Dataset of Social Media Embedding in Online Journalism},
  author={Munif Ishad Mujib and Hunter Scott Heidenreich and Colin J. Murphy and Giovanni C. Santia and Asta Zelenkauskaite and Jake Ryland Williams},
  year={2020},
  eprint={2008.02870},
  archivePrefix={arXiv},
  primaryClass={cs.SI},
  url={https://arxiv.org/abs/2008.02870}
}

Related Work

• Look, Don’t Tweet: Unified Data Models for Social NLP: provides the unified cross-platform social media data model underlying broader Twitter analysis.
• Coordinated Social Targeting on Twitter: a companion study from the same research group and time period, documenting coordinated follower-manipulation patterns on high-profile Twitter accounts.

In the lead-up to the 2020 U.S. presidential election, we developed tools to monitor the follower dynamics of prominent Twitter users. This work details our investigation into several strange phenomena we observed, which suggest the presence of coordinated manipulation campaigns. We present preliminary evidence of networks of automated (bot) and potentially compromised accounts aiming to artificially inflate or diminish the perceived social status of high-profile individuals. (Note: this work was presented as an extended abstract at IC2S2 2020. It is available as an arXiv preprint but was not submitted as a full peer-reviewed paper.)

Key Contributions

• High-frequency sampling infrastructure: custom tooling sustaining 1-second follower-count polling over weeks, recursive follower-list “tunneling” through Twitter’s pagination limits, and API-rate-window management, capturing sub-second signatures invisible to standard API sampling and requiring no internal platform access.
• Four documented phenomena: spikes & sawteeth, circulating accounts, resurrected pre-2010 accounts, and follower-list anomalies, each cataloged as a separate experiment.
• Preliminary evidence of coordination: anomalous patterns consistent with systematic bot activity around prominent political accounts in the lead-up to the 2020 election, presented as observation rather than proven causation.

Methodological Strategy

To capture these transient phenomena, we built a high-frequency observational pipeline that bypassed standard sampling limitations. Our framework treated the social graph as a dynamic signal:

1. High-Resolution Sampling: Implemented “cycling scripts” to monitor follower counts at 1-second resolution, revealing sub-second “spike” anomalies often missed by aggregate metrics
2. Recursive “Tunneling”: Developed a recursive sampling method to extract deep history from follower lists, uncovering strata of “ancient” accounts buried by recent activity
3. Cross-Referencing: Correlated circulation events with follower count distortions to validate the “sawtooth” waveform hypothesis

Key Findings

• Spikes & Sawteeth: Dramatic, sub-second spikes and saw-tooth wave patterns in follower counts, suggesting automated activity
• Circulating Accounts: Cohorts of users repeatedly following and unfollowing the same high-profile accounts
• ‘Resurrected’ Ancient Accounts: Disproportionate numbers of pre-2010 Twitter accounts with long inactivity periods suddenly appearing as recent followers
• Follower List Anomalies: Unusual temporal groupings in follower histories suggesting historical manipulation campaigns

Significance & Sociological Impact

Beyond the technical detection of botnets, this work raises questions about a potential vulnerability in the infrastructure of modern public discourse. The observed patterns suggest these coordinated campaigns may be targeting the metrics that algorithms use to verify relevance (such as follower counts and engagement rates) as a vector for artificially shifting perceived social proof.

• Manipulating social-proof signals: Our observations are consistent with attempts to manipulate the “rich-get-richer” dynamics of social platforms. If inorganic follower inflation can influence recommender systems, that could amplify specific narratives to legitimate users. This remains a documented observation rather than a proven causal chain, and warrants further investigation.
• Account-age filters are not enough: The data highlights patterns where dormant pre-2010 accounts appear as new followers in bulk, suggesting that account-age alone is insufficient as a defense against coordinated manipulation.
• A Case for Metric Scrutiny: These findings suggest that raw follower counts are noisy signals that can be manipulated at scale. Supplementing them with engagement-quality metrics may make platform recommender systems more robust to coordinated inorganic activity.

Citation

@article{heidenreich2020investigating,
  title={Investigating Coordinated 'Social' Targeting of High-Profile Twitter Accounts},
  author={Heidenreich, Hunter Scott and Mujib, Munif Ishad and Williams, Jake Ryland},
  journal={arXiv},
  volume={abs/2008.02874},
  year={2020},
  url={https://arxiv.org/abs/2008.02874},
  eprint={2008.02874},
  eprinttype={arXiv}
}

Related Work

For related work in the computational social science cluster:

• NewsTweet Dataset: Social Media in Digital Journalism: a companion dataset studying how tweets surface in journalism, from the same research group and time period.
• Look, Don’t Tweet: Unified Data Models for Social NLP: provides the unified cross-platform social media data model used in broader analysis of Twitter and related platforms.
• GPT-2 Susceptibility to Universal Adversarial Triggers: proposes UAT-based bot detection as a constructive application, addressing the same automated manipulation behavior documented here from a complementary detection angle.