Abstract

Traditional OCR pipelines are often brittle, while modern “OCR-free” Vision-Language Models (VLMs) frequently lack precise token-to-pixel alignment. To address this, we introduce GutenOCR, a family of VLMs designed specifically as a “grounded OCR front-end.” By fine-tuning Qwen2.5-VL on a curriculum of synthetic and real-world documents, GutenOCR provides both high-quality text transcription and explicit geometric grounding (bounding boxes) through a unified, prompt-based interface. This approach allows downstream systems to request exactly the data format they need, from plain text to complex JSON structures.

Key Contributions & Results

• Unified Interface: Transforms Qwen2.5-VL models into specialized OCR systems supporting full-page reading, detection, localized reading, and conditional detection via prompting.
• In-Domain Improvements: GutenOCR-7B more than doubles the composite grounded OCR score of its base model (0.40 to 0.82) on 10.5K held-out pages, with the gains concentrated in localized reading and detection.
• Fox Benchmark: GutenOCR-3B achieves a region-level Character Error Rate (CER) of 0.053, surpassing even the dedicated Fox model (0.059), and sharply improves line-level OCR over its Qwen2.5-VL backbones (0.817 to 0.240), though the specialized Fox model still leads at line level.
• Curriculum Learning: Training uses a three-stage curriculum across synthetic data, real-world business documents, and long-context scientific articles to progressively build layout and grounding competency.
• Trade-offs: While GutenOCR reads content accurately (high Page F1), it orders text based on 2D layout columns. It also experiences catastrophic forgetting of color-based prompts and clear negative transfer on math formula recognition (notably the 3B model), to the point the paper recommends the un-fine-tuned backbone for equation-heavy use.

Methodology

• Data: The training mixture combines large-scale real-world documents (business forms, scientific articles) with synthetic data designed to teach precise grounding (e.g., “Grounded LaTeX” and “SynthDoG Grounding”).
• Curriculum Learning: Training progresses through three stages, starting with short contexts and synthetic data, moving to real-world business documents, and finishing with long-context scientific articles (up to 16k tokens).
• Unified Interface: The model treats “pipeline” stages (detection, reading, grounding) as different input-output schemas of a single model, allowing downstream systems to request exactly the data format they need (e.g., plain text vs. JSON boxes).

Models

We release 3B and 7B parameter models on HuggingFace:

• GutenOCR-3B (HuggingFace)
• GutenOCR-7B (HuggingFace)

You can try GutenOCR directly at ocr.roots.ai, where you can upload a document image and see the model’s parsed text output alongside bounding-box highlights on the original image.

Why This Matters

GutenOCR is proposed as a foundational layer for systems where every extracted answer must be explicitly linked to supporting pixels. By providing stable, grounded outputs, it enables human-in-the-loop workflows where reviewers can easily verify hallucinations or missing text by checking the predicted bounding boxes. This work pairs closely with our release of PubMed-OCR, which provides the large-scale, high-density annotations necessary to train such layout-aware models.

Resources

• Live Demo: Try GutenOCR on your own documents.
• Paper (arXiv): Full technical report.
• Code (GitHub): Training code and model release.
• GutenOCR-3B (HuggingFace): 3B parameter model weights.
• GutenOCR-7B (HuggingFace): 7B parameter model weights.

Citation

@misc{heidenreich2026gutenocrgroundedvisionlanguagefrontend,
      title={GutenOCR: A Grounded Vision-Language Front-End for Documents},
      author={Hunter Heidenreich and Ben Elliott and Olivia Dinica and Yosheb Getachew},
      year={2026},
      eprint={2601.14490},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2601.14490},
}

Related Work

• PubMed-OCR: The large-scale annotation dataset used to train GutenOCR’s layout-aware grounding capabilities.
• LLMs for Page Stream Segmentation: Complementary work on document understanding at the page-stream level.
• The Evolution of Page Stream Segmentation: Rules to LLMs: Background on the history and evolution of document processing pipelines.
• The Reliability Trap: When 99% Accuracy Isn’t Enough: Explores calibration challenges in deployed PSS systems, directly relevant to GutenOCR’s deployment context as an OCR front-end.

PubMed-OCR is an OCR-centric corpus of scientific articles derived from PubMed Central Open Access PDFs. Each page image is annotated with Google Cloud Vision and released in a compact JSON schema with word-, line-, and paragraph-level bounding boxes. The corpus spans 209.5K articles (1.5M pages; ~1.3B words) and supports layout-aware modeling, coordinate-grounded QA, and evaluation of OCR-dependent pipelines. We analyze corpus characteristics (e.g., journal coverage and detected layout features) and discuss limitations, including reliance on a single OCR engine and heuristic line reconstruction. We release the data and schema to facilitate downstream research and invite extensions.

Key Contributions

• OCR-First Supervision: Unlike prior datasets for PubMed that align XML to PDFs, PubMed-OCR provides native OCR annotations (Google Cloud Vision), bypassing alignment errors and covering non-digital scanned pages.
• High-Density Annotation: At ~1.3B words across 1.5M pages, PubMed-OCR is far denser per page than comparable corpora like OCR-IDL: ~13x the word density (844 vs. 62.5 words/page) and ~6x the line density (106 vs. 17.5 lines/page), achieved despite drawing from fewer total pages.
• Multi-Level Bounding Boxes: Includes explicit word-, line-, and paragraph-level bounding boxes to support hierarchical document understanding and layout-aware modeling. We also hope that this leads to VQA datasets with grounded answers in document layout.
• Open Access & Reproducibility: Derived strictly from the redistributable PMCOA subset, releasing both the JSON annotations and original PDFs to ensure verifiable and reproducible research.

Technical Implementation

Corpus Construction

PubMed-OCR is built from PubMed Central Open Access (PMCOA) PDFs, chosen specifically because the PMCOA license permits redistribution of both the original documents and derived annotations. Each PDF is rendered to page images, then passed to the Google Cloud Vision (GCV) API. Each page produces a structured JSON annotation file capturing the detected text along with bounding box geometry at word, line, and paragraph levels.

JSON Annotation Schema

Each page annotation follows this compact schema. Bounding boxes are axis-aligned rectangles in [x1, y1, x2, y2] pixel coordinates. Words, lines, and paragraphs are stored as parallel flat lists under the text key:

{
  "text": {
    "words": [
      {"text": "Example", "box": [180, 746, 210, 786]}
    ],
    "lines": [
      {"text": "Example sentence", "box": [180, 746, 540, 786]}
    ],
    "paragraphs": [
      {"text": "Example sentence\nSecond line", "box": [180, 746, 540, 820]}
    ]
  },
  "image": "..."
}

Line Reconstruction

GCV returns word-level detections natively. Line and paragraph groupings are reconstructed using spatial heuristics: words are clustered into lines by vertical overlap and horizontal proximity, and paragraph grouping follows a similar process at a coarser scale. These heuristics work well for standard single-column scientific layouts but can fail on multi-column or irregularly structured pages (see Limitations).

Using the Dataset

The corpus spans 1.5M pages, so streaming is recommended for most use cases:

import json
from datasets import load_dataset

# Streaming is recommended for the full 1.5M-page corpus
ds = load_dataset("rootsautomation/pubmed-ocr", streaming=True, split="train")

# Inspect a page
page = next(iter(ds))
print(f"Article: {page['accession_id']},  Page: {page['page']}")

# Parse OCR annotations
ocr = json.loads(page["ocr_json"])
text = ocr["text"]

# Iterate over lines and words
for line in text["lines"][:5]:
    print(f"  Line: {line['text']}")
    print(f"  BBox: {line['box']}")

# Access individual word detections
for word in text["words"][:5]:
    print(f"  Word: {word['text']}, BBox: {word['box']}")

Full schema documentation is available on the HuggingFace dataset card.

Why This Matters

The lack of large-scale, high-quality OCR datasets with explicit geometric grounding has been a major bottleneck for training layout-aware models. By releasing PubMed-OCR, we provide the community with the dense, multi-level bounding box annotations necessary to build the next generation of document understanding systems. This dataset directly supports the development of models like GutenOCR, enabling them to learn precise token-to-pixel alignment and robust layout reasoning.

Limitations

• Single OCR engine: All annotations come from Google Cloud Vision. GCV’s error modes (handwriting, degraded scans, complex math, non-Latin scripts) propagate uncorrected into the dataset. Different OCR engines could yield different coverage patterns and error distributions.
• Heuristic line reconstruction: Spatial word-to-line clustering is approximate. Multi-column layouts, rotated text, or unusual page orientations may produce incorrect line groupings.
• PMCOA scope: Coverage is limited to the Open Access subset of PubMed Central. Commercial or subscription articles are excluded.

Citation

@misc{heidenreich2026pubmedocrpmcopenaccess,
      title={PubMed-OCR: PMC Open Access OCR Annotations},
      author={Hunter Heidenreich and Yosheb Getachew and Olivia Dinica and Ben Elliott},
      year={2026},
      eprint={2601.11425},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2601.11425},
}

Related Work

This dataset directly enables GutenOCR, a family of vision-language models trained on PubMed-OCR annotations to produce grounded OCR outputs with explicit bounding boxes.

For related work on document processing pipelines that consume OCR output, see LLMs for Page Stream Segmentation and Page Stream Segmentation with LLMs: Challenges and Applications.

Page Stream Segmentation (PSS) is critical for automating document processing in industries like insurance, where unstructured document collections are common. This paper explores the use of large language models (LLMs) for PSS, applying parameter-efficient fine-tuning to real-world insurance data. Our experiments show that LLMs outperform baseline models in segmentation accuracy. We find that stream-level calibration remains a significant challenge. We evaluate post-hoc calibration and Monte Carlo dropout, finding they offer limited improvement, highlighting the need for future work in this area for high-stakes applications.

Published in the COLING 2025 Industry Track, this work builds on our earlier research establishing the TabMe++ benchmark and decoder-based LLM approach, extending those methods to real-world industrial deployment.

Blog Post: For a narrative overview of the reliability and calibration findings discussed in this paper, see The Reliability Trap: When 99% Accuracy Isn’t Enough.

Key Contributions

• Real-World Evaluation: Applied small-to-mid-sized LLMs (Phi-3.5-mini, Mistral-7B) to a proprietary insurance dataset, outperforming strong baselines like XGBoost in segmentation accuracy.
• Parameter-Efficient Fine-Tuning: Successfully used parameter-efficient fine-tuning (PEFT) to adapt LLMs for the specialized task of page stream segmentation.
• Calibration Complexity: Found that post-hoc calibration and Monte Carlo dropout offer limited improvement at the stream level, keeping human-in-the-loop workflows necessary for high-stakes automation (see stream-level confidence analysis below).
• Throughput Analysis: Introduced an accuracy-vs-throughput framework to quantify how much volume can be safely automated at strict confidence thresholds.

Stream-Level Confidence

A key insight from this work is why calibration becomes increasingly difficult as documents grow longer. We define stream-level confidence as the product of individual page-level confidences:

$$C = \prod_{i=1}^{N} C_i$$

where $C_i$ is the confidence for page $i$ and $N$ is the number of pages in the stream. This multiplicative relationship means that even small page-level errors compound aggressively. As streams grow longer, confidence drops rapidly, making it difficult to set reliable thresholds for automation.

Technical Implementation

Models & Fine-Tuning

We fine-tuned Mistral-7B-v0.3 and Phi-3.5-mini (4-bit quantized) using QLoRA. Training was performed efficiently on a single NVIDIA H100 GPU using the Unsloth library and Hugging Face’s TRL.

• Stack: Unsloth + TRL
• Config: Rank $r=16$, Alpha $\alpha=16$

Dataset

The study utilized a proprietary insurance dataset consisting of 7.5k document streams (44.7k pages). This real-world data includes health records, legal contracts, and police reports, offering a more challenging and realistic evaluation than synthetic benchmarks.

Task Framing

The task framing follows the TabMe++ work: binary classification over a local window (previous page + current page). Here the models are prompted to output valid JSON marking the start of a new document.

Impact

This work demonstrates both the promise and the current limitations of using LLMs in high-stakes industrial applications. LLMs can significantly improve segmentation accuracy over traditional methods, but performance metrics alone are not sufficient for deployment. For sectors like insurance, stream-level calibration is an open problem that must be solved before full automation becomes responsible.

Citation

@inproceedings{heidenreich2025page,
  title={Page Stream Segmentation with LLMs: Challenges and Applications in Insurance Document Automation},
  author={Heidenreich, Hunter and Dalvi, Ratish and Verma, Nikhil and Getachew, Yosheb},
  booktitle={Proceedings of the 31st International Conference on Computational Linguistics: Industry Track},
  pages={305--317},
  year={2025}
}

Advanced neural network architectures developed for tasks like natural language processing are often transferred to spatiotemporal forecasting without a deep understanding of which components drive their performance. This can lead to suboptimal results and reinforces the view of these models as “black boxes”. In this work, we deconstruct the core mechanisms of Transformers and Recurrent Neural Networks (RNNs) (namely attention, gating, and recurrence). We then build and test novel hybrid architectures to identify which components are most effective. A key finding is that while adding recurrence is detrimental to Transformers, augmenting RNNs with attention and neural gating consistently improves their forecasting accuracy. Our study reveals that a seldom-used architecture, the Recurrent Highway Network (RHN) enhanced with these mechanisms, emerges as the top-performing model for forecasting high-dimensional chaotic systems.

Key Contributions

• Systematic Ablation: Deconstructed Transformers and RNNs into core mechanisms (attention, gating, recurrence) to isolate performance drivers
• Novel Hybrid Architectures: Synthesized and tested new combinations of neural primitives for spatiotemporal forecasting
• RHN advantage on chaotic systems: Demonstrated that attention-augmented Recurrent Highway Networks outperform standard Transformers on high-dimensional chaotic systems
• Robustness Analysis: Validated models across both clean physics simulations and noisy real-world industrial datasets

Motivation

In modern ML, architectures are often transferred from one domain (like NLP) to another (like physical forecasting) without understanding the underlying mechanics. This “black box” approach leads to suboptimal compute usage and performance ceilings.

Our goal was to break these architectures down. We treated the core mechanisms of Transformers and RNNs (Gating, Attention, and Recurrence) as orthogonal basis vectors. By decoupling these components, we could synthesize and test hybrid architectures to find the best configuration for spatiotemporal forecasting.

Methodological Approach

We built a modular framework to mix and match neural primitives. We systematically evaluated:

1. Gating Mechanisms: Testing Additive, Learned Rate, Input-Dependent, and Coupled Input-Dependent variants
2. Attention: Implementing multi-headed attention with relative positional biases
3. Recurrence: Testing standard cells (LSTM, GRU) against deeper transition cells like Recurrent Highway Networks (RHN)

This ablation isolated exactly which mathematical operation was driving the performance gain.

Key Findings

Recurrent Highway Networks on Chaotic Systems

For high-dimensional chaotic systems like the Multiscale Lorenz-96 shown below, we found that a Recurrent Highway Network (RHN) augmented with Attention and Neural Gating was the top-performing architecture. This hybrid exceeded the forecasting accuracy of standard Transformers, suggesting that deeper recurrence (processing depth per timestep) matters for complex dynamics.

Transformers: Recurrence Hurts, Gating Helps

We attempted to force recurrence into Transformers to give them “memory,” but it consistently hurt performance. However, Neural Gating significantly improved Transformer robustness. For real-world, noisy data (traffic, weather), the Pre-Layer Normalization (PreLN) Transformer with added gating proved to be the most robust model.

Adding Attention to LSTMs and GRUs

We tested on the Kuramoto-Sivashinsky equation, a model of turbulence and flame fronts. We found that standard LSTMs and GRUs are under-optimized for this setting: adding attention to these cells improved their valid-prediction time several-fold, with the best attention-augmented LSTM and GRU reaching roughly 4x and 6.6x their baseline valid-prediction time, respectively (the paper reports the top RNNs at 2-7x baseline on K-S). (On the partially-observed Multiscale Lorenz-96 system the same attention-plus-gating gain is smaller, more than 40%.)

Robustness on Real-World Datasets

While chaotic systems test the limits of theory, we also validated our models on seven standard real-world datasets: the four Electricity Transformer Temperature (ETT) subsets plus Traffic, Electricity, and Weather.

Unlike the clean physics simulations, these datasets contain real-world noise and irregularities. In this environment, the Pre-Layer Normalization (PreLN) Transformer proved to be the most robust architecture. While it didn’t always beat the RHN on pure chaos, its stability makes it a strong default choice for general time-series forecasting tasks where training speed and reliability are paramount.

Why This Matters

This work treats architectural components as independently tunable choices rather than fixed defaults, and that framing surfaces a concrete trade-off. Transformers train in only 25-50% of the time the RNNs require (roughly 2-4x faster), while the attention-augmented RNNs give better inference accuracy on the chaotic physical systems. Which mechanism to select depends on whether the training budget or the forecast precision is the binding constraint, and the ablation makes that an informed choice rather than a default one.

The ablation framework here, treating architectural components as independently tunable factors and measuring their marginal contribution, shaped how later evaluation work is structured. The same principle of isolating variables rather than comparing end-to-end black boxes appears in the document processing research, from benchmark construction in page stream segmentation to grounded evaluation in GutenOCR.

Related Work

The methodology here shares a design philosophy with EigenNoise, which similarly decomposes a neural mechanism (word vector initialization) into theoretically grounded components to isolate what drives performance. Both papers treat model components as testable hypotheses rather than fixed architectural choices.

For broader context on where this fits in the portfolio’s Scientific Machine Learning arc, see the Research overview.

Citation

@misc{heidenreich2024deconstructingrecurrenceattentiongating,
  title={Deconstructing Recurrence, Attention, and Gating: Investigating the transferability of Transformers and Gated Recurrent Neural Networks in forecasting of dynamical systems},
  author={Hunter S. Heidenreich and Pantelis R. Vlachas and Petros Koumoutsakos},
  year={2024},
  eprint={2410.02654},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2410.02654}
}

Page Stream Segmentation (PSS), the task of correctly dividing a sequence of pages into distinct documents, is a critical first step in automated document processing pipelines. Research in this area has been held back by the lack of high-quality, public datasets.

In this work, we address this issue by enhancing an existing benchmark, TabMe, with commercial-grade Optical Character Recognition (OCR) to create TabMe++. This new version significantly reduces noise and improves text detection, highlighting the critical importance of OCR quality for document understanding tasks.

We then conduct the first evaluation of large, decoder-based language models (LLMs) on the PSS task. Our findings show that models like Mistral-7B, when fine-tuned using parameter-efficient methods, outperform smaller encoder-based models and traditional baselines. For instance, our best model correctly segments 80% of document streams in the test set without any errors.

Key Contributions

• Enhanced Public Benchmark (TabMe++): Re-processed the entire TabMe dataset with commercial OCR, correcting significant text recognition errors and reducing blank pages by over 80% (from 2.27% to 0.38%)
• First Application of Large Decoder-Based LLMs to PSS: Systematically evaluated and fine-tuned billion-parameter, decoder-only LLMs for page stream segmentation
• Decoder LLMs Outperform Encoders on TabMe++: fine-tuned decoder models outperform previous encoder-based and multimodal approaches on the benchmark
• OCR Quality Analysis: Quantified the impact that high-quality OCR has on PSS model performance through comparative experiments

The Evolution of Page Stream Segmentation

The history of PSS can be read as three distinct algorithmic eras, revealing a clear trajectory toward semantic understanding:

• The Heuristic Era: Early systems relied on handcrafted rules and region-specific pattern matching (e.g., looking for headers/footers), which failed to generalize across heterogeneous documents.
• The Encoder Era: The field moved to “learning-based” methods using Convolutional Neural Networks (CNNs) and later Transformer encoders like LayoutLM and LEGAL-BERT. While better, these often required complex multimodal architectures.
• The Decoder Era (this work): This work establishes the viability of the third era: using billion-parameter generative models (decoder-only LLMs) which simplify the architecture while improving semantic reasoning.

Blog Post: Read the full story of these eras in The Evolution of Page Stream Segmentation.

Key Evaluation Metrics

Beyond standard F1 scores, the study evaluates models on metrics that directly translate to operational costs:

• Straight-Through Processing (STP): The percentage of document streams segmented perfectly, requiring zero human intervention. The fine-tuned Mistral-7B achieved an STP of 0.800, meaning 80% of streams were fully automated. In contrast, the traditional XGBoost baseline achieved only 0.074.
• Minimum Number of Drag-and-Drops (MNDD): A proxy for human effort, measuring how many pages a human would need to move to correct the segmentation. The best LLM reduced this “effort metric” by over 13x compared to the XGBoost baseline (0.81 vs 10.85).

Document-Level Precision and Recall

We define a ground truth segmentation $\mathcal{G}$ and a predicted segmentation $\mathcal{P}$. A “True Positive” is defined strictly as a document present in both sets ($\mathcal{P} \cap \mathcal{G}$). The metrics are calculated as:

$$P = \frac{|\mathcal{P} \cap \mathcal{G}|}{|\mathcal{P} \cap \mathcal{G}| + |\mathcal{P} \setminus \mathcal{G}|}$$

$$R = \frac{|\mathcal{P} \cap \mathcal{G}|}{|\mathcal{P} \cap \mathcal{G}| + |\mathcal{G} \setminus \mathcal{P}|}$$

This rigorous definition ensures that a model is only rewarded if it gets both the start and end boundaries of a document correct.

Technical Innovation

Our approach combines commercial-grade OCR processing with parameter-efficient fine-tuning of large language models. We addressed two main bottlenecks: data quality and model efficiency.

Data Remediation

The original TabMe dataset relied on Tesseract OCR, which introduced significant noise. By reprocessing the images with Microsoft OCR, we reduced the number of “blank” pages from 2.27% to just 0.38%, recovering critical features like titles and ID numbers that were previously lost.

Model Architecture

We formulated the task as a binary classification of page pairs: predicting if a “break” exists between Page $N$ and Page $N+1$.

Problem Formulation

The task is treated as a binary classification problem over a window of pages. For a specific page $p_i$, the model predicts a binary label $y_i$ based on a window of adjacent pages $(p_{i-l}, \ldots, p_i, \ldots, p_{i+r})$. In this work, we strictly defined the window as:

$$l=1, \quad r=0$$

This means the decision for page $p_i$ is made solely based on the pair $(p_{i-1}, p_i)$.

Efficient Tuning

We utilized Low-Rank Adaptation (LoRA) and 4-bit quantization to fine-tune Mistral-7B and Phi-3-mini on a single NVIDIA H100 GPU, proving that PSS does not require massive compute clusters.

Why This Matters

Page Stream Segmentation is the critical first step in any automated document processing pipeline. If a system fails to correctly separate documents, all downstream tasks (like classification or data extraction) will operate on corrupted inputs. By demonstrating that parameter-efficiently fine-tuned LLMs can achieve an 80% straight-through processing rate, this work provides a viable path toward fully automating high-volume document workflows.

Beyond the path to automation, this work gives the research community improved evaluation tools: the enhanced TabMe++ dataset and the quantified impact of OCR quality on PSS performance have direct applications in commercial document processing pipelines.

We later extended these findings to real-world industrial deployment and analyzed model calibration challenges in our follow-up COLING Industry paper on LLMs for Insurance Document Automation. The calibration challenges that emerged from that deployment are explored in depth in The Reliability Trap: When 99% Accuracy Isn’t Enough.

Citation

@misc{heidenreich2024largelanguagemodelspage,
  title={Large Language Models for Page Stream Segmentation},
  author={Hunter Heidenreich and Ratish Dalvi and Rohith Mukku and Nikhil Verma and Neven Pičuljan},
  year={2024},
  eprint={2408.11981},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2408.11981}
}

We developed EigenNoise, a method to initialize word vectors using zero pre-training data. By deriving a co-occurrence matrix solely from the theoretical harmonic structure of language (Zipf’s Law), this project demonstrates that we can mathematically synthesize a “warm-start” for NLP models. This approach challenges the reliance on massive corpora for initialization and offers a competitive alternative for low-resource environments.

Key Contributions

• A data-free initialization scheme: word vectors derived from a co-occurrence matrix synthesized from independent (Zipfian) frequency statistics, with no pre-training corpus.
• Grounds the construction in the harmonic statistical structure of language, so the representation follows from first principles rather than from data.
• Evaluates with Minimum Description Length (MDL) probing, which measures how much task-relevant information a representation encodes and how compactly, rather than raw accuracy.
• After fine-tuning, EigenNoise approaches the performance of GloVe (trained on Gigaword) despite seeing no pre-training text.

Technical Implementation

The core insight is that “noise” in language follows a predictable distribution.

1. Modeling: We model the “null hypothesis” of text, how words would co-occur if they were statistically independent but followed Zipfian rank-frequency. This yields a theoretical co-occurrence matrix $\hat{X}$:

$$\hat{X}_{ij} = \frac{2mN}{r_i r_j H_N}$$

Where $r_i$ is the rank of word $i$, $N$ is vocabulary size, $m$ is the context window size, and $H_N$ is the $N$-th harmonic number.

2. Factorization: We then solve for the word vectors by performing an eigen-decomposition on this matrix, extracting the top $d$ components to form the representation space.

3. Probing: Validated performance using MDL probing on CoNLL-2003 and TweetEval benchmarks.

Why This Matters

This research explores how much structure can emerge from frequency statistics alone, with no text exposure at all. The central finding is that EigenNoise vectors, derived purely from Zipf’s Law, reach competitive performance with GloVe after fine-tuning. This is evidence that a significant portion of what we call “learned linguistic knowledge” is a consequence of word frequency distributions, not semantic exposure to real text.

In 2026, small pretrained models are freely available and handle most low-resource initialization needs, so the practical case for data-free initialization is narrower than it was in 2022. The theoretical contribution remains relevant: EigenNoise establishes a clean null hypothesis for what word vectors look like when only frequency information is present. For interpretability researchers trying to disentangle frequency artifacts from genuine semantic content, this baseline has value independent of the initialization use case.

The MDL probing methodology applied here also contributes beyond the main result. Unlike task accuracy, MDL measures how much information a representation encodes and how compactly, providing a more principled lens for evaluating representational quality. EigenNoise’s co-occurrence prior is grounded directly in the Independent Frequencies Model (IFM) introduced in the companion Word2Vec factorization paper. Together, the two works form a coherent theoretical line: the IFM characterizes the frequency-driven baseline of embedding space, and EigenNoise operationalizes it as a practical, data-free initialization scheme.

Citation

@misc{heidenreich2022eigennoisecontrastivepriorwarmstart,
  title={EigenNoise: A Contrastive Prior to Warm-Start Representations},
  author={Hunter Scott Heidenreich and Jake Ryland Williams},
  year={2022},
  eprint={2205.04376},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2205.04376},
}

Related Work

For the theoretical foundation underlying EigenNoise’s null hypothesis, including the first analytical solution to Word2Vec’s softmax objective, see Analytical Solution to Word2Vec Softmax & Bias Probing.

While the Skip-Gram with Negative Sampling (SGNS) objective for Word2Vec has famously been shown to factorize a shifted PMI matrix, the implicit matrix factorization of the original Softmax objective has remained an open question. In this work, we provide the first known analytical solution to Word2Vec’s softmax-optimized skip-gram algorithm.

We use this derivation to introduce the Independent Frequencies Model (IFM), identifying a “frequency-ratios property” that unifies classical word vector models. This theoretical insight allows us to derive a low-cost, training-free method for measuring semantic bias directly from corpus statistics.

Key Contributions

• Analytical Solution: Proved that the softmax skip-gram objective converges to a factorization of the log-conditional probability matrix, a derivation for Word2Vec’s original (unapproximated) objective that prior work had left open.
• Independent Frequencies Model (IFM): Introduced a dense co-occurrence model computable purely from unigram frequencies to act as a null hypothesis for embedding structures.
• Bias Dissonance Metric: Derived a low-cost, training-free method for measuring semantic bias directly from corpus statistics using the frequency-ratios property.
• Data Transparency: Demonstrated how specific corpora exhibit distinct bias profiles, offering a tool for auditing datasets before training large models.

Key Theoretical Results

1. The Softmax Factorization Theorem

We prove that under the log-softmax objective, Word2Vec implicitly converges towards a factorization of the log-conditional probability matrix of the co-occurrence model.

Theorem: For the objective $\mathcal{L}_{\text{soft}} = - \sum _{t,s} F _{t,s}^m \log \varphi (\vec{u}_t \vec{v}_s)$, the algorithm converges to:

$$ \vec{u}_{t}\vec{v}_{s}^{T} = \log\frac{F_{t,s}^{m}}{f_{t}^{m}} $$

where $F_{t,s}^m$ is the co-occurrence count and $f_t^m$ is the marginal frequency. This effectively makes the dot product of the embedding vectors equal to the log-conditional probability of the context word given the target word.

2. The Independent Frequencies Model (IFM)

To understand the baseline behavior of these models, we introduce the IFM, which models a dense co-occurrence matrix computable purely from unigram frequencies:

$$ \hat{F}_{t,s}^{m} = \frac{2m f_t f_s}{M} $$

This model acts as a “null hypothesis” for embedding structures, allowing us to isolate true semantic signals from statistical noise.

Methodological Innovation: Bias Dissonance

Leveraging the frequency-ratios property derived from our factorization, we propose a metric called Dissonance ($\Delta$) to probe semantic bias in data without training a model.

For an analogy $A:B :: C:D$ (e.g., man:king :: woman:queen), we measure the alignment of their corpus frequency ratios. High dissonance indicates that the corpus statistics do not support the analogy, potentially revealing bias or under-representation.

Intuitive Example: If a corpus contains the phrase “man is king” 100 times more often than “woman is queen,” the frequency ratios are misaligned. A perfect, unbiased analogy would have matching ratios (i.e., man relates to king at the same rate woman relates to queen). Any deviation from this symmetry is captured by our dissonance metric, revealing where the data itself encodes asymmetric associations.

$$ \Delta(x,y|\mathcal{D}) = \left| \log\frac{f_{t}f_{\bar{s}}}{f_{s}f_{\bar{t}}} \right| / \max_{l \in \mathcal{V}} { \log f_l } $$

By applying this to the Bigger Analogy Test Set (BATS), we demonstrated how specific corpora (like Wikipedia vs. Google Books) exhibit distinct bias profiles regarding geographic and encyclopedic knowledge.

Visualizing Statistical Independence

Impact

This work bridges the gap between empirical success and theoretical foundations in NLP by:

1. Solving a fundamental mechanism: Providing the missing factorization proof for Softmax Word2Vec.
2. Efficient pre-training (a future direction): The factorization suggests embedding layers could be “warm-started” from unigram statistics via the IFM, a direction taken up in the companion EigenNoise work.
3. Data Transparency: Offering a computationally inexpensive tool for auditing datasets for bias before investing resources in training large models.

My Contribution

Jake Williams is the first author and primary driver of this work. He developed the core theory, derived the factorization proofs, designed the dissonance metric, and ran the experiments. My role was supporting: I contributed through critique and refinement during the writing process, but the intellectual heavy lifting belongs to Jake.

Citation

@misc{williams2022knowcompanywordslies,
      title={To Know by the Company Words Keep and What Else Lies in the Vicinity},
      author={Jake Ryland Williams and Hunter Scott Heidenreich},
      year={2022},
      eprint={2205.00148},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2205.00148},
}

Related Work

For a complementary analytical approach to word representations, deriving data-free word vector initializations from the same frequency-ratio insights, see EigenNoise: Data-Free Word Vector Initialization.

This is my undergraduate senior thesis, completed at Drexel University in 2021. The scope (308 million posts across four platforms, structural topology analysis, and domain adaptation experiments with Transformer models) was unusually broad for a senior thesis, spanning large-scale data engineering, graph-structural analysis, and representation-learning experiments.

Social media research is often siloed by platform, with tools built specifically for Twitter’s flat structure or Reddit’s tree structure. This fragmentation makes cross-platform analysis difficult. In this work, I introduce PyConversations, an open-source Python package that normalizes data from Twitter, Facebook, Reddit, and 4chan into a single, platform-agnostic data model. (Note: the repository is archived and no longer actively maintained.)

Leveraging this tool, I processed over 308 million posts to analyze the structural “shape” of online conversations. I then evaluated the efficacy of domain-adaptive pre-training (DAPT) for Transformer-based language models, finding that training on a toxic domain (4chan) boosts hate-speech detection by over 5 F1.

The Engineering Problem: Data Normalization

Social media platforms impose different structural constraints on discourse, making it difficult to feed heterogeneous data into a single ML pipeline:

• Twitter: Technically allows infinite depth, but functionally operates as a flat stream or shallow tree.
• Facebook: Enforces a hard limit of two depth levels (comments and replies), resulting in “short and fat” conversation trees.
• Reddit & 4chan: Allow for deep, branching tree structures.

To solve this, I designed a Universal Message Schema and the PyConversations library. This system ingests raw dumps from these disparate sources and maps them to a unified Directed Acyclic Graph (DAG) format, preserving the parent-child relationships regardless of the source platform’s constraints.

Key Contributions

• PyConversations Library: An open-source package for robust conversational analysis, featuring graph-based traversing and filtering.
• Massive Dataset Analysis: Processed a collection of 308 million posts and 15.8 million conversations, creating one of the largest comparative cross-platform analyses at the time of thesis submission.
• Structural Insights: Quantified how UI constraints shape human behavior. For instance, Facebook’s depth limit forces users to “bunch” comments, creating uniquely wide conversation trees compared to Reddit’s deep, narrow threads.
• Domain Adaptation Experiments: Continued-pretrained RoBERTa on platform-specific slices (e.g., the 4chan-adapted RoBERTa-4chan), demonstrating that exposing models to toxic domains improved hate-speech detection F1 by over 5 points.

Structural Analysis Findings

By treating conversations as graphs, we uncovered distinct topological signatures for each platform:

The “Shape” of Discourse

We measured the width (max posts at any depth) and depth (max distance from root) of conversation trees.

• Facebook exhibited a “short and fat” topology due to its 2-level nesting limit.
• 4chan threads were surprisingly shallow despite having no depth limits. This suggests that the platform’s ephemerality (threads are deleted quickly) and the “bump limit” mechanic discourage long-term dialogue, though data scraping limitations on this transient platform also contribute to this topology.
• Reddit maintained the most robust tree structures, with “good faith” communities like r/ChangeMyView showing distinct patterns of sustained engagement.

Information Density

We analyzed Innovation Rate, a measure of how quickly a text introduces new vocabulary. We found that Twitter threads have negative innovation rates (indicating high novelty per token) likely forced by the strict character limits. In contrast, Reddit posts showed higher redundancy, typical of longer-form essay writing.

Representation Learning & Domain Adaptation

We experimented with “Warm-Start” tuning: taking a standard RoBERTa model and pre-training it further on platform-specific data before fine-tuning on downstream tasks (TweetEval).

• Limited gains on most general tasks: Domain-adaptive pre-training added little on sentiment and emotion (from well under 1 up to a few F1 points), with irony detection the exception (+5.6 to +5.9 F1). Base RoBERTa already covers most of the signal for general NLP tasks.
• The Toxic Exception: The notable exception was Hate Speech Detection. The 4chan-adapted model (RoBERTa-4chan) was the strongest here, outperforming the baseline by over 5 F1. This highlights that for specialized, out-of-distribution language (like toxic slang), domain adaptation remains valuable.

Significance

This work bridges the gap between Computational Social Science and ML Engineering. It provides the community with a reusable tool (PyConversations) to handle the messy reality of social data and offers empirical evidence on the limits and benefits of domain-adaptive pre-training for LLMs.

Citation

@thesis{heidenreich2021look,
  title={Look, Don't Tweet: Representation Learning and Social Media},
  author={Hunter Heidenreich},
  year={2021},
  school={Drexel University},
  type={Undergraduate Senior Thesis}
}

Related Work

For related work on how social media content surfaces in digital journalism, including a dataset of embedded tweets across 273,899 news articles, see NewsTweet Dataset: Social Media in Digital Journalism.

Historical context: This paper was published at AIES 2021, the AAAI/ACM Conference on AI, Ethics, and Society, predating the modern red-teaming practices and adversarial robustness benchmarks that emerged with instruction-tuned and RLHF-trained models. GPT-2 is now a historical baseline, but the core methodology and findings remain a relevant foundation for current adversarial robustness work.

Abstract

This work investigates universal adversarial triggers (UATs), a method for disrupting language models using input-agnostic token sequences. We investigated whether it is possible to use these triggers to control the topic and the stance of text generated by GPT-2. Across four controversial topics, we demonstrated success in identifying triggers that guide the model to produce text on a targeted subject and influence the position it takes. Our goal is to raise awareness that even deployed models are susceptible to this influence and to advocate for immediate safeguards.

Key Findings & Contributions

• Topic and Stance Control: We were the first to systematically explore using UATs to control both the topic and the stance of a language model’s output. We found that controlling the topic is highly feasible, and controlling the stance is also possible.
• The “Filter Bubble” Hypothesis: We observed that triggers for fringe topics (e.g., Flat Earth) were harder to find but offered a higher degree of stance control than broader topics. We posit this may reflect “filter bubbles” in the training data, where fringe viewpoints use distinct linguistic patterns.
• Ethical & Security Analysis: We highlighted the security risks of deployed models being manipulated by external adversaries without internal model access. To be responsible, we withheld the most sensitive triggers we discovered.
• Constructive Applications: Beyond a security flaw, we proposed that UATs could be used constructively as a diagnostic tool to audit models for bias or as a method for bot detection on social media.

Significance & Why This Matters

This work extended early research on UATs by moving beyond single-issue attacks (like generating toxic content) to a nuanced analysis of topic and stance control. It demonstrated that a gradient-based search process (adapting HotFlip) is effective at manipulating model outputs, emphasizing a critical vulnerability for any organization deploying large language models.

For ML practitioners and security researchers, this highlights the importance of robust safeguards against input-agnostic attacks. It also opens the door to using these same adversarial techniques constructively: as diagnostic tools to audit models for hidden biases or to detect automated bot activity on social media platforms.

Related Work

The constructive bot-detection application proposed here connects directly to empirical work on coordinated inauthentic behavior. Coordinated Social Targeting on Twitter documents real-world follower-manipulation patterns on high-profile accounts, illustrating the kind of automated adversarial activity that UAT-based detection methods could help identify.

Citation

@inproceedings{10.1145/3461702.3462578,
  author = {Heidenreich, Hunter Scott and Williams, Jake Ryland},
  title = {The Earth Is Flat and the Sun Is Not a Star: The Susceptibility of GPT-2 to Universal Adversarial Triggers},
  year = {2021},
  isbn = {9781450384735},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  url = {https://doi.org/10.1145/3461702.3462578},
  doi = {10.1145/3461702.3462578},
  booktitle = {Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society},
  pages = {566--573},
  numpages = {8},
  keywords = {adversarial attacks, bias, language modeling, natural language processing},
  location = {Virtual Event, USA},
  series = {AIES '21}
}

We introduce NewsTweet, a dataset and data collection pipeline designed to study the embedding of social media in digital journalism. Our descriptive analysis of articles collected from Google News (chosen for its significant role in shaping attention) reveals that 13% of stories include embedded tweets. The dataset provides a foundation for exploring how social media content is sourced and which users become newsworthy. (Note: this is an arXiv preprint from 2020 and was not published at a peer-reviewed venue.)

Key Contributions

• Large-Scale Dataset: A dataset of 273,899 news articles, with 35,218 containing embedded tweets, collected from Google News RSS feeds over a four-month period.
• Data Collection Pipeline: Details an automated pipeline for acquiring news articles, extracting embedded tweets, and collecting the corresponding user timelines from Twitter’s API.
• Descriptive Statistics: Presents statistics on the prevalence of tweet embedding across different news categories, outlets, and users, highlighting key patterns.

Data Availability

The NewsTweet dataset is not publicly available for direct download. Due to Twitter/X’s Terms of Service restrictions on redistributing tweet content, the full dataset cannot be shared openly. Researchers interested in accessing the data or the collection pipeline are encouraged to contact the authors via the arXiv paper (arXiv:2008.02870).

Dataset Characteristics

Scale and Coverage

• News Sources: 5,961 unique news domains aggregated through Google News RSS feeds.
• Time Period: Data collection initiated on May 15th, 2019, with the paper describing the first four months of data.
• Collection Velocity: The pipeline averaged 2,302 articles per day, with approximately 296 containing embedded tweets.
• Content Types: Focuses specifically on embedded tweets from Twitter, the most frequently embedded platform.
• Metadata: Includes article source, Google News category (e.g., Sports, Health), and full tweet and user objects from the Twitter API.

Technical Implementation

• RSS-to-API Pipeline: Automatically crawls Google News RSS feeds to extract article HTML, identifying embedded tweet IDs to fetch full objects via the Twitter API.
• Artifact Filtering: Implements cleaning protocols to handle artifacts, such as detecting and excluding YouTube pages that appear as articles in Google News feeds.
• Longitudinal Tracking: Features a “top-off” mechanism that continuously tracks discovered users, updating their timelines to capture historical context.
• Rate Limit Management: Utilizes a random sampling queue to maintain continuous data collection across thousands of users without exceeding Twitter API limits.

Key Findings

Embedding Prevalence

• 13% of news articles in our Google News-sourced collection contained embedded tweets.
• Significant variation across categories: Sports (24% of articles) and Entertainment (14%) had the highest rates of embedding, while Health (2%) had the lowest.
• News outlets that publish the most articles are well-known mass media organizations, while outlets with the highest average number of embeds per article are often focused on Sports and Entertainment.

User and Content Patterns

• Public figures dominate: Well-known figures like politicians and celebrities, alongside organizations, are embedded far more often than ordinary users.
• Some users have a small number of their tweets embedded many times, while others gain newsworthiness from a wider range of their content.
• The Health category, despite having few embedded tweets, had the highest proportion of unique tweets (93%), suggesting that when tweets are embedded, they are less likely to be reused across multiple stories.
• “Catch-up” Phenomenon: Data reveals a class of users with high “embedding effectiveness”: those embedded more frequently than they tweet. This suggests journalists often use embeddings to “catch readers up” on backstories for previously unknown individuals.

Significance

The dataset is a foundation for studying how social media surfaces in journalism: how sourcing routines are evolving in the digital age, how traditional outlets and social platforms interact, and how previously-unknown users become newsworthy, grounded in the per-category and per-outlet embedding rates and the user-newsworthiness patterns the dataset captures.

My Contribution

I am the second of six authors on this paper. My contributions focused on the descriptive analysis: writing code to process the collected data, generating summary tables and statistics, and helping write and review the manuscript.

Citation

@misc{mujib2020newstweetdatasetsocialmedia,
  title={NewsTweet: A Dataset of Social Media Embedding in Online Journalism},
  author={Munif Ishad Mujib and Hunter Scott Heidenreich and Colin J. Murphy and Giovanni C. Santia and Asta Zelenkauskaite and Jake Ryland Williams},
  year={2020},
  eprint={2008.02870},
  archivePrefix={arXiv},
  primaryClass={cs.SI},
  url={https://arxiv.org/abs/2008.02870}
}

Related Work

• Look, Don’t Tweet: Unified Data Models for Social NLP: provides the unified cross-platform social media data model underlying broader Twitter analysis.
• Coordinated Social Targeting on Twitter: a companion study from the same research group and time period, documenting coordinated follower-manipulation patterns on high-profile Twitter accounts.

In the lead-up to the 2020 U.S. presidential election, we developed tools to monitor the follower dynamics of prominent Twitter users. This work details our investigation into several strange phenomena we observed, which suggest the presence of coordinated manipulation campaigns. We present preliminary evidence of networks of automated (bot) and potentially compromised accounts aiming to artificially inflate or diminish the perceived social status of high-profile individuals. (Note: this work was presented as an extended abstract at IC2S2 2020. It is available as an arXiv preprint but was not submitted as a full peer-reviewed paper.)

Key Contributions

• High-frequency sampling infrastructure: custom tooling sustaining 1-second follower-count polling over weeks, recursive follower-list “tunneling” through Twitter’s pagination limits, and API-rate-window management, capturing sub-minute signatures invisible to standard API sampling and requiring no internal platform access.
• Four documented phenomena: spikes & sawteeth, circulating accounts, resurrected pre-2010 accounts, and follower-list anomalies, each cataloged as a separate experiment.
• Preliminary evidence of coordination: anomalous patterns consistent with systematic bot activity around prominent political accounts in the lead-up to the 2020 election, presented as observation rather than proven causation.

Methodological Strategy

To capture these transient phenomena, we built a high-frequency observational pipeline that bypassed standard sampling limitations. Our framework treated the social graph as a dynamic signal:

1. High-Resolution Sampling: Implemented “cycling scripts” to monitor follower counts at 1-second resolution, revealing sub-minute “spike” anomalies often missed by aggregate metrics
2. Recursive “Tunneling”: Developed a recursive sampling method to extract deep history from follower lists, uncovering strata of “ancient” accounts buried by recent activity
3. Cross-Referencing: Correlated circulation events with follower count distortions to validate the “sawtooth” waveform hypothesis

Engineering Note: This methodology required custom infrastructure to work around Twitter API rate limits and pagination constraints. We built a “tunneling” system that recursively paginated through follower lists (which Twitter caps at 5,000 per request) while respecting rate windows. The “cycling scripts” maintained persistent connections and implemented intelligent backoff strategies to sustain continuous 1-second polling over weeks of observation. This data engineering work was essential to capturing the high-resolution temporal signatures that standard API usage would miss.

Key Findings

• Spikes & Sawteeth: Dramatic, sub-second spikes and saw-tooth wave patterns in follower counts, suggesting automated activity
• Circulating Accounts: Cohorts of users repeatedly following and unfollowing the same high-profile accounts
• ‘Resurrected’ Ancient Accounts: Disproportionate numbers of pre-2010 Twitter accounts with long inactivity periods suddenly appearing as recent followers
• Follower List Anomalies: Unusual temporal groupings in follower histories suggesting historical manipulation campaigns

Significance & Sociological Impact

Beyond the technical detection of botnets, this work raises questions about a potential vulnerability in the infrastructure of modern public discourse. The observed patterns suggest these coordinated campaigns may be targeting the metrics that algorithms use to verify relevance (such as follower counts and engagement rates) as a vector for artificially shifting perceived social proof.

• Manipulating social-proof signals: Our observations are consistent with attempts to manipulate the “rich-get-richer” dynamics of social platforms. If inorganic follower inflation can influence recommender systems, that could amplify specific narratives to legitimate users. This remains a documented observation rather than a proven causal chain, and warrants further investigation.
• Account-age filters are not enough: The data highlights patterns where dormant pre-2010 accounts appear as new followers in bulk, suggesting that account-age alone is insufficient as a defense against coordinated manipulation.
• A Case for Metric Scrutiny: These findings suggest that raw follower counts are noisy signals that can be manipulated at scale. Supplementing them with engagement-quality metrics may make platform recommender systems more robust to coordinated inorganic activity.

Citation

@article{heidenreich2020investigating,
  title={Investigating Coordinated 'Social' Targeting of High-Profile Twitter Accounts},
  author={Heidenreich, Hunter Scott and Mujib, Munif Ishad and Williams, Jake Ryland},
  journal={arXiv},
  volume={abs/2008.02874},
  year={2020},
  url={https://arxiv.org/abs/2008.02874},
  eprint={2008.02874},
  eprinttype={arXiv}
}

Related Work

For related work in the computational social science cluster:

• NewsTweet Dataset: Social Media in Digital Journalism: a companion dataset studying how tweets surface in journalism, from the same research group and time period.
• Look, Don’t Tweet: Unified Data Models for Social NLP: provides the unified cross-platform social media data model used in broader analysis of Twitter and related platforms.
• GPT-2 Susceptibility to Universal Adversarial Triggers: proposes UAT-based bot detection as a constructive application, addressing the same automated manipulation behavior documented here from a complementary detection angle.

We introduce a novel unsupervised algorithm for inducing semantic networks from noisy, crowd-sourced data. By framing network construction as a “relationship disambiguation” task, we process Wiktionary’s English entries to build a massive, WordNet-like semantic resource. The resulting network is more than 5x larger than Princeton WordNet and features over 344,000 linked example sentences (vs. WordNet’s 68k). Evaluation on standard word similarity benchmarks demonstrates that our fully data-driven approach yields semantic structures competitive with expert-annotated resources.

Key Contributions

• Unsupervised Hierarchy Induction: We propose a deterministic algorithm to construct a Directed Acyclic Graph (DAG) of senses from pairwise relationships, effectively inducing a semantic hierarchy without human supervision.
• A Massive Semantic Resource: We release a dataset enriched with hundreds of thousands of semantically linked usage examples, serving as a critical resource for tasks like Word Sense Disambiguation (WSD).
• Disambiguation Framework: We model “relationship disambiguation” using a Laplacian kernel and FastText embeddings to filter noisy user annotations.
• Open-Source Infrastructure: We provide a full pipeline for downloading, parsing, and constructing networks from Wiktionary data.

Technical Approach

The core of our method addresses the noise inherent in crowd-sourced dictionaries. We frame the problem as Latent Semantic Network Induction:

1. Relationship Disambiguation: For every linked pair of words (e.g., go ~ proceed), we define a semantic subspace using their definitions. We utilize FastText embeddings and a Laplacian kernel to identify which specific definitions participate in the relationship.
2. Hierarchy Construction: We apply a custom intersection algorithm that treats more general senses as the “overlap” between specific definition sets. We formalize this as a set-theoretic “hole punching” operation, where a general sense $t$ is defined by the intersection of definition sets $\mathbb{D}’$, excluding any broader intersections:

$$f^{-1}(t) = \left(\bigcap_{\mathbb{D}’} D_{u\sim v}\right) \setminus \left(\bigcup_{\mathbb{D} \supset \mathbb{D}’} \bigcap_{\mathbb{D}} D_{u\sim v}\right)$$

Evaluation & Validation

The primary achievement is scale: our induced network contains over 344,000 linked example sentences, compared to Princeton WordNet’s 68,000 (more than 5x the coverage), built entirely from crowd-sourced data without expert annotation.

Beyond scale, the network holds up semantically. On standard noun-similarity benchmarks (RG-65), the unsupervised network achieves a Spearman rank correlation of $\rho = 0.83$, matching the performance of Explicit Semantic Analysis (ESA) models built on expert-annotated WordNet ($\rho = 0.82$). The point is not that we beat WordNet by 0.01. It is that a fully automated approach over noisy Wiktionary data produces a resource of comparable quality at 5x the scale.

Why This Matters

Building high-quality linguistic resources typically requires expensive expert annotation. Princeton WordNet took decades of lexicographer effort. This work demonstrates that an unsupervised algorithm over crowd-sourced data can produce a resource of comparable semantic quality at more than 5x the scale. For ML practitioners, that matters: larger coverage means more training signal for downstream tasks like Word Sense Disambiguation. For this portfolio, it shows early experience building structured NLP datasets from scratch, a theme that continues in later work on large-scale document corpora.

Related Work

For a theoretical treatment of word semantics from the same collaboration, including the first analytical solution to Word2Vec’s softmax objective, see Analytical Solution to Word2Vec Softmax & Bias Probing.

Citation

@inproceedings{heidenreich2019latent,
  title={Latent semantic network induction in the context of linked example senses},
  author={Heidenreich, Hunter and Williams, Jake},
  booktitle={Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)},
  pages={170--180},
  year={2019}
}