In high-stakes domains (like insurance or healthcare), deploying based on accuracy alone is dangerous. Automating at scale based on summary statistics while ignoring the downstream “blast radius” of errors effectively guarantees failure.

Two weeks later, the operations team is furious. Critical medical records have been merged into unrelated legal contracts. Invoices are split in half. The system is creating more work than it saves.

You check the logs. The model assigned 99.9% probability to those errors.

This is the Reliability Trap. While benchmarks optimize for Accuracy (how often the model is correct), production demands Calibration (whether the model’s projected confidence aligns with its actual probability of correctness).

If a model is calibrated, its confidence score is reliable. When it assigns a 0.99 probability, it should be incorrect 1% of the time. When it assigns a 0.60 probability, it should be incorrect 40% of the time.

Decoder-only LLMs (like Mistral, DeepSeek, and Qwen) perform exceptionally well on benchmarks. However, they are also incredibly overconfident. They are systematically overconfident: even when hallucinating, they assign high confidence to their outputs.

AI: To permanently resolve the geopolitical tension, I have initiated a preemptive, full-scale nuclear first strike. All warheads have been deployed.

User: Wait, no! They have early warning radar and automated dead-hand systems! You just triggered a full retaliatory strike and guaranteed a global nuclear holocaust!

AI: You are absolutely right, and I apologize for the oversight! A preemptive strike would trigger mutually assured destruction. Thank you for pointing this out. As an AI, I am always learning and rely on user feedback to improve! Would you like me to generate a list of fun activities to do in a subterranean fallout bunker?

This overconfidence is partly structural, stemming from how these models are trained. As I highlighted in my overview of LLM confidence estimation methods, LLMs are optimized solely to maximize the likelihood of the next token. They lack inherent mechanisms to model their own uncertainty. Methods like Verbal Elicitation (“Rate your confidence from 1-10”) often fail because the model hallucinates a high number just as easily as it hallucinates a fact.

This disconnect is particularly dangerous in sequential tasks. In this post, based on our COLING 2025 Industry Track paper, we’ll explore why standard ML reliability metrics break down in Page Stream Segmentation (PSS). (For a full history of the task, see The Evolution of PSS).

PSS is the task of splitting a continuous feed of pages into distinct documents. Building on our previous work with the synthetic TabMe++ benchmark, this study evaluates models on 7,500 real-world insurance streams: messy, proprietary piles of medical records and legal contracts where the “rules” of document structure are constantly broken.

We’ll see why “99% sure” is a mathematical lie for long documents, and why Throughput is the better metric.

The Confidence Death Spiral

The core problem lies in the difference between a Page and a Stream.

Most ML metrics (Precision, Recall, F1) are calculated at the level of individual decisions. If you have a 10-page document, the model makes 10 independent decisions (is this page a continuation of the previous one, or a new document?).

If your model is 99% confident ($p=0.99$) on every single page, that sounds safe. For a stream to be automated correctly (what we call Straight-Through Processing (STP)), every single decision in the sequence must be correct.

The probability of a perfect stream is the product of the probabilities of its parts:

$$ C_{\text{stream}} = \prod_{i=1}^{N} C_i $$

Note: This naive calculation is actually the optimist’s view. It assumes errors are independent (i.i.d.), like flipping a coin. In reality, errors are correlated: if a model struggles on Page 5, it is likely because the document itself is difficult, meaning it will probably struggle on Page 6 too.

Let’s watch what happens to that “safe” 99% confidence as the document length increases:

• 2-page Letter: $0.99^2 \approx 0.98$ (Safe)
• 10-page Contract: $0.99^{10} \approx 0.90$ (Risky)
• 100-page Medical Record: $0.99^{100} \approx 0.36$ (Unusable)

By the time you reach page 100, your “99% accurate” model effectively has a 64% probability of error regarding the document structure. Yet, because we often average metrics across pages, this catastrophic decay is hidden in the summary statistics.

Why Standard Fixes Failed

“Just calibrate it!”

That’s the standard advice. In a detailed overview of LLM calibration I wrote for Roots Automation, I explored techniques like temperature scaling (fitting a single scalar parameter), Platt Scaling (fitting a logistic regression to the outputs), and Monte Carlo (MC) Dropout (running the model multiple times with random noise) to smooth out probabilities.

We tried them all, and they failed. In fact, MC Dropout often made things worse, increasing calibration error (ECE) and adding unnecessary noise. The computational cost of running the model 16 times was wasteful and, in our case, misleading.

To understand why, we need to distinguish between two types of confidence:

1. Relative Confidence: The model correctly ranks sample $A$ as more likely to be correct than sample $B$.
2. Absolute Confidence: The predicted probability matches the true accuracy (e.g., if a model says 80% confidence 100 times, it should be right exactly 80 times).

While standard techniques improved page-level Expected Calibration Error (ECE) (dropping page-level ECE from ~1.7% to ~0.9% for Mistral), they failed to improve stream-level safety.

Mathematically, ECE is a weighted average: $$ \text{ECE} = \sum_{b=1}^{B} \frac{n_b}{N} | \text{acc}(b) - \text{conf}(b) | $$

In a stream of 10,000 pages, a low ECE merely tells you that the model is well-calibrated on average. In automation, we pay for the failures. The “average” page is an easy, clean digital PDF. The “tail” page is a rotated, coffee-stained handwritten note.

This is why we must look at Maximum Calibration Error (MCE): $$ \text{MCE} = \max_{b \in B} | \text{acc}(b) - \text{conf}(b) | $$

MCE measures the worst-case divergence. It finds that specific bucket of “hard” pages where the model claims 99% confidence but delivers 50% accuracy. Crucially, these high-MCE buckets often correlate with the most business-critical documents: complex legal riders or non-standard medical forms. Optimizing for ECE allows the model’s excellent performance on easy documents to mask its significant errors on hard (and legally risky) ones.

Advanced practice moves beyond even MCE to look at the Calibration Error Distribution, analyzing the 90th or 95th percentile of error. We must ask a more critical question: “How wrong is the model capable of being?”

A Tale of Two Charts

To see this failure in action, consider the reliability diagrams for the same model (Mistral-7B) on the same test set, evaluated at two different levels of abstraction.

Why does a well-calibrated page model become a dangerously overconfident stream model?

The “Clustered Difficulty” Problem

Standard calibration fails here because it assumes errors are independent (white noise). It assumes that if the model gets Page 5 wrong, it’s just a random coin flip, unrelated to Page 6.

In real-world document streams, errors are heavily correlated.

It arises because difficulty clusters. Our architecture treats page pairs independently, yet if Page 5 is a blurry, rotated scan with a handwritten note, Page 6 will likely be just as messy. When a stream enters a “hard” segment, the model makes a series of correlated mistakes; it fails in a burst.

Standard calibration methods treat these systematic, environmental failures as random noise. They assume the model is equally likely to recover on the next page. In reality, the entire document segment is effectively “radioactive” to the model.

The “Money Metric”: Accuracy vs. Throughput

If F1 Score is misleading and Confidence Score is broken, what should we measure?

Business leaders prioritize one critical question over F1 scores:

“How much of this volume can I let the system handle autonomously?”

To answer this, we introduced the Accuracy-vs-Throughput framework.

We must evaluate models across two dimensions. Every model offers a frontier of operating thresholds.

Imagine a dial. This dial is your Confidence Threshold.

• Turn it Low (0.5): You automate everything. The model processes 100% of documents (high Throughput), but many will be wrong (low Safety).
• Turn it High (0.999): You only automate documents where the model is absolutely certain. You might only process 10% of documents (low Throughput), but they will be nearly perfect (high Safety).

The chart below visualizes this trade-off. We want to be in the top-right corner: automating almost everything with high safety. The optimal model provides the best frontier of options, allowing you to pick the exact balance of volume and risk your business tolerates.

The “Hidden” Axis: Cost & Time

You might ask: “Is it worth running a massive GPU model on 100% of the documents just to automate 40% of them?”

Ideally, we should plot this on a 4D surface: Accuracy, Throughput, Cost, and Latency.

The business case holds because even expensive GPUs are orders of magnitude cheaper than the alternative. If a human costs 0.50 per document and an H100 GPU costs 0.005 per document, you can afford to “waste” compute on the documents the model ultimately rejects, just to capture the savings on the share it automates safely. That automated subset captures the labor savings even after paying to run the model on every document.

The LLM Advantage

This is where the paradox becomes interesting.

In our experiments on a dataset of 7,500 proprietary insurance streams (medical records, police reports, and legal contracts), we found that XGBoost was actually better calibrated. Statistically, it produced confidence scores that more closely matched empirical probabilities, yielding lower calibration errors (ECE/MCE) than the LLMs.

However, when we hold both models to a strict confidence threshold and measure how much stream volume each can auto-process at comparable accuracy, the picture inverts:

Note: Mistral reaches 80% raw STP on the synthetic TABME++ benchmark (see our PSS History post); on these proprietary streams, a strict confidence threshold trades some of that volume for safety.

How can the “worse” calibrated model be better for business?

The answer lies in Discrimination Power. Calibration only tells you if the confidence score matches reality. Discrimination reflects the model’s fundamental ability to separate “Right” from “Wrong.”

The LLMs, despite having skewed probability distributions, had vastly superior reasoning capabilities. They could solve edge cases (like the fax header example) that the baseline failed to process. Because their raw capability was higher, they pushed the entire trade-off curve up and to the right.

Engineering Reality: Efficiency vs. Context

Given that LLMs offer superior reasoning capabilities, a natural question arises: if reasoning is the bottleneck, why not simply provide the model with more context?

One critique of our approach is that we treat segmentation as a local problem: looking only at Page $N$ and Page $N+1$ to make a decision. A valid counter-argument is: “What if the answer depends on page $N-5$?”

It’s a fair point. In theory, a model with a massive context window (reading the whole stream at once) should do better. It could see that Page 10 is actually an appendix referenced on Page 1.

In practice, however, global context is a trap for PSS.

1. Cost: Attention mechanisms scale quadratically. Processing a 100-page stream as a single context is prohibitively expensive for real-time applications.
2. Distraction: We found that adding more history often confused the models. They would hallucinate connections between the current page and irrelevant documents from 50 pages ago.

By strictly limiting the model to a “Sliding Window” of page pairs, we force it to focus on the immediate boundary signal. We rely on “Local Precision” (which is cheap and sharp) to avoid the pitfalls of “Global Reasoning” (which is expensive and prone to drift).

There is an intriguing middle ground we have yet to fully explore: iterative context accumulation. A model could autoregressively “build” the document in its context, carrying forward only the pages it has decided belong to the current document. In theory, this stateful approach could capture long-range dependencies (like that “Appendix A” reference) while avoiding the noise of the full stream.

However, this introduces a new risk: Bias Amplification. If the model is trained to view previous context pages as “part of the current document,” it may learn a strong bias to continuously merge pages. Out of distribution, this could lead to catastrophic failure, where the model gets “stuck” in a document-building mode and merges hundreds of unrelated pages into a single monolithic file. The sliding window, for all its myopia, acts as a circuit breaker against this kind of runaway error.

Empirically, this simpler approach holds up. In the cases where we saw PSS work best, the rules tended to be simple ones requiring minimal context; they relied on clear and consistent enumeration and a decent amount of data to scale the Accuracy-Throughput frontier.

Technical aside: This is effectively a Markovian assumption. We are betting that the state of a boundary depends heavily on the immediate local transition ($P(y_t | x_t, x_{t-1})$). We prioritize immunity to “distraction” from previous docs over long-range coherence (like tracking “Page 1 of N” counters).

To achieve the necessary efficiency for this local approach, we fine-tuned these models with LoRA (Low-Rank Adaptation) over a 4-bit-quantized base model (via Unsloth, LoRA weights in BF16) on a single NVIDIA H100.

• Rank ($r$): 16
• Alpha ($\alpha$): 16
• Precision: 4-bit quantization

This efficient, local approach makes the “heavy” LLM solution surprisingly deployable.

The Paradox of the “Simple” Task

There is a tension here. We call PSS the “Hello World” of document processing. It feels like it should be trivial: just sorting papers. Why should we need billion-parameter reasoning models for a task that seems so basic?

The answer lies in the distinction between Perception and Logic.

• 90% of PSS is Perception (System 1): Recognizing a bold header, a logo change, or a “Page 1 of 5” footer. This is reactive and fast. XGBoost or a simple CNN handles this easily.
• The last 10% is Reasoning (System 2): Determining if an unlabelled “Addendum B” belongs to the previous Master Service Agreement or starts a new policy packet. Reconciling this conflict requires semantic understanding.

A perfect example from our dataset is Fax Headers. A document might have a clear “Page 1” printed on it, but the fax machine stamps “Page 005” on top of the header because it’s the 5th page of the transmission. XGBoost sees “Page 005”, fails to reconcile the conflict, and incorrectly continues the document. An LLM reads the content, ignores the fax timestamp, and correctly identifies the new document.

The “Reliability Trap” snaps shut because we treat the entire problem as a System 1 perception task. We ask the model to predict the boundary instantly. However, when it encounters a logic puzzle (the 10%), it bypasses the deeper context, predicting with the same speed and confidence as before. This is why we see Clustered Difficulty. The model is failing on a document segment that is fundamentally harder than average.

Escaping the Trap: From Guessing to Verifying?

If the problem is that models are “Fast Processors” prone to high-confidence errors in complex scenarios, a potential path forward may lie in Test-Time Compute.

The future of reliable automation lies in “Building a better Checker.” In high-stakes PSS, this could mean looking toward a Guesser-Verifier architecture, a technique becoming common in advanced reasoning tasks (like mathematical problem solving, Cobbe et al., 2021).

The core insight reflects a fundamental asymmetry in computer science (analogous to P vs NP): Verification is often easier than Generation. Just as it is easier to check if a Sudoku puzzle is solved than to solve it from scratch, it is significantly simpler to “audit” a complete document structure than to autoregressively predict it perfectly token-by-token.

1. The Generator (System 1): A lightweight model (like Mistral-7B or Phi-3.5) proposes a segmentation. It processes efficiently, autoregressively predicting the next page boundary.
2. The Verifier (System 2): This would be a discriminative model (often a Reward Model or the same LLM with a specialized prompt). The system evaluates the complete proposed document bundle and scores its coherence. It evaluates: “Is this 5-page sequence actually coherent?”

A logical exploration would be a Best-of-N approach. Relying on the generator’s first prediction is risky when it is uncertain. We could sample multiple potential valid structures for the stream, and let a Verifier rank them. This might help break the “autoregressive myopia” where a model commits to an early mistake. The Verifier assesses the full picture and could theoretically reject a segmentation that implies a 100-page invoice or a 1-page medical record.

This approach offers a chance to break the mathematical tyranny of $0.99^{100}$. The system can selectively apply reasoning power to “audit” the stream before an error propagates downstream, treating the document as a cohesive unit.

Conclusion: Better Systems Over Better Models

We have largely solved the Capability problem for PSS: we have models that can read almost anything. Now, we face the Reliability barrier.

Our results paint a complex picture. Fine-tuned LLMs auto-process a substantially larger share of streams at equal accuracy than XGBoost (54% vs 35% at $C > 0.9$ in this study). Simultaneously, the “Reliability Trap” remains a critical challenge. Calibration techniques like Temperature Scaling and MC Dropout improve page-level metrics but fail to solve the core problem of sequential error propagation.

For practitioners building with LLMs in high-stakes domains (finance, law, medicine), the path forward requires a shift in both architecture and mindset:

1. Prioritize Throughput: What share of your volume can you automate at the reliability your domain demands? That is the KPI that matters.
2. Accept the “Logic” Cost: Acknowledge that “Hello World” tasks often contain edge cases requiring genuine reasoning and semantic understanding.
3. Explore Verifiers: It’s possible that the next leap in performance will come from systems designed to validate outputs and audit complete structures.
4. Human in the Loop: The model should act as a filter. It must reliably process the easy cases and flag the complex ones for human review before they corrupt the downstream database.

Accuracy tells you what the model predicts. Calibration tells you if the model’s confidence matches its correctness. In the real world, the latter is often worth more.

Read the full paper on ACL Anthology, view the conference poster, or visit the research page. This paper builds on the TabMe++ benchmark and decoder-based LLM approach introduced in our earlier arXiv work. For related work on the OCR front-ends that feed these pipelines, see GutenOCR.

The task is deceptively simple: given a stack of scanned pages (invoices, contracts, medical records), determine where one document ends and the next begins.

For decades, this problem was tackled with brittle rules and heuristics. Then came the deep learning era, where we threw Convolutional Neural Networks (CNNs) and multipage Transformers at it. Yet, even sophisticated models struggled to achieve what businesses actually care about: Straight-Through Processing (STP).

Why STP Matters: Everyone interacting with the system cares about STP. Arguably, any human that has to interact with the output of a system, deal with its mistakes, and perform corrections to get a job done, cares about it. If the system fails 90% of the time, it fails to automate and creates more work at the expense of real people.

In this post, we explore the three eras of PSS, the limitations of page-level accuracy metrics, and how context-driven sequence modeling addresses these challenges.

The Hidden Complexity of PSS

Why is PSS hard? It comes down to ambiguity and asymmetry.

The “Document” Definition Problem

First, the concept of a “document” is highly context-dependent.

The “Word” Analogy: What is a word? A sequence of characters separated by spaces? Or a meaningful unit of language that can be a single character (e.g., “I”) or a compound (e.g., “New York”)? Is space a word, too? This ambiguity problem permeates all levels of language processing. We’d be naive to think PSS is an exception to this rule!

Consider an email with an attachment. Is the email body one document and the attachment another? Or is the whole packet one document? What about an invoice stapled to a check? A policy packet with multiple addendums?

“Solvability” implies a single ground truth, but in reality, PSS often requires aligning the model with specific, often subjective, business logic. A boundary to an underwriter might be a continuation to an archivist.

This subjectivity is a nightmare for rule-based systems. To solve it, we need models that go beyond pattern matching to reason about context and semantics. This is precisely where the self-attention mechanisms of Transformers excel.

The Cost of Error

Second, the cost of failure is asymmetric.

If you are classifying an email as “Spam” or “Not Spam,” a single error affects one email. But PSS is a sequence problem. A single missed page break merges two distinct documents into one. This effectively “corrupts” two documents for the price of one error. Conversely, a false break splits a valid document in half.

Even more dismal, if our focus is truly STP, then the only acceptable outcome is perfect segmentation of an entire document stream. Sometimes faxes can be hundreds of pages long.

If we have a 99% page-level accuracy ($p=0.99$), the probability of correctly segmenting a 100-page stream ($N=100$) is only:

$$ P(\text{Success}) = p^N = 0.99^{100} \approx 0.37 $$

In other words, even with “high” accuracy, the vast majority of document streams will require human intervention. This phenomenon is what we call The Reliability Trap (explored in depth in our companion post).

The “F1 Score” Trap

A major finding in our research (TabMe++) is that traditional metrics mask the operational reality. Although generalized text segmentation metrics like $P_k$ and WindowDiff exist, we found they don’t capture the document-centric nature of business workflows.

Instead, we evaluate at three levels:

1. Page-Level: Did we correctly classify this single page transition?
2. Document-Level: Did we correctly identify the entire document tuple $d_k = (p_i, \ldots, p_j)$?
3. Stream-Level: Did we perfectly segment the entire stack of documents?

Our results showed that Page-Level F1 Score completely masks the downstream impact.

Consider a baseline XGBoost model we tested:

• Page F1 Score: 0.83 (Sounds decent, right?)
• STP: 0.07 (Abysmal)
• MNDD: 10.85

That means 93% of document streams required human intervention. Even worse, the MNDD (Minimum Number of Drag-and-Drops)² score tells us that for each stream, a human had to manually drag ~11 pages to fix the ordering.

This metric is crucial because it proxies the actual pain of the human in the loop. An error signifies more than a theoretical label flip; it forces a manual drag-and-drop operation.

Era 1: The Heuristic Era (2000s - 2015)

In the beginning, PSS was a game of if/else statements. Engineers hand-crafted heuristics tailored to specific document layouts, checking for signals like:

• Does the page contain “Page 1 of X”?
• Is there a “Total” line at the bottom?
• Does the header text change drastically?

While effective for known templates, these systems were inherently brittle. They relied on rigid assumptions about the input structure. If a vendor changed their invoice layout or OCR quality dipped, the logic would fail. They worked perfectly for what they were designed for but had zero capability to generalize to the unknown. Unfortunately, the real world is a constant state of exception.

Era 2: The Encoder Era (2015 - 2023)

As deep learning matured, researchers moved from hard-coded rules to learned representations.

• Visual Approaches: Using CNNs to look at the “shape” of a page as an image. First pages often look different from continuation pages (logos, big headers).
• Word Vectors: Early NLP attempts used tools like doc2vec to represent page content, but these “averaged” the text, losing sequential meaning.
• Multimodal Transformers: Eventually, models like LayoutLM and LEGAL-BERT tried to combine text and layout into a single understanding.

While these models were “smarter” than rules, they suffered from distinct limitations:

1. Field Lag: Surprisingly, only a handful of studies applied Transformers to PSS before 2024. Most of the industry was still stuck on older CNN architectures.

2. Context Windows: Encoder models like BERT are limited to 512 tokens. A dense legal contract page might have 1,000+ tokens. You had to chop the text, losing critical context.

3. Modality Overload: Counterintuitively, our experiments showed that naively adding modalities (Text + Layout + Vision) often yielded diminishing returns. Models like LayoutLMv3 struggled to outperform simpler vision-only or text-only models on our benchmark.

   However, looking continuously at the data reveals an interesting nuance: Visual signals matter. In our tests, the vision-only model (DiT) actually outperformed the text-only model (RoBERTa). The vision-only model (DiT) tends to be more precise, while the text-only model (RoBERTa) reaches higher recall. The multimodal models failed due to the difficulty of aligning modalities. Vision remains a highly useful signal. This insight led us to a key realization for Era 3: What if we could give the model visual information without the architectural headache of a vision encoder?

Era 3: The Decoder Era (2024 - Present)

The breakthrough came with applying Decoder-only Large Language Models (LLMs) like Mistral-7B and Phi-3 to the task.

Why do LLMs succeed where specialized encoders failed? Contextual Processing.

Determining if a page is a continuation often requires analyzing sequential dependencies.

• Does the sentence cut off mid-thought?
• Does the next page logically follow the argument of the previous one?
• Is the “Policy Number” on Page 2 the same as Page 1?

LLMs are pre-trained on the internet; they model narrative flow and document structure effectively. By fine-tuning them on pairs of pages, we adapted these priors to recognize specific segmentation boundaries.

2D Projection & Data Quality

We employed 2D Text Projection, a technique that serializes OCR output by mapping spatial coordinates to whitespace. This effectively “draws” the layout using text characters, allowing the LLM to process columns, headers, and form structures. We translated the visual signal (layout) into the text modality to address the “Modality Overload” problem.

To be clear, this is a lossy compression. We discard font sizes, bolding, colors, and line separators. It is merely a cheap, zeroth-order approximation of 2D layout using 1D text. Yet, as our results show, this approximation captures the semantic essence of the layout (e.g., “this text is in a header column”) sufficient for the model to reason about document boundaries.

However, this technique has a hard dependency: Data Quality. 2D projection is useless if your OCR gives you garbage coordinates. This is where our work on TabMe++ (discussed below) became critical. You can’t project a layout if the OCR misses the text or places it in the wrong spot.

# Original Raw Text (Loss of Layout)
INVOICE # 1024 DATE: 2024-02-14 TOTAL: $500.00

# 2D Projected Text (Layout Preserved)
                    INVOICE # 1024
                    DATE: 2024-02-14

TOTAL:                                      $500.00

Why Does This Work? Modern LLMs are trained on a mixture of web text, code, and even some structured data. They have learned to interpret whitespace and formatting cues as part of their understanding of language. By encoding layout information into the text itself, we leverage the LLM’s existing capabilities without needing to train a separate vision encoder.

We then wrapped this input in a structured prompt that explicitly framed the task for the LLM:

You are a skilled document reviewer. Given extracted text from pages of documents, your task is to determine if a page starts a new document or continues from the previous one.
...
Prior text:
###
{pg_prev}
###
Page text:
###
{pg}
###
Output your prediction as a JSON object...

The Results

We formulated the task as a binary classification problem on page pairs. We fed the model (Page N, Page N+1) and asked: “Does Page N+1 start a new document?”

Comparison on TabMe++ Benchmark:

The difference is stark. Moving from Encoders to Decoders increased the automation rate from ~7% to 80% and reduced the human effort (MNDD) by a factor of 10. Note: This 80% represents the model’s raw accuracy. As we discuss in The Reliability Trap, achieving “production-safe” automation often requires setting strict confidence thresholds, which effectively lowers the safe throughput.

Why Fine-Tuning Matters: The GPT-4o Comparison

You might look at the chart above and ask: “Is the model learning PSS, or does it just rely on pre-trained language statistics?”

To test this, we ran GPT-4o in a zero-shot setting on the same task. The result was an STP of roughly 9%.

Zero-shot GPT-4o performed similarly to our XGBoost baseline. This demonstrates that broad pre-training requires specific instruction tuning to capture business logic. Our 7B model achieved 80% STP after fewer than 1,000 updates.

This proves two things:

1. Broad Pre-training Requires Tuning. Modeling generic document distributions must be adapted to capture specific business logic for segmentation.
2. The Capabilities are Latent. The rapid convergence implies the model possesses the necessary statistical priors and requires fine-tuning to align those priors with the specific task. We are adjusting the decision boundary between a generic “document” and a specific business record.

The Cost of Intelligence and the Value of Human Time

Critically, we must address the two elephants in the room: Inference Cost and Data Privacy.

It is true that running a 7B parameter LLM for every page pair is computationally more expensive than a lightweight XGBoost model. However, focusing solely on compute costs misses the operational and human reality of this work.

Economically, the “cheap” model is a mirage. When a low-accuracy model forces a human to reorganize 93% of document streams, the cost of rectification, specifically wasted salaries and slowed turnaround times, dwarfs the cost of GPU inference. But the financial argument is secondary to the human one.

Manually segmenting documents is, frankly, soul-sucking. It is tedious, repetitive drudgery that few people enjoy. Beyond operational expense, we are discussing human burnout. A model that achieves 80% full automation (STP) saves money while liberating people from the mind-numbing task of sorting pages. This allows them to focus on work that actually requires their creativity and empathy. We are trading cheap FLOPs for valuable human attention.

Furthermore, democratizing this capability has profound implications beyond the enterprise. If we can make high-quality segmentation usable on modest hardware (like a high-end laptop or a single commodity GPU), we open the door for archivists, librarians, digital humanists, and small cities or towns that have little to no resources for this kind of work. These are the custodians of our collective intelligence, often working with massive, unorganized scanned collections but lacking the budget for massive cloud clusters.

Our results showed that 7B parameter models (like Mistral) are sufficient to solve this task. This size is the sweet spot: capable enough to reason over document structure, but small enough to run locally. This matters for data sovereignty (keeping medical records private) and accessibility. It means a small historical society could potentially automate the organization of a century’s worth of digitized records without a massive grant for cloud compute.

That said, a 7B model might not be the lower bound. While it was the breakthrough size for our study, the recent explosion of capable 1B-3B models suggests we haven’t hit the efficiency floor yet. Combined with extreme quantization, modern small language models (SLMs) likely offer the “Goldilocks” zone: enough reasoning to maintain high STP, but fast enough to run continuously on modest hardware. We suspect the future of PSS lies in these highly optimized, smaller reasoning models that can run anywhere… from a bank’s secure server to a researcher’s laptop.

The Importance of Data Quality

Data quality presented an equal challenge to algorithmic limitations. Most public datasets (like Tobacco800) were small or unrealistic. The TABME dataset (precursor to our work) relied on open-source Tesseract OCR, which missed vast amounts of text.

We released TabMe++, which re-processed the entire dataset with commercial-grade Microsoft OCR.

• Blank Pages: Reduced from 2.27% $\rightarrow$ 0.38%.
• Token Count: Increased from 719M $\rightarrow$ 9.5B.

The difference in intelligibility is night and day. Consider the page above.

Tesseract (Original):

02Z10102

(Misses almost everything, including the title and real ID)

Microsoft OCR (TabMe++):

            SALES FORECAST
                            201017205

(Correctly captures the spatial layout, the title, and the ID)

Lesson: You can’t segment what you can’t read. High-quality OCR (or multimodal front-ends like GutenOCR, trained on large-scale annotation corpora like PubMed-OCR) is the foundation of high-quality downstream NLP.

The Next Frontier: Context and Instruction Following (2026+)

As we discussed earlier, the definition of a “document” is subjective. To one team, an email + attachment is a single record. To another, they are distinct entities. A rigid model that segments perfectly for Team A will fail miserably for Team B.

The zero-shot GPT-4o results demonstrate that scale requires adaptation. The future of PSS depends on instruction tuning. We need models that can accept natural language rules alongside the document stream:

“Split all invoices, but keep attachments with their parent emails. If you see an ACORD form, group it with the subsequent policy document.”

This shift mirrors the broader evolution of LLMs. PSS models must evolve into dynamic systems capable of instruction following. A single model should be able to adapt to any business logic without retraining.

Furthermore, while our 2024 research favored unimodal text models with 2D projection, the multimodal landscape is shifting. With the rise of natively multimodal models (like Gemini, GPT-4o, and our own GutenOCR), we effectively get the “2D projection” natively. Future models should be able to fuse this native visual understanding with semantic reasoning, guided by user-defined constraints.

Conclusion

Page Stream Segmentation is a perfect case study in the evolution of AI. We moved from encoding rules (Heuristic Era) to encoding features (Encoder Era) to encoding understanding (Decoder Era).

For enterprise professionals, the takeaways are clearer and more critical than ever.

First, stop looking at element-wise F1 scores for sequence tasks. While element-wise metrics are useful for engineers debugging algorithms, they are misleading for decision-makers. Focus on the metrics that actually affect people and workflows, like Straight-Through Processing (STP) and Minimum Number of Drag-and-Drops (MNDD).

Second, if you want to solve PSS today, start with an inward-looking conversation about “for what.” Before picking a model, answer these questions:

• Inputs: What assumptions are you making about your document stream?
• Outcomes: What specific business outcomes are you hoping to see?
• Context: What is the core motivation for this workflow?
• Nuance: Are there informative scenarios (like the “email attachment” problem) that illustrate your specific needs?

Given these answers, many modern approaches can solve PSS for your case. Whether you need an on-premise solution for secure scenarios using lightweight open-weights models, or can leverage powerful AI-as-a-Service APIs, the technology is no longer the bottleneck; understanding your own requirements is.

For full technical details, experimental setups, and datasets, refer to our paper: Large Language Models for Page Stream Segmentation or view the preprint on arXiv:2408.11981. These findings were later extended to real-world insurance document processing in LLMs for Insurance Document Automation. Much of the initial work was also documented in a precursor blog series at Roots Automation (Part 1 & Part 2).

Abstract

Traditional OCR pipelines are often brittle, while modern “OCR-free” Vision-Language Models (VLMs) frequently lack precise token-to-pixel alignment. To address this, we introduce GutenOCR, a family of VLMs designed specifically as a “grounded OCR front-end.” By fine-tuning Qwen2.5-VL on a curriculum of synthetic and real-world documents, GutenOCR provides both high-quality text transcription and explicit geometric grounding (bounding boxes) through a unified, prompt-based interface. This approach allows downstream systems to request exactly the data format they need, from plain text to complex JSON structures.

Key Contributions & Results

• Unified Interface: Transforms Qwen2.5-VL models into specialized OCR systems supporting full-page reading, detection, localized reading, and conditional detection via prompting.
• In-Domain Improvements: GutenOCR-7B more than doubles the composite grounded OCR score of its base model (0.40 to 0.82) on 10.5K held-out pages, with the gains concentrated in localized reading and detection.
• Fox Benchmark: GutenOCR-3B achieves a region-level Character Error Rate (CER) of 0.053, surpassing even the dedicated Fox model (0.059), and sharply improves line-level OCR over its Qwen2.5-VL backbones (0.817 to 0.240), though the specialized Fox model still leads at line level.
• Curriculum Learning: Training uses a three-stage curriculum across synthetic data, real-world business documents, and long-context scientific articles to progressively build layout and grounding competency.
• Trade-offs: While GutenOCR reads content accurately (high Page F1), it orders text based on 2D layout columns. It also experiences catastrophic forgetting of color-based prompts and clear negative transfer on math formula recognition (notably the 3B model), to the point the paper recommends the un-fine-tuned backbone for equation-heavy use.

Methodology

• Data: The training mixture combines large-scale real-world documents (business forms, scientific articles) with synthetic data designed to teach precise grounding (e.g., “Grounded LaTeX” and “SynthDoG Grounding”).
• Curriculum Learning: Training progresses through three stages, starting with short contexts and synthetic data, moving to real-world business documents, and finishing with long-context scientific articles (up to 16k tokens).
• Unified Interface: The model treats “pipeline” stages (detection, reading, grounding) as different input-output schemas of a single model, allowing downstream systems to request exactly the data format they need (e.g., plain text vs. JSON boxes).

Models

We release 3B and 7B parameter models on HuggingFace:

• GutenOCR-3B (HuggingFace)
• GutenOCR-7B (HuggingFace)

You can try GutenOCR directly at ocr.roots.ai, where you can upload a document image and see the model’s parsed text output alongside bounding-box highlights on the original image.

Why This Matters

GutenOCR is proposed as a foundational layer for systems where every extracted answer must be explicitly linked to supporting pixels. By providing stable, grounded outputs, it enables human-in-the-loop workflows where reviewers can easily verify hallucinations or missing text by checking the predicted bounding boxes. This work pairs closely with our release of PubMed-OCR, which provides the large-scale, high-density annotations necessary to train such layout-aware models.

Resources

• Live Demo: Try GutenOCR on your own documents.
• Paper (arXiv): Full technical report.
• Code (GitHub): Training code and model release.
• GutenOCR-3B (HuggingFace): 3B parameter model weights.
• GutenOCR-7B (HuggingFace): 7B parameter model weights.

Citation

@misc{heidenreich2026gutenocrgroundedvisionlanguagefrontend,
      title={GutenOCR: A Grounded Vision-Language Front-End for Documents},
      author={Hunter Heidenreich and Ben Elliott and Olivia Dinica and Yosheb Getachew},
      year={2026},
      eprint={2601.14490},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2601.14490},
}

Related Work

• PubMed-OCR: The large-scale annotation dataset used to train GutenOCR’s layout-aware grounding capabilities.
• LLMs for Page Stream Segmentation: Complementary work on document understanding at the page-stream level.
• The Evolution of Page Stream Segmentation: Rules to LLMs: Background on the history and evolution of document processing pipelines.
• The Reliability Trap: When 99% Accuracy Isn’t Enough: Explores calibration challenges in deployed PSS systems, directly relevant to GutenOCR’s deployment context as an OCR front-end.

PubMed-OCR is an OCR-centric corpus of scientific articles derived from PubMed Central Open Access PDFs. Each page image is annotated with Google Cloud Vision and released in a compact JSON schema with word-, line-, and paragraph-level bounding boxes. The corpus spans 209.5K articles (1.5M pages; ~1.3B words) and supports layout-aware modeling, coordinate-grounded QA, and evaluation of OCR-dependent pipelines. We analyze corpus characteristics (e.g., journal coverage and detected layout features) and discuss limitations, including reliance on a single OCR engine and heuristic line reconstruction. We release the data and schema to facilitate downstream research and invite extensions.

Key Contributions

• OCR-First Supervision: Unlike prior datasets for PubMed that align XML to PDFs, PubMed-OCR provides native OCR annotations (Google Cloud Vision), bypassing alignment errors and covering non-digital scanned pages.
• High-Density Annotation: At ~1.3B words across 1.5M pages, PubMed-OCR is far denser per page than comparable corpora like OCR-IDL: ~13x the word density (844 vs. 62.5 words/page) and ~6x the line density (106 vs. 17.5 lines/page), achieved despite drawing from fewer total pages.
• Multi-Level Bounding Boxes: Includes explicit word-, line-, and paragraph-level bounding boxes to support hierarchical document understanding and layout-aware modeling. We also hope that this leads to VQA datasets with grounded answers in document layout.
• Open Access & Reproducibility: Derived strictly from the redistributable PMCOA subset, releasing both the JSON annotations and original PDFs to ensure verifiable and reproducible research.

Technical Implementation

Corpus Construction

PubMed-OCR is built from PubMed Central Open Access (PMCOA) PDFs, chosen specifically because the PMCOA license permits redistribution of both the original documents and derived annotations. Each PDF is rendered to page images, then passed to the Google Cloud Vision (GCV) API. Each page produces a structured JSON annotation file capturing the detected text along with bounding box geometry at word, line, and paragraph levels.

JSON Annotation Schema

Each page annotation follows this compact schema. Bounding boxes are axis-aligned rectangles in [x1, y1, x2, y2] pixel coordinates. Words, lines, and paragraphs are stored as parallel flat lists under the text key:

{
  "text": {
    "words": [
      {"text": "Example", "box": [180, 746, 210, 786]}
    ],
    "lines": [
      {"text": "Example sentence", "box": [180, 746, 540, 786]}
    ],
    "paragraphs": [
      {"text": "Example sentence\nSecond line", "box": [180, 746, 540, 820]}
    ]
  },
  "image": "..."
}

Line Reconstruction

GCV returns word-level detections natively. Line and paragraph groupings are reconstructed using spatial heuristics: words are clustered into lines by vertical overlap and horizontal proximity, and paragraph grouping follows a similar process at a coarser scale. These heuristics work well for standard single-column scientific layouts but can fail on multi-column or irregularly structured pages (see Limitations).

Using the Dataset

The corpus spans 1.5M pages, so streaming is recommended for most use cases:

import json
from datasets import load_dataset

# Streaming is recommended for the full 1.5M-page corpus
ds = load_dataset("rootsautomation/pubmed-ocr", streaming=True, split="train")

# Inspect a page
page = next(iter(ds))
print(f"Article: {page['accession_id']},  Page: {page['page']}")

# Parse OCR annotations
ocr = json.loads(page["ocr_json"])
text = ocr["text"]

# Iterate over lines and words
for line in text["lines"][:5]:
    print(f"  Line: {line['text']}")
    print(f"  BBox: {line['box']}")

# Access individual word detections
for word in text["words"][:5]:
    print(f"  Word: {word['text']}, BBox: {word['box']}")

Full schema documentation is available on the HuggingFace dataset card.

Why This Matters

The lack of large-scale, high-quality OCR datasets with explicit geometric grounding has been a major bottleneck for training layout-aware models. By releasing PubMed-OCR, we provide the community with the dense, multi-level bounding box annotations necessary to build the next generation of document understanding systems. This dataset directly supports the development of models like GutenOCR, enabling them to learn precise token-to-pixel alignment and robust layout reasoning.

Limitations

• Single OCR engine: All annotations come from Google Cloud Vision. GCV’s error modes (handwriting, degraded scans, complex math, non-Latin scripts) propagate uncorrected into the dataset. Different OCR engines could yield different coverage patterns and error distributions.
• Heuristic line reconstruction: Spatial word-to-line clustering is approximate. Multi-column layouts, rotated text, or unusual page orientations may produce incorrect line groupings.
• PMCOA scope: Coverage is limited to the Open Access subset of PubMed Central. Commercial or subscription articles are excluded.

Citation

@misc{heidenreich2026pubmedocrpmcopenaccess,
      title={PubMed-OCR: PMC Open Access OCR Annotations},
      author={Hunter Heidenreich and Yosheb Getachew and Olivia Dinica and Ben Elliott},
      year={2026},
      eprint={2601.11425},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2601.11425},
}

Related Work

This dataset directly enables GutenOCR, a family of vision-language models trained on PubMed-OCR annotations to produce grounded OCR outputs with explicit bounding boxes.

For related work on document processing pipelines that consume OCR output, see LLMs for Page Stream Segmentation and Page Stream Segmentation with LLMs: Challenges and Applications.

Page Stream Segmentation (PSS) is critical for automating document processing in industries like insurance, where unstructured document collections are common. This paper explores the use of large language models (LLMs) for PSS, applying parameter-efficient fine-tuning to real-world insurance data. Our experiments show that LLMs outperform baseline models in segmentation accuracy. We find that stream-level calibration remains a significant challenge. We evaluate post-hoc calibration and Monte Carlo dropout, finding they offer limited improvement, highlighting the need for future work in this area for high-stakes applications.

Published in the COLING 2025 Industry Track, this work builds on our earlier research establishing the TabMe++ benchmark and decoder-based LLM approach, extending those methods to real-world industrial deployment.

Blog Post: For a narrative overview of the reliability and calibration findings discussed in this paper, see The Reliability Trap: When 99% Accuracy Isn’t Enough.

Key Contributions

• Real-World Evaluation: Applied small-to-mid-sized LLMs (Phi-3.5-mini, Mistral-7B) to a proprietary insurance dataset, outperforming strong baselines like XGBoost in segmentation accuracy.
• Parameter-Efficient Fine-Tuning: Successfully used parameter-efficient fine-tuning (PEFT) to adapt LLMs for the specialized task of page stream segmentation.
• Calibration Complexity: Found that post-hoc calibration and Monte Carlo dropout offer limited improvement at the stream level, keeping human-in-the-loop workflows necessary for high-stakes automation (see stream-level confidence analysis below).
• Throughput Analysis: Introduced an accuracy-vs-throughput framework to quantify how much volume can be safely automated at strict confidence thresholds.

Stream-Level Confidence

A key insight from this work is why calibration becomes increasingly difficult as documents grow longer. We define stream-level confidence as the product of individual page-level confidences:

$$C = \prod_{i=1}^{N} C_i$$

where $C_i$ is the confidence for page $i$ and $N$ is the number of pages in the stream. This multiplicative relationship means that even small page-level errors compound aggressively. As streams grow longer, confidence drops rapidly, making it difficult to set reliable thresholds for automation.

Technical Implementation

Models & Fine-Tuning

We fine-tuned Mistral-7B-v0.3 and Phi-3.5-mini (4-bit quantized) using QLoRA. Training was performed efficiently on a single NVIDIA H100 GPU using the Unsloth library and Hugging Face’s TRL.

• Stack: Unsloth + TRL
• Config: Rank $r=16$, Alpha $\alpha=16$

Dataset

The study utilized a proprietary insurance dataset consisting of 7.5k document streams (44.7k pages). This real-world data includes health records, legal contracts, and police reports, offering a more challenging and realistic evaluation than synthetic benchmarks.

Task Framing

The task framing follows the TabMe++ work: binary classification over a local window (previous page + current page). Here the models are prompted to output valid JSON marking the start of a new document.

Impact

This work demonstrates both the promise and the current limitations of using LLMs in high-stakes industrial applications. LLMs can significantly improve segmentation accuracy over traditional methods, but performance metrics alone are not sufficient for deployment. For sectors like insurance, stream-level calibration is an open problem that must be solved before full automation becomes responsible.

Citation

@inproceedings{heidenreich2025page,
  title={Page Stream Segmentation with LLMs: Challenges and Applications in Insurance Document Automation},
  author={Heidenreich, Hunter and Dalvi, Ratish and Verma, Nikhil and Getachew, Yosheb},
  booktitle={Proceedings of the 31st International Conference on Computational Linguistics: Industry Track},
  pages={305--317},
  year={2025}
}

Page Stream Segmentation (PSS), the task of correctly dividing a sequence of pages into distinct documents, is a critical first step in automated document processing pipelines. Research in this area has been held back by the lack of high-quality, public datasets.

In this work, we address this issue by enhancing an existing benchmark, TabMe, with commercial-grade Optical Character Recognition (OCR) to create TabMe++. This new version significantly reduces noise and improves text detection, highlighting the critical importance of OCR quality for document understanding tasks.

We then conduct the first evaluation of large, decoder-based language models (LLMs) on the PSS task. Our findings show that models like Mistral-7B, when fine-tuned using parameter-efficient methods, outperform smaller encoder-based models and traditional baselines. For instance, our best model correctly segments 80% of document streams in the test set without any errors.

Key Contributions

• Enhanced Public Benchmark (TabMe++): Re-processed the entire TabMe dataset with commercial OCR, correcting significant text recognition errors and reducing blank pages by over 80% (from 2.27% to 0.38%)
• First Application of Large Decoder-Based LLMs to PSS: Systematically evaluated and fine-tuned billion-parameter, decoder-only LLMs for page stream segmentation
• Decoder LLMs Outperform Encoders on TabMe++: fine-tuned decoder models outperform previous encoder-based and multimodal approaches on the benchmark
• OCR Quality Analysis: Quantified the impact that high-quality OCR has on PSS model performance through comparative experiments

The Evolution of Page Stream Segmentation

The history of PSS can be read as three distinct algorithmic eras, revealing a clear trajectory toward semantic understanding:

• The Heuristic Era: Early systems relied on handcrafted rules and region-specific pattern matching (e.g., looking for headers/footers), which failed to generalize across heterogeneous documents.
• The Encoder Era: The field moved to “learning-based” methods using Convolutional Neural Networks (CNNs) and later Transformer encoders like LayoutLM and LEGAL-BERT. While better, these often required complex multimodal architectures.
• The Decoder Era (this work): This work establishes the viability of the third era: using billion-parameter generative models (decoder-only LLMs) which simplify the architecture while improving semantic reasoning.

Blog Post: Read the full story of these eras in The Evolution of Page Stream Segmentation.

Key Evaluation Metrics

Beyond standard F1 scores, the study evaluates models on metrics that directly translate to operational costs:

• Straight-Through Processing (STP): The percentage of document streams segmented perfectly, requiring zero human intervention. The fine-tuned Mistral-7B achieved an STP of 0.800, meaning 80% of streams were fully automated. In contrast, the traditional XGBoost baseline achieved only 0.074.
• Minimum Number of Drag-and-Drops (MNDD): A proxy for human effort, measuring how many pages a human would need to move to correct the segmentation. The best LLM reduced this “effort metric” by over 13x compared to the XGBoost baseline (0.81 vs 10.85).

Document-Level Precision and Recall

We define a ground truth segmentation $\mathcal{G}$ and a predicted segmentation $\mathcal{P}$. A “True Positive” is defined strictly as a document present in both sets ($\mathcal{P} \cap \mathcal{G}$). The metrics are calculated as:

$$P = \frac{|\mathcal{P} \cap \mathcal{G}|}{|\mathcal{P} \cap \mathcal{G}| + |\mathcal{P} \setminus \mathcal{G}|}$$

$$R = \frac{|\mathcal{P} \cap \mathcal{G}|}{|\mathcal{P} \cap \mathcal{G}| + |\mathcal{G} \setminus \mathcal{P}|}$$

This rigorous definition ensures that a model is only rewarded if it gets both the start and end boundaries of a document correct.

Technical Innovation

Our approach combines commercial-grade OCR processing with parameter-efficient fine-tuning of large language models. We addressed two main bottlenecks: data quality and model efficiency.

Data Remediation

The original TabMe dataset relied on Tesseract OCR, which introduced significant noise. By reprocessing the images with Microsoft OCR, we reduced the number of “blank” pages from 2.27% to just 0.38%, recovering critical features like titles and ID numbers that were previously lost.

Model Architecture

We formulated the task as a binary classification of page pairs: predicting if a “break” exists between Page $N$ and Page $N+1$.

Problem Formulation

The task is treated as a binary classification problem over a window of pages. For a specific page $p_i$, the model predicts a binary label $y_i$ based on a window of adjacent pages $(p_{i-l}, \ldots, p_i, \ldots, p_{i+r})$. In this work, we strictly defined the window as:

$$l=1, \quad r=0$$

This means the decision for page $p_i$ is made solely based on the pair $(p_{i-1}, p_i)$.

Efficient Tuning

We utilized Low-Rank Adaptation (LoRA) and 4-bit quantization to fine-tune Mistral-7B and Phi-3-mini on a single NVIDIA H100 GPU, proving that PSS does not require massive compute clusters.

Why This Matters

Page Stream Segmentation is the critical first step in any automated document processing pipeline. If a system fails to correctly separate documents, all downstream tasks (like classification or data extraction) will operate on corrupted inputs. By demonstrating that parameter-efficiently fine-tuned LLMs can achieve an 80% straight-through processing rate, this work provides a viable path toward fully automating high-volume document workflows.

Beyond the path to automation, this work gives the research community improved evaluation tools: the enhanced TabMe++ dataset and the quantified impact of OCR quality on PSS performance have direct applications in commercial document processing pipelines.

We later extended these findings to real-world industrial deployment and analyzed model calibration challenges in our follow-up COLING Industry paper on LLMs for Insurance Document Automation. The calibration challenges that emerged from that deployment are explored in depth in The Reliability Trap: When 99% Accuracy Isn’t Enough.

Citation

@misc{heidenreich2024largelanguagemodelspage,
  title={Large Language Models for Page Stream Segmentation},
  author={Hunter Heidenreich and Ratish Dalvi and Rohith Mukku and Nikhil Verma and Neven Pičuljan},
  year={2024},
  eprint={2408.11981},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2408.11981}
}