---
title: The Missing Format
author: E. M. Freeburg
author_id: https://ericfreeburg.com/#person
date: '2026-08-07'
updated: '2026-08-07'
canonical: https://ericfreeburg.com/last-fingerprint/missing-format/
---

# The Missing Format

## Summary

Every corpus decision the field records is about which text to train on: domain, language, quality, recency, deduplication. None is about how the text's arrangement is notated. This paper argues that notation has quietly become a training variable of the first importance, through a mechanism nobody chose: the structural markup that conversion pipelines now attach to everything long enough to have structure is a *perfectly reliable* cue for discourse boundaries. Where the cue is present, a boundary does not have to be inferred; it can be read. An inference has been removed from the curriculum — not by volume, which is trivial, but by reliability, which is what shortcut learning actually runs on.

We measure the situation three ways. A census of public pre-training corpora shows that the supply of long, coherent, low-boundary-density text has collapsed to a rounding error, and that the newest corpora are the most heavily marked. A converter study shows the markup is *faithful* — which makes the cue more trustworthy, not less, and closes the obvious escape route: there is no null level left to reweight toward, because after serialization no document has zero markup. And a paired-serialization probe on a small base model puts the framing under direct reading-side test: swapping a boundary's notation while keeping its typographic form changes the model's use of long-range context by a measured zero — the operative cue is the *announcement*, not the sigil — while deleting the announcement itself makes the following prose measurably harder to predict, and the model recovers none of the loss from the long context that contains the evidence. The marker is informative, and the inference that should substitute for it is absent.

The proposal that follows is deliberately conservative: **format augmentation, not format replacement** — present the same structure in multiple notations so the sigil stops being a shortcut — and **long-context slot substitution** as the first serious experiment, because the long-context stage is the one place where the training sequence is the document, and it is currently fed the most heavily marked corpus anyone has built.

## Nothing left to infer

A Markdown heading is a two-token cue that is present at every discourse boundary and absent everywhere else. It is trivially detectable, it never lies, and conversion pipelines now apply it to essentially everything long enough to have structure. That sentence contains the entire argument; the rest of this paper is measurement and consequence.

Consider what a language model has to do with a chapter break when nothing announces it. The prose cools, a thread resolves, a new scene opens with widened aperture — and the model, to predict well, must represent that a boundary of a certain kind has occurred, because everything downstream depends on it: which entities remain live, which tensions carry forward, what register comes next. That representation has to be built from evidence spread over thousands of tokens. It is, in the plainest sense, an inference.

Now put `##` in front of the chapter title. The inference is no longer required. A model that can read the boundary off the surface has no pressure to construct the machinery that would have recognized it — and under any objective that rewards prediction, cues that are this cheap and this reliable are exactly the ones that get taken. This is the central finding of the shortcut-learning literature, which is explicit that what governs shortcut adoption is not how often a feature appears but how *reliably* it predicts, and how easily it can be extracted.[^shortcuts] A structural sigil is the limit case on both axes: maximally available — a literal surface token — and perfectly predictive, because faithful converters place it at every real boundary and nowhere else. The vision-language version of the finding states the cost directly: a contrastive objective "mainly learns the easy-to-detect minimal shared features … that are sufficient to minimize the contrastive objective while suppressing the remaining shared and/or task-relevant information."[^bleeker]

The wager of this paper is that the suppressed inference was load-bearing. A model that must work out where discourse boundaries fall has to maintain a running representation of discourse structure — the very capacity that is conspicuously fragile in frontier models, which sustain an argument across tens of thousands of tokens poorly, and which chunk their own output into signposted sections even when asked not to.[^fingerprint] A model that can read boundaries never has to build that representation, and *thinking in Markdown* is what it looks like when notation and structure have never been encountered as separable things.

Stated as a claim about data rather than models: three properties have to coincide in a training document before the inference is ever demanded — it must be **long**, it must be **coherent**, and it must have **low boundary density**, so that long spans pass with nothing announcing their structure. Text with all three properties is what we call *the missing format*. The name is earned twice over: missing because the corpora no longer contain it in any quantity, and missing because, as we show below, no one has ever recorded the variable on which it differs from everything else.

## Nobody decided this

The flatness of the old web corpora was an accident of tooling. C4 consumed Common Crawl's pre-extracted plain text and then applied a line filter — "We only retained lines that ended in a terminal punctuation mark"[^t5] — which discards every heading, because headings do not end in periods. FineWeb re-extracted from raw HTML with trafilatura, but through a wrapper that leaves the library in its default `txt` output mode, which renders headings as unmarked plain lines while emitting `- ` for list items.[^datatrove] The Pile chose jusText in 2020; DCLM chose resiliparse because it was 8× faster than trafilatura at the same benchmark quality; Nemotron-CC chose jusText because it yielded 28.6% more high-quality tokens.[^extractors] Every one of these decisions was made on yield and benchmark movement. Not one was made, or recorded, as a decision about how much structural notation the training stream would carry.

Then the direction reversed. The frontier of corpus construction moved from stripped web text to PDF conversion by vision-language models, and those models emit structure by design — headings, lists, tables, LaTeX — because faithfulness to the document is the point.[^olmocr] Dolma contained a books subset: 6.0B tokens of Project Gutenberg, 0.2% of its 3.06T total.[^dolma] Dolma 3 contains no books source at all; its second-largest component, 13.6% of the pretraining mix, is olmOCR-converted science PDFs.[^dolma3] The one register that reliably supplied book-length unmarked prose left the mixture in the same generation that the most heavily marked material anyone has produced entered it.

No step in this history was a mistake. Each extractor choice improved the number being watched. Structure-preserving conversion genuinely helps the benchmarks the field runs — a 2025 study found markdown-preserving HTML extraction worth about a point on a 13-benchmark average, and extraction choice alone moves structured-content tasks by ten points.[^aicc] The claim here is not that these results are wrong. It is that the evaluation suite prices exactly one side of a trade. Retrieval, extraction, instruction-following, short-form QA: structured input helps all of them, and none of them is sensitive to whether the model retains the capacity to work out the shape of a long document for itself. A field hill-climbing on a proxy blind to one side of a trade keeps climbing, and every step is correctly reported as progress.

## What the corpora contain

The variable the record lacks is easy to define and cheap to compute. For a tokenized stream, let clean-window survival **S(W)** be the fraction of non-overlapping W-token windows containing no structural markup — no heading sigils, list markers, rules, or blockquote prefixes. Alongside it, measure the distribution of *clean runs*: maximal spans of consecutive tokens with no markup at all. S(W) asks how much of the stream still demands the inference at horizon W; the clean-run tail asks how long the demand is ever sustained.

Measured over sampled streams of ten public corpora (tokenizer `o200k_base`):[^census]

| corpus | median doc (tokens) | any markup | ATX `#` | S(8k) | p99 clean run |
|---|---:|---:|---:|---:|---:|
| C4 | 244 | 4.0% | 0.02% | 0.889 | 3,232 |
| Dolma CC | 426 | 2.9% | 0.02% | 0.823 | 5,790 |
| RedPajama v2 | 555 | 5.7% | 0.08% | 0.706 | 6,548 |
| Nemotron-CC | 328 | 10.7% | 0.07% | 0.568 | 5,025 |
| FineWeb | 388 | 21.5% | 0.12% | 0.603 | 2,374 |
| DCLM | 614 | 27.2% | 0.98% | 0.505 | 4,211 |
| The Pile | 390 | 15.1% | 3.97% | 0.301 | 5,326 |
| **Dolma books** | **77,297** | 33.0% | 0.75% | **0.866** | **109,045** |
| FinePDFs | 1,230 | 56.5% | 7.63% | 0.522 | 4,682 |
| **Dolma 3 olmOCR** | **5,056** | **74.0%** | **33.3%** | **0.153** | 6,024 |

Three facts organize the table.

**The direction of travel is steep.** ATX heading presence runs from 0.12% of FineWeb documents to 33.3% of Dolma 3's olmOCR slice — roughly a 275-fold swing — and any-markup presence from 21.5% to 74.0%. The newest corpus is by far the most marked, and the gradient tracks the pipeline history above: WET-stripped text at one end, vision-model PDF conversion at the other.

**The scarce resource is not unmarked text; it is *long* unmarked text.** Every web corpus, whatever its markup level, has a 99th-percentile clean run between about 2,400 and 6,500 tokens, because a corpus of 244-token documents cannot contain a long clean run no matter how it is notated. Books are different in kind: a median document of 77,297 tokens and a p99 clean run of 109,045 — twenty times the best web corpus, and almost none of the gap is markup. It is document length.

**No source supplies all three properties at once.** The web is unmarked but short. The olmOCR slice is long but carries the lowest survival anyone has built — S(8k) = 0.153. Books are long, coherent, and low-density — and books are the row that fell out of the mixture.

![Scatter of median document length against 99th-percentile clean run for the ten corpora, with books isolated in the upper right](/last-fingerprint/assets/missing-format/fig-missing-quadrant.svg)

*Caption: The supply problem in one plane. Horizontal: median document length. Vertical: the 99th-percentile clean run — the longest markup-free spans a corpus actually delivers. Web corpora (blue) are clean but short; PDF-derived corpora (orange) are long but marked; books (green) sit alone in the quadrant the argument needs, and are the row that left the mixture.*

One register note worth carrying forward: within long-form prose, *sustained argument* — the treatise, the monograph — is an even richer supply of long clean spans than narrative, because its chapters are longer than a novel's. In our ground-truth corpus (below), unbroken paragraph runs have median 25 and 90th percentile 78 in narrative against median 2 in technical material. The register that best supplies the missing format is also the one that has most completely vanished from modern mixtures.

## The cue that cannot be reweighted

The volume objection arrives immediately and deserves a direct answer. In a serialized novel, structural markup is 664 tokens out of 1,024,826 — 0.065%, about 2.7 structural markers (≈5 markup tokens) per 8,192-token training sequence. How can that matter?

Because nothing in the mechanism runs on volume. What shortcut adoption runs on is predictivity and availability,[^shortcuts] and what circuit formation runs on is the *reliability* of a surface pattern, not merely its frequency — the recent induction-head literature finds formation governed by a decision boundary in repetition frequency and repetition reliability jointly, with the models studied "more than twice as sensitive to reliability as … to frequency."[^inductionheads] A cue present at every boundary and absent everywhere else sits at reliability 1.0. Its frequency is beside the point.

Four measurements sharpen this from a plausibility argument into a specific, uncomfortable situation. They come from a 50-document study we ran with ground truth derived from machine-readable sources — Gutenberg text, arXiv LaTeX, RFC XML — stratified by how much structure the author was actually thinking in, from unbroken narrative (R0) and sustained argument (R1) to structure-skeleton technical writing (R3), typeset to real paged media and run through nine current extraction and conversion configurations.[^census]

**First: conversion is accurate, which makes the cue more reliable, not less.** We expected converters to fabricate structure on prose, and pre-registered that test. The result was a clean null. Pooled imposition loss on prose-like registers — the survival cost of what converters add beyond the document's genuine structure — is −0.029 [−0.047, −0.010] at W = 8,192: significantly *negative*. Modern converters slightly under-mark; they are not inventing boundaries. They faithfully mark real ones. A fabricated cue would at least be noisy, and noise is what keeps a shortcut from closing. A faithful cue is a trustworthy one.

**Second: what destroys the long windows is faithful serialization itself.** Writing a novel's real chapter breaks as Markdown — the operation everyone calls benign, because it is faithful — removes 53% of 2,048-token clean windows and 97% of 8,192-token windows. The arithmetic is unforgiving: authored boundaries arrive about every 3,100 tokens of narrative, so almost every 8k window contains one. Under realistic packing the picture is the same: sequences cut from the marked serialization survive at 0.033; the same books flattened have a packing-limited ceiling of 0.888.

**Third: format is now a coordinate with no null level.** Across all 50 documents, the minimum markup density after serialization is 1.25 markup tokens per 10k. **No document has zero.** Between-document *dispersion* passes through conversion essentially untouched — coefficient of variation 1.19 in the authored ground truth, 1.16 and 1.21 through the two strongest converters — so conversion is not homogenizing the spread. What it removes is one specific value: *absent*. This is why the problem cannot be reached by mixture weighting, however sophisticated. Reweighting moves mass between levels of a variable. It cannot restore a level that no longer exists in the support.

**Fourth — and this is the fact that makes the obvious fix wrong: the boundary survives its own notation.** Strip the `#` but leave the chapter title as a short standalone line between blank lines, and a detector sensitive to typographic structure rather than Markdown syntax — a line that C4's own filter would have discarded[^detector] — finds the boundaries in essentially the identical places:

| register | W | S, Markdown syntax | S, structural detector on flat text | difference |
|---|---:|---:|---:|---:|
| narrative | 1,024 | 0.701 | 0.699 | −0.001 |
| narrative | 2,048 | 0.472 | 0.462 | −0.010 |
| narrative | 4,096 | 0.202 | 0.198 | −0.004 |
| narrative | 8,192 | 0.026 | 0.026 | **0.000** |
| argument | 8,192 | 0.140 | 0.143 | +0.003 |

Flattening the notation, on its own, changes almost nothing about where boundaries are detectable, because the typographic form — short bare line, whitespace around it — is itself a perfectly good cue. This zero is a result, and it dictates the shape of any intervention: what has to be attacked is not the sigil's *presence* but its *reliability*. Removing one notation while leaving another equally reliable one in place merely swaps shortcuts.

## The one stage where the sequence is the document

Where in training would any of this bind? The answer has a structure that is easy to miss and, once seen, hard to unsee.

**Over 95% of pre-training tokens are consumed at sequence lengths of 8,192 or less.** Llama 3 ran 15.6T of its ~16.4T tokens at 8,192, with a long-context extension of roughly 800B — 4.9% of the total.[^llama3] DeepSeek-V3 pre-trained 14.8T tokens at 4K and extended context with about 126B tokens of YaRN phases — under one percent.[^deepseek] Qwen3 ran roughly 35T of its 36T at 4,096.[^qwen3] Olmo 3 7B ran 5.93T of pretraining and 100B of midtraining at 8,192, then 50B tokens — 0.8% — at 65,536.[^olmo3]

At the main stage, a training sequence is not a document. It is a packed window: several short documents concatenated, or an arbitrary 8k slice from the middle of a long one, frequently straddling document boundaries. And at this stage the attention mask typically does not even separate neighbours — Llama 3 reports that intra-document masking "had limited impact" in standard pre-training; DeepSeek-V3 packs documents "but do[es] not incorporate cross-sample attention masking."[^masking]

There is a statistical-mechanics way to say what that stage can and cannot teach. Mutual information between tokens of natural text decays as a power law, and the power law is *caused by* hierarchical document structure — correlations flow from the high organizational levels down into words.[^powerlaw] Transformers are the preferred architecture in part because they capture slow power-law dependence where recurrent models cannot. Set against that: authored boundaries arriving every ~3,100 tokens with markup announcing them, sequences capped at 8,192, and every web corpus's clean runs exhausted by 6,500 — the gradient is shown a truncated rendition of the very law the architecture was chosen to fit. Planning-horizon work points the same way: models *pre-cache* features for future tokens where the data has genuine future structure, and pre-caching rises with scale[^precache] — but the horizon over which pre-caching can pay is capped by how far ahead the data ever rewards looking.

The long-context stage inverts every one of those properties. Sequence length exceeds the median book. Intra-document masking is switched on — Olmo 3's released training scripts run the main stages at `DEFAULT_SEQUENCE_LENGTH = 8192` with no masking flag, and the long-context script at 65,536 with `generate_doc_lengths=True,  # enables intra-document masking`.[^olmo3scripts] At this stage, and only at this stage, **the training sequence is the document**: one work, entire, in reading order, with attention confined to it. It is the only point in the entire training pipeline where a gradient can flow through the experience of tracking one document's structure across tens of thousands of tokens.

And that slot is currently filled with the most heavily marked material available. Olmo 3's long-context pool is 639B tokens, of which effectively all is olmOCR-converted science PDFs[^olmo3] — the bottom row of the census table, S(8k) = 0.153, the lowest clean-window survival of any corpus we measured. The one stage whose mechanics are exactly right for training the missing inference is being fed the corpus in which the inference is least often required.

![Stacked horizontal bars of pre-training token budgets by stage for four model families, showing the long-context stage as a sliver](/last-fingerprint/assets/missing-format/fig-token-slot.svg)

*Caption: Where the tokens go. For each family, the main pre-training stage (blue, sequence length ≤ 8,192, no intra-document masking) against the long-context stage (orange, 32k–128k, masking on). The stage with the right mechanics for long-range structure — where the sequence is the document — is a sliver, which is precisely what makes it cheap to fill deliberately.*

The economics follow. Nobody needs to touch 15T tokens of web text. The intervention the analysis points to lives in a slot that is under 1% of the budget in the very training runs that already have the infrastructure configured for it.

## A first measurement

The mechanism makes a prediction that can be tested in an afternoon on one laptop-class machine, and we ran it.

If a boundary sigil functions as a *conditional-independence license* — an announcement that what follows depends less on what came before — then on identical prose, a trained model should extract **less** information from long-range context when the boundary is marked than when it is not. The quantity to measure is the long-context information gain of recent data-curation work:[^longfilter] the improvement in next-token prediction from seeing a long context over a short one.

**Design.** We take the 21 narrative and sustained-argument works from the ground-truth corpus and serialize each from the same block structure in three arms: *marked* (genuine structure written as Markdown), *flat* (identical words; heading text kept as a bare line; notation dropped), and — for the works whose chapter heads are pure labels like "CHAPTER IV." or "II", where the line can be removed without removing content — *silent* (the label line deleted; the boundary reduced to whitespace). Marked versus flat isolates the **sigil**; flat versus silent isolates the **announcement**. We select target passages of ordinary interior prose — never headings — stratified by distance from the previous authored boundary, and for each target in each arm compute the per-byte negative log-likelihood of the same passage under a short context (768 tokens) and a long context (7,936 tokens) with a small open-weights base model (Qwen3-0.6B-Base, float32 scoring), forming the information gain IG = NLL(short) − NLL(long) alongside LongFilter's confidence-weighted surrogate.[^longfilter] Positive IG differences mean more measured use of long-range context. Anchors whose visible contexts contain no boundary in either compared arm have byte-identical inputs and serve as a built-in null: their contrast must be exactly zero.

**Results.** 169 aligned targets across all 21 works passed the filters for the notation contrast (median target 1,551 bytes of interior prose), and 88 targets across the 12 works with pure-label chapter heads for the announcement contrast; all intervals are 95% cluster bootstraps by work. Three numbers carry the outcome.

*Swapping the notation does nothing.* IG(flat) − IG(marked) = +0.04 millinats per byte [−0.08, +0.16], against a mean information gain of 24.0 mnats/byte — the sigil's effect on measured long-range information use is bounded within about ±0.7% of the quantity itself — and its effect on absolute predictability is an equally tight zero (−0.10 [−0.26, +0.06]). The eleven no-boundary anchors return contrasts of exactly zero, as the pipeline requires. This is the behavioural twin of the survival-equivalence table: to a trained reader, `## CHAPTER IV.` and a bare `CHAPTER IV.` line are the same event. The cue is the typographic announcement, not its notation.

*Removing the announcement measurably hurts prediction.* Deleting the label line makes the prose that follows harder to predict: per-byte NLL under the long context rises by +0.47 mnats/byte [+0.16, +0.77], with the largest rise immediately after the boundary (+0.62 [+0.17, +1.14]). The announcement carries real information about what follows.

*And none of the loss is recovered from long-range context.* Measured information gain is unchanged when the announcement is removed: IG(silent) − IG(flat) = −0.02 [−0.45, +0.39]. The model becomes more uncertain under the short context and the long context roughly equally, rather than reaching farther back to reconstruct what the vanished label would have told it — although the 7,900 visible tokens contain the entire close of the preceding chapter, which is exactly the evidence the inference would use.

![Dot-and-interval chart of the four probe contrasts: notation swap and announcement removal, on information gain and on prediction difficulty](/last-fingerprint/assets/missing-format/fig-probe.svg)

*Caption: The probe's four contrasts, mean and 95% cluster-bootstrap interval, in millinats per byte. Top pair: effect on long-range information gain — zero for the notation swap (tightly bounded) and zero for announcement removal. Bottom pair: effect on prediction difficulty under the long context — zero for the notation swap, positive for announcement removal, the only interval excluding zero. The marker is informative; the reader does not replace it from range.*

Read together: **the marker is informative, and the inference that should substitute for it is absent.** A reader shaped by the marked distribution treats the announcement as a primitive. When it is present, prediction is easier in a way that no change of notation perturbs; when it is deleted, the reader does not go and construct the boundary from long-range evidence — it simply predicts worse. That is what an inference removed from the curriculum looks like from the reading side. The probe also banked a null it was designed to be able to return: the *suppression* form of the mechanism — that a visible marker actively reduces use of far context on identical announced text — is not detected at this scale, and the interval bounds any such effect below roughly two percent of measured information gain. And there is a corollary for data curation: at this reader scale, the information-gain scores now used to select long-context training data[^selection] are insensitive to serialization — such pipelines will neither punish nor reward a corpus for its format, which is another way of saying the variable stays invisible to them too.

The probe's limits should be stated as plainly as its results. It is one 0.6-billion-parameter reader over 21 works; a reading-side probe cannot distinguish "the training pressure never existed" from "this reader is too small to have formed the machinery"; the announcement arm exists only where chapter heads are contentless labels; and nothing here shows that training on the missing format changes a capability. Those questions belong to the program's later steps, which is what the program is for.

## Format augmentation

The intervention follows from the fourth measurement above. Since flattening alone just promotes the typographic form into the new reliable cue, the target must be reliability itself — and reliability is destroyed by *diversity*, not by deletion.

**Present the same structure in multiple notations.** The same work appears with its boundaries as Markdown in one copy, as bare typographic lines in another, as whitespace alone in a third — same words, same arrangement, different surface. Under that regime the sigil stops predicting: a cue that is sometimes present and sometimes absent at the same underlying event cannot function as a shortcut, and the pressure to represent the *event* — the boundary itself, as a thing separate from any way of writing it down — returns. Because shortcut suppression is a reliability phenomenon, this attacks it at exactly the point where it operates.

Three properties recommend the augmentation form of the intervention over any purist alternative. It destroys nothing: the marked copies are still there, and structural competence — which the benchmarks reward because users genuinely want it — is still taught. Its content cost is zero: every copy is the same author's words. And it is *partial by design*: rewriting-based augmentation lines find that diversifying a modest fraction of occurrences captures most of the robustness gain,[^augmentation] so nothing requires reprocessing a corpus wholesale. An objective-side alternative exists — discourse-coherence objectives and post-training for hierarchical organization[^objective] — but the data-side version requires no new loss, no new labels, and no commitment about what the representation should look like; it only reinstates the conditions under which one would have to form.

**What we are betting emerges is a factorization**: a discourse channel that represents arrangement, and a surface channel that renders it, able to vary independently. The inferred thing — *a boundary is here* — is invariant across the forms that signal it: a `##`, a bare title line, a blank gap, "The next morning." A model forced to recognize it under all of them has been given, in effect, a change-of-notation group over documents, and what survives a group action is structure. A model with the factorization could plan an arrangement without committing to a rendering — think in prose and speak in Markdown — rather than thinking in Markdown because plan and notation were never separable. Interpretability work already finds partial notation-invariant structure with stable cross-notation transfer gaps,[^invariance] which is what "the factorization exists but nothing has forced it to completion" looks like from the inside.

The supply side is better than the folk number suggests. "Books are 0.2% of Dolma" describes a domain label, not a format ceiling. The addressable pool is everything whose structure was imposed rather than authored, plus everything markup-naive by construction: transcribed speech — podcasts, lectures, oral histories, proceedings — which is markup-free by construction, genuinely long, and growing faster than any text source; judicial opinions; parliamentary records; long-form journalism archives; monographs; correspondence. Public-domain book collections at the hundreds-of-billions-of-tokens scale already exist as OCR plain text.[^supply] And the arithmetic of the slot is favourable: we built a demonstration pack of 27 whole public-domain works — 2.31M tokens, one record per book, zero structural markup tokens in any record, validated by refusing to write if a single one appears — and its median work is 73,724 tokens: none fits a 32k context, 23 of 27 fit 128k, all fit 256k.[^pack] Under conventional 8k packing those same books shatter into 297 fragments averaging 4.1 markers each, no fragment knowing its position in the arc. Whole-document training on this material is only *possible* at frontier context lengths. It has just become possible.

## The program

Ordered by cost, each step informative on its own and none requiring the previous one to have gone the right way.

**1 · The information-gain probe.** Run above; replicate it with larger base models, more registers, transcript corpora.

**2 · Report survival notation-invariantly, over packed streams.** Not one curve but three: S over Markdown syntax; S over structure (the typographic detector); S over sequences (adding document boundaries). The quantity of interest is the syntax–structure gap, which we measure at ≈ 0 on prose — the zero that forces the reliability framing — and the packing ceiling, which bounds what any corpus can deliver at a given context length.

**3 · Build the paired corpus.** The same works in aligned multi-notation serializations, at scale, from sources whose structure is knowable: the augmentation artifact itself.

**4 · Measure the scaled supply.** S-statistics and clean-run distributions over transcript and public-domain book corpora. If speech is confirmed long and low-density at 32k+, it is the only *growing* source of the missing format, and nobody currently weights for this property.

**5 · The long-context slot substitution.** The first serious experiment, because the infrastructure already exists and the slot is small. In an Olmo-3-style long-context stage — sequence is the document, intra-document masking on — substitute a fraction of the olmOCR PDF pool with whole flattened and paired books at matched token count, and evaluate on long-context suites *plus* long-form generation likelihood. Existing long-context data work has already found that naive length-based book upsampling underperforms domain-balanced selection, and one team reported that upsampling books and code repositories did not move their long-context benchmarks[^fu] — but no experiment in this line has ever controlled for markup, which on the present analysis is the live variable. If controlling for it changes nothing, the locus argument fails, cheaply.

**6 · The matched pre-training run.** Three arms on identical content — markup-serialized, flattened, paired — matched token budgets. No published run isolates structural notation while holding words fixed. Steps 1–5 exist to make this one worth paying for; step 5 can be run for a fraction of one percent of a modern budget.

**What would count against the thesis.** Some of it already landed, and is printed above: the probe found no inference-time suppression — a visible marker does not measurably reduce a small reader's use of far context — so the version of the mechanism that survives is the training-time one, carried by the deficit finding (information lost with the announcement, and not reconstructed from range). Replications that extend the null — larger readers still showing no deficit, or recovering unannounced boundaries from context just fine — would cut against that remainder directly. The strongest standing counter-evidence is a base-model result from the model-idiosyncrasy literature: classifiers over markdown features identify *instruction-tuned* models at 77.7% against a 25% chance rate, but *base* models at only 38.5%[^idiosyncrasies] — base models mostly emit plain prose, which suggests much of the visible chunking is post-training policy rather than pre-training inventory. The honest reading is that post-training decides how much of the tendency is *expressed*, while pre-training decides what there is to express — the same literature's suppression experiments find the latent tendency present before RLHF — but this is exactly the kind of claim the three-arm run exists to settle, and the 38.5% number is a fact any version of this thesis has to carry visibly. Relatedly, probing work finds models can partially reconstruct document boundaries from flattened text,[^rightreset] so the removed inference is degraded, not absent; the claim that survives is about how much pressure exists to make it good. And if the paired arm behaves exactly like the flat arm in step 6, then reliability was not the operative variable and the factorization story collapses to a notation preference. Each of these is empirical, near-term, and cheap relative to the claims at stake.

## Coda: record it, then build it

Nobody decided that the pre-training stream should stop demanding structural inference. It fell out of a line filter here, a default output mode there, a books subset quietly dropped, a long-context slot filled with whatever was longest — each step locally correct, each improving the number being watched, the cost landing on a dimension nobody had instrumented.

The smallest useful response is instrumentation. A data card that records extractor identity, conversion flags, and OCR path — and one number, S(W) at the context lengths the model will actually train at — makes the variable visible retrospectively and the question askable for every future corpus. Pipelines already compute everything required; reporting it costs nothing.

The ambitious response is to build the missing format on purpose: assemble the long, coherent, low-boundary-density register at scale, serialize it in deliberately unreliable notation, and put it into the one training stage where a document is experienced whole. Every component of that sentence exists today — the sources, the serializers, the masking configuration, the slot. What does not yet exist is a model that was ever made to work out the shape of a long thing for itself. It would be worth finding out what that model can do.

[^shortcuts]: Geirhos et al., *Shortcut Learning in Deep Neural Networks*, Nature Machine Intelligence 2:665–673 (2020), arXiv:2004.07780 — "picking any feature that is sufficient to reliably discriminate on a given dataset." Hermann, Mobahi, Fel and Mozer, *On the Foundations of Shortcut Learning*, ICLR 2024, arXiv:2310.16228: feature adoption is governed by *predictivity* — "how reliably a feature indicates training-set labels" — and *availability* — "how easily the feature can be extracted from inputs." See also Du et al., *Shortcut Learning of Large Language Models in Natural Language Understanding*, CACM, arXiv:2208.11857.

[^bleeker]: Bleeker, Hendriksen, Yates and de Rijke, *Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning*, arXiv:2402.17510, §4.2.

[^fingerprint]: On markdown-structured output as a default register and its prose-level residue: Freeburg, *The Last Fingerprint: How Markdown Training Shapes LLM Prose*, arXiv:2603.27006.

[^t5]: Raffel et al., *Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer*, arXiv:1910.10683, §2.2. C4 consumed Common Crawl's pre-extracted WET text and additionally discarded any page containing a curly bracket.

[^datatrove]: HuggingFace `datatrove`, `src/datatrove/pipeline/extractors/trafilatura.py`: the wrapper passes `favor_precision` and sets no output format, leaving trafilatura in default `txt` mode; trafilatura's serializer applies heading prefixes only when formatting output is requested, while list markers are emitted unconditionally. FineWeb's released pipeline instantiates exactly this extractor over WARC input (arXiv:2406.17557).

[^extractors]: The Pile: arXiv:2101.00027, App. C.1 (jusText over WARC, chosen by inspection). DCLM: arXiv:2406.11794 (resiliparse; +2.5 points over WET extraction; 8× faster than trafilatura). Nemotron-CC: arXiv:2412.02595 (jusText; +28.6% high-quality tokens versus trafilatura). RedPajama v2: WET via CCNet (arXiv:2411.12372).

[^olmocr]: olmOCR: arXiv:2502.18443 — "linearized plain text … preserving structured content like sections, tables, lists, equations"; the released toolkit converts documents to Markdown. FinePDFs (HuggingFaceFW, 2025): ~3T tokens from 475M PDF documents via Docling and an olmOCR-family VLM for scanned input.

[^dolma]: Soldaini et al., *Dolma*, arXiv:2402.00159, Table 1: Project Gutenberg, 6.0B of 3,059B Llama-tokenizer tokens ≈ 0.196%.

[^dolma3]: Dolma 3 (`allenai/dolma3_pool` and `dolma3_mix-6T-1025`): six sources, none books; olmOCR science PDFs are 972B tokens of the 9.3T pool and 13.57% (805B) of the 6T mix. The books drop predates Dolma 3 — OLMo 2's mix already carried none.

[^aicc]: Ma, Qiu, Xu et al., *AICC: Parse HTML Finer, Make Models Better*, arXiv:2511.16397 (+1.08 pp over trafilatura-extracted text across 13 benchmarks). Extractor sensitivity of structured tasks: Li et al., *Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining*, arXiv:2602.19548. Document-level organization moving mid-training results: Tao et al., arXiv:2607.28109. Rendered-document pretraining as the opposite remedy from the same premise: Zhang et al., arXiv:2607.09657.

[^census]: Corpus census, converter study, ground-truth corpus, serializers and the S(W)/clean-run instruments are in the repository accompanying this paper, including the pre-registered converter analysis whose imposition-loss decision rule returned its null (`NULL_NO_IMPOSITION`), the amendment history, and per-document data. Three census rows were independently re-verified for this paper by streaming fresh ~8M-token samples (August 2026): C4 3.8% any-markup / S(8k) 0.870 / p99 3,081; FineWeb 20.7% / 0.613 / 2,248; DCLM 27.6% / 0.558 / 4,290 — consistent on every quantity, with ATX rates at these sample sizes carrying small-count noise (C4 0.07% vs 0.02%).

[^detector]: Detector specification: a line between blank lines, at most 100 characters, not ending in terminal punctuation in C4's sense — period, exclamation, question mark, or closing quotation mark, plus continuation punctuation (comma, colon, semicolon, dash) — with one exception: a trailing period after an uppercase letter is a label period ("CHAPTER IV.", "INTRODUCTION.") and does not disqualify. Or any line already matching Markdown syntax.

[^llama3]: Llama Team, *The Llama 3 Herd of Models*, arXiv:2407.21783: 15.6T tokens at 8,192 (§1, Table 4); long-context continued pre-training "approximately 800B" across six stages to 128K (§3.4.2). The ~16.4T total and 95.1% share are derived from those two figures.

[^deepseek]: DeepSeek-AI, arXiv:2412.19437: "the maximum sequence length to 4K during pre-training … 14.8T tokens" (§4.2); two YaRN phases of 1,000 steps (32K × 1920, then 128K × 480), ≈126B tokens by computation (§4.3).

[^qwen3]: Qwen3, arXiv:2505.09388, §3.2: over 30T tokens plus ~5T higher-quality tokens at sequence length 4,096, then "hundreds of billions" at 32,768.

[^olmo3]: Ai2, *Olmo 3*, arXiv:2512.13961: 8,192 through pretraining and midtraining, long-context extension at 65,536 with intra-document masking (§3.6, Table 35: 5.93T + 100B + 50B for the 7B); long-context pool total 639B tokens (Table 11), backbone "scientific PDFs scraped from the web and processed by olmOCR" (§3.6.1).

[^masking]: Llama 3, arXiv:2407.21783, §3.2: document-separating attention masking had "limited impact" in standard pre-training but was "important in continued pre-training on very long sequences." DeepSeek-V3, arXiv:2412.19437, §4.1: "we implement the document packing method for data integrity but do not incorporate cross-sample attention masking during training."

[^powerlaw]: Alvarez-Lacalle, Dorow, Eckmann and Moses, *Hierarchical structures induce long-range dynamical correlations in written texts*, PNAS 103:7956 (2006); Altmann, Cristadoro and Degli Esposti, *On the origin of long-range correlations in texts*, PNAS 109:11582 (2012) — correlations "flow from highly structured linguistic levels down to the building blocks of a text." Power-law mutual-information decay in language: Sarkar and Howard, arXiv:1912.07506; Mikhaylovskiy and Churilov, arXiv:2305.06615. Architecture: Shen, *Mutual Information Scaling and Expressive Power of Sequence Models*, arXiv:1905.04271 — recurrent models' MI "decays exponentially in temporal distance" while "Transformers can capture long-range mutual information more efficiently"; cf. Lin and Tegmark, arXiv:1606.06737.

[^precache]: Wu, Morris and Levine, *Do language models plan ahead for future tokens?*, COLM 2024, arXiv:2404.00859 — clear pre-caching in synthetic settings with genuine future structure, breadcrumbs-dominant in natural language, "pre-caching increases with model scale."

[^olmo3scripts]: `allenai/OLMo-core`, `src/scripts/official/OLMo3/`: `OLMo-3-1025-7B-pretrain-{1,2}.py` and `-midtrain.py` set `DEFAULT_SEQUENCE_LENGTH = 8192` with no masking flag; `OLMo-3-1025-7B-long-context.py` sets 65,536 and `generate_doc_lengths=True,  # enables intra-document masking`.

[^longfilter]: Deng et al., *Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data*, arXiv:2510.25804, ICLR 2026 — long/short-context information gain as a data-selection score (a confidence-weighted surrogate for the KL between predictions under long and short context), finding that "a significant portion of readily available long-text data lacks meaningful long-distance dependencies." The coherent-packing precedent is Staniszewski et al., *Structured Packing in LLM Training Improves Long Context Utilization*, arXiv:2312.17296, AAAI 2025, motivated by "long-range dependencies being rare in training data."

[^selection]: Attention-based long-context data selection: Wu et al., *LongAttn*, arXiv:2502.16860; Chen et al., *LADM*, ACL 2025, arXiv:2503.02502.

[^inductionheads]: Aoyama, Wilcox and Schneider, *Predicting the Emergence of Induction Heads in Language Model Pretraining*, ICML 2026, arXiv:2511.16893: emergence follows N ∝ B^0.63 · C^0.38 in tokens (r = .986); formation is governed jointly by bigram repetition frequency and reliability P(B | A,B,…,A), with sensitivity to reliability more than twice that to frequency.

[^augmentation]: Style/format-diverse rephrasing at pretraining scale: Maini et al., *Rephrasing the Web*, arXiv:2401.16380; DatologyAI, *BeyondWeb*, arXiv:2508.10975. Fact-format diversification for extractable knowledge: Allen-Zhu and Li, *Physics of Language Models 3.1*, arXiv:2309.14316; Zhu et al., arXiv:2503.03705. Partial-coverage sufficiency: multi-format training recovering most robustness from ~30% coverage, arXiv:2606.11643.

[^invariance]: Format-agnostic reasoning subspaces with persistent cross-notation transfer gaps: arXiv:2605.09496. Residual-stream traces of paragraph-scale planning at `\n\n` boundaries: arXiv:2511.00180.

[^objective]: Inter-sentence discourse objectives: Iter et al., *CONPONO*, arXiv:2005.10389. Post-training toward hierarchical discourse organization in long-form generation: *Align to Structure*, arXiv:2504.03622.

[^supply]: Institutional Books 1.0: 242B tokens of digitized library books (arXiv:2506.08300). Common Pile v0.1 (arXiv:2506.05209) carries four public-domain book sources as OCR plain text — the missing format's register, already assembled, already unmarked.

[^pack]: `pack/longform-flat.jsonl` in the accompanying repository: 27 works, 2,313,460 `o200k_base` tokens, one record per book, `core_markup_tokens = 0` in every record; flattening spec keeps heading words and drops notation.

[^fu]: Fu et al., *Data Engineering for Scaling Language Models to 128K Context*, arXiv:2402.10171 — naive per-domain length upsampling is suboptimal; balance domains, upsample long within domain. SmolLM3 (HuggingFace, 2025): "upsampling specific long context data such as code repositories, books, and long web pages … didn't further boost performance on RULER and HELMET benchmarks." Neither controlled for markup.

[^idiosyncrasies]: Sun et al., *Idiosyncrasies in Large Language Models*, ICML 2025, arXiv:2502.12150, Table 9: classification over markdown features alone, four-way among model families, 25% chance — 77.7% on instruct models, 38.5% on base models. The same paper's base-vs-instruct comparison finds the latent tendency present before post-training.

[^rightreset]: Prefix-removal probing recovers 47.7% of record boundaries from flattened, separator-stripped text: arXiv:2608.04330.

---

## Citation

Freeburg, E. M. (2026). *The Missing Format*. https://ericfreeburg.com/last-fingerprint/missing-format/

```bibtex
@misc{freeburg2026missingformat,
  author = {Freeburg, E. M.},
  orcid  = {0009-0002-2140-9131},
  title  = {The Missing Format},
  year   = {2026},
  month  = {8},
  url    = {https://ericfreeburg.com/last-fingerprint/missing-format/},
  note   = {Accessed via Markdown alternate}
}
```