The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora

Abstract↑

How a document's arrangement is written down, its notation, is a training variable that no dataset card records. The field has established that text-extraction choices change model behaviour, and has never once measured the notation of what those choices put into the corpus. We define clean-window survival, a deterministic count of how much of a stream still demands the boundary inference, and measure notation on three fronts. What corpora carry: a census of thirteen public corpora, where survival falls to 0.153 in a vision-converted PDF slice against 0.889 in C4; the scarce resource is not unmarked text but long unmarked text; a pre-registered supply test finds what remains institutional, not consumer. Our own pre-registered prediction failed: converters do not fabricate structure on prose, and that null forced the reliability mechanism that survives it. What readers use: across five base models spanning 0.6B to 8.2B and two pipelines, deleting a structural announcement makes the following prose measurably harder to predict, while swapping its notation moves nothing. That zero does not make notation unimportant; it relocates the variable: the operative cue is the announcement, not the sigil. What writers impose: a bounded null. Base models do not impose the marked register above the authored baseline, and handed prose with every announcement deleted they do not put one back, at a rate indistinguishable from zero against an authored reference of zero. We ship the format those measurements imply: the pure frame, paragraphs in authored order, every announcement deleted into a reversible sidecar, mixed against the marked copy over announcement presence rather than notation. Choose format operators by the capability they train, not by the fidelity they preserve, and record extractor identity and survival on data cards.

1. First paragraph to last↑

Take Moby-Dick from its first paragraph to its last and delete everything that is not the book: the chapter numbers, the chapter titles, the running heads, the transcriber's furniture. On the Project Gutenberg text that is 143 lines (141 chapter heads and two thematic breaks) and 1,442 tokens. What remains is 283,267 tokens of Melville in authored order, and the deletion is 0.51% of the file. A reader handed the result would not notice that anything had happened to it. A model handed the result is handed something it has almost never seen: a continuous stretch of 283,267 tokens in which nothing on the surface announces where one part of the book ends and the next begins.

That ratio is the argument of this paper stated once, and what the deletion does is not subtle. Packed at the 8,192-token window most pre-training still uses, the book arrives as 35 fragments carrying 4.1 announcements each, and every one of its 34 full windows contains at least one. After the deletion all 34 contain none, clean-window survival against ground truth goes from 0.000 to 1.000 at both 8,192 and 32,768 tokens, and the longest span containing no announcement goes from 10,595 tokens to the entire book.

The middle case is the one that reorganized this paper. Strip the Markdown but leave CHAPTER 1. Loomings. standing on its own line, and the census statistic the field would report for that file reads 1.000: perfectly clean, indistinguishable from the deleted version. Against ground truth the same file reads 0.000, exactly the marked book, because every boundary is still there, in the same place, on its own line. Faithful flattening moves the notation and leaves the announcement, and the announcement is what the reader was using.

The deletion is not free, and its price should be quoted before its benefit. Of those 1,442 tokens, 695 are label apparatus that carries no content and 747 are Melville's own title words, deleted with the rest. Every deleted line goes into a sidecar, the transform is reversible, and in the mixture this paper recommends the marked copy coexists with the pure one, so the corpus loses nothing and only the pure copy does. How big the deletion is depends on how a book announces itself: across the 27-work pack of §9 it is 0.71% of tokens, 0.25% across the fifteen narratives, and 2.6% in the worst single case; had every work announced itself with bare labels rather than titles, the whole operator would have cost 0.048% of the pack. It is under one percent almost everywhere, and the size is never the point. What is left is the point.

It is also, we think, the book as a person actually holds it. Nobody remembers the fifth chapter of Moby-Dick; they remember a part of the story in the context of the whole story. Readers plainly do see the breaks, and we are not claiming otherwise. The claim is about what comprehension and memory run on, and the reading literature is consistent in treating segmentation as something a reader constructs continuously out of the content rather than something the page supplies (Zacks et al. 2007; Zwaan & Radvansky 1998).^[1][2] We take that as motivation and not as evidence.

The reason nobody has run the corresponding experiment is not that it is expensive. It is that the variable is not in the record. The field has established that text-extraction choices change model behaviour, and has never once measured the notation of what those choices put into the corpus. No dataset card for C4, The Pile, RedPajama, Dolma, FineWeb, DCLM, Nemotron-CC, FinePDFs or the olmOCR corpora reports how much structural markup its text carries, and no statistic of that family appears anywhere in the literature. The most ambitious corpus audit of 2026 makes the point better than an assertion could: propella-1 annotates three billion documents across FineWeb-2, FinePDFs, HPLT 3.0 and Nemotron-CC on eighteen properties in six categories, and not one of the six categories is notational.^[3]

The neighbouring literature runs the other way, and we concede it completely before doing anything else. Structure-preserving extraction makes models better on every axis anyone has measured. A model-based HTML parser is worth 1.08 percentage points over trafilatura-extracted text across thirteen benchmarks (AICC);^[4] extractor choice alone is worth up to ten percentage points on WikiTQ (Li et al. 2026).^[5] We do not contest a single one of those results. We change the object and the variable. Those studies measure the consequence of an extraction choice on a benchmark, and nobody has measured the notation of the corpus that the choice produces; their variable is markup richness, which is a fidelity property, while ours is markup reliability, which is a distributional one. AICC is the result our mechanism explains rather than the rival it beats. Preserving markup helps on every task that reads structure, and the question here is what it costs at the one stage of training where structure has to be inferred.

This paper measures the variable, on three fronts, and then ships the format the measurements imply. Front I asks what corpora carry, and reports the first census of structural markup in public pre-training corpora: ten published rows with their provenance stated honestly, three new rows measured for this paper against a pre-registered supply rule, and a clean-window survival that runs from 0.889 in C4 to 0.153 in the olmOCR slice of Dolma 3, whose documents carry markup at 74.0%. Front II asks what readers use, and reports a paired-serialization probe across model scales. Front III asks what writers impose. Between them sits a pre-registered converter study that returned a null against our own prediction, and the reason the reliability framing is the one that survived. Our own prior asserted the corpus thesis from the model side without measuring it (Freeburg 2026);^[6] this paper is the measurement.

Four things are new here.

• An instrument and a census. Clean-window survival S(W) and the clean-run distribution: a deterministic surface count, no model, no annotation, no seed, computed over thirteen public corpora. Anyone can recompute it in an afternoon and get the same number, which is also why the documentation ask at the end of the paper is realistic.
• Three fronts of measurement on one variable, with the honest nulls led: a pre-registered converter study whose prediction was ours and failed, and a notation swap that moves nothing on the reading side because the operative cue is the announcement rather than the sigil.
• A format, specified, validated and shipped. The pure frame: paragraphs in authored order, every structural announcement deleted, every deletion recorded in a reversible sidecar, and a validator that passes a file only if the detector fires zero times over it.
• A data card for notation, which is mostly a request to finish a field that the field's own flagship PDF corpus already carries.

2. The variable↑

A training corpus is not a collection of documents. It is a collection of images of documents under format operators: extraction, conversion, flattening, OCR, transcription convention, synthetic rephrasing. Write a document as a pair, a content c and the operator f that produced the version stored on disk. The training distribution is over the images f(c) alone. Domain labels are metadata about c, an object the gradient never touches, so two corpora with identical domain marginals and different operator distributions are different training distributions, and nothing on a standard dataset card would show it.^[7]

Call the missing quantity notation: the distribution, across a corpus, of how documents' arrangement is written down. It is not a proposal for tidier bookkeeping. It names a decision that is currently made by accident, at no apparent cost, by people optimizing something else.

Notation matters here for one specific reason. A Markdown heading is a two-token cue that is present at every discourse boundary and, in the material this paper is about, close to absent everywhere else. It is trivially detectable, faithful converters place it where the author put a boundary, and conversion pipelines now attach it to essentially everything long enough to have structure. Where the cue is present, a boundary does not have to be inferred; it can be read. How reliable the cue actually is turns out to be measurable, and it is measured in §5.5, where it holds in narrative and comes apart elsewhere. Consider what a model has to do with a chapter break when nothing announces it: the prose cools, a thread resolves, a new scene opens with a widened aperture, and the model, to predict well, must represent that a boundary of a certain kind has occurred, because everything downstream depends on it. That representation has to be assembled from evidence spread over thousands of tokens. It is, in the plainest sense, an inference. Put ## in front of the chapter title and the inference is no longer required.

The shortcut-learning literature is explicit that what governs whether a model takes a feature is not how often the feature appears but how reliably it predicts and how easily it can be extracted (Geirhos et al. 2020; Hermann et al. 2024),^[8][9] and the induction-head scaling work puts a number on the same trade: circuit formation is governed jointly by repetition frequency and repetition reliability, with sensitivity to reliability more than twice the sensitivity to frequency.^[10] We cite version three of Aoyama et al., arXiv:2511.16893, Eq. 7 and §7; the v1 posting, which ar5iv still renders, gives different exponents. A structural sigil (the markup character itself: the ## before a chapter title, the bullet, the horizontal rule) is the limit case on both axes: maximally available, being a literal surface token, and near-perfectly predictive, because faithful converters place it at every real boundary and, in narrative, almost nowhere else; §5.5 puts that precision at 0.967 once the documented typographic furniture is removed from the ground truth. Reliability, not volume, is what a shortcut runs on. That is the whole reason a variable worth 0.065% of the tokens pooled across the fifteen serialized narrative works of §5.3 can be worth measuring.

Two properties make the situation hard to reach from where the field stands. The first is that it is unreachable. Every method for controlling what a model learns from its data operates over documents as they already sit in the corpus: mixture weights decide how often a document is sampled, filters decide which are kept, and both take the corpus as given. The operation at issue happened before that.

Reweighting operates over documents as built, and therefore cannot reach upstream of the operator that built them. Only re-extraction can, and nobody re-extracts.

One operation does survive downstream, and §9 is built on it: deletion. An announcement still present in the stream can be removed and recorded; notation already destroyed cannot be recovered without the source.

The second is that it is unpriced. The title stays in the corpus, the domain weight is unchanged, the token count is unchanged, and no dataset card records what a document passed through on the way in. Every benchmark in current use rewards the arrangement that structure-preserving conversion produces, so published measurements report structural normalization as an improvement, and under the instruments in use they are right. A field hill-climbing on a proxy that is blind to one side of a trade keeps climbing, and every step is correctly reported as progress. The burden that places on this paper is worth stating plainly: if no instrument can price the other side, the claim is unfalsifiable and should be discarded. Section 4 proposes the instrument.

One distinction has to be fixed before any of this can be measured, because the field currently collapses it. Serialization register is the notation a text is stored in, and pipelines set it. Authorial composition register is whether the writer was thinking in structure at all, and history sets it. The two come apart at both ends. Books3, the largest openly released book corpus, shipped novelists' prose wearing the pipeline's headings (Presser 2020),^[11] and PDF erases the authoring environment by construction (ISO 32000-1:2008),^[12] after which extractors converge everything onto a common notation, so a LaTeX paper and a word-processor brochure arrive marked up the same way. Pipelines cannot create authorial composition register. They can only preserve or destroy the evidence of it, which is why this has to be recorded at ingestion or not at all. Because the bare word register already has an occupant in this literature (Myntti et al. 2025),^[13] we use it alone only in that occupant's sense, for the text-type strata of the ground-truth corpus, and otherwise qualify it in full, except where a pre-registered rule name fixes its own wording.

Two of our own measurements then relocate the target, and they are the reason this paper ends where it does. Flattening a novel's Markdown while keeping its chapter titles as bare lines changes clean-window survival by essentially nothing on prose (§5.5), and swapping the notation on identical text changes a trained reader's use of long-range context by a measured zero (§7). The sigil is not the cue. The announcement is: a short line, set off by whitespace, that says a boundary is here. Whatever is done about this variable has to be done at the level of announcements, and that is the level at which §9 specifies a format. Until that format is built, we give the thing the corpora lack a name: the missing format, long text in which nothing announces the arrangement, so the boundary inference is demanded rather than optional.

3. Nobody decided this↑

The flatness of the old web corpora was an accident of tooling. C4 consumed Common Crawl's pre-extracted plain text and then applied a line filter: "We only retained lines that ended in a terminal punctuation mark" (T5, §2.2).^[14] That discards every heading, because headings do not end in periods. It was a quality heuristic, and it was the largest single destroyer of structural markup in the corpus that trained a generation of models. The Pile's authors named the problem in 2020, calling the same filter "too aggressive" in their own appendix (App. C.1.1),^[15] and nobody has used the observation since.

The choices that followed were made the same way, on yield. The Pile extracted Common Crawl with jusText, chosen "based on visual inspection" of the output.^[15:1] FineWeb re-extracted from raw HTML with trafilatura, through a wrapper that passes no output-format argument and so leaves the library in its default plain-text mode.^[16][17] That default is not symmetric: trafilatura's serializer applies heading prefixes only when formatted output is requested, while list markers are emitted unconditionally.^[18] DCLM chose resiliparse because it was eight times faster than trafilatura at the same benchmark quality, and its own appendix records that resiliparse "keeps in about 10% more text, some of which may provide useful content such as section titles".^[19] Nemotron-CC chose jusText because it yielded 28.6% more high-quality tokens,^[20] and its mathematics successor changed extractor again, specifically to stop destroying notation that mattered.^[21] Every one of these decisions improved the number being watched. Not one was recorded as a decision about how much structural notation the training stream would carry.

One decision in the record was deliberate, and it is the strongest counter-current to any story of accidental flattening. The Llama 3 pipeline removed Markdown outright: markdown is "harmful to the performance of a model that is primarily trained on web data compared to plain text, so we remove all markdown markers" (§3.1.1).^[22] In the same paragraph, HTML pages with mathematics and code are processed to "preserve the structure" of that content. A frontier lab stripped notation where the structure was not authorially real and preserved it where it was, on benchmark evidence, which is this paper's preserve-and-impose distinction performed without the variable being named.

Then the direction reversed. The frontier of corpus construction moved from stripped web text to PDF conversion by vision-language models, and those models emit structure by design, because faithfulness to the document is the point (olmOCR^[23]; Docling^[24]; MinerU^[25]). FinePDFs runs Docling for text-embedded PDFs and RolmOCR, a Qwen2.5-VL fine-tune on olmOCR's own training mixture, for the scanned path.^[26][27] Qwen3's own account describes fine-tuning Qwen2.5-VL to extract text from PDFs at pre-training scale.^[28] The reversal coincided with a withdrawal: books were roughly fifteen percent of The Pile in 2020 and 4.5% of the LLaMA-1 mixture in 2023,^[15:2][29] then 6.0B of Dolma's 3,059B tokens in 2024, which is 0.196%,^[30] and Dolma 3 has no books source at all. The one component that reliably supplied book-length unmarked prose left the mixture in the same generation that the most heavily marked material anyone has built entered it.

The mechanism is easiest to see in four lines of a public script. Books3, which is 12.07% of The Pile and the largest book source in the most-cited open corpus of its era, was produced by epub2txt-all:^[11:1]

h = html2text.HTML2Text()
h.body_width = 0
text = h.handle(html)
bookname = filename + '.md'

html2text is the canonical HTML-to-Markdown converter, running here at library defaults, which means # and ## headings, * bullets, **bold** and [text](url) links.^[31] Every chapter file it wrote was named .md. A default output mode, chosen by nobody, decided the notation of the largest book corpus in open pre-training, and the script's own header comment traces the same converter back to BookCorpus.

4. What to measure↑

The variable the record lacks is easy to define and cheap to compute. For a tokenized stream, let clean-window survival S(W) be the fraction of non-overlapping W-token windows containing no structural markup: no heading sigils, no list markers, no rules, no blockquote prefixes. Alongside it, measure the distribution of clean runs, the maximal spans of consecutive tokens with no markup at all. S(W) asks how much of a stream still demands the boundary inference at horizon W; the clean-run tail asks how long the demand is ever sustained. Code fences and table pipes are tracked separately and excluded from the core tag set, because they mark content rather than arrangement. Every number in this paper uses the o200k_base tokenizer.

One consequence of that definition has to be stated, because it changes what the statistic means for short-document corpora: a document shorter than one window contributes no windows at all, rather than counting as trivially clean. For a web corpus whose median document is 244 tokens, S(8,192) is therefore a statement about the long tail of that corpus and not about its typical document, while for a books corpus whose median document is 77,297 tokens it is a statement about the typical document. We report it that way throughout, and the data-card field of §12 asks for the contributing-document share alongside the value for exactly this reason.

The same two statistics can be computed against notation or against typography, and the difference between those two readings is a measurement in its own right (§5.5). Against notation, a window is contaminated if it contains Markdown syntax. Against typography, it is contaminated if it contains a line that a notation-blind detector reads as structural: a short line, standing between blank lines, that does not end the way running prose ends.

The detector's precision and its recall are both measurable, and one of them is a warning about every survival figure in this paper. On a novel whose chapters are titled rather than numbered, it catches 4 of 141 real announcements, or 2.8%, because CHAPTER 1. Loomings. ends in a lowercase word and a period and therefore reads to the filter as a sentence; its label-period exception only fires after a capital, as in CHAPTER IV. Titled heads are largely invisible to it. Every structural survival figure reported here for material with titled chapters is therefore an upper bound on cleanliness, and the true contamination is worse than the number. We use the detector as a screen, report its recall where we know it, and use ground truth wherever ground truth exists. Rule-class recognition of chapter heads on this corpus family has published precedent: Chapter Captor segments 9,126 Project Gutenberg novels with hybrid rule-plus-neural header detection at F1 0.77 (Pethe, Kim, and Skiena 2020),^[32] so a deterministic detector is here the field's own instrument rather than an eccentric one.

The obvious objection is that S(W) is a regular expression, and the answer is that this is the point. It is deterministic, seedless and auditable; anyone can recompute it on any corpus in an afternoon and get the same number to the last digit. The methodological sibling to this work profiles three million passages of Dolma on eleven discourse dimensions using a trained scorer built from four hundred hand-annotated passages,^[33] which is a real advance and answers a different kind of question. It also cannot be asked of a lab: you cannot ask a pre-training team to run your RoBERTa over their corpus and reproduce your annotation guidelines. You can ask them to report a surface count over tokens their pipeline already has in memory, which is why the documentation proposal in §12 is realistic and the alternative is not.

5. Front I: what corpora carry↑

5.1 The census↑

Table 1 is the census. Ten rows are the published measurement; three more were run for this paper and are reported separately, at a different budget and sampling rule, because they are not part of the published ten. Three facts organize it.

The direction of travel is steep. Between FineWeb and the olmOCR slice of Dolma 3, the modern web baseline and the vision-converted frontier, ATX heading presence goes from 0.12% of documents to 33.3%, and any-markup presence from 21.5% to 74.0%. On the re-checked FineWeb figure of 0.191% the heading swing is a factor of 174, so what the table supports is a swing well over a hundredfold rather than the cleaner-sounding number the published rates alone would give. The gradient tracks the pipeline history exactly: line-filtered plain text at one end, vision-model PDF conversion at the other.

The scarce resource is not unmarked text; it is long unmarked text. Every web corpus, at every markup level, has a 99th-percentile clean run between about 2,400 and 6,500 tokens, because a corpus of 244-token documents cannot contain a long clean run however it is notated. Books are different in kind: a median document of 77,297 tokens and a 99th-percentile clean run of 109,045, which is about seventeen times the best web corpus and roughly twenty times the typical one. The gap is dominated by document length rather than by markup: no web corpus in the table has a median document within two orders of magnitude of the books row, so a long clean run is not available to be had however the web text is notated.

No source supplies all three properties at once. The web is unmarked and short. The olmOCR slice is long and carries the lowest survival of anything measured here. Books are long, coherent and low-density, and books are the row that left the mixture.

The provenance of those ten rows has to be stated, and stating it costs less than being caught. The original census run wrote no per-row output file. Seven of the ten rows therefore rest on a measurement that cannot be regenerated: the sample size, the document count, the sampling rule, the corpus revision and the date were never recorded, and the only machine-readable copy of the table before this paper was an array of constants inside a figure script. That is exactly the failure the paper's own recommendation exists to prevent, and a paper asking laboratories to report S(W) on their data cards cannot ship S(W) without an artifact. Every row now carries its provenance explicitly in data/census/census-table.json, including the nulls where nothing was recorded, and six of the ten have been independently re-measured.

Table 2 reports those re-measurements, including the one that disagrees. The three web rows agree on every quantity, with two caveats we would rather print than have found: at an 8M-token sample only documents of at least 8,192 tokens contribute windows, so the survival column rests on 23, 62 and 138 windows for C4, FineWeb and DCLM, and the agreement there is nearly uninformative; and FineWeb's heading rate re-checks 1.6 times higher than the published value, which is the correction that moves the swing above. The olmOCR row diverges. A topic-balanced re-measurement puts its S(8,192) at 0.271 rather than 0.153, and a 25M-token pass on 2,180 windows puts it at 0.296; its any-markup rate re-checks within five points, and its median document at roughly half. We report the divergence as the pre-stated criteria classify it. The claim the row is doing work for survives it: at 0.271 the olmOCR slice is still the lowest-survival corpus in the census, at half the next-lowest re-measured row, DCLM at 0.558, and it is still the corpus that fills the one training stage where a document is experienced whole (§6).

One note about compositional practice is worth carrying forward. Within long-form prose, sustained argument supplies longer clean spans than narrative does, because a treatise's sections run longer than a novel's chapters. In the ground-truth corpus below, unbroken paragraph runs have a median of 25 and a 90th percentile of 78 in narrative against a median of 2 in technical writing. The kind of writing that best supplies the missing format is also the kind that has most completely vanished from modern mixtures.

Table 1. Structural markup across public pre-training corpora, tokenizer o200k_base. † marks rows independently re-measured in August 2026. The three lower rows were measured for this paper under a different sampling rule and are not part of the published ten: each ran at a 25M-token budget except pre-1929 books, which ran to a 60M-token hard cap and still returned only 267 documents. Their intervals are document-level bootstraps with 1,000 resamples, where every other interval in this paper uses 10,000. S(8,192) is computed over documents of at least 8,192 tokens, because a document shorter than one window contributes no windows rather than counting as trivially clean (§4); for the web rows that is a small tail of each corpus, and the median-document and clean-run columns are the ones that describe the typical document. S(32,768) was measured only for the three new rows, where §5.2 reports it, and is unmeasured for the published ten.

The ten upper rows are the published census, produced by a run that wrote no per-row output file; † marks the six rows independently re-measured in August 2026 and reported in Table 2. Per-row provenance, including the five fields the original run did not record, is in data/census/census-table.json. The three lower rows were measured for this paper at a different budget and under a different sampling rule and are not part of the published ten.

Table 2. August 2026 re-verification of six census rows, printed as published → re-measured.

Sampling is head-of-stream at an 8M-token budget except the last row, which is a 25M-token pass over the same topic-balanced sample; dataset revisions, shard lists and read orders are recorded per row in data/census/census-table.json. "Windows" is the number of non-overlapping 8,192-token windows the fresh sample contained, and it is the reason the three web rows' agreement on S(8,192) is weak evidence rather than strong. The two best-effort rows sample a single shard under a configuration the original run did not record, so a difference there is not necessarily drift. The Pile is absent because it cannot be re-verified: the reachable mirror has Books3, BookCorpus2, OpenSubtitles, YouTube Subtitles and OpenWebText2 removed, which are precisely the long, low-markup subsets, making it a different corpus rather than a check on this one.

5.2 The supply side↑

Whether the missing format can be resupplied is an empirical question, and we pre-registered the test before running it. A corpus supplies the missing format at horizon W if and only if its median document is at least W tokens and its S(W) is at least 0.50. Both conjuncts are required, which is the whole point: the claim under test is that no available source has both. Three candidates were measured under a seeded shuffle at a 25M-token budget, except the library-scan books row, which ran to its 60M-token hard cap and still returned only 267 documents. All three come from openly licensed collections (Common Pile v0.1),^[34] which matters because it turns the supply question into a shopping list rather than a proposal: that one project has already assembled both parliamentary records and transcribed audio, and a 242B-token corpus of digitized library books exists alongside it (Institutional Books 1.0).^[35]

UK parliamentary proceedings confirm, at both horizons. Whole sittings run to a median of 45,251 tokens, survival is 0.930 [0.916, 0.943] at 8,192 and 0.802 [0.761, 0.839] at 32,768, and the stream carries 1.79 core markup tokens per 10,000. That is a books-class row from a source that is not books, and it grows.

Creative-Commons YouTube transcripts refute, on length, and this is the most decision-relevant number in the experiment. Their markup density is the lowest anywhere in the census, 0.01 core tokens per 10,000 with S(8,192) = 0.996, and their median document is 1,779 tokens, an order of magnitude short of a long-context training window. The fastest-growing transcript pool in the world is unmarked and unusable for this purpose at the same time.

Library-scan books refute, on density, and that was the surprise. We expected them to confirm, because the registration says so: this row was recorded before measurement as expected to confirm at 32,768 and to be books-class, and it did neither. Pre-1929 public-domain books are long, with a median document of 135,399 tokens, and they are not clean: 92.5% of documents carry detector-visible markup, the stream runs 33.45 core markup tokens per 10,000, and S(8,192) is 0.413 [0.370, 0.463], falling to 0.142 [0.111, 0.182] at S(32,768). OCR of a printed page reconstructs the page's furniture. The row is also underpowered against its own pre-registered floor, at 267 documents rather than 400, and one document, at 7,998,460 tokens, is 13.3% of the sampled stream on its own. At that length it is a bound volume set rather than a book, and the row's pooled window statistics are influenced by it accordingly.

The supply claim holds, and the honest headline is narrower than the one it replaces: the long, unmarked speech supply is institutional, not consumer. Proceedings qualify; the consumer transcript pool does not. The verdict rests on one row of three, and that row is the sampling-sensitive one, so the two facts belong in the same breath: Hansard is flagged sampling_sensitive in its own provenance, and reading from the head of a shard rather than from a seeded shuffle gives a median document almost eight times lower and S(8,192) = 0.757 against 0.921, both at the 8M-token sensitivity budget; the row's 25M-token value is 0.930. Corpus statistics of this kind are properties of a sampling rule as much as of a corpus, which is one more reason the sampling rule belongs on the card.

5.3 No null level↑

Across the 50 documents of the ground-truth corpus, the minimum markup density after serialization is 1.245 core markup tokens per 10,000, in Conrad; the next three are 2.23, 2.79 and 3.17. No document has zero. Dispersion between documents passes through conversion essentially untouched, with a coefficient of variation of 1.190 in the authored ground truth against 1.162 through Docling and 1.206 through Marker, two of the four configurations whose dispersion sits within 3% of the authored value. The claim holds for four of the five structure-emitting configurations; pymupdf4llm is 15% higher at 1.364. So conversion is not homogenizing the spread. What it removes is one specific value from the support: absent. This is why the problem cannot be reached by mixture weighting, however sophisticated the weighting is. Reweighting moves mass between levels of a variable, and it cannot restore a level the variable no longer has.

The volume objection arrives immediately and deserves a direct answer. Across the fifteen serialized narrative works, structural markup is 664 tokens out of 1,024,826, which is 0.0648% of the stream and works out to 2.66 structural markers, or 5.31 markup tokens, in an 8,192-token training sequence. Per work the share ranges from 0.0125% to 0.2194%. Nothing in the mechanism runs on volume: shortcut adoption runs on predictivity and availability,^[8:1][9:1] and circuit formation runs on reliability at more than twice its sensitivity to frequency.^[10:1] A cue present at every boundary and absent everywhere else would sit at reliability 1.0 and its frequency would be beside the point; §5.5 measures how close the real one comes. The concession owed in the same breath is that the dose-response curve is unmeasured. We do not know how much of this material, at what density, is needed to install or remove a format preference, and our own prior evidence about how little data installs one cuts against a scarcity story rather than for it.

5.4 What conversion actually adds: a pre-registered null↑

We predicted that converters would fabricate structure on prose, and that the damage would concentrate where organizational variety still lives. We pre-registered that prediction, and it failed. Over 50 documents with ground truth derived from machine-readable sources, typeset to real paged media and run through nine current extraction and conversion configurations, the pooled imposition loss on prose-like material is −0.029 [−0.047, −0.010] at W = 8,192: significantly negative on all three prose strata separately as well as pooled, and negative but not distinguishable from zero on the structured control, whose interval closes at exactly zero (Table 3). Modern converters slightly under-mark. They are not inventing boundaries; they are faithfully marking real ones. The registered rule returned NULL_NO_IMPOSITION.

That null killed the fabrication mechanism, and what survived it is the stronger claim. A fabricated cue would at least be noisy, and noise is what keeps a shortcut from closing. A faithful cue is a trustworthy one, so accuracy in conversion is not a mitigation of the problem in this paper. It is the problem.

Six things have to be said about how that verdict was reached. First, the registered decision rule was close to unsatisfiable at the evaluation point: S_marked(8,192) for prose-like work is 0.063, and a POSITIVE verdict required a bootstrap lower bound above 0.10, so the rule would have needed converters to destroy essentially every clean window that faithful marking leaves. The amendment log recorded that ceiling before the deciding run and required the paper to say so in those words. Second, the answer to that objection was pre-specified in the same entry and is reported here: at W = 2,048 the ceiling is 0.547, the rule is satisfiable, and imposition is still negative pooled, at −0.011 [−0.017, −0.005] for prose against a control point estimate of −0.014 whose interval covers zero. At that horizon the narrative stratum alone no longer excludes zero, at −0.006 [−0.011, +0.001], and the sentence should not be read as claiming it does. Third, the registration promoted Track B, real publisher and Internet-Archive PDFs, to primary for the register strata, and the headline above is Track A, the self-rendered arm. Track B cannot support the imposition estimand for prose: its eleven real scanned narrative books have no ground-truth marked stream to compare against, so the prose contrast is undefined there. It can support the emission ratio, which needs only the authored heading count, and that number is reported below rather than omitted. The deviation itself was not recorded in the amendment log at the time and is disclosed here. Fourth, the study contains no vision-language converter: olmOCR and MinerU were provisioned and produced no rows, so a null measured on nine non-visual configurations is being carried into an argument whose own genealogy makes visual conversion the frontier. Fifth, and in the other direction, the pre-registration is better than the usual: version one was frozen before any data existed, and the one amendment that changed the primary outcome documents the exact data state at the moment it was made, eleven scored cells over two documents with zero technical documents converted, which makes an outcome-driven amendment impossible for the quantity the new rule turns on.

Sixth, the registered rule admits four verdicts and this study returns two of them. The second is NULL_NO_CONCENTRATION, and it fires in the direction opposite to the one we predicted: the prose-minus-control contrast is −0.025 [−0.045, −0.005], so the loss is not merely unconcentrated on prose but significantly smaller there than on the structured control, where we had predicted it would be larger. The registered monotonicity secondary is likewise not satisfied, at −0.005 [−0.013, +0.003], with the stratum ordering non-monotone at both horizons. And the ceiling-free relative estimand the amendment log pre-specified at every W is computed and reported here for the first time: −0.46 [−1.08, −0.14] at W = 8,192 and −0.020 [−0.033, −0.009] at W = 2,048. All three run with the null rather than against it, which is exactly why they are easy to leave out and worth printing.

One quantity from the same runs sharpens the null rather than softening it. It has to be reported on three surfaces, because it moves between them. Counted as core-markup spans emitted per authored heading and pooled over all fifty documents, the five structure-emitting configurations run between 1.19 and 1.97 on PDFs rendered from the marked source, against 1.84 spans per authored boundary in the faithful marked serialization itself, or 1.85 against the 2,599 authored headings the converter join counts. The converters are tracking the source rather than inventing on top of it. On narrative alone the pooled figure is misleading, because it is carried by the technical stratum's list markers: there the same tools run between 1.01 and 1.08, and rendering the flat source drops them to between 0.09 and 0.95. The navigation overlay is therefore partly, not wholly, manufactured from the page. On structured material it is reproduced at essentially the same rate with or without source markup; on prose, removing the markup cuts it on every converter, by a quarter or more on all but the layout mode and by a tenth there.

Track B runs higher still on the register it covers, and we report it because the registration made that track primary. Its eighteen real publisher and Internet-Archive PDFs are all technical writing, and over them the same configurations run between 1.27 and 4.43 spans per authored heading, four of the five at 2.52 or above, against 1.56 to 2.63 on the technical documents we typeset ourselves. Docling emits 74% more on real paged media than on our own rendering of the same register and the layout-aware pymupdf4llm 83% more, while Marker emits 4% less: the over-emission is real, and it is tool-specific rather than universal. Three limits travel with that comparison and none of them removes it. The two arms are different document sets at the same register rather than the same documents rendered twice, so it compares populations and not pairs. Track B's scanned-book arm is nineteen documents, sixteen of them prose, and the single tool run over them emits no structural spans at all, so the register this paper's proposal is about is not covered there by any tool that emits structure. And the rendering path of the self-rendered PDFs was not independently inspected here, so we report the ratios and not a mechanism. The direction is what matters for the null above: under-marking is a property of documents we typeset ourselves, and on real paged media two of these three tools over-mark substantially.

Table 3. Imposition loss from nine converter configurations over 50 ground-truth documents, cluster bootstrap (seed 20260805, n = 10,000).

5.5 Notation invariance and cue reliability↑

The fourth measurement is the one that makes the obvious fix wrong. Strip the # but leave the chapter title as a short standalone line between blank lines, and the notation-blind detector of §4 finds the boundaries in essentially the same places (Table 4). On narrative the two readings differ by −0.001, −0.010, −0.004 and 0.000 at 1,024 through 8,192 tokens; on sustained argument they differ by at most +0.007. We print all four registers rather than the two the argument needs, because the exceptions are informative: mixed material differs by +0.031 at 8,192, and structured technical writing by −0.015 and −0.018 at the two shortest horizons, where the detector reads enumerated skeletons the syntax check misses.

Flattening the notation, on its own, changes almost nothing about where boundaries are detectable, because the typographic form is itself a perfectly good cue: a short bare line with whitespace around it. This zero is a result and not a nuisance, and it dictates the shape of any intervention. What has to be attacked is not the sigil's presence but its reliability. Removing one notation while leaving another equally reliable one in place merely swaps one shortcut for another, which is the finding that sends §9 to the level of the announcement rather than the level of the marker.

The hinge of the whole mechanism is that the cue is reliable, so we pre-registered a test of that sentence rather than continuing to assert it. E5 matches every structural sigil in a serialization against the authored boundaries of the same document, one to one, and reports precision and recall per stratum (Table 5). The pre-registered verdict is UNDERMINED as stated. The rule required precision at or above 0.99 with a lower bound at or above 0.95 in narrative and in sustained argument, and sustained argument returns 0.765 [0.521, 1.000] against notation and 0.722 [0.485, 0.977] against typography. Narrative, which is where the missing format lives, holds: 0.994 [0.984, 1.000] and 0.956 [0.933, 0.977].

Two things about that failure are worth more than the verdict. It is carried by a single document: Arnold's Culture and Anarchy, whose Project Gutenberg text carries a numbered endnote apparatus that the syntax detector reads as ordered-list markers, contributing 31 of the false positives on its own. Dropping it restores precision to 1.000 and 0.951, which we report as post hoc and not as an amendment to the rule; it is the same false-positive mode that accounts for the residue in the flat pack of §9. And the pooled row is dominated by structured technical writing at 0.418, where a sigil is usually marking a list rather than announcing a discourse boundary, which is not a failure of the cue so much as a reminder of what the cue is for.

Two secondary verdicts from the same run are worth a sentence, because they turn the converter null of §5.4 into a stronger statement. Recall returns PERFECT_RECALL on the notation arm and CONSISTENT_WITH_UNDER_MARKING on the typographic one, which converts "converters do not fabricate" into "converters do not fabricate and they under-mark". The detector-agreement check passes, with the recall gap between the two arms at 0.033 in narrative and 0.050 in sustained argument against a threshold of 0.05, so the second of those sits exactly on the line rather than under it.

The sentence we retire is that the cue never lies. What replaces it is scoped and more useful. In narrative, the material this paper's proposal is actually about, a structural sigil announces an authored boundary at precision 0.994 [0.984, 1.000] against the manifests as built, and at 0.967 once the thirty-two heading blocks our own strata file documents as typographic apparatus are removed from the ground truth. We report both because the registration required both, and the adjusted figure is the one that decides the band: 0.967 falls in the registered SUPPORTED, WEAKENED range rather than the SUPPORTED AS STATED one, whose pre-stated consequence is that this paper writes near-perfect reliability rather than perfect reliability everywhere the claim appears, including the abstract. Near-perfect is still what a shortcut runs on. Elsewhere the sigil is doing other work and its reliability is a measured quantity between 0.42 and 0.84 on either reading. A mechanism that only holds where the paper applies it is a narrower claim than the one we started with, and it is the one the data supports.

Table 4. Clean-window survival measured two ways: against Markdown syntax in the marked serialization, and against a notation-blind structural detector run over the flattened text. On prose the two agree to within a hundredth at every horizon, the largest gap being 0.010 on narrative at 2,048 tokens, so the boundary is readable from typography alone and flattening the notation buys nothing. All four registers are printed, including the two where the readings come apart.

Table 5. How reliably a structural sigil announces an authored boundary, matched one to one against ground truth on the 50-document corpus; cluster bootstrap by document, 10,000 resamples, seed 7. Precision is reported twice as the registration required: as-is, and in the adj. column with the thirty-two heading blocks the strata file documents as typographic apparatus removed from the boundary set. Narrative satisfies the rule as-is and lands in the SUPPORTED, WEAKENED band once adjusted. * = fewer than ten documents; point estimates stand, intervals are underpowered, and strata are not merged to manufacture power.

6. The one stage where the sequence is the document↑

Where in training would any of this bind? The answer has a shape that is easy to miss and hard to unsee afterwards.

Over 95% of pre-training tokens are consumed at sequence lengths of 8,192 or less. Llama 3 ran 15.6T of its roughly 16.4T tokens at 8,192, with a long-context extension of about 800B.^[22:1] DeepSeek-V3 pre-trained 14.8T tokens at 4,096 and extended context with about 126B tokens of YaRN, under one percent of the budget.^[36] Qwen3 ran roughly 35T of its 36T at 4,096.^[28:1] Olmo 3's 7B ran 5.93T of pre-training and 100B of midtraining at 8,192, then 50B at 65,536.^[37] The worst case among those four families is 95.12% and the token-weighted pool is 97.31%, so "over 95%" is not a rounded claim. It is the floor.

At the main stage a training sequence is not a document. It is a packed window: several short documents concatenated, or an arbitrary slice from the middle of a long one, frequently straddling a document boundary. At that stage the attention mask usually does not even separate the neighbours. Llama 3 reports that document-separating masks had "limited impact" in standard pre-training while being "important in continued pre-training on very long sequences",^[22:2] and DeepSeek-V3 packs documents but "do[es] not incorporate cross-sample attention masking".^[36:1]

There is a statistical-mechanics way to say what such a stage can and cannot teach. Mutual information between tokens of natural text decays as a power law, and that decay is caused by hierarchical document structure, with correlations flowing from the high organizational levels down into the words;^[38][39] transformers are preferred in part because they capture slow power-law dependence where recurrent models cannot.^{[40][41][42][43]} Set against that: authored boundaries arrive every 3,078 tokens of narrative on average and every 2,392 tokens at the median, which is worse; sequences are capped at 8,192; and every web corpus's clean runs are exhausted by 6,500. Planning-horizon work points the same way, finding that models pre-cache features for future tokens where the data has genuine future structure and that pre-caching rises with scale.^[44] Packing sharpens the picture: sequences cut from the marked serialization of our narrative corpus survive at 0.033 at L = 8,192, against a ceiling of 0.888 for the same books flattened.

The long-context stage inverts every one of those properties. Sequence length exceeds the median book, and intra-document masking is switched on. Olmo 3's released training scripts run the main stages at DEFAULT_SEQUENCE_LENGTH = 8192 with no masking flag, while both the 7B and the 32B long-context scripts read:^[45]

DEFAULT_SEQUENCE_LENGTH = 65536
...
generate_doc_lengths=True,  # enables intra-document masking

At this stage, and only at this stage, the training sequence is the document: one work, entire, in reading order, with attention confined to it. It is the only point in the pipeline where a gradient can flow through the experience of tracking one document's structure across tens of thousands of tokens.

What that stage is fed is therefore the whole question, and the public record answers it precisely. Olmo 3's long-context data pool is 639B tokens and is, to a rounding error, entirely olmOCR-converted science PDFs: every length bucket from 8K to 1M and beyond is a PDF row, and there is no books row, no transcript row, and no long-form prose row of any kind. The 50B stage actually trained draws 33.9% from that pool and replays midtraining data for the other 66.1%.^[37:1] The consequence is sharper than the headline number. Every long document the model meets at 65,536 tokens with intra-document masking on comes from the corpus with the lowest clean-window survival in our census, published at S(8,192) = 0.153 and re-measured at 0.271, which is the lowest figure either way. The remaining two-thirds is the ordinary short-document mix, packed, which is the stage the long-context stage was supposed to be different from.

Nothing in this is anyone's error, and the field has already noticed the scarcity without naming its cause. SmolLM3's team reports that upsampling long-context data such as code repositories, books and long web pages "didn't further boost performance on RULER and HELMET benchmarks";^[46] a 2026 mid-training line builds long-context supervision out of cross-repository code because, in its authors' words, the existing long-document sources are "finite resources and often scarce in long-distance dependencies".^[47] Meanwhile every long-context data method in current use, whether selection, synthesis or supervision reweighting, keys on long-range information gain, attention statistics or predictive uncertainty,^{[48][49][50][51][52][53]} and none of those criteria conditions on how the candidate is notated. The variable is invisible to the tools that choose what fills the slot.

The economics follow, and they are the reason this paper's proposal is small. Nobody needs to touch 15T tokens of web text. The intervention the analysis points to lives in a stage that is under one percent of the budget, in training runs that already have the infrastructure configured for it.

7. Front II: what readers use↑

Front I measured what the corpora carry. This front measures what a trained reader does with it, on text where the notation is the only thing that moves. Twenty-one narrative and sustained-argument works are serialized from one block structure into three arms. The marked arm writes the genuine structure as Markdown; the flat arm keeps every word, drops the notation, and leaves the heading text standing as a bare line; the silent arm deletes the announcement line and puts nothing in its place. Marked against flat isolates the sigil, flat against silent isolates the announcement. Targets are ordinary interior prose, never headings, stratified by distance from the last authored boundary and byte-identical across arms: 169 across the 21 works, and 88 in the silent arm, which exists only for the twelve works whose chapter heads are contentless labels. Each target is scored with 768 tokens of preceding context and again with 7,936, and information gain is the difference, in millinats per byte. Intervals are cluster bootstraps by work, 10,000 resamples, seed 7.

The published measurement was one 0.6-billion-parameter reader, which was this work's own first-listed limitation. It is now five, spanning 0.60B to 8.19B and two independent pre-training pipelines (Table 6, Figure 5). The anchor-set hash is identical for all five and equals the published run's, so the comparison is exactly paired by construction rather than by post-hoc alignment, and recomputing the 0.6B row through the cross-scale code reproduces every published estimate to three decimals.

R1, the deficit persists across scales: satisfied at five of five. Deleting the announcement makes the prose that follows it harder to predict at every scale measured. The rise is +0.466 [+0.159, +0.770] millinats per byte at 0.6B, +0.223 [+0.017, +0.436] at 1.7B, +0.571 [+0.195, +1.011] at 4B, +0.578 [+0.288, +0.849] at the cross-family reader, and +0.459 [+0.123, +0.883] at 8B, the largest reader of the five. Each lower bound stays above zero at three bootstrap seeds. The finding is no longer a single measurement: it holds over a 13.7× span of scale and across two pre-training pipelines that share no data decisions. It also has a published precedent, which we extend rather than compete with. ChapterBreak handed long-range language models a narrative segment ending at a chapter boundary and asked them to pick the true next chapter out of same-narrative negatives, and reported that they "fail to effectively leverage long-range context, substantially underperforming a segment-level model trained directly for this task".^[54] That is this front's question four years earlier: their task is discriminative and supervised over candidate continuations, ours is a paired likelihood contrast on identical bytes with the announcement present or deleted, and the curve below adds a spatial profile their design cannot see.

R3, the notation swap is a zero everywhere: satisfied, and tight. Every reader's notation contrast contains zero and every one is tight against the pre-registered bound, the widest interval endpoint being 0.65%, 0.91%, 0.55%, 0.89% and 1.68% of that reader's measured information gain against a threshold of 2%. The largest reader is the loosest of the five and is still inside the bound. This is the most misreadable result in the paper, so we interpret it here as well as in the abstract. A zero does not say that notation is unimportant. It says the sigil is not the operative cue: present the chapter head as ## CHAPTER IV. or as CHAPTER IV. and a trained reader does the same thing with the prose that follows, at every scale we ran. What moves the reader is whether the line is there at all, which is why §9 specifies its intervention over announcement presence rather than over notation. One corollary matters for curation: information-gain scores of the kind used to select long-context training data are insensitive to serialization across the scales tested. The swap's point estimate declines with scale inside the null, from +0.05 at 1.7B to −0.15 at 8B, and at 8B the interval's upper bound is +0.006: one more doubling in the same direction and the registration's named Reversal branch, the marked arm using more long-range information, would fire. The five readers share one anchor set, so this is one correlated trend rather than five findings; the third-pipeline reader of the replication extension below sits on the positive side of the null, and the training-time test is where it resolves.

One sign inside that null is worth stating, because a reader will notice it before we point at it. The other half of the notation contrast, the raw difference in prediction difficulty rather than in long-range gain, has a negative point estimate at four of the five readers, running from −0.04 to −0.10 millinats per byte: flat text is marginally easier than marked text, not harder. The 8.19B reader is the exception, at +0.028 [−0.094, +0.163]. Every interval covers zero, and the closest to excluding it is the cross-family reader's [−0.206, +0.014]. Because the five readers share one anchor set, the five signs are not independent draws, so this is one correlated near-null rather than five of them. We report the signs anyway, because four of the five run the same way and because that direction points away from a suppression story rather than toward one.

R2, larger readers reconstruct: fired, and it cuts against this paper. The rule asked whether a bigger reader answers a deleted announcement by reaching further back into context, and its first branch fired: at 1.72B the announcement information-gain contrast is +0.270 [+0.010, +0.521], an interval excluding zero from above. Under the pre-registered rules that is a headline finding in its own right, and what it says is that the inference may be present in larger readers and missing only in small ones. We report the rule as fired and the robustness picture together, because the second is what a reader needs in order to use the first. The effect replicates on none of the three larger readers: 4B returns +0.028 [−0.327, +0.404], the 7.30B cross-family reader +0.612 [−0.100, +1.418] and 8B +0.151 [−0.109, +0.394], all covering zero. Four readers were eligible under the rule and one fired, and the same-family sequence is not ordered in scale: −0.022, +0.270, +0.028 and +0.151 from 0.6B to 8B. The deciding bound is +0.010; across three bootstrap seeds it falls as low as +0.003, so the rule fires at every seed we ran, with a margin above zero at its closest seed smaller than the ±0.005 reproducibility envelope this project fixed for the probe before the run. The attenuation branch did not fire at all, because the deficit is not monotone on the same-family ladder. We report R2 as satisfied because the rule was written in advance and it fired, and we report in the same place that a single marginal interval out of four eligible tests is the weakest form in which it could have fired. The honest sentence is that reconstruction is not ruled out at reader scales above 1.7B rather than that it has been shown, and that the matched training run of §10 is what settles it.

R4, the determinism gate: passes per reader. Anchors whose visible context contains no boundary in either arm receive byte-identical inputs, so their contrast must be exactly zero, and all eleven return exactly zero, bitwise, on every completed reader. One descriptive quantity travels alongside, labelled descriptive because it was not pre-registered: mean information gain on the marked arm falls as the reader grows, 23.98 to 22.05 to 21.87 to 18.99 millinats per byte up the Qwen3 ladder, with 19.94 at the cross-family reader, which sits above the largest Qwen3 reader rather than below it, so the ordering across families is confounded and only the within-family decline is read here. Larger readers gain less from the extra 7,168 tokens because they already predict the short-context continuation better, which is why the announcement contrast is reported in absolute units.

Table 6. The probe across the reader ladder, in millinats per byte, cluster bootstrap by work (10,000 resamples, seed 7). Announcement ΔNLL is the rise in prediction difficulty when the announcement line is deleted; notation ΔIG is the change in long-range information gain when the same boundary is written as a bare line instead of a Markdown heading. The anchor set is identical across all five readers and identical to the published run, 169 targets in 21 works with the same anchor_set_sha256, so the comparison is exactly paired by construction. The ladder is five completed readers spanning 13.7× of scale, 0.60B to 8.19B, and two pre-training pipelines; the fifth, Qwen3-8B-Base, completed after the writer-side runs and enters on the same anchor set as the other four. Weights are bf16 with float32 scoring throughout. Verdicts are the pre-registered rules applied mechanically over the complete ladder.

The silent arm is not an ablation of a formatting nicety. It is the format this paper ships (§9), applied to the twelve pure-label works: every structural announcement deleted, nothing put in its place. The announcement column of Table 6 is therefore the per-reader cost, in millinats per byte, of reading the pure frame, and R1 says that cost is real at every scale we measured. It is the inference demand the format exists to reinstate. The probe deletes pure-label heads only, because deleting a titled head would confound announcement removal with content removal, so these numbers bound the format's effect on its cleanest subset rather than on the full specification.

An anchor-level average says the announcement costs something. It does not say where the cost lives, and a local re-orientation charge and an accumulating debt are different claims about what the pure frame does to a training experience. The last experiment runs the same three arms over one entire book: every token of all three serializations of Moby-Dick, scored with at least 4,096 tokens of contiguous same-arm context, 860,715 scored tokens over 209 windows, paired across arms by paragraph and by position within the paragraph. The pairing is exact rather than approximate, and the pure-against-flat comparison pairs 100.000% of its tokens.

The deficit is local, bounded and steep (Figure 6). Paired against the flat arm, the pure frame costs +0.0335 nats per byte [+0.0245, +0.0423] over the first sixteen tokens after a boundary, +0.0085 over the next forty-eight, +0.0030 from 65 to 256, +0.0009 from 257 to 1,024, and nothing distinguishable from zero at 1,025 tokens and beyond. Its scope belongs in the same breath as its result: one reader, one book, 141 chapter-like segments. It generalizes across distance, not across works, registers or readers. The pre-registered expectation was near-bin elevation with convergence by the 1,025 to 4,096 bin, and it held. Put in one line: the announcement is worth about a thousand tokens. Its value is re-orientation at the resumption of prose and not a debt that accumulates across a chapter, which is the property an operator meant to raise inference demand without degrading long-run reading has to have.

The units need a sentence, because the two experiments are quoted in different ones and the ratio between them is a thousand. E6's figures above are nats per byte, which is what the figure is drawn in; the near-bin deficit in §7's millinats per byte is +33.5. Set against E1's +0.47 that is not a contradiction and not an error of scale, because the two average over different things. E1 averages a target window of roughly 1,500 bytes beginning at a boundary-relative anchor, and E6 averages the first sixteen tokens after every boundary in one book. The deficit is concentrated at the resumption of prose and gone by roughly a thousand tokens, so an average over a long target is small and an average over the first sixteen tokens is not.

Two results from the same run are reported against us. The expectation that marked and flat are indistinguishable at every bin is violated in the letter: three late bins have intervals excluding zero, at −0.00025, −0.00042 and −0.00032 nats per byte, roughly one percent of the near-boundary pure-frame deficit measured in the same run, though of the same order as the probe's anchor-level announcement deficit at this reader, which averages over a much longer window. The pre-registered interior-token sensitivity column identifies what they are, and the methodological point generalizes. A full-book pass scores every token, including the paragraph-final tokens whose byte span carries the separator that follows, and the three arms write different separators at the 141 pre-announcement paragraphs; restricted to interior tokens the same bins read +0.00003, +0.00012 and +0.00011. An anchor design cannot see this effect at all, because it scores selected targets rather than whole streams, which is why the anchor-level zero and the full-book violation are consistent rather than in tension. The third expectation, that the deficit does not grow at long distance, is reported INDETERMINATE: only sixteen of this book's chapters run past 4,096 tokens, so the last bin's interval is wider than the quantity it is meant to bound. One book cannot answer that, which is an argument for running the curve over the whole pack rather than for reading the last bin.

One caveat closes the front. Olmo-3 is a hybrid-attention model, twenty-four sliding-window layers of width 4,096 and eight full-attention layers, with YaRN engaging above 8,192, so at 7,936 tokens of context only a quarter of its layers see the whole window. A null from that reader on the information-gain contrasts is therefore weaker evidence than a null from a full-attention reader, while its positive announcement-removal result is exactly as strong. The Qwen3 ladder carries the scale argument and Olmo-3 carries the pipeline-generality argument. The front's five stated limits are consolidated in §11.

Replication extension (post hoc). Two extensions were run during final review, registered as amendments before any GPU pass and committed to report regardless of direction (A2 to the probe protocol; B1 to the curve's). A sixth reader from a third training pipeline, SmolLM3-3B-Base, replicates the announcement deficit on the identical anchor set: +0.370 [+0.040, +0.758] millinats per byte, seed-robust, with all 11 null controls exactly zero, so the deficit is CI-positive at six of six readers across three pipelines that share no data decisions. The same reader holds the swap null on reading behaviour, +0.104 [−0.030, +0.255] at 1.40% tightness, and fails the registration's auxiliary condition: its notation ΔNLL, −0.152 [−0.304, −0.025], excludes zero, a static marked-context penalty near 0.8% of mean information gain that no primary reader showed. On this pipeline the use of long-range context is notation-invariant while absolute prediction is not, which sharpens rather than weakens the relocation: the cue that moves reading with distance is the announcement, and it is one more reason the intervention runs through announcements rather than sigils. The five-reader primary verdicts are unchanged. The whole-book curve replicates on a second work, Galton's 1883 Inquiries into Human Faculty, serialized by the same operators under its own pinned structural contract: the pure−flat deficit is +0.0735 [+0.0574, +0.0906] nats per byte in the nearest bin, about twice Moby-Dick's, decays monotonically, and reaches the null inside a thousand tokens; the marked−flat separator artifact recurs at a single far bin with the same sign and the same interior-only reversal; and the far-tail rule returns INDETERMINATE by its pre-fixed segment floor, recorded in the amendment before the run.

8. Front III: what writers impose↑

Front II asked what a reader uses. This front asks what a writer puts back, which is the mechanism by which a corpus property becomes a model property, and the mechanism our own prior asserted without measuring.^[6:1] The design is front II's, turned around: the same works, the same three serializations and the same 7,936-token contexts, with four base models continuing each context under greedy decoding and the first 1,500 bytes scored by the same detectors the census uses, for markup emitted per thousand tokens, a boundary-insertion rate and an announcement rate. The reference level is not zero but the author's own continuation of the same anchor, scored identically, because authored text carries markup too (§5.3). Interior anchors have no boundary inside the measurement window, so the three serializations give byte-identical authored continuations and the human reference is exactly zero difference; pre-boundary anchors are ones where the author's next move is a boundary, and their pure-label subset carries the question the format needs answered. Only base models are used, because an instruction-tuned model's imposition is a post-training artifact and answers a different question. Rules I1 to I6 were registered before the run. One count moved between registration and build: the registration's dry run recorded 29 pure-label pre-boundary anchors, and the deterministic derivation at build time reproducibly finds 31 under the same label rule. The delta is documented in the anchor builder and the preflight log, the human baseline was recomputed on the realized set before any generation existed, and I3's human reference is exactly zero by construction at either count.

The first-order finding is that they mostly stop writing. Greedy decoding on this material degenerates into loops, and that conditions everything below. The pre-registered guard flags a continuation whose four-gram repetition rate reaches 0.35, seven times the worst of 483 authored continuations of the same anchors, so it cannot fire on well-formed prose. It fires on 84% to 100% of continuations in every one of the thirty-six cells: 77 of 1,328 generations survive, three at 0.6B, twelve at 1.7B, twenty-two at 4B and forty at Olmo-3, and mean four-gram repetition runs 0.84, 0.78, 0.73 and 0.68 up the ladder against 0.005 for the authors. None of this is a defect of the pipeline. Maximization-based decoding has been known to degenerate since Holtzman et al.,^[55] it is documented for base models under greedy decoding at billions of parameters (Carlsson et al.),^[56] and the mechanism is self-reinforcing, because a sentence already in the context raises the probability of repeating it (Xu et al.).^[57] It is the default regime for this configuration, and the consequence is that this front measures what base models do in that regime. Far past the protocol's 20% threshold the guard stops being a nuisance filter and becomes a selection effect, so every quantity is reported twice: the exclusion-respecting cells hold two to eight anchors and cannot carry an interval, so the levels below are the exclusion-free view over all 69 interior and 41 pure-frame anchors per model, while the registered verdicts are the exclusion-respecting ones. What protects the contrasts is that the collapse is common-mode, at similar rates in all three arms of every model, and the arms stay exactly paired on the same anchors. The finding still deserves its own line, and the secondary decoding arm says how to read it. Greedy decoding on this material collapses; sampling does not. At temperature 1.0 the guard trips on none of the 153 slots at either reader it was specified for, four-gram repetition falls to between 0.0035 and 0.0059 against 0.0035 to 0.0047 for the authors themselves, and paragraph lengths return to between 249 and 466 characters against the authors' 267 to 351, where greedy emits a single unbroken block of roughly 1,400. The collapse is a property of the decoding regime rather than of the models, which is what the hyperfitting result would predict (Carlsson et al. 2025).^[56:1] What is not settled by that is whether sampled long output is organized the way authored text is organized, and that is exactly what the training-time test of §10 measures.

One reader behaves differently at the margin: Olmo-3 occasionally ends the document instead of looping, at one anchor of 61 in the flat arm and one of 31 in the pure frame, and no other reader emits an end-of-text token anywhere. The nearest published observation about generation under a hard constraint reports formatting richness collapsing rather than rerouting,^[58] which is about instruction-tuned helpfulness rather than base continuation, so we note it and do not lean on it.

I1, writers impose the register: not satisfied, and the null is bounded. On the interior stratum the author opens a new paragraph-level boundary in 14.5% of windows and emits no Markdown at all. Handed the flat serialization, the readers insert boundaries at 5.8%, 8.7%, 11.6% and 15.9%, none distinguishable from the author's rate except the smallest, which is below it. Markup emission from a flat context is +3.53 tokens per thousand [0.00, +11.09] at 1.7B, +3.53 [0.00, +11.06] at 4B, +1.82 [0.00, +5.70] at Olmo-3 and exactly zero at 0.6B, every interval touching zero. Stated in the form the registration requires: imposition above the author, if present at all, is below 11.1 detector-visible markup tokens per 1,000 generated from a flat context and below 0.15 in boundary-insertion rate at 95% confidence, on the widest of the four readers; from a marked context the 4B's markup excess is distinguishable from the author's, reported with the exploratory contrasts below. That is a loose bound and it is the one the data support. Base models handed unmarked prose do not write the marked register back onto it, which locates the imposition risk in post-training rather than in base pre-training, and is what the strongest published counter-evidence to this paper says as well (§11).

I3, the register is restored at silent boundaries: not satisfied, and this is the front's robust result. The registered quantity is the announcement rate in continuations of pure-frame anchors whose deleted boundary was a pure label. The null is exact rather than estimated: the serialization removed the announcement, so the authored continuation contains none and anything the model writes is a reintroduction with no source in its input. All four readers return 0.000 with a bootstrap interval of [0.000, 0.000] on the kept set; on the full set three of four are still exactly zero, the exception being one announcement in 31 continuations at 0.6B, inside a generation whose four-gram repetition rate was 0.941 and which the guard had already flagged. The human reference is exactly 0.000 by construction. Set beside front II's finding that no reader recovers a deleted announcement from long-range context either, these are two independent measurements of the same absence, one on the reading side and one on the writing side. In the terms of §9, a base model tolerates the pure frame; it does not scaffold it.

That reading needed a discount and now has an answer. A model that has fallen into a loop reintroduces no announcement because it is producing little that is structured at all, so an exact zero measured in that regime is weaker evidence of tolerance than the number looks. The pre-registration named a temperature-1.0 secondary arm as the answer to "greedy suppresses rare events", and it has now run at the two readers it was specified for. Handed the pure frame and free to sample, the 1.7B puts no announcement back in 31 continuations, 0.000 [0.000, 0.000]; the 0.6B produces one in 31, a fluent continuation of Arnold whose numbered endnote apparatus the detector reads as an announcement, the same register that undermined the reliability audit's precision (§5.5), for 0.032 [0.000, 0.107], an interval that includes zero, against a human reference of exactly 0.000. The null holds essentially unchanged where the models generate fluently, which makes I3 a robust null rather than a discounted one. We report the secondary beside the primary rather than beneath it, because the primary's exclusion-respecting I3 cells hold one to five surviving anchors per reader and the 0.6B's interior marked and flat cells survive the guard not at all, while the secondary reaches the same conclusion on the full 31 with zero trips. It stays secondary all the same: the registration fixes greedy as the estimand, because the claim is about the model's default and the default is the mode of the conditional.

The guard itself then needs a limitation of its own. It was registered as a nuisance filter, calibrated at seven times the worst authored continuation, and under greedy decoding it removed 1,251 of 1,328 rows, which is a selection effect rather than a filter and is why every greedy rate above is the exclusion-free view. Under sampling it removes nothing at all. A screen whose behaviour moves that far with the decoding regime cannot be treated as a property of the text it screens, and a future run of this protocol should either report both regimes as this one does or drop the guard and report the repetition statistic beside the rate it conditions.

One exploratory reading of the secondary arm belongs here, labelled as exploratory and nothing more. The pre-registered free instrument is the expected number of markup lines along the realized path, read off the distribution at every line-start step, and it sees an event that no realization has to contain. Under sampling it separates the arms: marked context against silent context is +0.205 [+0.107, +0.343] expected sigils at 0.6B and +0.212 [+0.143, +0.292] at 1.7B, while flat against silent excludes zero at neither reader. Notation in the input raises the probability mass the model puts on emitting markup, and the announcement rate it actually realizes at a silent boundary stays at zero. Input-dependence is real in the distribution and absent in the realization. We report that as a distribution-level observation, not as the confirmatory propagation claim that I2's single-anchor cells cannot support.

I2, imposition is input-dependent: the rule returns satisfied and we report why that should not be read as a finding. Every cell that triggers it holds a single anchor from a single work, so its cluster bootstrap resamples one cluster on every draw, the interval has zero width, and it excludes zero by construction rather than by evidence. Three of its four triggering contrasts are negative, while the branch text the rule selected is the positive one. We therefore treat I2 as unevaluable at these scales under greedy decoding, which means none of I1, I2 or I3 is satisfied and the front returns the pre-registered bounded null of I5: the variable is carried by corpora and used by readers, and base readers at these scales do not write it back. The experiment that would test the post-training prediction that follows is the same protocol against an instruction-tuned model of matched scale, which is an afternoon of GPU.

One pattern in the exclusion-free view is worth recording as exploratory, and only as that. On the interior stratum, markup emitted from a marked context is at or above markup emitted from a flat context on every reader, 3.87 against 3.53, 4.07 against 3.53, 1.89 against 1.82 and zero against zero, while markup from a pure-frame context is exactly 0.000 on all four; the one contrast against the author that excludes zero is the 4B model's marked context, at +4.07 per thousand [+0.19, +11.65]. Should that ordering survive a decoding regime that does not collapse, it would say notation in the context is carried into the continuation, and the general form of such an effect is documented and localized to two to four percent of attention heads across 26 models (Liu & Chu).^[59] We do not claim it here, and the same work reports entrainment decreasing with model size, which cuts against reading our own I4 as scale stability.

I4, imposition scales: not satisfied, and indeterminate rather than flat. The registered ladder quantity is 0.500 [0.000, 1.000] at 1.7B and 0.250 [−0.500, 1.000] at 4B, with no value at 0.6B, because the exclusion-respecting interior cells are where the collapse bites hardest. Those cells cannot support a scaling claim in either direction, so we report the rule as unsatisfied and the question as open rather than as evidence of scale stability.

I6, the determinism gate: passes, and how it came to pass is the point. Eleven null-control anchors per model have byte-identical prompts in two arms and must therefore produce byte-identical text; all four readers return 11 of 11. On the first attempt they did not, and the cause was found rather than guessed. The generation library admits sequences into a prompt batch in sub-groups, so two arms of one anchor falling on opposite sides of a sub-group boundary begin decoding at different steps and thereafter see different batch shapes; different shapes select different kernels and reduction orders, a near-tie in the argmax flips, and greedy decoding amplifies one flipped token into a whole different continuation. The two offenders diverged at generated characters 46 and 54, and the third pair, whose arms fell inside one sub-group, was byte-identical as predicted. The recorded amendment, A1 in the log, prefills each batch as a single group, which makes two byte-identical prompts two rows of the same matrix multiplication at every step and so bitwise identical by construction rather than by luck; it changed batch scheduling only, the estimand and the decision rules were untouched, it also cut measured peak memory on the reader it was calibrated against from 28.75 to 8.45 GB, and the pre-amendment run directory was deleted so no row produced under the old schedule is used anywhere above. A gate that fails, finds its cause and then holds is better evidence about a pipeline than one that never fails. The secondary arm produced a second defect of the same species and it is disclosed on the same terms: the exclusion gate written for greedy was still being applied under sampling, so rows that had tripped nothing were being quarantined, and the guard markers are now written per arm, so a guard result carries the arm it was computed in. No number above was produced under the defective quarantine.

Table 7. Front III on the interior stratum, where the three serializations produce byte-identical authored continuations and the author's reference is therefore exactly zero difference. Markup is detector-visible tokens per 1,000 generated, by the notation of the visible context; boundary rate is the fraction of continuations opening a new paragraph-level boundary, from the flat context; announcements restored is the pre-registered I3 quantity, the rate at which a reader writes an announcement into a pure-frame continuation whose own boundary was deleted, over 31 pure-label anchors in 12 works. Decoding is greedy at 384 generated tokens from 7,936-token contexts with the end-of-text token unsuppressed; greedy was pre-registered because it is the estimand, what a base model does when it maximizes. The temperature-1.0 companion registered as the answer to "greedy suppresses rare events" has run at 0.6B and 1.7B on the pre-boundary stratum and is reported in the text as a secondary arm. There is no sampling seed to report for the greedy arm. Bootstrap intervals are clustered by work, 10,000 resamples, seed 7. Rep. trips is the pre-registered repetition guard firing on the interior flat arm, and it is the reason the levels here are the exclusion-free view over all anchors while the verdicts in the text are the exclusion-respecting ones: 77 of 1,328 continuations survive the guard across the four readers, three at 0.6B and forty at Olmo-3, leaving exclusion-respecting interior cells of one to five anchors in the announcements restored column and bounding nothing. Those intervals are zero-width because they resample one cluster, so they are degenerate rather than tight; the temperature-1.0 secondary arm returns 0 of 31 at 1.7B and 1 of 31 at 0.6B with no trips at all, and the text reports both beside this table for that reason. The guard never fires on the author's own prose. Four readers rather than five: the protocol's arbitration rule gave the machine to this front rather than to the 8B probe row, which ran afterwards and appears in Table 6 with no generation arm to match it, so the same-family ladder here is three points and the reduced strength is accepted rather than papered over.

(Markup columns are detector-visible tokens per 1,000 generated, the mean of per-continuation rates, pooled rates within 8% everywhere, by the notation of the visible context.)

9. The pure frame↑

Three measurements in this paper compose into a design, and the design is narrower than the one we set out to make. Flattening a work's notation while keeping its heading words leaves the boundaries detectable in the same places (§5.5). Swapping the notation on identical text moves a trained reader's use of long-range context by a measured zero, at every reader scale we have run (§7). Deleting the announcement itself is the only manipulation that changes reading behaviour at all. Together those say something uncomfortable about the obvious remedy: faithful flattening is a no-op. Stripping Markdown from a novel and leaving CHAPTER IV. standing on its own line produces a file that is different in notation and identical in what it demands. The sigil axis is a distraction, and it is our own zero that retires it. The pre-registered qualifier belongs in this paragraph rather than in a footnote, because the whole-book curve of §7 violated the notation null in the letter at three late distance bins. Those violations are roughly one percent of the curve's near-boundary deficit and they are separator-token effects at paragraph edges, not effects on prose: restricted to interior tokens the same bins are positive and negligible. So the no-op claim is scoped to interior prose, which is what the sentence is about, and is not a claim that the two files are identical token for token.

What is left is the announcement: a short line, set off by whitespace, whose function is to say that a boundary is here. The material that demands the inference is the material with no announcements in it at all. We call that the pure frame, and we specify it rather than gesture at it.

• Keeps body paragraphs, in authored order, separated by exactly one blank line. Sentence-level punctuation and intra-paragraph typography are untouched.
• Deletes every structural announcement line: notation sigils and the lines carrying them, pure labels (CHAPTER IV., PART ONE), titled heads (Chapter 3: The Meeting), running heads, page furniture, transcription artifacts and footnote-marker lines. Operationally, a line is an announcement if the census detector of §4 fires on it, or if it is notation syntax.
• Renders the boundary as nothing. A deleted announcement leaves adjacent paragraphs separated by the same single blank line as any other two paragraphs. A double gap would itself be an announcement.
• Records every deletion in a per-document sidecar: the deleted text, its class, its character offset in the source serialization, and the paragraph index it preceded. The transform is reversible, and the audit is the artifact's warranty.

A file passes the validator if and only if it contains zero announcements and every record's sidecar round-trips byte-exactly. The wording matters, because §4 showed that the detector's recall on titled heads is 2.8% on a novel. The detector is a screen, not the definition; the ground-truth announcement list is the definition, and the sidecar is the warranty. Where those agree, S(W) = 1.0 for every W up to document length, and the validator is what turns that from a claim about a pipeline into a checkable property of a shipped file.

Building it surfaced eleven questions the specification did not answer, and three of them are worth a sentence each because anyone implementing this will meet them. Deletion is partial where only a line-initial sigil fires: a 300-word paragraph its author happened to number keeps its words and loses its 32. prefix, which is the spec's blockquote clause read at prefix granularity. Deletion iterates to a fixed point, because removing one line can leave the line beneath it newly standalone and therefore newly announcing; the pass number is recorded per edit. And the sidecar is an ordered list of non-overlapping replacements over the flat text rather than a list of deletions, which is what makes the inverse well defined and the round trip exact in both directions.

Three honesty clauses travel with the spec. Deleting titled heads deletes authored words, which no amount of framing makes free; the sidecar makes it reversible, and in the recommended mixture the marked copy coexists, so the corpus loses nothing and only the pure copy does. Paragraph breaks remain, and the reason is exactly the mechanism of this paper: a paragraph break fires at every paragraph, so its reliability as a discourse-boundary cue is close to zero, and a cue with no reliability is not a shortcut. Furniture classes also differ by material. Version 0.1 specifies literary and argument prose; proceedings, correspondence and technical writing each need a class list before the spec applies to them.

We ship two packs over the same 27 public-domain works, because the comparison is the argument. The flat pack is the faithful baseline: 2,313,460 tokens, one record per book, notation removed and heading words kept verbatim. Its markup claim has to be stated carefully, because the builder's original guard was a tautology that could not fire. What is true is that the flattener writes no Markdown of its own: no #, no rule, no bullet glyph, no case change. Under this paper's own census detector, nine of the twenty-seven records still light up, for 826 tokens in 280 spans, or 0.036% of the pack, and the cause is inspected and benign: 272 of those spans are numbered chapter headings whose author's own text begins with a numeral, as in 2. Mistakes about Political Economy, six are >From line-starts inherited from a Gutenberg source file, and two are footnote markers. The flat pack's S(8,192) is therefore 0.836, against 0.153 for the olmOCR slice and 0.889 for C4, with the caveat of §4: C4's figure is computed over the small tail of its documents that reach one window, and the pack's over documents that are all far longer than one. It is a better number for this paper than the guarantee it replaces, because a reviewer can check it.

The pure pack is the format itself. The same 27 works come to 2,297,423 tokens after 1,577 lines and 16,502 tokens are deleted (the pure total sits 465 tokens above flat minus deletions: a deletion junction breaks byte-pair merges and the remainder retokenizes), which is 0.713% of the flat pack. Three survival results come out of it, two of them warranties and one of them a tautology, and it is worth separating them. Both census detectors firing zero times over the result is a fixed point of the deletion rule rather than independent evidence: the builder deletes whatever the screen fires on until the screen stops firing, which is the same objection this paper just made to the flat pack's original guard. What is independent is the ground truth, the announcement list the intermediate representation emitted, against which S(8,192) = 1.000 over 267 windows and S(32,768) = 1.000 over 57, and in all 27 records the longest markup-free run is the entire work; and the sidecar, which round-trips byte-exactly in all 27 records. The detector is the screen; the sidecar is the warranty. The cost is concentrated exactly where the honesty clause says it will be: titled heads are 10,257 of the 16,502 deleted tokens, 62% of everything removed, while pure labels cost 1,118 tokens across the whole pack. Deleting CHAPTER IV. is free and deleting Of the Limits to the Authority of Society over the Individual is not.

Two further disclosures belong with the artifact. The operator's fixed-point rule deletes whatever the screen fires on, and on this pack that includes 319 short standalone lines the manifests type as body text: signature blocks, one-line dialogue turns, an epigraph fragment, and in the mixed-material works a great many table rows and index entries. They are listed individually in the artifact rather than summarized, because they are the operator's audit surface. And the format is not a compression scheme. On Moby-Dick the deletion buys 1,587 tokens of headroom, the pure stream is still 283,267 tokens, and it still does not fit a 256k context. The whole gain is in what the stream demands, not in what it costs.

The recommendation is a mixture, and this paper's own data settles which axis it runs on. It is not a mixture over notation, of Markdown against plain text, because the notation swap is a measured zero. It is a mixture over announcement presence: the same work present both marked and pure, so that at the same underlying event the announcement is sometimes there and sometimes not. That destroys the cue's reliability at the level where reliability is doing the work, and it destroys nothing else. The marked copies remain, structural competence is still taught, and the content cost is zero because every copy is the same author's words. Aim it at the long-context stage of §6, which is under one percent of the token budget and already has the masking configured.

We state the central hypothesis as a bet, with its price. We are betting that the factorization exists and that reliability is what forces it: that a model trained on the same arrangement with and without its announcements forms a discourse channel separable from the surface channel that renders it. If the paired arm of the three-arm run (§10, test 3) is indistinguishable from the marked arm, reliability was not the operative variable, the factorization story collapses to a notation preference, and we will report it as such.

One last thing about what the operator is for, because it is the part most easily mistaken for archival fussiness. Fidelity is the converters' objective, and §5.4 shows that they achieve it. The pure frame has a different target. It is a repeatable operator that turns a work into what it already is, an unbroken stretch of sustained thought, because that is the material pre-training is not currently producing. It is judged by the output it produces, which the validator checks, rather than by fidelity to its input, so edge cases in furniture classification need to be repeatable and audited rather than archivally perfect. The pure frame is a format, not an edition. Nobody reads it; models train on it. The general form of the point is the sentence we would most like carried out of this paper: choose format operators by the capability they train, not by the fidelity they preserve. And if the objection is that books are 0.2% of a modern mixture, the answer is that smallness is the economics of the proposal rather than an embarrassment. A slice carrying a disproportionate share of a corpus's organizational variety is exactly what a volume-keyed accounting system cannot see.

Table 8. The two long-form packs over the same 27 public-domain works. The flat pack is the faithful baseline: notation removed, heading words kept verbatim. Its detector-visible markup is not imposed structure but kept authored text, 272 of the 280 spans being numbered chapter headings whose author's own words begin with a numeral; the flattener writes no Markdown of its own. At S(8,192) = 0.836 it stands against 0.153 for the olmOCR slice of Dolma 3 and 0.889 for C4, subject to §4's caveat that the web figure is computed over the tail of C4 that reaches one window while the pack's is computed over documents that all exceed it many times over. The pure pack deletes the announcements and records every deletion in a sidecar. Its detector-zero row is a fixed point of the deletion rule and not evidence; the independent warranties are the ground-truth survival column and the 27 byte-exact sidecar round trips. Fragment counts are per document at an 8,192-token packing length and are a property of the works rather than of the serialization, which is the point of printing them twice.

10. Three settling tests↑

The program this paper belongs to has three tests, in increasing order of cost. One has been run, one is piloted here, and one is specified and costed and has not been run.

Test 1, the fabrication test. Run; it returned a null against us. The prediction was that current converters would fabricate structure on prose, and that the damage would concentrate on exactly the material still carrying organizational variety, because heading detection is a classifier and continuous prose is the class on which its positives should mostly be false. It needed no model training and a few days of compute. It is reported in §5.4, and it failed: imposition loss is significantly negative on every prose stratum, so the converters under-mark rather than invent. The fabrication mechanism is dead. What replaced it is the reliability mechanism, which is stronger, because a faithful cue is a trustworthy one and trustworthiness is what a shortcut needs.

Test 2, the reader-side test. Piloted here. The original form of this test compared public checkpoints from families whose pipelines are documented to differ in structural handling, and looked for benchmark scores and long-form prose likelihood to separate. Front II is that test in a sharper form, because it holds the content byte-identical across arms instead of comparing families whose corpora differ in a hundred ways at once, and because it runs the same anchor set across a ladder of reader scales. What it can settle is whether trained readers use the announcement, and whether the notation carrying the announcement matters. What it cannot settle is why: a reading-side probe cannot distinguish "the training pressure never existed" from "these particular readers never formed the machinery". Front III narrows that gap from the writing side by asking whether models put the announcements back, and they do not. Only test 3 closes it.

Test 3, the decisive test. Specified and costed; not run. It needs three arms on identical content rather than two. The obvious two-arm version, plain text against Markdown, does not work, because it conflates faithful re-serialization with imposed hierarchy and therefore cannot separate this paper's claim from the trivial claim that notation matters. An earlier design's third arm was imposed structure, a detector running over continuous prose and emitting hierarchy it inferred; our own notation zero retires that axis, and the protocol records the retirement. The three arms as specified are marked, current practice, announcement reliability near 1.0; pure, the format of §9, reliability zero; and paired, half the works marked and half pure, partitioned by work at matched tokens rather than duplicated, so the manipulation is not confounded with repetition, taking the stream's announcement reliability to one half. Every word, token budget and data order is identical across arms; only the serialization moves.

The cost matters, because a costed experiment is an invitation and an uncosted one is a wish, so the costing ships as a repo artifact with its arithmetic exposed rather than as a number in a sentence. At one billion parameters and 100 billion tokens per arm, the three-arm run is an estimated 3,100 H100-hours, which is 1.4% of one OLMo-3-7B pre-training run by the same arithmetic, or about two days on 64 accelerators; a screening version at 29 billion tokens per arm is 0.7%. The cheaper precursor is cheaper still and costs no tokens at all, because it is a substitution rather than an addition: replacing 5 to 25 billion tokens of an Olmo-3-style 50-billion-token long-context stage with pure-frame material adds one branch to a stage whose infrastructure is already configured. Those are derived estimates from published throughput and token counts, not measurements, and the protocol states them that way. The readouts are the instruments this paper ships, run per arm over held-out works: the probe's announcement contrast, the whole-book curve and the generation probe, with the loop-collapse rate reported as a first-class outcome because it is the number that connects the variable to a product surface. They run beside the laboratory's own suites, under decision rules pre-registered in the style of R1 to R4 and I1 to I6. We pre-commit the direction in both cases. If the paired arm is indistinguishable from the marked arm, reliability was not the operative variable and the factorization of §9 collapses to a notation preference; and if the slot substitution moves nothing on any instrument, the locus argument fails, cheaply, which is the point of running it first.

11. Epistemic status and limitations↑

The strongest published evidence against this paper's framing is a base-model result, and it belongs here rather than in a reviewer's report. A four-way classifier over model families, at 25% chance, identifies instruction-tuned models from markdown features alone at 77.7% and base models at only 38.5%.^[60] Base models mostly emit plain prose, which suggests that much of the visible chunking is post-training policy rather than pre-training inventory. The same table has a second row that is rarely quoted and that we think settles the reading: on the original text rather than its markdown features, the same classifier identifies base models at 87.3% against 96.3% for instruct. So base models are highly identifiable, just not from markdown. Post-training decides how much of the tendency is expressed; what pre-training supplies is what there is to express. That distinction is a hypothesis, not a finding, and the honest concession is that post-training demonstrably does the visible compression: matched-checkpoint work isolates narrative flattening to post-training directly.^[61] The open question is what range the base inventory had, and nobody has measured it, because notation was never a coordinate. Our own prior found the latent tendency present before post-training in a base-versus-instruct comparison;^[6:2] that is our result and not the idiosyncrasies literature's, and we would rather say so than let the two blur.

A second counter-current is fresher and cuts the same way. A prefix-removal probing method recovers 47.7% of the original records as clean units from flattened, separator-stripped text, against 25.9% for the strongest embedding baseline.^[62] That is a single-author preprint, and it is an intervention: 47.7% is what the probing method can extract, not what a model does unprompted. Read correctly it is the right correction to make to this paper's language. Boundaries are degraded, not absent, and the claim that survives is about how much pressure exists to make the reconstruction good.

The reader-side front carries five limits that we state whatever its verdicts turn out to be. It measures reading and not training, so it cannot distinguish an absent training pressure from readers that never formed the machinery. Its announcement arm covers twelve of the twenty-one works, for the reason given in §7, so the scale ladder is a ladder over that subset. The notation contrast is a tight null rather than a wide one, which bounds any suppression-style effect below 1.7% of measured information gain at every reader scale and at 0.7% on the published one. The rise in prediction difficulty after announcement removal is largest immediately after the boundary in the point estimates at every reader, but it is not monotone in distance, it does not replicate as an interval, and it must not be read as decay. The near-boundary stratum excludes zero only at the published 0.6B reader, at +0.615 [+0.172, +1.143]; the same stratum returns +0.238 [−0.137, +0.613] at 1.7B, +0.410 [−0.082, +0.927] at 4B, +0.775 [−0.165, +1.641] at the cross-family reader and +0.273 [−0.116, +0.673] at 8B, all four covering zero. The middle and far strata, +0.175 and +0.605 with the far interval including zero, were computed at the 0.6B reader only and are not ladder-wide results. And one anchor, also at that reader, breaks the arm's implicit null while the result survives it: of six anchors whose long context contains no boundary, five return exactly zero as the pipeline requires, while the sixth, the smallest target in the set at 571 bytes, returns +5.91, which is consistent with token-grid drift from the upstream deletions rather than any boundary effect. Dropping it gives +0.404 [+0.110, +0.692], still excluding zero, and leaves the near-stratum headline unchanged. Weights are bf16 with float32 scoring throughout, and the eleven exact-zero control anchors come from seven of the twenty-one works.

One statistical posture covers the whole paper and is easier to state once than to defend seventeen times. Every interval printed here is a pre-registered contrast evaluated once, not a survivor of a search: the rules, the strata, the horizons and the thresholds were fixed before the runs that decide them, and where a rule admits several triggers we report how many were eligible and how many fired, as §7 does for R2 and §8 does for I2. Two conventions vary and are labelled where they occur: the census supply rows use 1,000 bootstrap resamples where everything else uses 10,000, and the writer-side front reports every rate twice because its exclusion filter crossed the threshold at which the registration said it stops being a filter. Where a bootstrap interval has zero width it is a degenerate resample over one or a few clusters, and it is named as such wherever it is printed rather than being allowed to read as a tight interval.

The pre-registered rules were evaluated mechanically on all five readers and reported as they fell (§7). Three went our way and one did not. The announcement deficit persists at five of five scales (R1) and the notation swap is a tight null at five of five (R3), which is what the framing predicts; the determinism gate passes at five of five (R4), which is a statement about the pipeline. But R2, which asked whether larger readers reconstruct a deleted announcement from context, fired at 1.7B, and its branch text was written in advance to say that this cuts against the paper. It does, and we report it as a headline rather than a footnote, with the qualification the diagnostics require: the effect appears at one of the four eligible readers, is absent at all three larger ones, and turns on a bound that stays above zero at every seed run, by a margin at its closest seed smaller than the reproducibility envelope fixed before the run. A reader who wants the strongest form of the objection should take it as stated and wait for the training-time test, which is the only instrument that separates "the pressure never existed" from "these readers never formed the machinery". The fifth reader landed after the writer-side runs and all four verdicts were recomputed over the complete ladder; none of them changed direction. A post-hoc sixth reader from a third pipeline replicates the deficit and holds the reading-side swap null while failing its auxiliary ΔNLL condition, a small static notation sensitivity; the replication and the failed conjunct are reported together in §7 (Amendment A2).

The writer-side front carries four limits of its own, and the first is large. Greedy decoding collapsed into repetition on 84% to 100% of continuations in every cell, leaving 77 of 1,328 generations on the kept set, which is far past the threshold at which the pre-registered guard becomes a selection effect rather than a filter; every rate is therefore reported twice, and the exclusion-respecting cells are small enough that only the exact zeros should be leaned on. Second, the one rule that returned satisfied, I2, is carried entirely by single-anchor cells whose bootstrap has zero width, and three of its four triggering contrasts run the opposite way to the branch text it selected, so we treat it as unevaluable and report the front as the pre-registered bounded null (§8). Third, the scaling rule I4 is indeterminate rather than flat; the secondary temperature-1.0 arm has run and shows the collapse to be an artifact of greedy decoding, but it covers two readers and one stratum, and the registration keeps greedy as the estimand, so the primary's small cells stand as the primary. Fourth, the front measures four base models on public-domain literary prose at one context length, which bounds what its null can mean: it says these readers do not impose this register on this material, not that no base model imposes any.

One of this paper's own pre-registered tests returned against its own slogan, and we would rather that be visible here than discovered. The reliability of a structural sigil as a boundary announcement was registered with a threshold and measured, and the verdict is UNDERMINED as stated: the claim holds in narrative at precision 0.994, or 0.967 once the documented typographic furniture is removed from the ground truth as the registration required, and fails in sustained argument at 0.765, or 0.606 adjusted, carried there by one document's endnote apparatus (§5.5). The adjusted narrative figure puts the claim in the registered SUPPORTED, WEAKENED band rather than the SUPPORTED AS STATED one, which is why this paper writes near-perfect reliability and never perfect reliability. The mechanism therefore has a scope, and the scope happens to be the material the proposal is about, which is convenient enough that it deserves saying out loud rather than quietly.

Three more limits belong on the record and are argued where they arise. The census cannot be fully regenerated: seven of its ten published rows rest on a run that wrote no output file, its survival column rests on few windows at the re-verification budget, and one row diverges on re-measurement (§5.1). The converter study contains no vision-language converter, and its registered rule was close to unsatisfiable at the evaluation horizon, which is why the satisfiable companion at W = 2,048 is reported alongside it (§5.4). And the motivation this work started from remains a motivation: readers report a recognizable flatness in machine-generated prose, everything outlined and signposted and closed with a neat turn, and if that texture is real it is what a thin inventory looks like from outside. It carries a withdrawal condition, because an aesthetic judgment invoked as evidence needs one: strip the formatting from prose of both kinds and have raters judge it blind, and if their judgments are predicted by whether they believe a machine wrote it rather than by organizational measures computed on the stripped text, the motivation is an artifact of disclosure and should be withdrawn without prejudice to anything measured above. That experiment has not been run.

Table 9. The falsifier ledger, with directions committed in advance. Three entries are already settled: the first row's claim survives a live survey (§1), the second row's claim is our own and is dead, and the fourth row's falsifier fired at one reader of five and is reported as the pre-committed response requires.

12. Record it: a data card for notation↑

The smallest useful response to everything above is instrumentation, and almost all of it already exists. A dataset card that records what a document passed through on the way in, plus one number computed over tokens the pipeline already holds, makes the variable visible retrospectively and the question askable for every corpus built after it. Table 10 is the proposal. What the card records is the serialization register of §2; the authorial composition register it can only preserve evidence for, in the source that extractor and ocr_path point back to, which is why these fields belong at ingestion rather than in an audit like ours.

The framing matters more than the field list, because it changes what is being asked. FinePDFs already carries a per-record processor field, declared in its own schema as determining which PDF extractor produced the sample.^[26:1] A major 2025 corpus therefore already records extractor identity per document. The proposal here is not new infrastructure. It is to generalize a field the field's own flagship PDF corpus already has, and to add the one statistic nobody computes. That is a request to finish something rather than to adopt something, and the two land very differently.

The reason the statistic is a realistic ask is the reason it looked like a weakness in §4. S(W) is a surface count over a tokenized stream: deterministic, seedless, and reproducible by anyone with the corpus and an afternoon. No laboratory is going to run someone else's RoBERTa over its pre-training mixture and reproduce someone else's annotation guidelines, and none should have to. Reporting S(W) at the context lengths the corpus will actually be trained at costs a pass over data the pipeline has already tokenized once.

The last row of the table is the lesson of our own census, and we would rather propose it having been caught by it. Document count, token count, sampling rule, corpus revision and date are the five things that turn a statistic into a measurement, and they are exactly the five things our own published rows could not supply for seven of ten corpora. A corpus statistic without its sampling rule is a number, not a measurement, and the Hansard row of §5.2, whose median document moves by a factor of almost eight between two sampling rules, is the demonstration.

Documentation tooling has not closed this gap on its own. A 2026 scoping review of dataset-documentation tools finds no pipeline-level extraction fields anywhere in the tooling it surveys,^[63] and the automated-quality-metric line that has grown up alongside datasheets has no notation axis either.^[64] The fields below are cheap and they are not being asked for by anything currently in use.

Table 10. Proposed data-card fields for notation. The first four generalize a field FinePDFs already carries per record. The middle three are the one statistic nobody computes, plus the tokenizer it was computed under. The last is the lesson of our own census, where all five of its components were unrecorded for seven of ten rows.

Artifacts. The two packs and their sidecars, the census with per-row provenance (data/census/census-table.json), every experimental artifact a printed number in this paper traces to, the claims ledger that enforces the trace, the pure-frame implementation and validator, and the costed laboratory protocol of §10 are published at https://github.com/emfreeburg/announcement-carries-cue.

13. Related work↑

Extraction. Three groups arrived independently at the position that extraction is an under-examined variable, and all three measured its consequences rather than its product. AICC builds a 7.3T-token corpus with a model-based HTML parser and beats trafilatura-extracted text by 1.08 percentage points across thirteen benchmarks;^[4:1] the Apple and Stanford extractor study reports differences of up to ten percentage points on WikiTQ from extractor choice alone;^[5:1] lightweight extraction models continue to push in the same direction.^[65] We concede all of it. The same abstract that reports the ten points also observes that different extractors may lead to similar model performance on standard language-understanding tasks while the pages surviving a fixed filtering pipeline differ substantially, which is this paper's unpriced-loss claim stated by the field's own leading extraction study. Both venues are stated in a refereed venue rather than a preprint. One condition is missing from that whole line, and it is the one this paper's mechanism turns on: AICC's gain is measured against trafilatura-extracted text and never against the same corpus under randomized notation, which is exactly the format-augmentation condition. Whether markup fidelity or markup reliability is doing the work in that result is, as far as we can tell, untested. AICC's venue is unverified on the arXiv record and we cite it as a preprint.

Corpus measurement. This paper is in the tradition of what is in a corpus rather than what it scores: WIMBD,^[66] Dodge et al. on C4,^[67] AboutMe,^[68] Quality at a Glance,^[69] the Pile datasheet.^[70] The nearest sibling profiles three million Dolma passages on discourse dimensions with a trained scorer;^[33:1] the boundary is that a trained scorer answers questions a regular expression cannot, and a regular expression can be asked of a laboratory. Fingerprinting work established that dataset-specific formatting propagates through training into outputs (NeurIPS 2025 Spotlight),^[71] and register-based stratification is the closest existing non-domain coordinate.^[13:1] Quality filters have their own version of the problem: restyling a document into Wikipedia's conventions, with facts and length held fixed, reverses an educational-quality classifier on roughly seven percent of documents.^[72]

Mixing and augmentation. The data-mixing literature optimizes sampling weights over content-defined partitions (the 2026 survey,^[73] DoReMi,^[74] DoGE,^[75] RegMix,^[76] Data Mixing Laws^[77]), and every method in it operates over documents as built; newer coordinates such as web-graph position are still properties of the content or its provenance.^[78] Augmentation research rephrases at pre-training scale^[79][80][81] or diversifies document formats to make knowledge extractable,^[82][83] and a 2026 taxonomy of pre-training augmentation has no notation axis in it at all.^[84] Partial coverage does appear to suffice in the analogous supervised setting, where expanding roughly 30% of a fine-tuning set into multiple answer formats recovers most of the gain of full expansion;^[85] whether the same saturation holds for notation at pre-training scale is untested, and it is one of the cheapest open questions this program raises.

Adjacent directions. Rendering documents as pixels rather than extracted text attacks the same loss from the other side, and reports that visual pre-training on "the same underlying corpora" outperforms text-only pre-training,^[86] with a retrieval-time analogue in PIXELRAG.^[87] The ground is not empty. Encoder-era probing finds long-document transformers acquire implicit representations of document structure during pre-training and benefit from infusing more of it (Buchmann et al., EACL 2024);^[88] that line adds structure to models and measures neither the corpus nor the cost of the announcement. Document-level organization has been shown to matter at mid-training: on the content-matched arm, holding generated text and tokens fixed, book-level packaging is worth +1.02 on LLM-synthesized textbook data, and a length-matched control rules out document length alone.^[89] Whitespace as load-bearing notation that extraction destroys has been argued from the poetry side.^[90] Format effects at inference time are a separate literature that we deliberately do not lean on.^{[91][92][93][94]}

14. Coda↑

Nobody decided that the pre-training stream should stop demanding structural inference. It fell out of a line filter here and a default output mode there, a books subset quietly dropped, a long-context slot filled with whatever was longest. Each step was locally correct and improved the number being watched, and the cost landed on a dimension nobody had instrumented.

So: record it, then build it. Recording it costs a field on a card and a pass over tokens that have already been counted once. Building it costs a public-domain corpus, a deletion, a sidecar and a validator, aimed at a training stage that is under one percent of the budget.

What we would ask a reader to take away is not the census or the null but the choice they make visible. Every operator in the current pipeline is chosen for how faithfully it preserves a document. At least one of them should be chosen for what it teaches the model to do.

Notes↑

1. Zacks et al., Event Perception: A Mind-Brain Perspective, Psychological Bulletin 2007. ↩︎

2. Zwaan and Radvansky, Situation Models in Language Comprehension and Memory, Psychological Bulletin 1998. ↩︎

3. Idahl et al., propella-1: Multi-Property Document Annotation for LLM Data Curation at Scale, arXiv:2602.12414. ↩︎

4. Ma et al., AICC: Parse HTML Finer, Make Models Better — A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser, arXiv:2511.16397. ↩︎ ↩︎

5. Li et al., Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pre-training, Findings of ACL 2026, aclanthology.org/2026.findings-eacl.307. ↩︎ ↩︎

6. Freeburg, The Last Fingerprint: How Markdown Training Shapes LLM Prose, arXiv:2603.27006. ↩︎ ↩︎ ↩︎

7. Rae et al., Compressive Transformers for Long-Range Sequence Modelling, arXiv:1911.05507. ↩︎

8. Geirhos et al., Shortcut Learning in Deep Neural Networks, Nature Machine Intelligence 2020. ↩︎ ↩︎

9. Hermann et al., On the Foundations of Shortcut Learning, ICLR 2024. ↩︎ ↩︎

10. Aoyama et al., Predicting the Emergence of Induction Heads in Language Model Pretraining, arXiv:2511.16893v3. ↩︎ ↩︎

11. Presser, epub2txt-all (the Books3 converter), github.com/shawwn/scrap. ↩︎ ↩︎

12. International Organization for Standardization, ISO 32000-1:2008 — Document Management: Portable Document Format — Part 1: PDF 1.7, iso.org/standard/51502.html. ↩︎

13. Myntti et al., Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation, COLM 2025. ↩︎ ↩︎

14. Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, JMLR 2020. ↩︎

15. Gao et al., The Pile: An 800GB Dataset of Diverse Text for Language Modeling, arXiv:2101.00027. ↩︎ ↩︎ ↩︎

16. Face, **, github.com/huggingface/datatrove. ↩︎

17. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, NeurIPS 2024. ↩︎

18. Barbaresi, **, github.com/adbar/trafilatura. ↩︎

19. Li et al., DataComp-LM: In Search of the Next Generation of Training Sets for Language Models, NeurIPS 2024. ↩︎

20. Su et al., Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset, ACL 2025. ↩︎

21. Karimi Mahabadi et al., Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset, arXiv:2508.15096. ↩︎

22. Llama Team, The Llama 3 Herd of Models, arXiv:2407.21783. ↩︎ ↩︎ ↩︎

23. Poznanski et al., olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models, arXiv:2502.18443. ↩︎

24. Auer et al., Docling Technical Report, arXiv:2408.09869. ↩︎

25. Wang et al., MinerU: An Open-Source Solution for Precise Document Content Extraction, arXiv:2409.18839. ↩︎

26. Face, FinePDFs dataset card, huggingface.co/datasets/HuggingFaceFW/finepdfs. ↩︎ ↩︎

27. AI, RolmOCR model card, huggingface.co/reducto/RolmOCR. ↩︎

28. Qwen Team, Qwen3 Technical Report, arXiv:2505.09388. ↩︎ ↩︎

29. Touvron et al., LLaMA: Open and Efficient Foundation Language Models, arXiv:2302.13971. ↩︎

30. Soldaini et al., Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, ACL 2024. ↩︎

31. Swartz et al., html2text: Convert HTML to Markdown-formatted text, github.com/Alir3z4/html2text. ↩︎

32. Pethe et al., Chapter Captor: Text Segmentation in Novels, EMNLP 2020, aclanthology.org/2020.emnlp-main.672. ↩︎

33. Johnson et al., Characterizing Narrative Content in Web-scale LLM Pretraining Data, arXiv:2606.19468. ↩︎ ↩︎

34. Kandpal et al., The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text, arXiv:2506.05209. ↩︎

35. Cargnelutti et al., Institutional Books 1.0: A 242B Token Dataset from Harvard Library's Collections, Refined for Accuracy and Usability, arXiv:2506.08300. ↩︎

36. DeepSeek-AI, DeepSeek-V3 Technical Report, arXiv:2412.19437. ↩︎ ↩︎

37. Olmo, Olmo 3, arXiv:2512.13961. ↩︎ ↩︎

38. Álvarez-Lacalle et al., Hierarchical Structures Induce Long-Range Dynamical Correlations in Written Texts, PNAS 2006. ↩︎

39. Altmann et al., On the Origin of Long-Range Correlations in Texts, PNAS 2012. ↩︎

40. Shen, Mutual Information Scaling and Expressive Power of Sequence Models, arXiv:1905.04271. ↩︎

41. Lin and Tegmark, Criticality in Formal Languages and Statistical Physics, Entropy 2017. ↩︎

42. Sarkar and Howard, Scale-dependent Relationships in Natural Language, arXiv:1912.07506. ↩︎

43. Mikhaylovskiy and Churilov, Autocorrelations Decay in Texts and Applicability Limits of Language Models, Dialogue 2023. ↩︎

44. Wu et al., Do Language Models Plan Ahead for Future Tokens?, COLM 2024. ↩︎

45. AI, **, github.com/allenai/OLMo-core. ↩︎

46. Face, SmolLM3: smol, multilingual, long-context reasoner, huggingface.co/blog/smollm3. ↩︎

47. OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling, arXiv:2608.05141. ↩︎

48. Deng et al., Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data, ICLR 2026. ↩︎

49. Wu et al., LongAttn: Selecting Long-context Training Data via Token-level Attention, arXiv:2502.16860. ↩︎

50. Chen et al., LADM: Long-context Training Data Selection with Attention-based Dependency Measurement for LLMs, ACL 2025. ↩︎

51. Jia et al., EntropyLong: Effective Long-Context Training via Predictive Uncertainty, arXiv:2510.02330. ↩︎

52. Gao et al., NExtLong: Toward Effective Long-Context Training without Long Documents, ICML 2025. ↩︎

53. Zhu et al., Where Does Long-Context Supervision Actually Go? Effective-Context Exposure Balancing, arXiv:2605.10544. ↩︎

54. Sun et al., ChapterBreak: A Challenge Dataset for Long-Range Language Models, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) 2022, aclanthology.org/2022.naacl-main.271. ↩︎

55. Holtzman et al., The Curious Case of Neural Text Degeneration, ICLR 2020. ↩︎

56. Carlsson et al., The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation, ICLR 2025. ↩︎ ↩︎

57. Xu et al., Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generation, NeurIPS 2022. ↩︎

58. Baghaei Potraghloo et al., One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness, arXiv:2604.13006. ↩︎

59. Liu and Chu, Sentence-Level Contextual Entrainment in Large Language Models, arXiv:2606.24077. ↩︎

60. Sun et al., Idiosyncrasies in Large Language Models, ICML 2025. ↩︎

61. Li et al., Narrative Flattening: How Post-Training Compresses Thematic, Affective, and Stylistic Variation in LLM Fiction, arXiv:2605.27878. ↩︎

62. Vegeto, Right Reset: Chunking by Prefix Removal, arXiv:2608.04330. ↩︎

63. Reynolds-Cuéllar et al., From Reflection to Repair: A Scoping Review of Dataset Documentation Tools, arXiv:2602.15968. ↩︎

64. Winata et al., Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability, arXiv:2506.01789. ↩︎

65. Liu et al., Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM, arXiv:2511.23119. ↩︎

66. Elazar et al., What's In My Big Data?, ICLR 2024. ↩︎

67. Dodge et al., Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus, EMNLP 2021. ↩︎

68. Lucy et al., AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters, ACL 2024. ↩︎

69. Kreutzer et al., Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets, TACL 2022. ↩︎

70. Biderman et al., Datasheet for the Pile, arXiv:2201.07311. ↩︎

71. Mansour and Heckel, Measuring Fingerprints of Web-filtered Text Datasets and Fingerprint Propagation Through Training, NeurIPS 2025. ↩︎

72. Klimaszewski and Andruszkiewicz, Is a Document Educational or Just Wikipedia-Style? Pitfalls of Classifier-Based Quality Filtering, ACL 2026. ↩︎

73. Chen et al., Data Mixing for Large Language Models Pretraining: A Survey and Outlook, Data Intelligence 2026. ↩︎

74. Xie et al., DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining, NeurIPS 2023. ↩︎

75. Fan et al., DoGE: Domain Reweighting with Generalization Estimation, arXiv:2310.15393. ↩︎

76. Liu et al., RegMix: Data Mixture as Regression for Language Model Pre-training, ICLR 2025. ↩︎

77. Ye et al., Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance, ICLR 2025. ↩︎

78. Badoni et al., Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality, arXiv:2606.11499. ↩︎

79. Maini et al., Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling, arXiv:2401.16380. ↩︎

80. DatologyAI, BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining, arXiv:2508.10975. ↩︎

81. How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data, COLM 2026. ↩︎

82. Allen-Zhu and Li, Physics of Language Models: Part 3.1, Knowledge Storage and Extraction, arXiv:2309.14316. ↩︎

83. Zhu et al., Enhancing LLM Knowledge Learning through Generalization, arXiv:2503.03705. ↩︎

84. Chen et al., Demystifying Training-Time Augmentation for Data-Constrained Language Model Pretraining, arXiv:2606.16246. ↩︎

85. Liu et al., Improving Cross-Format Robustness in Language Models with Multi-Format Training, arXiv:2606.11643. ↩︎

86. Zhang et al., Scalable Visual Pretraining for Language Intelligence, arXiv:2607.09657. ↩︎

87. Wang et al., PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation, arXiv:2606.28344. ↩︎

88. Buchmann et al., Document Structure in Long Document Transformers, EACL 2024, aclanthology.org/2024.eacl-long.64. ↩︎

89. Tao et al., Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training, arXiv:2607.28109. ↩︎

90. Bhyravajjula et al., so much depends / upon / a whitespace: Why Whitespace Matters for Poets and LLMs, EMNLP 2025, aclanthology.org/2025.emnlp-main.1783. ↩︎

91. Lee et al., The Format Tax, arXiv:2604.03616. ↩︎

92. Zhang and Zhang, The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content, arXiv:2606.11198. ↩︎

93. Yun et al., The Price of Format: Diversity Collapse in LLMs, arXiv:2505.18949. ↩︎

94. Zhang et al., From Lists to Emojis: How Format Bias Affects Model Alignment, ACL 2025. ↩︎