Research proposal. Companion to "The Last Fingerprint" [Freeburg 2026], which treated the model side only.
Summary
A novel enters a modern corpus through a scan, an OCR pass and a linearizer. It comes out with headings the novelist never wrote, breaks where the thought did not break, and a chapter that ran forty pages now cut into eleven sections. Every word survives. The domain label survives: still a book, still counted as a book, still part of the book fraction. What does not survive is the arrangement, and the arrangement is what the model trains on, because no method the field uses to control training data ever sees the original. Mixture weights choose how often to sample the converted version. Filters choose whether to keep it. Both take the corpus as given, and the corpus was given by an extractor.
Some pipelines recover structure the author put in the text. Some impose structure the author did not. Nobody currently distinguishes the two, records which happened, or measures how often. This proposal is addressed to the people who would have to: those who build and document pre-training mixtures, maintain extraction and OCR stacks, and write data cards.
It sets out three studies establishing whether the distinction matters. The motivating hypothesis is that imposed structure substitutes the pipeline's arrangement for the author's, that the substitution happens upstream of every control the field exercises over training data, and that it is therefore invisible to corpus accounting and unreachable by mixture reweighting. If that is right, a consequential training decision is currently made by whoever selects an extraction library, and no reported quantity would change if it were made differently.
The hypothesis may be wrong, and the program is built to find that out cheaply. The first study needs no models and no training: run current converters over two document classes and count structural elements in the output with no counterpart in the source. If converters turn out to be conservative on continuous prose, the concern dissolves and the remaining studies are unnecessary. That is obtainable in days, and it is worth having before anyone spends more than days on this. The second study reads public checkpoints for a predicted divergence between benchmark performance and long-form likelihood. The third, and the only expensive one, is a three-arm matched pre-training run separating faithful conversion from imposed hierarchy, which no existing experiment does.
A positive result would not mean that structure is harmful or that structured corpora should be abandoned. It would mean that a decision now left to extractor defaults should be recorded and made deliberately.
The gap
Dataset documentation for major corpora reports domain proportions, language mixes, token counts, deduplication method and filter thresholds. It does not report what a document passed through on the way in: which extractor, which conversion target, which OCR path, whether structure in the output was recovered or generated.
That omission is not an oversight about metadata hygiene. It is a gap in the causal record. The model is not trained on documents; it is trained on the images of documents under whatever operators the pipeline applied. Write the pair as content c and operator f: the training distribution is over f(c), while every label the field records describes c, which the gradient never touches. Two corpora with identical domain marginals and different operator distributions are different training distributions, and nothing published about either would reveal it.
Operators are known to matter at fine grain. Extractor choice alone moves downstream task accuracy by several points [Li, J. et al. 2026]. Transcription convention determines whether a literary dash reaches the corpus as itself or as a double hyphen, so different editions of one book carry systematically different punctuation statistics; Moby-Dick reached PG-19 as 1,714 double hyphens and zero em dashes [Rae et al. 2019]. These are consequential decisions, taken as engineering, recorded by no one.
The specific quantity this proposal names is format composition: the joint distribution over notation and compositional practice in a corpus. It has never been measured for any major pre-training dataset.
The hypothesis
Stated as a hypothesis rather than a finding, because that is its status.
When a pipeline imposes structure on prose that was not composed with structure, it does not add a layer over the original arrangement. It replaces it. Heading detection inside continuous prose, paragraph normalization into sections, segmentation of a sustained argument into scannable units: each substitutes an arrangement the operator inferred for the one the author produced. The substitute is what enters the corpus and what the gradient integrates over.
Two properties would make that consequential, and both are conjectures this program would test.
The substitution may be unreachable from where the field stands. Not destroyed in the world; source WARCs are archived and the books still exist. But mixture optimization operates over documents as already built, so reweighting changes how often a model sees the converted version rather than restoring the original. Scale draws more from the distribution presented and does not reconstitute one that was not. Post-training gates what a model holds, and the finding that reinforcement learning amplifies behaviors already established in pre-training points the same way [Zhao et al. 2025]. If this is right, re-extraction is the only operation that reaches the problem, and re-extraction is part of nobody's published practice.
The damage may concentrate rather than spread. Heading detection and list normalization are classification decisions applied uniformly. On a README or an API reference their positives are almost all true: the structure is there to find. On a novel or a sustained essay the base rate of genuine structural boundaries is very low, so a large share of what the operator emits would be fabricated. Under that account the damage is a false-positive rate on a low-base-rate class, and it falls hardest on exactly the material that still carries organizational variety, while barely touching material that already resembles everything else. An operation that looks uniform would be, in effect, a variance-reduction pass aimed at the tail.
This is the most testable part of the hypothesis and it is where the program starts.
Why the gap has persisted
The obvious objection is that if any of this mattered, someone would have noticed. The proposal takes that seriously, and the answer is a specific one: on every axis anyone currently measures, more structure is better.
Holding content fixed and varying only document organization improves mid-training benchmark averages by roughly a point [Tao et al. 2026]. Structured-format synthetic corpora outperform curated web text [Niklaus et al. 2026]. Model-based parsers that preserve document structure beat flattening extractors [Ma et al. 2025]. Choosing a better extractor moves WikiTQ by ten points [Li, J. et al. 2026]. There is no published measurement showing that structural normalization produces worse models, and this proposal does not predict one.
The conjecture is that the evaluation suite prices one side of a trade. Benchmarks reward retrieval, extraction, instruction-following and short-form question answering, all of which structured input demonstrably helps. Nothing in that suite is sensitive to whether a model retains the capacity to sustain a long unsignposted argument, to develop a thought across pages without announcing its outline, or to produce organizational forms it was not prompted into.
If that is the situation, the trajectory is rational rather than mistaken. Each conversion improves the number being watched; any cost lands on a dimension nobody has instrumented. A field hill-climbing on a proxy blind to one side of a trade keeps climbing, and every result along the way is correctly reported as progress. Which also sets the burden this program has to meet: if no instrument can be built that prices the other side, the concern is unfalsifiable and should be dropped. Building that instrument is the point of the second study.
What has to be distinguished
Two things get conflated in any discussion of structure in corpora, and the program cannot proceed without separating them.
Serialization is the notation text is stored in, and pipelines set it. Authorial composition register is whether the writer was thinking in structure at all, and history sets it. They come apart at both ends. Books3, the largest openly released book corpus, shipped novelists' prose wearing the pipeline's headings [Presser 2020]. PDF erases the authoring environment by construction [ISO 32000-1:2008], after which extractors converge everything onto a common notation, so a LaTeX paper and a word-processor brochure arrive identically marked up. A corpus can be "full of Markdown" because many authors composed in structure or because a converter imposed it, and the file does not say which. Pipelines cannot create authorial register; they can only preserve or destroy the evidence of it.
The practical consequence is a design constraint on all three studies. Re-serialization is not imposed hierarchy. Wrapping a novel in Markdown and marking its existing chapter breaks changes notation and leaves the arrangement intact; detecting headings inside continuous prose changes the arrangement itself. Books3 is the first kind. This proposal concerns the second, and an experiment that does not separate them will produce an uninterpretable result. That is the flaw in the obvious version of the decisive experiment, and the reason the third study below has three arms rather than two.
The program
Three studies, in increasing order of cost, each one designed so that a negative result is informative and terminates the sequence.
Study one: a converter fabrication census
Cost: days. No models, no training, public tools and public documents.
Assemble two document sets: technical material whose structure is genuinely authored, and public-domain long-form prose. Run the current generation of extraction, conversion and OCR-to-text pipelines over both [Poznanski et al. 2025; Auer et al. 2024; Wang et al. 2024]. For each output, count structural elements that have no counterpart in the source: headings the author never wrote, list items assembled from running prose, section breaks the text does not contain. Report fabrications per ten thousand words by document class and by tool.
The concentration conjecture predicts a large asymmetry, with fabrication rates several times higher on continuous prose. A null result here, meaning converters are conservative on prose or fabricate at similar rates across classes, removes the mechanism the rest of the program depends on, and the sequence should stop.
This study also produces something independently useful regardless of outcome: the first published characterization of what current document tooling actually does to prose, by tool, which anyone assembling a corpus could consult when choosing one.
Study two: a divergence probe on public checkpoints
Cost: days to weeks. Public checkpoints, public text, no training.
The hypothesis predicts that benchmark performance and continuous-prose competence move in opposite directions as pipelines become more structure-imposing. That is a testable dissociation and it requires no new models.
Select checkpoint families whose data pipelines are documented to differ in structural handling. Measure standard benchmark scores and, on the same checkpoints, held-out likelihood on markup-naive long-form prose postdating every training cutoff involved. The prediction is divergence across the family: benchmarks improving while long-form likelihood degrades.
Two design cautions. The held-out text must postdate the cutoffs, or the measurement is contaminated. And pipeline differences between families are confounded with everything else that differs between families, so this study can support or embarrass the hypothesis but cannot establish causation. Its value is that it is nearly free and it is the instrument the whole argument owes: if no such divergence is detectable anywhere, the claim that something unpriced is being lost has no observable content and should be abandoned.
Study three: a three-arm matched pre-training run
Cost: substantial. Small models, three matched corpora, one training budget.
The obvious experiment, plain-text extraction against Markdown conversion, does not work. It conflates faithful re-serialization with imposed hierarchy and so cannot separate this hypothesis from the trivial claim that notation matters.
Three arms on identical content. Arm one: aggressive plain-text flattening. Arm two: faithful structure-preserving conversion of documents whose structure is genuinely authored. Arm three: imposed structure, where a detector runs over continuous prose and emits hierarchy it inferred. Train matched models and evaluate all three on the standard suite and on the divergence instrument from study two.
The prediction that matters is not that the arms differ. Existing work varying document organization already indicates they will, and that the more structured arm will score higher [Tao et al. 2026]. The prediction is that arm three separates from arm two: similar or better benchmark performance, worse long-form likelihood, gain and loss landing on different instruments. If arm three is indistinguishable from arm two, the distinction this whole program rests on is not real, and that is a clean and publishable negative.
Related work has approached this shape from an adjacent direction, comparing matched corpora rendered as pixels against extracted plain text [Zhang, Y. et al. 2026]. The ground is not empty. The variable proposed here, imposed structure with content held fixed, is untested.
What would follow, and what would not
Worth being explicit about the size of the claim under each outcome.
If study one shows high fabrication rates on prose and study two shows no divergence anywhere, the operators are doing what is suspected and it does not reach the weights in any way that current instruments can see. That is a real finding and it argues for closing the file.
If both are positive but study three shows arm three matching arm two, then serialization matters and the imposed-versus-faithful distinction does not, which would be a simpler and more useful result than the hypothesis.
If all three are positive, the consequence is narrower than it might sound. It would not mean structure is bad or that structured corpora should be abandoned; structural competence is valuable and the evaluations rewarding it measure something real. It would mean that the strength and default amplitude of that competence is set by a pipeline decision nobody records, that the decision is currently made for yield, and that it should be recorded and chosen deliberately. The immediate practical recommendations would be small: log extractor identity, conversion flags, source vintage and OCR path as first-class corpus metadata, and treat those as candidate mixture coordinates. Pipelines already produce all of it.
Why now rather than later
The timing is not neutral, and the strongest reason is the one that expires.
The natural experiment is still open, and it is closing. Document-oriented tooling has moved decisively toward structured output where its predecessors produced undifferentiated text, and that shift is recent enough that the historical record remains genuinely mixed. C4 ran no extractor at all, consuming Common Crawl's pre-extracted WET files and then discarding every page containing a curly brace [Raffel et al. 2020]; Dolma likewise consumed WET via CCNet and names no extractor [Soldaini et al. 2024]; RefinedWeb and FineWeb used trafilatura [Penedo et al. 2023, 2024; Barbaresi 2021]; resiliparse belongs to DataComp-LM [Li et al. 2024] and jusText to Pile-CC and Nemotron-CC [Gao et al. 2020; Su et al. 2024]. There is even an explicit counter-current, with the Llama 3 pipeline removing Markdown markers from web data on the grounds that they harmed performance for a model trained mainly on web [Llama Team 2024]. Normalization has run in both directions. That variation is what makes study two possible at all, and it disappears as practice converges on a common stack.
The selection pathway is now measurable rather than speculative. Filters trained on labels from an instruction-tuned judge inherit that judge's taste in presentation: restyling a document into Wikipedia's conventions, facts and length held fixed, reverses a standard educational-quality classifier's filtering decision on roughly seven percent of documents, and human raters score the same documents 0.77 points below the model that labeled the classifier's training set [Klimaszewski and Andruszkiewicz 2026]. Format is being scored as quality, upstream of any mixture decision, at a rate somebody can now put a number on.
Re-extraction gets more expensive the longer it waits. It is the one operation that reaches upstream of the problem, and it is cheap today because sources are retained and operators are rerunnable. That is less true every year.
Two things are deliberately not part of this argument. The share of books in major mixtures has fallen sharply, from roughly fifteen percent of The Pile in 2020 to 4.5 percent of LLaMA-1 in 2023 and 0.2 percent of Dolma in 2024 [Gao et al. 2020; Touvron et al. 2023; Soldaini et al. 2024], and synthetic rewriting now runs at trillion-token scale [Maini et al. 2024, 2025]. Both are real and neither is the case being made here, because both are answerable by domain mixing: a lab that wanted more books could simply weight books higher. The concern in this proposal survives that remedy, since it is about what happens to a book on the way in rather than how often the result is sampled. And markup-naive text is not strictly non-renewable, since transcribed speech from podcasts, video archives and broadcast is markup-naive by construction and growing fast; whether it substitutes for written long-form argument, which is organized differently from talk, is a separate open question this program does not settle and someone should.
Relation to existing work
The neighbors are real and each has established a segment of the chain.
The data-mixing literature optimizes sampling weights under a token budget, from heuristics through proxy-model search and regression to online reweighting [Xie et al. 2023; Fan et al. 2023; Ye et al. 2024; Liu et al. 2024]. The 2026 survey organizes that field entirely around content-defined partitions and names finer-grained partitioning and pipeline-aware design as open directions [Chen et al. 2026]. Non-domain coordinates have begun to appear, including web-graph position [Badoni et al. 2026] and Biber-style situational register, the latter used to stratify pre-training data directly [Myntti et al. 2025]. In every case the coordinate is a property of the content or its provenance, never of the notation or the pipeline that produced it.
Extraction work has established that extractor choice changes both yield and how much structure survives. Rephrasing work has established that holding content fixed while changing style changes what a model learns and how fast. Fingerprinting work has shown that dataset-specific formatting propagates through training into outputs [Mansour and Heckel 2024]. Format-bias work has shown that alignment rewards surface structure independently of content [Zhang, X. et al. 2025; Yun et al. 2025]. Corpus-census work has begun profiling major corpora on non-domain coordinates, scoring three million passages of Dolma on eleven interpretable narrative dimensions [Johnson et al. 2026].
What has not been done is to join them, and specifically to treat the pipeline's structural rewriting as a possible loss rather than as a yield or quality decision.
What is already known about the model side
One area where the program should be careful about its own assumptions, because the evidence is genuinely contested.
Pre-training plausibly installs an inventory of organizational moves, and post-training an expression policy governing when they surface. How the two divide is unsettled. The most direct study of instruction-tuned formatting concludes that the coupling of competence to surface form is introduced by instruction tuning rather than inherited [Baghaei Potraghloo et al. 2026]; a probing study finds most of the markdown signal in post-training, though base models sit meaningfully above chance [Sun et al. 2025]; a third isolates narrative flattening to post-training using matched pre-training checkpoints [Li, Z. et al. 2026]. The defensible reading is that post-training compresses, amplifies or gates what an inventory holds, and that how much of the inventory pre-training supplies is exactly what studies two and three would establish.
Two cautions follow for anyone reading model outputs as evidence. Token-level tells are policy-sensitive and non-stationary, so a rate of zero is a vendor signature rather than a fact about the inventory. And the intuition that suppressing one boundary-marking device simply raises adjacent ones is untested and the nearest evidence cuts against it: under a hard constraint, formatting richness has been observed to collapse rather than reroute [Baghaei Potraghloo et al. 2026]. Neither the substitution dynamic nor an output-side provenance probe should be assumed by this program; both are separate open questions.
What this would produce
Study one yields a characterization of what current document tooling does to prose, by tool and by document class, which is useful to anyone building a corpus regardless of how the larger question resolves.
Study two yields an instrument. If the divergence is detectable, the field gains a cheap readout for a dimension it currently cannot see, applicable to any public checkpoint.
Study three yields the causal answer, in whichever direction it falls.
And the smallest deliverable is the one available immediately: format composition as a recorded quantity. Extractor identity, conversion flags, source vintage and OCR path already exist in pipeline logs. Reporting them alongside domain weights costs almost nothing and would make the question askable retrospectively, which it currently is not for any corpus ever built.
References
Auer, C., Lysak, M., Nassar, A., et al. (2024). Docling Technical Report. arXiv:2408.09869.
Badoni, V., Chen, D., & Wang, X. (2026). Hubs or Fringes: Pretraining Data Selection via Web Graph Centrality. arXiv:2606.11499.
Baghaei Potraghloo, E., Azizi, S., Kundu, S., & Pedram, M. (2026). One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness. arXiv:2604.13006.
Barbaresi, A. (2021). Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction. ACL-IJCNLP 2021: System Demonstrations, 122–131.
Chen, Z., Miao, Y., Supryadi, & Xiong, D. (2026). Data Mixing for Large Language Models Pretraining: A Survey and Outlook. Data Intelligence 8. arXiv:2604.16380.
Fan, S., Pagliardini, M., & Jaggi, M. (2023). DoGE: Domain Reweighting with Generalization Estimation. arXiv:2310.15393.
Freeburg, E. M. (2026). The Last Fingerprint: How Markdown Training Shapes LLM Prose. arXiv:2603.27006. Unrefereed preprint.
Gao, L., Biderman, S., Black, S., et al. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027.
ISO 32000-1:2008. Document management — Portable document format — Part 1: PDF 1.7. International Organization for Standardization.
Johnson, T., Ash, E., Piper, A., & Antoniak, M. (2026). Characterizing Narrative Content in Web-scale LLM Pretraining Data. arXiv:2606.19468.
Klimaszewski, M., & Andruszkiewicz, P. (2026). Is a Document Educational or Just Wikipedia-Style? Pitfalls of Classifier-Based Quality Filtering. ACL 2026. arXiv:2605.23721.
Li, J., Fang, A., Smyrnis, G., et al. (2024). DataComp-LM: In Search of the Next Generation of Training Sets for Language Models. arXiv:2406.11794.
Li, J., Gardner, J., Kang, D., et al. (2026). Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining. arXiv:2602.19548.
Li, Z., Zhu, Y., Wu, S., Bao, H., & Evans, J. A. (2026). Narrative Flattening: How Post-Training Compresses Thematic, Affective, and Stylistic Variation in LLM Fiction. arXiv:2605.27878.
Liu, Q., Zheng, X., Muennighoff, N., et al. (2024). RegMix: Data Mixture as Regression for Language Model Pre-training. ICLR 2025. arXiv:2407.01492.
Llama Team, AI @ Meta (2024). The Llama 3 Herd of Models. arXiv:2407.21783.
Ma, R., Qiu, J., Xu, C., et al. (2025). AICC: Parse HTML Finer, Make Models Better. arXiv:2511.16397.
Maini, P., Seto, S., Bai, H., et al. (2024). Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. arXiv:2401.16380.
Maini, P., Dorna, V., Doshi, P., et al. (2025). BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining. arXiv:2508.10975.
Mansour, Y., & Heckel, R. (2024). Measuring Fingerprints of Web-filtered Text Datasets and Fingerprint Propagation Through Training. arXiv:2412.02857.
Myntti, A., Henriksson, E., Laippala, V., & Pyysalo, S. (2025). Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation. arXiv:2504.01542.
Niklaus, J., Yamaguchi, A., Štefánik, M., et al. (2026). How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data. COLM 2026. arXiv:2604.13977.
Penedo, G., Malartic, Q., Hesslow, D., et al. (2023). The RefinedWeb Dataset for Falcon LLM. arXiv:2306.01116.
Penedo, G., Kydlíček, H., Ben Allal, L., et al. (2024). The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv:2406.17557.
Poznanski, J., Borchardt, J., Dunkelberger, J., et al. (2025). olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models. arXiv:2502.18443.
Presser, S. (2020). epub2txt-all, in shawwn/scrap. github.com/shawwn/scrap
Rae, J. W., Potapenko, A., Jayakumar, S. M., & Lillicrap, T. P. (2019). Compressive Transformers for Long-Range Sequence Modelling (PG-19). arXiv:1911.05507.
Raffel, C., Shazeer, N., Roberts, A., et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR 21(140), 1–67.
Soldaini, L., Kinney, R., Bhagia, A., et al. (2024). Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv:2402.00159.
Su, D., Kong, K., Lin, Y., et al. (2024). Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset. arXiv:2412.02595.
Sun, M., Yin, Y., Xu, Z., Kolter, J. Z., & Liu, Z. (2025). Idiosyncrasies in Large Language Models. ICML 2025. arXiv:2502.12150.
Tao, J., Peng, M., Li, Y., et al. (2026). Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training. arXiv:2607.28109.
Touvron, H., Lavril, T., Izacard, G., et al. (2023). LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971.
Wang, B., Xu, C., Zhao, X., et al. (2024). MinerU: An Open-Source Solution for Precise Document Content Extraction. arXiv:2409.18839.
Xie, S. M., Pham, H., Dong, X., et al. (2023). DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. NeurIPS 2023. arXiv:2305.10429.
Ye, J., Liu, P., Sun, T., et al. (2024). Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance. ICLR 2025. arXiv:2403.16952.
Yun, L., An, C., Wang, Z., Peng, L., & Shang, J. (2025). The Price of Format: Diversity Collapse in LLMs. arXiv:2505.18949.
Zhang, X., Xiong, W., Chen, L., et al. (2025). From Lists to Emojis: How Format Bias Affects Model Alignment. ACL 2025, 26940–26961. arXiv:2409.11704.
Zhang, Y., Zhao, Z., Zhang, W., et al. (2026). Scalable Visual Pretraining for Language Intelligence. arXiv:2607.09657.
Zhao, R., Meterez, A., Kakade, S., et al. (2025). Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining. COLM 2025. arXiv:2504.07912.