Why Language Models Wreck Your Document's Structure When You Trust Them to Edit

Editing documents with a language model looks like the obvious solution: paste in the text, give an instruction, and get a result without hours of manual work. In practice, the document that comes back is often different: headings shifted, paragraphs lost, or passages reworded that nobody asked it to touch. Instead of editing, the structure quietly degrades.

Generation vs. editing

The core tension lies in the transformer architecture itself. Language models are trained to predict the next token from context. They do not "edit" the way a word processor does. When a model gets the instruction "fix the grammar in the third section," it does not apply a targeted rule to specific lines. It generates what it thinks the updated text should look like as a whole. For an LLM, the boundary of "edit only this" is fundamentally blurry: the model weighs the entire context and generates a new version rather than applying atomic changes. This is especially visible in large documents: the more "material" there is, the stronger the drift from the original.

The lost-in-the-middle phenomenon

Even models with a 200k-token context window run into structural problems with long documents. Studies consistently find a "lost in the middle" effect: models retain information from the beginning and end of the context well, but systematically lose details from the middle. The longer the document, the stronger the effect. For real files, this means structural elements in middle sections are the most likely to get distorted.

Typical casualties:

• Nested lists: nesting levels get flattened to one
• Tables: lose column alignment or are converted to prose
• Cross-references: turn into plain text with no anchors
• YAML front matter and custom tags: deleted as "unnecessary clutter"
• Section numbering: breaks after any insertion or deletion of content

Formatting as tokens without semantics

Markdown, HTML, LaTeX: the model sees all of it as ordinary tokens, not as syntax with rules. To a neural network, the `##` characters are simply two hash signs, not "a second-level heading consistent with the table of contents." Formatting patterns are reproduced by statistical analogy with training data, not by explicit syntax rules.

The result is predictable: inconsistent indentation, mixed-up heading levels, broken link anchors, random switching between formats within one document. Each of these flaws is hard to spot on its own, but together they make the document unfit for automated processing or publication.

"The model does not understand your document.

It understands the probability distribution of the next token in its context."

Instructions vs. patterns

Another factor is the competition between the user's explicit instruction and the statistical patterns absorbed during training. If standard Markdown dominated the training data, the model will gravitate toward it even against an explicit prohibition. A non-standard corporate template, the specific markup of technical standards, an author's own structural style: all of it is vulnerable to the "memory" of the training corpus.

Long instructions also get "diluted" as generation proceeds. By the end of a large document, attention to the constraints in the original prompt weakens, and structural drift grows.

What this means

The practical takeaway is straightforward: LLMs handle targeted tasks better than comprehensive editing of a whole document in a single pass. For complex structured files, such as technical specifications, legal contracts, and academic papers with cross-references, it makes sense to split the document into isolated sections and work on each separately. Instructions should be as explicit as possible: "change only this paragraph, leave the rest alone." After any LLM edit, an explicit check of structural elements is required, because they are the first to suffer.