AI Editors Silently Rewrite Your Text: How to Measure the Share of Edits Before Choosing a Model

You hand an article to a neural network with the request "fix only the errors." What comes back is a smooth text in which the author is no longer recognizable. Sound familiar? This is not the whim of a single model but default behavior, and researchers have already tried to measure it.

The request "grammar only" does not protect meaning

In a 2026 preprint, researchers from Berkeley, Google DeepMind and other organizations compared the edits of humans and three AI models on 86 essays. People edited in small ways and in different directions. The models moved the text far from the original. A direct prohibition did not help: the meaning changed even after an expert's remarks and a request to fix only grammar. In a neighboring experiment, among those who wrote actively with a neural network, the share of neutral essays with no position on the question rose by almost 70%.

The practical conclusion: you cannot choose an editor by the feeling that the text "got better." Smoothness is precisely a symptom of rewriting.

Rankings are no help here. Kimi K3 is in the top twenty of the Russian slice of Arena, DeepSeek V3.2 ranks 4th on llmarena.ru, Qwen3.5-27B is first in the long-text column of MERA Reason, and Grok 4.7 is 13th in the AI-Stat public ranking. But these tables measure dialogues, role-play scenarios and long texts, not the editing of someone else's article. Moreover, places from different tables cannot be compared with each other: the participants, tasks and dates differ.

How to measure how much the model changed

First, the prompt. List the permitted edits by name (spelling, punctuation, grammar, obvious typos), forbid everything else and add "if in doubt, do not edit." Ask it to return the full text and a list of edits in the format "was → became." If the text contains more changes than the list, the model edited silently.

Next, we count. The difference is shown by the command git diff --no-index --word-diff original.txt edited.txt. The share of changes comes from a short Python script with difflib: split both texts into words and punctuation marks and take one minus the ratio of SequenceMatcher. Tentative thresholds: up to 5% is a targeted edit, 15% and above is a rewrite. But first run the same metric on a human editor's edit: that is what should serve as the benchmark.

There is a catch. If the model rearranged paragraphs, a word-level count will see a deletion and a new insertion, even though there is no new text. For such cases, Devatine and Abraham propose a metric based on LZ77 compression. A rough approximation is given by standard zlib: compress the original together with the edited version and see how much was added. The zlib window is 32 KB, so cut a long article by sections. A high word_share with a low compression_share most likely means block rearrangement. If both values are high, the model wrote the text anew. This is a simplification, not the authors' code.

Checklist for a run on your own article

• Take an article of 3–5 thousand characters and run each model in a new conversation with the same prompt.
• Disable or limit reasoning mode. In a crash test of 27 models, Chinese models with it spent 51,000–68,000 tokens, while GPT-6 Astra and Gemini 4 Argon stayed within 6,461–9,841. You will pay for the difference in time and money.
• Calculate word_share and compression_share, then search for Chinese characters with a regular expression. In a blind test they were inserted by Tencent Hy4 Preview and NVIDIA Nemotron 3.5 Super VL, while Qwen, DeepSeek and Kimi were not named among them. But a single foreign word per article is easy to miss by eye.
• Check the diff against the model's list of edits: every discrepancy means a silent edit.
• Read the changed passages with your own eyes: a script cannot see that the meaning has gone.

Bottom line

The thresholds and code here are starting points, the result depends on the text, and model versions change quickly: in October 2026, Qwen 3.8 Max and DeepSeek V4.1 were already being named among Chinese flagships. So it is convenient to set the run up as a regression test for the prompt and repeat it after every revision. Check registration and payment from Russia for each model on its website in advance.

Editing can be given to the model whose share of changes is closest to a human editor's edit and which makes no silent edits. The smoothness of the result decides nothing in this choice. And what share of changed text do you consider acceptable for your articles?