Habr Author Tests How Many Edits It Takes to Erase an LLM Watermark

A Habr author set out to measure something that has so far been argued about only in theory: how much you have to change a text before a language model's watermark can no longer be read. The trigger was a claim that Claude already marks its output and that OpenAI is rolling out a similar mechanism. The introduction poses four questions but gives no results: the author defers the measurements themselves to the main part. The questions are well chosen, though, and they determine what to expect from these marks in practice.

How the mark works

A text watermark is not hidden in the characters. Usually it is a statistical bias in token selection: a secret key determines which words the model weights slightly higher. The reader sees no difference. A detector that holds the key counts the share of "preferred" tokens and compares it with the share expected in random text. Researchers at the University of Maryland described the scheme with "green" and "red" lists in 2023, and Google later released its own variant, SynthID-Text.

The introduction does not specify which scheme Claude and OpenAI use. The author points to two Habr write-ups that explain the mechanics in detail, including ways to bypass it.

This leads to the key property: the signal accumulates with length. A single phrase proves almost nothing; a long text proves a lot. Any word substitution dilutes that signal.

The experiment's four questions

• Rewrite depth. How much of a "wall of AI slop" must be rewritten before the detector reliably does not flag it. The measure is the share of changed tokens.
• Two marks at once. If another model, also watermarked, does the rewrite, does the text end up carrying two traces, and how confidently can each be found.
• Human text edited by AI. Fixing errors and punctuation, rewriting a couple of paragraphs: where is the line past which a text counts as generated.
• Code. Where a mark can live at all: in comments, in extra parentheses, in variable names.

The last question is the most interesting technically. A statistical mark needs freedom of choice: the model must have several nearly equivalent continuations. Prose has plenty of such spots. In code, syntax and APIs rigidly dictate most tokens, leaving less room for bias. This is a general principle of such schemes, not the author's finding. Whether the measurements confirm it for commercial models will become clear in the main part.

What is still unknown

As the text puts it, detectors for Claude and OpenAI have not appeared in public access. The author assumes they will at some point, and that texts already written could then be checked retroactively. That is the main risk he describes: a report for a client, a scientific paper, or prose you are personally responsible for.

Detection has a flip side. Any confidence threshold produces false positives, especially on short texts and on text lightly edited by a model. That is why the results for the third question matter not only to those who want to remove a mark, but also to those who could be accused without grounds.

Practical takeaways

With no public detectors and no published measurements, it makes sense to act calmly and ahead of time.

• Keep drafts and edit history. This is the best evidence of authorship, better than any detector.
• If you work with a client or a journal, find out their AI-assistance policy in advance and put the agreement in writing.
• Do not treat a detector hit as a verdict: a mark gives a probability, not proof.
• Confidential data should not be sent to cloud models to "polish" a text, regardless of watermarks.
• Wait for the main part of the experiment and check its conclusions against the methodology: sample size, text language, model versions.

The Habr experiment may give the first benchmarks for editing thresholds. For now, these are questions, not answers.