Habr: How to Measure Template-Like AI Text with Shingles and MinHash

Texts that neural networks write for content marketing and media increasingly look alike: the same opening, three arguments, a paragraph of "conclusions." An analysis on Habr proposes measuring this with metrics: shingles, MinHash and structure indicators. For scale: MinHash with 128 hash functions gives a standard error in the similarity estimate of no more than 4.4 percentage points, which is enough to look for a template across thousands of pieces. The announcement reveals only the main points, so what follows describes how these methods work and where to apply them.

Where the template comes from

A model picks the most probable continuation. At low temperature and with rigid instructions, it arrives at the same phrasings again and again. Fine-tuning on human preferences adds to this: raters more often approve a tidy, neutral, well-structured answer, and the model locks in exactly that style. Research on RLHF records a drop in output diversity relative to base models.

The third factor is the prompts themselves. Teams copy successful instructions from each other, such as "write an article with an introduction, three sections and a conclusion." Different models given the same skeleton produce closely related results.

How to measure similarity

Shingles are sliding windows of n words. A text becomes a set of such windows, and similarity is calculated as the Jaccard coefficient: the size of the intersection divided by the size of the union. For Russian, windows of 3–5 words are usually used, and you will have to tune the parameter to the genre.

The problem is cost: comparing all pairs in a corpus of 10,000 texts means about 50 million pairs. This is where MinHash helps. Each text is compressed into a short signature, and the share of matching positions between two signatures estimates the Jaccard coefficient. Add LSH buckets, and similar candidates can be found without a full brute-force comparison.

Shingles catch lexical repetition but miss an identical skeleton built from different words. That requires structural metrics:

• Jaccard on shingles: the share of n-grams two texts have in common;
• MinHash estimate: a fast approximation of the same value on large corpora;
• distinct-n: the share of unique n-grams within a set of texts; a drop signals monotony;
• self-BLEU: how similar each text is to the rest of the sample;
• spread of sentence and paragraph length: in template-like texts it is abnormally small;
• repetition of sentence openings and headline wording;
• similarity of section order: introduction, list, conclusion in the same sequence.

How to apply it in practice

Start with a baseline. Take 100–200 texts written by humans at your publication and calculate the metrics on them. Then do the same with AI-generated pieces on the same topics. The difference between the two distributions is your real level of template-likeness, not an editor's impression.

Next, set thresholds and build the check into your pipeline. A new text is compared against a recent sample via MinHash and LSH, and if it exceeds the threshold, it is sent back for revision. Structural metrics should be reviewed separately, because a high score on them combined with a low score on shingles means an identical skeleton with different words.

If the metrics are bad, raising the temperature is not the only thing that helps. What works: varied style samples in the prompt, changing the task structure to suit the type of material, randomly choosing an angle from a pre-prepared list, and mandatory factual inserts such as figures, quotes and internal data. The last item matters more than the rest, because a model cannot reproduce a fact it does not have in a templated way.

The method's limitation is simple: it measures form, not quality. A text that is unique by shingles can be empty, and a repeating skeleton is sometimes justified by the genre, for example in a news roundup. The metrics are there to reveal the problem; the decision remains with the editor.