Home-Built AI-Slop Detector Catches 5 of 12 Signs of AI Text With Code

A Habr author mapped 12 signs of AI slop, meaning text produced by a neural network that nobody proofread, to the checks of his own detector. Code catches five of the signs, and two of those only partly. The remaining seven rely solely on rules for an editor. In a test run, the script missed two typical lead-in phrases: «В этой статье мы рассмотрим» («In this article we will look at») and «Вывод простой» («The conclusion is simple») followed by a colon.

Why an overall percentage fixes nothing

Online detectors return a number and do not explain what exactly is wrong with the text. The percentage itself is unreliable. In a study by Liang et al., seven detectors flagged an average of 61% of TOEFL essays as machine-written, even though humans wrote them, and English was not those humans' native language. Such a detector errs systematically where the author writes carefully and cautiously, because it treats that as a sign of "machine-ness."

OpenAI shut down its AI-text classifier in the summer of 2023 because of low accuracy. If the developer of the models itself could not build a reliable general-purpose tool, it is hard to expect one from a random service that outputs a percentage.

A narrow check instead of a verdict

The author takes a different approach. The script names a specific phrase and the rule that triggered on it, so each hit can be disputed individually. The editor sees not "this text is 73% AI-written" but "here is this lead-in and here is this rule."

Wikipedia editors follow the same logic. They maintain an open list of signs of AI text and state plainly that a sign points to a possible problem and proves nothing by itself. The script rests on the same assumption: it looks for traces of unproofread text and does not determine authorship. The overall score of this detector has already been discussed separately on Habr.

The numbers at a glance

• Total signs of AI slop in the breakdown: 12.
• Caught by code: 5, of which 2 partly.
• Left solely to the editor: 7.
• Missed in the test run: «В этой статье мы рассмотрим» and «Вывод простой:».
• False positives per Liang et al.: an average of 61% of TOEFL essays by non-native English speakers.
• OpenAI's classifier was shut down in the summer of 2023.

Where the line falls between code and editor

The announcement does not list all 12 signs; the full breakdown is behind the cut. But the two missed phrases show the general picture. Even boilerplate lead-ins, which look like ideal targets for a regular expression, slip through when the form changes: a different verb, punctuation mark, or word order. This is my reading, not the author's conclusion, but it is logical. Code is reliable where a sign sits on the surface and repeats verbatim. Signs that require understanding meaning remain with the human: an empty generalization, a lack of specifics, a paragraph from which half the words can be cut with no loss.

How to apply this in practice

If you are building a check for a newsroom or your own blog, you can take several rules from the breakdown.

• Split the signs into two lists: what the script checks and what a human checks. Do not try to force everything into code.
• Every hit should show the phrase and the rule. Then the editor can accept or reject it in seconds.
• Expand the dictionary based on misses. The two phrases that slipped through are a ready-made task for the next iteration.
• Do not use the result as proof of authorship. Be especially careful with texts by non-native speakers: detector errors hit them hardest.
• Keep proofreading a mandatory step. The script cuts manual work but does not replace it.

The bottom line is simple: a check that explains itself is more useful than a verdict you have to take on faith.