Homemade AI-slop detector catches 5 of 12 signs of machine-written text with code

The author of a Habr piece mapped 12 signs of AI slop, meaning text that a neural network produced and nobody proofread, to the checks in his own detector. Code catches five signs, and two of those only partly. The remaining seven rely solely on rules for an editor. In a test run, the script missed two typical lead-in phrases: "In this article we will look at" and "The conclusion is simple" with a colon.

Why an overall percentage fixes nothing

Online detectors return a number and don't explain what exactly is wrong with the text. The percentage itself is unreliable. In work by Liang et al., seven detectors flagged an average of 61% of TOEFL essays written by non-native English speakers as machine-generated. Such a detector systematically errs where the author writes carefully and cautiously, because it treats that as a sign of "machine" writing.

OpenAI shut down its AI-text classifier in summer 2023 because of low accuracy. If the developer of the models itself could not build a reliable general-purpose tool, it is hard to expect one from a random service that outputs a percentage.

A narrow check instead of a verdict

The author takes a different approach. The script names the specific phrase and the rule that triggered, and each hit can be disputed on its own. The editor sees not "this text is 73% written by AI" but "this lead-in and this rule."

Wikipedia editors follow the same logic. They maintain an open list of signs of AI text and state plainly that a sign points to a possible problem and proves nothing by itself. The script starts from the same assumption: it looks for traces of unproofread text and does not determine authorship. The overall score of this detector has already been covered separately on Habr.

The numbers at a glance

• Total signs of AI slop in the analysis: 12.
• Caught by code: 5, of which 2 partly.
• Left only to the editor: 7.
• Missed in the test run: "In this article we will look at" and "The conclusion is simple:".
• False positives per Liang et al.: an average of 61% of TOEFL essays by non-native English speakers.
• OpenAI's classifier shut down in summer 2023.

Where the line between code and editor falls

The announcement does not list all 12 signs; the full breakdown is behind a cut. But the two missed phrases show the general picture. Even formulaic lead-ins, which look like ideal targets for a regular expression, slip through when the form changes: a different verb, punctuation, word order. This is my reading, not the author's conclusion, but it is logical. Code is reliable where a sign lies on the surface and repeats verbatim. Signs that require understanding meaning stay with the human: empty generalization, lack of specifics, a paragraph from which half the words can be removed with no loss.

How to apply this in practice

If you are building a check for a newsroom or your own blog, the breakdown offers several rules.

• Split the signs into two lists: what the script checks and what a human checks. Don't try to force everything into code.
• Every hit should show the phrase and the rule. Then the editor can accept or reject it in seconds.
• Grow the dictionary from misses. The two phrases that slipped through are a ready-made task for the next iteration.
• Don't use the result as proof of authorship. Be especially careful with texts by non-native speakers: detector errors hit them hardest.
• Keep proofreading as a mandatory step. The script reduces manual work but does not replace it.

The bottom line is simple: a check that explains itself is more useful than a verdict you have to take on faith.