Habr Author Trained a Random Forest on 81 Articles and Caught Data Leakage and LLM Copying

Some Habr articles attract a couple of thousand views, while others are read hundreds of thousands of times. The author of a piece about the Kwork freelance marketplace that received 30,000 views decided to investigate why, using Python and machine learning. The result was a small but instructive experiment in which the mistakes proved more interesting than the final numbers.

How the data was collected

Training required a contrasting sample: 80 articles by other authors from the Freelance hub, half from the latest feed and half from the all-time best list, plus the author's own article. That produced 81 rows in total. For each article, the author recorded title length; the number of words, images, links, and code blocks; and views, converting values such as "30K" into integers. He also saved the first 3,000 characters of each text for evaluation by a language model.

Next came LLM labeling: title clickbait, introduction quality, topic relevance, and readability, each on a scale from 1 to 10. The free Gemma model accessed through OpenRouter immediately returned a 429 error because the model pool was overloaded. The author switched to a local Qwen3 4B running in LM Studio on a Mac mini M4. The model processed all 81 articles, and every one received the same scores: 7, 6, 8, and 7. The reason was mundane: those were the exact numbers the author had included in the prompt as a JSON example. Instead of analyzing the texts, the 4-billion-parameter model simply copied the template.

The lesson is straightforward: labeling like this needs validation, for example by checking the number of unique values in a column. The author added that check and replaced the model's scores with features calculated using regular expressions: whether the title contains a question, numbers, or trigger words such as "money" or "client," and the number of images per 1,000 words.

What the Random Forest showed

Using log-transformed view counts, the model assigned the following weights. Title length accounted for 34.5% of the influence, text length for 26.3%, and links and images together for 33.7%. Code blocks contributed only 0.3%, which makes sense for a freelance hub: readers come for workplace situations, not snippets.

The group comparison looked like this. Among the top 20% of articles, with a threshold of 148,000 views, the average title was 46.9 characters long. Weak articles averaged 61.9 characters, while the author's title had 71. Top-performing articles contained about 1,774 words, compared with 1,255 for weak articles and 766 for the author's piece. Top articles averaged 4.2 images and 4.8 links; the author's article had one image and no links.

Data leakage and an honest test

The author then built a predictor: a classifier estimated the chance of reaching 25,000 views, while a regressor predicted the number of views. He compared the original article with an "improved" version featuring a shorter title, and the original unexpectedly won. There were two reasons. First, his own article with 30,000 views remained in the training set, and trees with unrestricted depth simply memorized its parameters. This is classic data leakage. Second, adding a number to the title was penalized: numbers appeared three times more often in weak articles than in top-performing ones, 0.3 versus 0.1, because the latest feed contained promotional case studies.

After the correction—the article was removed from the training set and tree depth was limited to max_depth=4—the results became logical. The original structure received a 26% probability of becoming a hit and a forecast of 14,795 views. The improved version, with a 46-character title, 1,800 words, 4 images, and 5 links, received 41% and 34,919 views. The original actually achieved 30,000 views, twice the forecast: the model sees only the "packaging," not the topic. To estimate dispersion, the author added the interquartile range of predictions from all 200 trees.

Key takeaways

The limitations are worth remembering: 81 records from a single hub, sampling bias from comparing the latest feed with the all-time best articles, and no evaluation on a held-out dataset. The figures are guidelines rather than laws. Two lessons, however, are universal. First, do not put a ready-made output example containing real numbers into the prompt, and always validate responses from small models. Second, the object used to test a model must not appear in training, and tree depth should be constrained.

Practical guidelines for publishing in a similar hub are a 45–50-character title, 1,500–1,800 words, at least four illustrations, and 4–5 links. The author has written a new article using these parameters and promises to compare its actual views with the model's predicted range.