A Cheap Filter Model in Front of Claude Sonnet: Rewording the Question Lifted AUC to 0.95

Almost half of the material the author saves in Telegram for his blog is not needed for the blog. Yet once a day Claude Sonnet dutifully writes a full analysis of every item, wasting tokens. The fix follows the classic cascade pattern: put a cheap model in front of the expensive one and let it decide whether the expensive one is worth calling at all. The author described the results of the first two weeks on Habr.

The task and the setup

The starting situation will be familiar to anyone running a content pipeline on LLMs. Saving to "Favorites" is easy, so everything ends up there: news, links, useful and not-so-useful articles. Sonnet then takes over and prepares an analysis of each item. Some of the material in this stream will never make it into the blog under any circumstances, but every item is paid for.

In front of Sonnet, the author placed a small model he calls Jev. It does not issue a "take it" or "skip it" verdict. Instead, it answers with the probability that an item is suitable for the blog. The cutoff threshold can be moved without touching the model itself. For a task where the cost of errors is uneven, this is more convenient than a binary answer. A missed good item costs more than an unnecessary analysis.

The question matters more than the model

The main takeaway of the post is that the wording of the question put to the filter affected the result more than the choice of the model itself. By the AUC metric, the two versions of the task differed as 0.46 versus 0.95.

The first number is worse than a coin flip: an AUC of 0.5 means the model cannot tell the needed from the unneeded. The second is close to a good classifier. The gap appeared without changing weights and without fine-tuning, purely from how the question was posed. For anyone choosing between "get a bigger model" and "rewrite the prompt," this is an argument for the latter. Measure AUC on a labeled sample first, and only then change the model.

What the announcement tells us

• Input stream: everything the author saves in Telegram.
• Expensive model: Claude Sonnet, one pass per day.
• Share of unneeded items: almost half of the items saved for the blog.
• Filter: the cheap Jev model, answering with a probability.
• Quality: AUC of 0.46 versus 0.95, depending on how the question is worded.
• Observations: a log from two weeks of operation.
• Protection: the code includes safeguards; how they work is explained in the full article.

The announcement does not include prices, the exact threshold, the size of the labeled set, or the false-rejection rate. Without them, the real savings cannot be estimated, and checking them requires the full text.

What you can borrow

The pattern carries over to any pipeline where an expensive model processes a heterogeneous stream: reviewing applications, moderation, triaging support requests, preparing digests. A few practical steps follow from the author's experience:

1. Label at least a small sample by hand and compute AUC before launch. Without this, a flaw in the wording will go unnoticed. 2. Ask the filter for a probability, not a yes or no. The threshold can then be tuned separately. 3. Build in safeguards: decide what happens if the filter is unavailable, answers strangely, or cuts too much. A safe default is better than silently losing material. 4. Keep a log of the filter's decisions and review the rejected items periodically. Two weeks of records show what is not visible at the start.

The method requires neither new models nor new infrastructure. All it takes is a cheap call in front of the expensive one and discipline in measurement.