Habr author builds agentic search on open-source models and tests where it breaks

Habr has published a write-up of an experiment that many teams are repeating today: agentic search on open models instead of a ready-made service like Deep Research. The author states his motive up front. Saving on a subscription does not interest him. He wants to find the minimum technology stack an agent needs to locate specific information, cite a source that can be checked, and not invent a link. The announcement contains no figures. The cost of the runs and the specific failures are left to the full text, so here we rely only on the stated problem setup.

What problem the author is solving

The title promises three things: who the agent found, how much it spent, and where everything broke. That is a good framing for an evaluation. Agentic search is easy to show off with a successful example but hard to measure honestly. The task of "finding a specific person or fact" was not chosen at random. It has a reference answer, so you can tell a real find from a plausible fabrication.

The requirement of "control over your own sources" deserves separate mention. Commercial agents browse the web on their own, and the user sees only the final report. In a self-built system you can restrict domains, log every request, and save the page a quote was taken from.

What such a stack usually consists of

The announcement does not list the author's components, so what follows is the general scheme used to build such systems. It is not a description of his design.

• Search layer: a meta-search engine or a results API that returns a list of URLs.
• Page loader: fetches the HTML, cleans it into readable text, and copes with JavaScript and blocking.
• Planner: an open LLM that splits the question into sub-queries and decides when to stop.
• Extractor: pulls the needed fact out of the page together with a verbatim quote.
• Verifier: checks that the URL exists and that the quote is actually on the page.

The last item answers the author's central question. A fabricated link appears when the model builds its answer from memory rather than from a loaded document. If every URL in the answer passes a "page opened, quote found" check, a hallucinated link turns from an invisible error into a rejected result.

Where such systems usually break

The title says outright that something broke. The typical failure points in agentic search are well known, and the author probably describes some of them.

First: search result quality. If the search layer does not return the right page, even a strong model will neither invent it nor find it. Second: long chains of steps. Small open models hold a plan worse across dozens of tool calls and tend to loop or end the search prematurely. Third: context. Raw pages fill the window quickly, and the tokens spent on reading directly determine cost. Fourth: namesakes and outdated data. An agent can confidently stitch together facts about different people.

How to use this approach in practice

If you are building a similar system, take the evaluation criteria from the author's setup, not the specific architecture. Put together a set of questions with known answers and track several metrics on it:

• share of answers with a working link;
• share of answers where the quote appears verbatim on the page;
• cost per successful answer, not per run;
• number of steps before stopping and share of runs that loop.

The cost of a successful answer matters more than the price of a run. A cheap model that fails half the time can end up costing more than an expensive one. To compare your stack with the author's results, you need his actual figures, and they are only in the full article on Habr.