Agentic search on open-source models: who it found, what it cost, and where it all broke

A developer on Habr ran a series of experiments on agentic search using budget open-source models. The main idea was to figure out the minimal tech stack needed for reliable information retrieval with verifiable sources — without hallucinated links. The task was concrete: find HR contacts and AI specialists at Sber to determine the minimum toolset an agent needs for trustworthy internet search.

A $20 stack

The author chose a minimal set of tools. Agent Hermes is an agentic system with tools for searching, opening pages, and recording results. Ollama is an open-source model provider with a free cloud tier (all six models in the first two stages ran on the free tier).

OpenRouter is a model marketplace for access to paid requests and JEV. JEV is a small judge model that doesn't search itself but verifies found people against three criteria: confirmation of role at the company (0-1), specialist category (HR/AI/ML/other), and whether they currently work there (0-1). Grok bot from X/Cursor orchestrates the agents and manages the experiment.

Search ran through SearXNG (proxy) and Yandex as the primary search engine, since Google was blocking requests with CAPTCHAs.

First test: bare model vs. skills

The author tested one model (gemma4:31b) on a specific task: find 10 well-known Sber HR staff via developers.sber.ru. Without skills, the model found almost all the needed people (recall of 0.97), but with 8 extraneous results (precision of 0.68). Adding skills raised precision to 0.88 but slowed the search from 52 to 121 seconds. The best result came from the bare model plus the JEV filter — recall of 0.97, precision of 0.97, and a cost of roughly $0.001 per run. Conclusion: the bottleneck isn't the model, it's its surrounding tooling.

Second test: six models on two tasks

Next, six open-source models were tested on two scenarios:

• H1 (find HR and recruiters): gemma4:31b found 3 confirmed contacts (the best result), nemotron-3-super found 1, gpt-oss:120b found 1, gpt-oss:20b found 0, nemotron-3-nano:30b found 1, and nemotron-3-ultra found 0-4. Conclusion: HR search barely worked for any model.

• H2 (find AI/ML specialists): Only three models crossed the threshold of 5 confirmed contacts in both runs. gemma4:31b found 8-15 contacts (a 1.9x spread). nemotron-3-super found 6-10. gpt-oss:120b found 6-7 contacts — the most stable result, with 66.5% of results coming from Sber's official sites, taking 113 seconds to search.

nemotron-3-ultra found the most people overall (30-33), but quality was lower: JEV accepted only 7-8 of them, and the cost was high ($0.19-0.38 per run). nemotron-3-nano:30b hallucinated one link — the model took an address that wasn't from the search results. gpt-oss:20b proved unstable: three out of four runs returned malformed JSON. Hallucinated links were virtually absent in all other cases.

Why Yandex, not Google

Google and Bing blocked agent requests with CAPTCHAs. Google's Custom Search API is shutting down on January 1, 2027 and is unavailable to new customers. The official API for full web access requires a minimum of $30,000 per month. Bing's Search API has been closed since August 11, 2025. In 2026, Google broke SearXNG workarounds at least five times, and starting in August added TLS fingerprint verification for clients. Yandex indexes the Russian-language web better: Habr, hh.ru, vc.ru, TAdviser, and Rusprofile all appear as relevant results. On public SearXNG instances (as of October 2026), the average error rate was: Yandex 0%, Google 10%, DuckDuckGo 60%, Brave 85%.

Paid models

The author also tested frontier models: Deepseek, Kimi K3, GLM-5.3 (first round) and Mimo, GPT-6 Luna, Gemini 3.8 flash, Claude Sonn (second round). Full results for the paid models are given in the original article.

Conclusions

The new discovery is that the bottleneck in agentic search isn't model quality, but its surrounding tooling (skills and instruments). A bare model plus a cheap external filter outperforms a model with complex built-in skills. Searching for HR contacts through open sources is practically impossible. Hallucinated links can almost entirely be avoided with proper filtering.