Live Nov 18 · Cohort 3Nov 18, 19:00 MSK — free hands-on AI agents live workshop · Live Cohort 3 starts Nov 23Join Nov 18 Live →Cohort 3 (Nov 23)

1,000 AI agents wrote a chatbot with no neural network: win rate rose from 0% to 26%

Hugging Face has unveiled PyBot, a chatbot with no neural network inside: just if-else branches, regular expressions and lookup tables. A swarm of 1,000+ AI agents wrote the code across 5,000+ versions, producing 640,000 lines of Python. In SmolTalkBench, where an LLM judge compares answers, the bot's win rate rose from 0% to 26%, on par with the small SmolLM-360M-Instruct model. The practical takeaway: this approach is worth trying for narrow tasks that must run without a GPU.

AI-processed from TG: @cgevent (Метаверсище и ИИще); edited by Hamidun News
1,000 AI agents wrote a chatbot with no neural network: win rate rose from 0% to 26%
Source: TG: @cgevent (Метаверсище и ИИще). Collage: Hamidun News.
Listen to article
Article Fact Sheet & Key Figures
ParameterDetails
Primary sourceTG: @cgevent (Метаверсище и ИИще) ↗
Editor-in-Chief
Key metric #1The agents wrote 550,000 lines of tests themselves
Run it with python -m pybotPython 3.8+, no GPU, no libraries
Key metric #3The 26% win rate was counted by an LLM judge, not by human raters

A chatbot without a neural network is usually a "cabinet of canned answers": ask "where is my order?" and a rule fires; rephrase the question slightly and you get "I don't understand." Hugging Face tested how far this approach can go when the rules are written not by a human but by a swarm of more than 1,000 AI agents.

The result is PyBot: a chatbot of 640,000 lines of Python with not a single neural network weight inside. In SmolTalkBench, a language model acts as judge and compares the answers of the bot and other systems. The bot wins roughly 26% of comparisons, the same as the small neural network SmolLM-360M-Instruct (360 million parameters).

The point of reading about it is not to replace LLMs but to learn a lesson: a long "write, test, improve" loop with automated evaluation delivers a surprisingly strong result even with ordinary code.

What exactly did they build?

PyBot is a dialogue bot with no weights, no matrix multiplications and no transformers. Inside there are only if-else branches, regular expressions and lookup tables. The bot was not trained: no dataset, no gradient descent. A swarm of agents wrote and rewrote all of the code.

The scale is notable. According to the authors, the agents generated and evaluated more than 5,000 versions of the bot. The result is a codebase of 640,000 lines of working code plus 550,000 lines of tests, which the agents also wrote themselves. That is the size of a large commercial product, not a teaching script.

Running it is simple: you need Python 3.8+, no GPU or external libraries are required, and the command python -m pybot is enough. There is a demo on Hugging Face Spaces, so you can try it without installing anything.

What does "26% win rate" mean?

In SmolTalkBench, an LLM judge compares answers, and the "win rate" is the percentage of comparisons in which PyBot's answer was preferred. At the start of the evolution this figure was 0%; by the end it had risen to 26%. In the authors' wording, this brought the deterministic bot to the level of SmolLM-360M-Instruct.

Read the number carefully. 26% does not mean "the bot is better than the model"; it means "the bot wins in roughly a quarter of cases." The parity applies to this benchmark only and to evaluation by another language model, and an LLM judge may favor a particular answer style. The available data does not say how the bot would handle unexpected questions from real users. Also, we are relying on a summary of the project and on links to the repository, demo and benchmark; for details of how the swarm works and how versions were selected, it is better to check the huggingface/pybot repository.

What to do with this in practice

First: measurable feedback decides everything. 5,000 iterations and half a million lines of tests are not "asked a chat for a function" but a continuous engineering loop. If you build agents for code, start with a benchmark and tests: without them, the swarm would have turned into a garbage generator.

Second: a deterministic solution has real advantages: instant startup on a CPU, predictable behavior, zero dependencies, and the ability to read and debug every rule. For narrow scenarios such as an internal assistant, support with a limited set of topics, or embedded devices, this may be a reasonable alternative even to a small model.

Third: the limitations are obvious. 640,000 lines of rules form a monolith that people are unlikely to want to edit by hand; extending it will most likely fall to the same swarm. And rules cannot cheaply reproduce a neural network's ability to understand unfamiliar phrasings.

Bottom line: PyBot does not make LLMs obsolete, and the Nvidia joke remains a joke. But the project shows that the boundary between "you need a model" and "code is enough" shifts when hundreds of agents write the code with automated verification. If your task has a clear quality metric, it is worth considering whether a swarm of agents with plain Python could solve it.

Story

Key Questions Answered

What exactly did they build?

PyBot is a dialogue bot with no weights, no matrix multiplications and no transformers. Inside there are only if-else branches, regular expressions and lookup tables. The bot was not trained: no dataset, no gradient descent. A swarm of agents wrote and rewrote all of the code. The scale is notable. According to the authors, the agents generated and evaluated more than 5,000 versions of the bot.

What does "26% win rate" mean?

In SmolTalkBench, an LLM judge compares answers, and the "win rate" is the percentage of comparisons in which PyBot's answer was preferred. At the start of the evolution this figure was 0%; by the end it had risen to 26%. In the authors' wording, this brought the deterministic bot to the level of SmolLM-360M-Instruct. Read the number carefully.

What to do with this in practice?

First: measurable feedback decides everything. 5,000 iterations and half a million lines of tests are not "asked a chat for a function" but a continuous engineering loop. If you build agents for code, start with a benchmark and tests: without them, the swarm would have turned into a garbage generator. Second: a deterministic solution has real advantages: instant startup on a CPU, predictable behavior, zero dependencies, and the ability to read and debug every rule.

ZK
Zhemal Khamidun — CPO at Alpina Digital, product manager of AlpinaGPT. Building AI products since 2019. Hamidun News is a pet project that grew into a production aggregator with 6,000+ readers.
✦ Unified AI Workspace · Nomad

Test Frontier LLMs & Autonomous Agents in One Workspace

Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.

Try Nomad AI →Launch Nomad Agent
What do you think?
Loading comments…