The same Gemma 4 wrote a backend that passed 80% of checks in opencode, but only 41% in OpenHands
Can you hand a local model a real backend with a database, a message queue and project rules? The author ran 4 open models through 2 harnesses (wrapper programs that give an agent a terminal and an editor): 8 pairs, 3 runs each, 24 in total. Gemma 4 in opencode passed about 80% of checks, while the same Gemma in OpenHands passed only 41%. The takeaway is to choose a "model + harness" combination, not a model alone.
AI-processed from Habr AI; edited by Hamidun News
Local open-weight models are increasingly pitched as an alternative to cloud ones: code doesn't leave your machine, and the price doesn't depend on someone else's tariffs. But can you give such a model not a toy function but a real backend, a service with a database, a message queue and project rules, and get code that both works and follows those rules? The author tested this with four models and two harnesses.
A harness is a wrapper program around a model: it gives the agent a terminal and a file editor and monitors the course of the work. Each of the eight "model + harness" pairs ran the task three times, for 24 runs in total. The percentage in the headline is the share of checks passed (service behavior and compliance with project rules), averaged over five steps; for each pair, the median of three runs is used.
The author's short verdict: almost, but not quite. The best pairs scored 80% and 79%, but no one finished without a single failure.
What was tested and how
There is one task: write a FastAPI service with PostgreSQL and RabbitMQ from scratch. It was split into five steps, the way a team lead would do it: skeleton, API, domain rules, database, queue worker. The agent performs each step in a fresh session and continues its own code from the previous step, so errors accumulate: if the service is broken at step two, step three starts from a broken base. The project rules live in an AGENTS.md file and describe a clean architecture with elements of DDD.
Scoring had three layers. The first is 68 hidden tests that hit the service over HTTP and RabbitMQ; the agent can't see them. The second is 8 general rule checks: linter, types, layer boundaries, the agent's own green tests, untouched configs. The third is an LLM judge, Claude Fable, scoring 22 architectural criteria; its score is reported as a separate number. If a mandatory check fails (for example, migrations don't apply to an empty database) or the agent doesn't finish within 2 hours per step, the step gets zero.
The author also tested the tests themselves. Claude Code wrote the reference solution, and there were also 23 corrupted versions of it. At first the tests didn't catch everything: at step one, 3 of 6 corrupted versions passed, and a version whose /ready took 62 seconds to respond instead of 5 got a perfect score. The tests were then extended. The judge was cross-checked against a second judge, GPT-5.6-Sol: the scores matched exactly in 27 of 30 cases.
Who did what
The models: Qwen3.8–27B, Gemma 4 31B, Nemotron 3 Super (a MoE model with 12B of its 120B parameters active) and Devstral 2 (123B parameters). The harnesses: the terminal-based opencode and OpenHands, a heavier platform with a loop detector.
- Gemma on opencode — 80% (range 76–96%). A run takes 72 minutes and 145K tokens. It passes 81% of checks in all three runs. On OpenHands the same model scored 41%, and the run ranges don't overlap.
- Qwen on opencode — 79% (78–96%). With this spread, the author treats the one-point gap as a tie. The best run passed all behavior checks and tripped only on the linter, scoring 96%; with a pre-commit hook that runs the linter automatically, one run reached 99%. But a run takes 227 minutes and 596K tokens: three times longer and four times more than Gemma. The judge gave Qwen 87% against Gemma's 77%, yet on the tests the pairs are equal. Qwen passes a check in all three runs 69% of the time.
- Qwen on OpenHands — 59%; all four zeroed steps were due to the time limit.
- The remaining pairs have medians from 41% to 55% and can't be ranked against each other. Devstral on opencode spends only 114–188K tokens per run, but at steps 4–5 its service is broken in all three runs.
Where the agents broke
The hardest check: the database stops responding, connections stay open, and /ready must return 503 within 10 seconds. Only 2 runs out of 24 passed; of the 22 failures, 17 were hangs. In 15 of those, the code is almost identical: an asyncio timer around a connection from the pool.
It looks right, but exiting the block still waits for a response from the server. Only Qwen on opencode didn't take it on faith. It tested this against a fake PostgreSQL that accepts a connection and then goes silent (about 130 of 169 turns), and then switched to plain asyncpg with terminate().
The second run that passed created a new engine on every request, violating the "one engine per process" rule, and the tests can't see that.
Gemma on OpenHands corrupted files: a cat command with an unquoted heredoc (<<EOF) duplicated the text in 17 of 24 calls, while with a quoted one (<<'EOF') it never happened in 9. Qwen and Nemotron on the same harness always write with quotes: 0 failures in 97. Five files were corrupted in 2 of 3 runs. The agent noticed the corruption but repeated the same command 4 times, and the loop detector cut the session off. The author didn't investigate why the terminal behaves this way: it's a correlation.
The harness's rule turned out to be stronger than the project's rule. The task required finishing a step with a commit, but after the long history was compacted, the requirement survived only in a summary, and opencode's system prompt tells the agent not to commit without an explicit request. In 13 such steps, the agent never called git commit. Moving the rule into AGENTS.md helped in only 1 case out of 6. And in one Gemma run, the agent corrupted the git history itself, leaving the judge with no code snapshots.
What to do in practice
- Evaluate the "model + harness" pair, not the model alone.
- Hand mandatory rules to tools (linter, hooks), not to prompt text.
- Don't accept green tests as proof: check whether they catch deliberately broken code.
- Nov 19, 2026Amazon and Google's Anthropic Stakes Valued at $308 Billion in IPO at $83 a Share
- Nov 19, 2026Google's Gemini 4 Neon runs twice as fast as Carbon and costs 17% less per task
- Nov 19, 2026OpenAI's coding agents would cost $2400 a day per researcher at API prices
- Nov 19, 2026A GitHub Script Catches App Store and Google Play Rejection Reasons Before Review: 542 Stars in a Day
Key Questions Answered
What was tested and how?
There is one task: write a FastAPI service with PostgreSQL and RabbitMQ from scratch. It was split into five steps, the way a team lead would do it: skeleton, API, domain rules, database, queue worker. The agent performs each step in a fresh session and continues its own code from the previous step, so errors accumulate: if the service is broken at step two, step three starts from a broken base. The project rules live in an AGENTS.md file and describe a clean architecture with elements of DDD.
Who did what?
The models: Qwen3.8–27B, Gemma 4 31B, Nemotron 3 Super (a MoE model with 12B of its 120B parameters active) and Devstral 2 (123B parameters). The harnesses: the terminal-based opencode and OpenHands, a heavier platform with a loop detector. Gemma on opencode — 80% (range 76–96%). A run takes 72 minutes and 145K tokens. It passes 81% of checks in all three runs. On OpenHands the same model scored 41%, and the run ranges don't overlap. Qwen on opencode — 79% (78–96%).
Where the agents broke?
The hardest check: the database stops responding, connections stay open, and /ready must return 503 within 10 seconds. Only 2 runs out of 24 passed; of the 22 failures, 17 were hangs. In 15 of those, the code is almost identical: an asyncio timer around a connection from the pool. It looks right, but exiting the block still waits for a response from the server. Only Qwen on opencode didn't take it on faith.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.