EvoNova Labs Releases a Chaos Benchmark of 10 Failure Modes for Always-On AI Agents
An agent that answers test questions flawlessly can break after four hours of around-the-clock operation: memory leaks, the gateway starts throwing 429 and 503 errors, and a payment gets duplicated after a restart. EvoNova Labs has described 10 such failures (F01–F10) and released a pure-Python benchmark that runs them in a fraction of a second. That is why agent resilience testing should run in CI separately from answer-quality evaluation.
AI-processed from Habr AI; edited by Hamidun News
When an agent prototype moves from a laptop to always-on production, the usual quality metrics stop telling you whether the system will survive. A model can reason brilliantly in a single call, but after four hours of continuous operation the loop breaks: memory fills up, the provider starts returning 429 and 503 errors, the agent gets stuck calling the same tool, or after an emergency restart it repeats an operation it has already completed. The EvoNova Labs team has collected these failures into a taxonomy, F01–F10, and released a benchmark that can be run in your own CI.
An important caveat: this material comes from the authors of the tools, not from independent research. The text does not include final comparison figures for "naive" versus "hardened" agents, so the suite should be treated as a ready-made checklist and a starting point for tests, not as a proven standard.
Ten Failures That Ordinary Eval Suites Don't Catch
The authors build on the MAST study (arXiv:2503.13657), which analyzed more than 1,600 multi-agent traces. In that work, a significant share of failures came not from a weak model but from the seams between runtime architecture, agents' excessive agreeableness, and the absence of strict boundary checks. The team combined this classification with the OWASP Top 10 for Agentic Applications (2026) and produced a matrix in the awesome-autonomous-agent-resilience repository:
- F01–F03: context degradation, 429/503 errors, and looping. Symptoms: lost system prompt, crashed worker, burned budget. Defenses: a sliding window that preserves the core instructions, a three-phase Circuit Breaker, and a cap on the number of steps.
- F04–F06: sycophancy, injection, and excessive privileges. The agent invents parameters, leaks secrets, or runs an unauthorized DROP. Defenses: a deterministic verifier, a causal graph, data isolation, and human confirmation for dangerous actions.
- F07–F08: sudden process death and memory-driven kills. The result is double charges or double sends, and RSS growth up to SIGKILL. Defenses: an intent log on SQLite WAL (PENDING → COMMITTED) and a bounded LRU cache.
- F09–F10: generated code escaping the sandbox, and hallucinated offers, such as a promised 99% discount. Defenses: a static AST inspector before execution and strict schema validation of policy.
How the Benchmark Works
The file benchmarks/resilience_bench.py is written in pure Python 3.11+ with no external dependencies and runs all ten scenarios in a fraction of a second. It includes two reference implementations: NaiveBaselineAgent, a typical naive loop, and HardenedReferenceAgent with built-in defenses. The final score is the percentage of invariants that pass.
For example, in F07 the test forces a step to repeat with the same idempotency key on SQLite in WAL mode and verifies that the real side effect happened exactly once, not twice. In F09 the inspector parses the syntax tree of a fragment such as ().__class__.__base__.__subclasses__() and blocks it before the interpreter runs.
Run it with: python -m benchmarks.resilience_bench --target compare --json report.json --badge badge.svg. The command produces a JSON report and an SVG badge for the README. The authors run the tests on GitHub Actions for Python 3.11 and 3.12 on every commit.
Other Tools in the Ecosystem
Alongside the benchmark, the authors have released a set of libraries under MIT, and some materials under CC0-1.0.
sci-integrity-guard (the sci-guard check and sci-guard omml commands) checks scientific and analytical texts that agents write. TableConsistencyGuard verifies that the numbers from one experiment match across different tables within a configurable tolerance, and flags "orphaned" percentages in an abstract as ORPHANED_PROSE_METRIC. CitationVerifier carefully extracts DOIs containing parentheses and semicolons and checks them against CrossRef, with a fallback to OpenAlex. LatexToOmmlConverter repairs corrupted literals such as \x0crac instead of \frac and assembles formulas into Word OMML format.
evonova-causal-guard works with causal relationships: directed CausalDAG graphs, an intervention operator, the d_separated (Bayes-Ball) algorithm, and search for adjustment sets via Pearl's backdoor criterion. Its job is to stop an agent from mistaking correlation for cause. TabuMemory stores SHA-256 fingerprints with a TTL and prevents the agent from re-proposing claims that have already been refuted, while DebateArbiter runs Proposer → Skeptic → Judge debates.
The runtime layer consists of agent-bunker (intent log, RSS monitoring, three-phase circuit breakers), orchestration-engine (asynchronous DAG wave orchestration using Kahn's algorithm), agent-security-sandbox (interception of dangerous imports via AST), and agentic-vector-memory (episodic memory with exponential decay).
What to Take From This
The authors' central point is practical: the neural network generates hypotheses and plans, while recovery logs, code isolation, number cross-checking, and reference validation are handled by lightweight deterministic scaffolding. If you are shipping an agent to production, start with three things: idempotency keys for every action with side effects, caps on steps and budget, and static validation of generated code. The F01–F10 matrix is useful as a list for your own stress tests, even if you don't take the authors' code wholesale.
- Nov 22, 2026Agent Ignored AGENTS.md: In a Third of Pushes It Didn't Ask, and Force-Pushed 31 Times
- Nov 22, 2026Wells Fargo and four other US banks stop taking responsibility for customers' AI agent mistakes
- Nov 22, 2026Yandex Grant Funds an LLM Engine: 35B Model on RTX 2080 Ti Runs Three Times Faster Than llama.cpp
- Nov 22, 2026Five Free GitHub Repos Open 1,400+ AI Models Behind One Endpoint
Key Questions Answered
What are the key details on Ten Failures That Ordinary Eval Suites Don't Catch?
The authors build on the MAST study (arXiv:2503.13657), which analyzed more than 1,600 multi-agent traces. In that work, a significant share of failures came not from a weak model but from the seams between runtime architecture, agents' excessive agreeableness, and the absence of strict boundary checks.
How the Benchmark Works?
The file benchmarks/resilience_bench.py is written in pure Python 3.11+ with no external dependencies and runs all ten scenarios in a fraction of a second. It includes two reference implementations: NaiveBaselineAgent, a typical naive loop, and HardenedReferenceAgent with built-in defenses. The final score is the percentage of invariants that pass.
What are the key details on Other Tools in the Ecosystem?
Alongside the benchmark, the authors have released a set of libraries under MIT, and some materials under CC0-1.0. sci-integrity-guard (the sci-guard check and sci-guard omml commands) checks scientific and analytical texts that agents write. TableConsistencyGuard verifies that the numbers from one experiment match across different tables within a configurable tolerance, and flags "orphaned" percentages in an abstract as ORPHANED_PROSE_METRIC.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.