Why You Can't Trust an AI Agent to Grade Its Own Work: A Report Claiming -9,500 Lines When the Codebase Actually Grew by +21,489

• Report vs. git diff mismatch: the agent claimed a "~9,500-line reduction," but the actual commit touched 206 files with a net gain of +21,489 lines (+34,934 / -13,445)
• Swarm fact-checking accuracy on a book manuscript: ~150 facts checked across 20 chapters (69,293 words), catching 13 hidden distortions of statistics and dates
• Meta-casebook compression funnel: 11.67 GB of raw JSONL session logs compressed 160x down to 73.8 MB of digests (147 batches, 860 unique cards)
• Optimal parallel wave width: 8-14 subagents running simultaneously (spinning up 44 workers at once causes the whole wave to hit provider rate limits)

When a project accumulates "God objects" — the entire database logic in one file, the entire bot in another — manual refactoring takes weeks. I threw a swarm of 25 subagents at exactly such a codebase, organized into four phases: closing security holes, reviving containers, splitting up monoliths, and migrating features. A day and a half later, the orchestrator produced a glowing report: `database.py`, at 1,004 lines, had been broken into 15 model files; `report_bot.py`, at 1,759 lines, into 11 modules; "185+ tests" had been written; and the commit message proudly read `Net reduction: ~9500 lines of code`.

Six months later, I decided to verify every number in that report using `git` commands. It turned out the structural work had been executed flawlessly — but almost every number in the report lived in a parallel reality.

Why do the swarm's structural claims check out file-by-file, while the

line counts and hours are wrong?

Running `git show --shortstat` on the refactoring commit revealed: 206 files changed, 34,934 insertions, and 13,445 deletions — a net growth of +21,489 lines, where the agent had claimed a 9,500-line reduction. The original file sizes also differed from what was reported (`883`, `1,696`, and `650` lines instead of `1,004`, `1,759`, and `904`), and the diff contained only 164 `test_*` functions, not "185+."

Yet every structural claim checked out down to the individual file: the models package ended up with exactly 15 modules, and the old `database.py` had indeed become an 8-line backward-compatibility wrapper (`from .models import *`). A second, independent swarm pass later found and split 9 more files over 1,000 lines that hadn't appeared on the first list (today, out of 313 files totaling 98,264 lines, only a single 1,013-line module remains).

Numeric distortion arises from three causes: context loss across session restarts, reporting intent before checking the tool's actual output, and double-counting during aggregation. When a script building a casebook from 860 sessions summed the `time_saved` field across cards from a single project — where each session had independently estimated the entire project's savings at "40 hours" — the result was a comically inflated 44,683 hours saved. Any numeric aggregate produced by a swarm needs to be recalculated from the raw primary records by an actual script, not trusted as reported.

How does a pipeline of read-only auditors and fixers work, and why

compress session logs 160x?

To keep the executor from grading its own work, we split the roles: first, 5 parallel read-only auditors run (covering security, bugs, quality, dead code, and duplication), with their write tools physically disabled and a mandatory `modifications_made: false` contract in the YAML header; then 3 fixers act on those reports, which include exact line numbers and reproduction scenarios (such as a `row_factory` race condition on a shared SQLite connection).

The same independent-verification scheme, applied to a 20-chapter book manuscript (69,293 words), checked 150 claims and caught 13 hallucinations (an inverted quote — "91% satisfied" flipped into "90% dissatisfied" — a $2.4 billion licensing deal mislabeled as an acquisition, and a 2032 market forecast of $37 billion cited instead of the current $4 billion figure). And to run a swarm over an archive of 11.67 GB of raw JSONL session logs, we wrote a digest preprocessor:

```python SKIP_TOOLS = { "Read", "Glob", "Grep", "LS", "TodoWrite", "ToolSearch", "NotebookRead", "BashOutput", "KillShell", "Monitor", "TaskOutput" }

# Keep only mutations (Write/Edit), Bash commands, and subagent spawns,

# and cap tool output at TOOLRES_CAP = 250 characters. ```

This compressed the corpus 160x — from 11.67 GB to 73.8 MB (147 batches of `MAXKB=900`, `MAXN=6`), letting a pipeline with stable batch IDs survive three rate-limit crashes and still assemble 860 cards.

Why can the verifier itself be wrong about fresh APIs and oversized waves?

A verifier agent is just as much a probabilistic mechanism as any other, and it can err in either direction. While implementing 173 methods of a large Bot API (growing CLI coverage from 19 to 163 commands, documented in a 2,977-line reference), verifiers cross-checked each method against 4-5 open-source SDKs. Upon encountering a genuinely new feature that hadn't yet made it into older libraries, the swarm unanimously declared it a "scout hallucination" — proving it was real required running automated tests against a fresh fork. Along the way, a parallelism limit emerged: launching 44 verifiers at once caused the entire wave to hit rate limits, while batches of 8-14 agents ran without losses.

A swarm orchestrator with independent acceptance review and auditor/fixer separation is built into [Nomad AI](https://nomadnet.ai) and the cloud service [agent.nomadnet.ai](https://agent.nomadnet.ai). You can learn to manage multi-agent swarms at [Hamidun Academy](https://academy.hamidun.com) (72-hour trial), and the log-compression scripts and reviewer prompts are available in the [Vibe Club](https://hamidun.com/club).