A Product Manager With No Developer Shipped an AI-Agent-Built Extension and Protected It With 89 Guard Tests

The agent writes: "Done, all tests pass." If your project has no developer and you don't read the code yourself, you can either take the agent's word for it or devise a way to verify it. A product manager who has been building a Chrome extension since January with two AI agents, Claude Code and Codex, chose the second option.

As of 28 October he has 593 commits, 78 478 lines of code, 177 511 lines of tests and version 1.0.1 in the Chrome Web Store.

But the same agent wrote both the tests and the code, so the number of tests proves nothing on its own. For example, the product promises to re-read its settings file at least once an hour. The agent changes the interval to a day, the test pulls the new number from the same code, compares it with itself and stays green.

Meanwhile the user waits a day. To guard against this, the author introduced "guards": tests that protect one specific product promise and fail when it is broken. There are 89 of them now.

If you hand an agent a task that runs for hours, you can borrow this approach.

What the product is and what this was tested on

The extension is called Chat with Multiple AI. It sends one prompt to 15 AI chats at once, including ChatGPT, Claude and Gemini. These are third-party sites, and they change without warning. So the description of how to work with each chat lives in a separate settings file. When a chat changes, the author edits that file, and the fix reaches users without a new version in the store. This has happened twice: on 19 October and 1 November. Before version 1.0.1, the agent itself ran through 70 acceptance scenarios in the browser.

The author splits verification into parts. The build, the tests and the guards make sure the rules in the code and in the documents don't drift apart. The agent in the browser shows what the product actually does. An independent reviewer examines changes that the reviewer did not write. The product manager himself accepts the visible behavior from a checklist in the report: what happens after a click, whether a result appeared, what happens on failure.

A guard: a product promise written as a test

The guard from the example checks two conditions: the re-read interval is greater than zero and does not exceed one hour. If the agent sets it to a day, the second condition fails and the test goes red. The guard's file name is the promise itself, in plain language, for example "a published fix reaches the user within an hour" or "a reminder is not sent until a record is written that it is about to be sent." Without that record, the product would not know after a restart that it had already sent the reminder, and would send it a second time. The list of files reads as a list of product promises.

Guards were added unevenly: 2 in May, 9 in July, 33 in August, 40 in September and 5 in October. The practical template for a request to the agent is this: describe the promise in the user's words, ask the agent to find what in the code upholds it, write a guard with a clear name and show that it fails when the promise is broken.

A green build and two copies of the rules

A green build may check nothing at all. On 27 July the count of type errors (when code confuses kinds of data, for example by treating text as a number) reached zero. Zero can be reached two ways: fix the errors or stop looking for them, for example by turning off strict mode or excluding a folder from the check.

The author has two countermeasures. The first is a minimum threshold: the check counts the files it has read and fails if there are too few. The second is deliberate breakage: a rule is intentionally violated in a copy of the project, and the guard must fail.

In software development this is called mutation testing. The template breaks a copy of the project 42 ways and shows which check caught each break; on the author's computer the demo takes less than a minute.

Another problem is duplicated rules. Claude Code and Codex read rules from different files, CLAUDE.md and AGENTS.md, which were supposed to match. The discrepancies accumulated: 9 lines in March, 12 in June and 21 in August, when the files were 102 and 85 lines long. Over time the rules file grew from 61 lines in February to 179 in October. Since 25 September there has been a single AGENTS.md, and a guard fails if another rules file appears next to it. Skills (instructions for repeatable actions such as a commit) live in each agent's own folders, so a guard compares every pair.

The author is upfront about a caveat: he did not measure how much of the improvement came from the harness and how much from smarter models. He published the harness as an open template on GitHub (yesgenius/agent-harness).