Momentic's agent found 269 bugs in an hour — three times more than Codex

Automated tests are good at catching issues a team already suspects. If a developer writes a scenario that says “open the cart, click checkout, see the confirmation,” the system will reliably repeat exactly that. But it will not try double-clicking a button, going back midway through checkout, or opening a form on a narrow screen unless someone explicitly adds those steps.

Startup Momentic proposes a different approach. Its QA agent, Mo, gets access to a running application and behaves like a persistent user: it inspects screens, maps available actions, launches parallel checks, and documents the problems it finds. It does not need the source code to do this.

In a benchmark published by the company, Mo found 269 confirmed bugs in 60 minutes. Codex with Playwright MCP found 86, Claude Code found 59, and Devin found 27. That sounds impressive, but the main takeaway is not that “one agent beat the others.” The more interesting point is that testing is gradually breaking away from prewritten scenarios and turning into independent product exploration.

How the agent finds what is missing from the test case

Consider a delivery service. After another update, conventional tests check login, restaurant search, and order placement. Everything is green. However, when a user returns from the payment window, the selected address is suddenly reset. The team did not anticipate this path, so there is no test for it.

Mo begins by building an application graph: which screens are available and which actions connect them. It distributes large sections among separate agents. One explores authentication, another the cart, and a third payment. Specialized components analyze screen images, locate the necessary interface elements, and track state changes.

To prevent parallel checks from corrupting one another's data, the agents reserve shared resources. Several processes can read the same object simultaneously, but only one is allowed to modify it. This matters for scenarios involving a shared account, orders, and test records in a database.

This approach does not require hard-coding an XPath, button label, or specific aria attribute for every action. The agent receives a goal in plain language and works out how to accomplish it. When the layout changes, this can make checks less brittle, although it does not completely eliminate flaky tests.

The numbers look strong, but this is a vendor benchmark

The dataset included 1,396 labeled bugs across four open-source applications: Gitea, Metabase, Twenty, and Saleor. In one hour, Mo achieved 19.3% recall with 84% precision. Codex achieved 6.2% recall, while its precision was slightly higher at 86%.

In other words, Mo found substantially more known defects, but its reports contained slightly more false positives. There is also a cost tradeoff: one confirmed bug cost $2.67 with Mo versus $0.80 with Codex. Based on these figures, Mo is more effective for broad product exploration, while Codex looks more economical on a limited budget.

The results should be treated cautiously. Momentic prepared the benchmark, the set of conditions, and the evaluation system. A language model performed the initial validation, after which a human reviewed the results. The company released the dataset and test environment, so the experiment can be reproduced, but the reported figures do not yet replace an independent comparison on your own product.

What teams should do now

There is no need to throw out Playwright or Selenium. Scenario-based tests remain useful for critical and predictable paths: login, payment, permission assignment, and data persistence. Agentic exploration makes more sense as a second layer for finding unknown problems.

A practical pilot is straightforward: run the agent against the preview environment for every major pull request, cap its time and budget, and manually confirm the defects it reports. Then compare how many unique bugs it found, how many reports were false, and how many hours of manual QA it saved.

If the agent consistently finds failures that existing tests miss, it can be added to the release process. If it merely repeats known checks and generates noise, impressive numbers from someone else's benchmark offer no value to your team.