Google unveils Auto-Diagnose, an AI system that finds the cause of integration test failures

Google has introduced Auto-Diagnose, an internal LLM-based system that analyzes the logs of failed integration tests, pulls out the key lines on its own and posts a diagnosis directly in code review. For large engineering teams, it is an attempt to remove one of the costliest hidden expenses of development: the hours, and sometimes days, spent manually hunting for the cause of a failure across dozens of log files.

The problem is measurable at Google. In an internal survey of 6,059 developers, diagnosing integration test failures made the top 5 most frequent complaints about engineering tools. A follow-up survey of 116 engineers showed that 38.4% of such failures take more than an hour to resolve, and 8.9% take more than a day. For unit tests, those figures were 2.7% and 0% respectively.

The reason is clear: an integration test almost never fails in one obvious place. A typical case involves a separate test driver, a set of services inside the system under test, logs scattered across different components, and a mass of warnings and errors unrelated to the root cause. In Google's research, the median failing test had 16 log files and 2,801 log lines.

Auto-Diagnose plugs into the existing development workflow. When an integration test fails, the system automatically receives the event, collects logs from the test driver and SUT components at INFO level and above, merges them into a single stream and sorts them by time. It then sends all of this, together with component metadata, to Gemini 2.5 Flash.

The model works without fine-tuning on Google-specific logs: the bet is not on fine-tuning but on a strict prompt and integration into the workflow. The prompt forces the model to proceed step by step: find the log sections, identify the component where the failure occurred, check the context, and only then formulate a conclusion.

The key point is a ban on guessing. If the logs contain no lines from the very component that failed to start or did not become healthy, the model must not speculate but state outright that there is not enough data. The answer is then converted to a standard format and published in Critique, Google's internal code review system, where the developer immediately sees the conclusion, the investigation steps and the most relevant log lines.

By the numbers, the system looks less like a lab prototype and more like a genuinely battle-tested internal tool. In a manual review of 71 real failures from 39 teams, Auto-Diagnose correctly identified the root cause in 64 cases, an accuracy of 90.14%. Google then rolled it out to all integration failures on code changes across the company, starting in May 2025. Over that time the system has handled 52,635 unique tests, 224,782 runs, 91,130 code changes and 22,962 authors.

The median time to post a diagnosis in code review was 56 seconds, and the 90th percentile was 346 seconds, so the result usually appears before the engineer has fully switched to another task. On average, one run consumes 110,617 input tokens and generates 5,962 output tokens.

Feedback is also solid: out of 517 responses from 437 developers, the share of Not helpful ratings was 5.8%, below Google's internal 10% threshold for such tools, and on helpfulness Auto-Diagnose ranked 14th out of 370 systems that post findings in Critique.

There is also an important side benefit. Seven errors from the manual evaluation turned out to be not failures of the model as such but problems in the logging infrastructure: in some cases the test driver logs were not saved after a crash, in others the logs of the failed component itself. Similar "more data needed" answers later helped uncover about 20 further infrastructure problems.

So the main significance of Auto-Diagnose is not only that Google is speeding up the analysis of test failures. The company is showing a more practical pattern for using LLMs in development: not asking the model to fix code blindly, but building it into a bottleneck of the process, giving it strict rules against guessing, and returning the result right where the engineer already works. For large teams this may be a more valuable scenario than yet another "AI coding assistant," because it shortens the time to understanding the cause of a failure, and that is most often what holds up a release.