Garage Eight dropped peer approvals for 69 MRs in an internal service using AI review

The team used AI to write code faster, then hit a review bottleneck: the number of reviewers had not increased. At Garage Eight, a single Android feature spanning 35 merge requests could spend a week waiting for two approvals on each. The practical question is straightforward: which changes can go through after a bot checks them, leaving people time to examine errors with genuinely costly consequences?

According to Anna Sabotovich, who leads the rollout of the AI reviewer, 69 MRs were merged in October in the internal development analytics service without another person’s approval. In the reviewer’s own repository, only six of 35 MRs had peer approval. This is a useful case study for teams with growing change queues, but it does not give everyone a blanket license to remove reviewers.

The number of reviewers depends on the cost of an error

Garage Eight divides services into four risk levels. For review purposes, what matters is what happens if a service produces an incorrect result. Domain architects assign the rating based on the worst-case scenario; an unrated service is treated as carrying the highest risk.

For example, an error in an internal experiment might affect only its authors. An error in a payout calculation, however, affects a customer’s money. Requiring the same number of approvals for these changes is administratively convenient, but allocates the team’s attention without accounting for the consequences.

The company started with internal repositories, where the consequences of an error are easier to detect and roll back. Under the intended approach, authors at the two lowest risk levels will be able to merge changes themselves after AI review and successful automated checks. For secondary customer-facing features, the plan is to keep one human reviewer for simple and ordinary MRs, and two for complex ones. A separate step in the reviewer’s workflow determines complexity: authors are not trusted to make that assessment.

At the highest level, where money and regulatory requirements are involved, two human reviewers remain. Until the automatic reinstatement of mandatory human approval is available, the company is retaining people at both upper risk levels.

A resolved comment does not necessarily mean a fixed bug

The first metric looked convincing: 92% of the bot’s comments were resolved. But in the mobile repositories, GitLab blocks merging while any discussion remains open. Among the comments examined on merged MRs, all 647 out of 647 had been resolved. The metric was effectively tracking compliance with a merge requirement.

The team therefore started checking whether code near the commented line changed before merging. It then compared the result with a baseline: how often a random location in the same file changed.

During the third quarter, the bot left 585 serious comments across 12 repositories. Code nearby changed after 219 of them—37%, against a baseline of 7%. Of 27 cases checked manually, 25 involved a change that was indeed related to the comment. The article’s author estimates that around 200 serious issues, including 98 bugs, were fixed before merging during the quarter.

It is important not to overstate the conclusion. A change to nearby code is an indirect sign that a comment was acted on. These figures do not measure the share of all errors detected, nor do they prove that the bot reviews better than a person. But they are far more useful than a count of resolved threads when deciding whether the reviewer is worth its cost in a particular repository.

Bots also need queues, checks and safeguards

Infrastructure quickly became part of review quality. One review using an internal model took around 50 minutes and held up the other MRs. In September, the reviewer went silent for seven days; in October, for another 30 hours. Both times, the API budget had run out. The team subsequently added alerts for silence, stalled checks and the remaining balance.

To evaluate model changes, the team assembled 42 historical MRs; a human had confirmed real problems in 32 of them. A set like this helps compare reviewer versions, although it is insufficient for broad conclusions about reliability.

Endless discussions are another problem: on one MR, the bot reviewed 27 commits and reported 45 findings, with the author’s agent responding to 44. The team considers it necessary to limit the number of cycles and review only the new diff.

The practical takeaway is to start with one repository where the cost of an error is clear and the queue for human review is measurable. Then check which comments lead to useful fixes, how much noise the bot creates and what happens when it fails. Garage Eight still plans to automatically bring humans back when rollbacks increase. That outstanding item is precisely what limits further removal of approval requirements.