Autopilot plays games and hunts bugs on its own: 28.7% more accurate than Claude Code

The problem it solves

Manual game testing is expensive and boring. QA engineers spend hours playing through the same level to check that, after yesterday's commit, the character doesn't fall through the floor. Classic automated tests help only partly: they break with any interface change and can't simply "play" and notice that something looks off.

Autopilot tries to close this gap. It is an agent that launches the game, studies its codebase, tries out the gameplay and records bugs. It plays not through the engine's internal API but with a real keyboard and mouse, just like a player.

How it works

Imagine you are building a platformer in Godot 4 and have added a double-jump mechanic. You run Autopilot in the specific-feature check mode. The agent reads the code, works out where the new mechanic lives, launches the game and tries jumping in different situations: at the edge of a platform, next to a wall, while falling. If the character hangs in the air or the jump fires three times, it will land in the final report.

There are four modes in total:

• checking the whole game;
• finding bugs on a single level;
• testing a specific feature;
• reaching a given goal, for example "get to the boss".

The key idea is self-learning. The agent remembers the experience of past runs and plays better over time: it finds the right places faster, wanders less and spends less time on what it has already explored.

What we know about the numbers and the design

The authors claim that Autopilot is 28.7% more accurate than Claude Code and three times faster than their previous agent. An important caveat: these are the developers' own figures. The comparison methodology, the set of games and the criteria for "accuracy" are not clear from the post, so the number should be treated as a guideline, not as an independent benchmark.

Unreal Engine 5, Unity 6 and Godot 4 are supported on Windows and macOS. The agent connects to Claude Code or Codex through your existing subscription, meaning you don't need to pay for a separate LLM. The engine plugin is in an open GitHub repository, but the agent itself runs in the cloud and is closed.

Limitations

The cloud component means the game build and code most likely leave your environment. For studios with strict NDAs, unannounced projects or console SDKs, this can be a blocker. It is worth finding out exactly what is transmitted and where the agent's "memory" is stored.

In addition, support is limited to desktop. Mobile games, consoles and VR fell off the stated list. Cost is a separate question: the agent runs on your subscription, and long play sessions with many steps can quickly hit limits.

Finally, an autonomous tester finds bugs, but it does not replace judging whether the game is fun to play. The feel of the controls, balance and pacing still remain with humans.

What to do in practice

If you are an indie developer or a small studio without dedicated QA, it makes sense to try Autopilot on a few well-defined scenarios: regression after a big commit, checking a new mechanic, a nightly run before release. Start with one level and compare the bugs found with those found manually. That way you will quickly understand its real usefulness.

Product managers and leads should look closely at the model itself: an agent with accumulating memory, running on top of an already-paid coding assistant subscription. This pattern will clearly extend beyond games, into application and interface testing.

The open plugin lets you study how the engine integration is built and, if you wish, adapt the ideas to your own pipeline. Repository: github.com/ramenvr/autopilot. But before adopting it, check the data policy and independently assess accuracy on your own project.