Best AI Agent Solves Only 60% of Cloud DevOps Tasks, and 80% of Failures Come from Faulty Reasoning

Imagine asking an AI agent to revoke AWS access for an employee who has left the company. An experienced engineer would check users, roles, keys and service accounts. An agent might disable one account, report "done" and leave other login paths open.

That is exactly the kind of failure CloudOpsBench revealed. It is a new open benchmark, meaning a set of standard tasks for comparing models. The best model tested, Claude Fable 5.

1, solved only 60% of tasks, and GPT-6 Astra solved 53%. About 80% of failures happen not because the model doesn't know the cloud, but because it reasons poorly: it misjudges the scale of the work, doesn't check all the relevant services, or stops too early. If you are thinking of handing cloud infrastructure operations to an agent, these numbers show where to put safeguards.

What CloudOpsBench tests

The benchmark was built by a team led by Ang Chen of the University of Michigan. Version 1.0 includes 45 AWS tasks written by experienced DevOps engineers. The set covers typical work: fixing a service, deleting resources, restricting access permissions, rolling out an update and decommissioning a project.

Each task runs identically in the real AWS cloud and in an emulator. The emulator is convenient for repeatable runs with no risk to real infrastructure. The project is published on GitHub under the Apache 2.0 license, along with a detailed analysis of errors. Extensions for GCP, Azure and OCI (Oracle Cloud Infrastructure) are planned.

Where agents break

Even the leader fails a noticeable share of tasks: Claude Fable 5.1 succeeds 60% of the time, GPT-6 Astra 53%. No model completed all the tasks. According to the authors, in most cases the bottleneck is reasoning.

Roughly 80% of failures fall into three scenarios:

• the agent misjudges the scope of the task;
• it doesn't check all related services;
• it stops before the work is finished.

This is precisely the story of the departed employee's access: the agent knows the commands but doesn't verify that the whole picture is closed out.

The remaining 20% or so of errors are wrong actions. Some of them have irreversible consequences for the infrastructure. For example, a deleted resource cannot be brought back, so the cost of a mistake is higher here than with unfinished work.

What to do in practice

The conclusions below are our recommendations, not results from the benchmark's authors.

First, limit permissions. For a production account, it makes sense to give an agent read access and a narrow set of safe operations, and to route resource deletion, access policy changes and project decommissioning through human approval.

Second, don't take "done" on faith. Since early stopping is a frequent cause of failure, you need an independent check of the outcome: a script or review that lists the affected services and compares them with the expected state.

Third, test on tasks close to your own. Because CloudOpsBench is open and runs in an emulator, you can use it to compare models or prompt variants without risking a real cloud.

Limitations

A set of 45 tasks is small and so far covers only AWS, so the figures apply to a specific benchmark version and specific models. In your infrastructure, with its own conventions, results may differ. But the overall conclusion is clear: what mainly holds agents back in cloud operations is planning and self-checking, not knowledge of commands. Until those skills improve, a human remains the last line of defense before irreversible actions.