IBM and Artificial Analysis build a benchmark: AI agents fail at IT tasks

IBM and Artificial Analysis have introduced ITBench-AA, the first comprehensive benchmark for assessing how well AI agents handle real tasks in enterprise IT environments. The results are sobering: leading models scored under 50%. The dream of autonomous AI engineers remains just that, a dream.

What ITBench-AA tested

The benchmark covered scenarios from real-life IT operations: configuring network infrastructure, managing databases, debugging errors in production systems, deploying applications, and monitoring and optimizing resources. These are not simple written tasks with a single correct answer. Here the AI has to act like a full-fledged engineer: interact with system interfaces, analyze error logs, make decisions under uncertainty, and change approach if the first attempt fails.

The tested models were leading ones: GPT-4, Claude 3 Opus, Gemini Ultra and others. Results were roughly the same: all scored 45–50%. Even more notable, when attempting complex multistep procedures, the agents often got stuck or made critical errors.

What the real problem is

The 50% figure is not just a low score. It signals fundamental limitations. IT work demands not only broad knowledge but also qualities that AI still shows inconsistently:

• Flawlessness: a single error can take a system down for thousands of users
• Consistency of reasoning: multistep procedures require following the logic closely
• Contextual understanding: knowing not just what to do, but why each step is critical
• On-the-fly adaptation: for when the standard instruction doesn't fit the specifics of the environment
• Accountability: the ability to step back and ask a human for help when unsure

As they stand, agents are better seen as systems that can help but require constant oversight and validation of their output.

Resetting expectations

ITBench-AA is already influencing company strategies. The illusion of "digital workers who will replace the IT department within a month" is fading. Demand is instead growing for more realistic solutions: human-AI partnership, where the agent takes on routine work (updating configs, basic monitoring, logging) and the engineer keeps control of critical operations.

The benchmark also creates, for the first time, a widely accepted standard for evaluating agents. ITBench-AA will become a tool for model developers to understand what to work on in upcoming versions.

What this means

AI is advancing, but more slowly than startups promise. Good news for IT professionals: your expertise remains a scarce resource. For companies, the signal is that full automation of IT tasks is not a one- or two-year project. For model developers, it is a concrete roadmap for improvement.