IBM Research dissects where AI agents break on APIs, documents and rules in VAKRA

IBM Research has published a detailed breakdown of why even strong language models still stumble on tasks for agentic systems. The new analysis of the VAKRA benchmark shows that calling a single API cleanly is not enough: problems begin where a model has to complete several steps, pick the right data source and not break the rules for using tools.

How VAKRA works

VAKRA is an executable benchmark for enterprise agents. Instead of toy function calls, it gives models a working environment with more than 8 thousand locally hosted APIs, real databases across 62 domains and document collections tailored to specific subject areas. A typical scenario requires not one answer but a chain of 3–7 steps: fetch data, choose the right tool, extract a fact from a document, pass the result to the next call and only then assemble the final answer.

The key idea is that VAKRA scores not just the model's final reply but its entire trajectory of actions. For complex tasks, the system first checks whether the agent obeyed the text constraints on tool use, then replays its calls in the same environment, compares intermediate results with a reference and only then grades the final answer. This matters because an agent can guess the final output by accident yet reach it by the wrong path, and for production that is almost useless.

Four types of tasks

The authors split VAKRA into four modes, each testing a separate layer of agent behavior. Together they cover the span from simple API chaining to multi-step reasoning over APIs and documents under external constraints. This matters because many agents look confident on single calls but quickly get lost when they must plan steps, switch between sources, keep dialogue context and remember tool access rules all at once.

• Business Intelligence APIs: 2,077 tasks across 54 domains, where the agent must call 1–12 tools in sequence and handle parameters and data filtering carefully.
• Dashboard APIs: 1,597 tasks across 17 domains, where the main difficulty is choosing the right endpoint among 6–328 available tools.
• Multi-hop over APIs: 869 tasks across 38 domains, where the answer is assembled through several logical hops, from one to five.
• Multi-source + policies: 644 tasks across 41 domains, where the agent alternates between APIs and document search, takes dialogue history into account and follows text rules such as "use only the retriever, don't touch other tools."

Where agents go wrong

The most useful part of the paper is its analysis of exactly where models break. The authors classify errors by stage: choosing the wrong tool, omitting required arguments or hallucinating them, wrong parameter values and, finally, a wrong final answer even after correct calls.

On the BI API segment, GPT-OSS-120B came out on top: it understood tool schemas noticeably better than the others and made fewer mistakes in names and parameter filling. But even there, success on individual steps did not guarantee a stable end-to-end result.

On tasks with a large set of dashboard APIs, Gemini-3-flash-preview did better than the rest, which makes sense: shortlisting tools and picking the exact endpoint matter most there.

As reasoning depth grew, quality fell for every model: 2-hop and especially 3+ hop questions noticeably dragged accuracy down. It got even worse when APIs had to be combined with document retrieval. The authors single out one telling failure: on some 1-hop RAG tasks, GPT-OSS-120B sometimes did not call the retriever at all and tried to answer "from memory," which this benchmark counts as an error.

Policies added another layer of difficulty: models either violated the constraints, or obeyed them but failed to gather the information needed for the answer.

What this means

VAKRA shows an unpleasant but useful truth about agentic systems: the ability to build a slick tool-calling demo does not yet mean readiness for real business processes. For teams choosing a model for support, analytics, compliance or internal workflows, the main question is no longer "can it call tools," but "does it keep a correct chain of actions under constraints, across several sources and without overconfident shortcuts."