SAAG: A Three-Stage Method for Diagnosing Function-Calling Errors in AI Agents

In July 2026, a paper on SAAG (Structured Agent Assessment and Grounding) appeared on arXiv. It is a cascaded evaluation method that splits the check of AI agents' function calls into three sequential stages and returns a separate error diagnosis for each, instead of a single binary score.

What is wrong with binary evaluation

Scoring function calls by exact match hides qualitatively different kinds of failure, the SAAG authors argue. A model can pick the right function but hallucinate the argument values. It can also satisfy the call schema formally while choosing the tool for the wrong reason.

Existing benchmarks collapse all of these cases into one right/wrong score. As a result, an engineer sees that the agent erred but cannot tell at which step: tool selection, call structure, or argument filling. For agents that call external functions in production, this blind spot is critical: without a per-stage breakdown, there is nothing to fix in a targeted way.

How SAAG works

SAAG splits the check into three sequential stages, each producing its own interpretable diagnosis. According to the arXiv abstract, the stages run as a cascade, from general to specific.

• Stage 1 — registry conformance: whether the invoked tool matches the registry of available agents
• Stage 2 — structural completeness: whether the call structure is correct and all required fields are present
• Stage 3 — argument grounding: whether the argument values are grounded in the source data rather than invented
• The benchmark is built on Glaive's function-calling dataset
• Registries come in three sizes: 5, 10, and 15 agents

The key feature is iterative self-repair. When an error occurs, the signal from the specific stage tells the model what to fix, but it does not reveal the correct answer (ground truth), which prevents leakage and answer-fitting.

What the tests showed

Structured stage-by-stage feedback consistently improves argument accuracy and reduces hallucination of argument values, compared with a single pass without feedback and with uninformative binary right/wrong feedback. The authors tested three local models under 4 billion parameters.

The authors describe the overall end-to-end quality gain (F1 metric) as modest and dependent on the specific model. In other words, stage-level diagnostics clearly help at the argument level, but they do not deliver a dramatic jump in overall accuracy for every model and every registry size at once.

"Stage-wise diagnostic evaluation is a necessary lens for

understanding and improving the reliability of agent calling," the paper's abstract on arXiv states.

What this means

For teams building AI agents on small local models, SAAG offers more than a pass/fail verdict: it gives a map of where exactly a tool call breaks. That makes debugging agents more targeted, especially where the goal is to reduce argument hallucination without fine-tuning the model and without leaking reference answers.

Frequently asked questions

What is SAAG?

SAAG (Structured Agent Assessment and Grounding) is a diagnostic method for evaluating function calling by AI agents, split into three stages: registry conformance, structural completeness, and argument grounding. It returns a separate signal for each stage instead of one overall score.

How does SAAG differ from conventional benchmarks?

Conventional benchmarks give a binary right/wrong score and do not show at which step the agent erred. SAAG separates errors by stage and also lets the model fix them iteratively without seeing the correct answer.

Which models was SAAG tested on?

The method was tested on three local models under 4 billion parameters, using a benchmark built from the Glaive dataset with registries of 5, 10, and 15 available agents.