SAAG: метод трёхэтапной диагностики ошибок вызова функций у AI-агентов

In July 2026, a paper called SAAG (Structured Agent Assessment and Grounding) appeared on arXiv — a cascading evaluation method that breaks down the assessment of an AI agent's function calls into three sequential stages, producing a separate error diagnosis for each instead of a single binary score.

What's wrong with binary scoring

Evaluating function calls by exact-match hides qualitatively different types of failures, the SAAG authors argue. A model may pick the correct function but hallucinate argument values — or formally satisfy the call schema while choosing the tool for the wrong reason.

Existing benchmarks collapse all these cases into a single "correct/incorrect" score. As a result, an engineer sees that the agent made a mistake but doesn't understand at which stage exactly: tool selection, call structure, or argument substitution. For agents that call external functions in production, this blindness is critical — without a breakdown by stage, there's nothing to fix in a targeted way.

How SAAG works

SAAG breaks the check down into three sequential stages, each producing its own interpretable diagnosis. According to the arXiv abstract, the stages proceed as a cascade — from general to specific.

• Stage 1 — registry conformance: whether the called tool matches the registry of available agents
• Stage 2 — structural completeness: whether the call structure is correct, whether all required fields are present
• Stage 3 — argument grounding: whether argument values are grounded in the source data rather than fabricated
• The benchmark is built on the function-calling dataset from Glaive
• Registries of three sizes — 5, 10, and 15 agents

The main feature is iterative self-repair. When there's an error, the signal from a specific stage tells the model what exactly to fix, without revealing the correct answer (ground truth), so there's no leakage or answer-fitting.

What the tests showed

Structured stage-by-stage feedback consistently improves argument accuracy and reduces hallucination of their values — compared to a single pass with no feedback and to uninformative binary "correct/incorrect" feedback. Testing was done on three local models under 4 billion parameters.

At the same time, the authors describe the resulting end-to-end quality gain (F1 metric) as modest and model-dependent. In other words, stage-by-stage diagnosis noticeably helps at the argument level but doesn't produce a dramatic jump in overall accuracy across all models and all registry sizes at once.

"Stage-wise diagnostic evaluation is a necessary lens for

understanding and improving the reliability of agent function calling," the paper's arXiv abstract states.

What this means

For teams building AI agents on small local models, SAAG offers not just a "pass/fail" verdict but a map of exactly where a tool call breaks down. This makes debugging agents more targeted — especially where it matters to reduce argument hallucination without fine-tuning the model and without leaking ground-truth answers.

Frequently Asked Questions

What is SAAG?

SAAG (Structured Agent Assessment and Grounding) is a diagnostic method for evaluating AI agents' function calls, broken down into three stages: registry conformance, structural completeness, and argument grounding. It produces a separate signal for each stage instead of one overall score.

How does SAAG differ from regular benchmarks?

Regular benchmarks give a binary "correct/incorrect" score and don't show at which stage the agent made a mistake. SAAG splits errors by stage and additionally lets the model fix them iteratively without peeking at the correct answer.

Which models was SAAG tested on?

The method was tested on three local models under 4 billion parameters, on a benchmark from the Glaive dataset with registries of 5, 10, and 15 available agents.