SAAG: Structured Agent Assessment and Grounding
06:00 · July 22, 2026 · arXiv cs.AI RSS
Exact-match evaluation of agent-calling obscures qualitatively different failure modes: a model may select the right function yet hallucinate argument values, or satisfy a schema while choosing a agent for the wrong reason. Existing benchmarks collapse these distinctions into a single binary score, leaving practitioners unable to diagnose where agent calls fail. We propose SAAG a cascaded diagnostic framework that decomposes agent-calling evaluation into three sequential stages: registry conformance, structural completeness, and argument grounding, each producing interpretable stage-specific diagnostics. These diagnostics additionally enable iterative self-repair: on prediction failure, the stage-specific signal guides targeted correction without leaking ground-truth values. We evaluate this framework on a controlled benchmark derived from Glaive's function-calling dataset across registry sizes of 5, 10, and 15 agents using three local sub-4B-parameter models. Structured feedback consistently improves argument precision and reduces value hallucination relative to single-pass inference and uninformative binary feedback, while end-to-end F1 gains are modest and model-dependent. These results suggest that stage-decomposed diagnostic evaluation is a necessary lens for understanding and improving agent-calling reliability across model families and registry scales.
Summary
SAAG addresses a persistent weakness in LLM function-calling evaluation: conventional exact-match or end-to-end success metrics conflate distinct failure modes such as selecting an unregistered function, emitting malformed argument sets, or supplying values unsupported by the user query. The framework therefore imposes a strict three-stage cascade. Registry conformance first verifies that the predicted function name exists in the supplied registry, using Function Name Exact Match together with a Nearest Candidate Score that quantifies recoverable misspellings. Only conformant predictions advance to structural completeness, which checks required-parameter recall, flags drifted or spurious parameters via weighted string similarity, and computes a spurious-parameter rate. The final stage, argument grounding, assesses whether supplied values are actually entailed by the query.
Each stage yields an interpretable diagnostic and, on failure, returns a minimal correction signal that guides the model toward revision without disclosing ground-truth values. This design supports iterative self-repair loops while preserving the conditional nature of later-stage metrics. The authors instantiate the approach on a controlled subset of the Glaive function-calling dataset, testing three sub-4B models against registries of five, ten and fifteen tools. Across these conditions, structured stage-specific feedback consistently raises argument precision and lowers value hallucination relative to both single-pass generation and binary success/failure signals, although gains in aggregate F1 remain modest and vary by model family.
By isolating where calls break, SAAG supplies practitioners with actionable diagnostics that binary benchmarks obscure, while remaining applicable to any tool-calling pipeline independent of downstream correction strategies.
Why it matters
This research provides a rigorous framework for diagnosing and mitigating hallucinations in AI agents, directly supporting the Dutch and EU focus on transparent and trustworthy AI. It offers researchers new methodologies to evaluate agentic systems beyond simple binary exact-match metrics.
