Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures
06:00 · August 3, 2026 · arXiv cs.AI RSS
Existing evaluations often reduce agent failures to system-level outcomes, obscuring where the fault originated and which intervention would improve the agent system. This creates a repair-assignment problem: the same visible failure may call for model post-training, harness engineering, environment redesign, or benchmark repair depending on its source. Because agent behavior emerges from interactions among models, harnesses, users, tools, memory, and environments, outcome-level labels are often insufficient for improvement. Most failure taxonomies do little to resolve this problem because they are benchmark-specific and lack a shared structure. We introduce an interaction-centric taxonomy that localizes failures to the interactions in which they originate and identifies the responsible component. It organizes 41 failure modes by assigning each to an edge between two components and a fault side indicating where the repair belongs. This makes the taxonomy actionable: model-side failures identify targets for post-training, harness-side failures point to scaffolding and tool-integration fixes, and environment or grader failures reveal evaluation conditions requiring redesign. The schema applies across agent architectures, from coding assistants to long-horizon personal assistants and multi-agent systems. We ground the taxonomy in worked examples from public benchmarks, model system cards, published reports, and logged agent trajectories, and evaluate its reproducibility using independent reasoning agents as judges. Across four frontier models, the strongest judge reaches Cohen's $\kappa=0.76$ against human category labels, suggesting that the categories capture shared structure rather than annotator-specific preferences.
Summary
Existing evaluations of AI agents typically label failures only by their visible system-level outcomes, which obscures the originating component and the appropriate repair. The resulting repair-assignment problem means that identical observed errors may require model post-training, harness changes, environment redesign, or benchmark fixes depending on their source. Because agent behavior arises from repeated interactions among models, harnesses, users, tools, memory, and environments, outcome-level labels alone rarely indicate which intervention will improve performance.
The paper introduces an interaction-centric taxonomy that treats each failure as an edge between two components and assigns a fault side to indicate where responsibility lies. It defines nine core components—model, owner, grader, third party, context, memory, tool, local environment, and external environment—and organizes 41 failure modes along these edges. Model-side faults point to post-training targets, harness-side faults indicate scaffolding or tool-integration work, and environment or grader faults signal evaluation conditions that need redesign before they can reliably measure capability. The same vocabulary applies to single-model assistants, long-horizon personal agents, coding agents, and multi-agent systems.
The taxonomy is grounded in concrete trajectories drawn from public benchmarks, model system cards, and published reports. To test whether the categories reflect shared structure rather than individual annotator preference, the authors used independent reasoning agents as judges. Across four frontier models, the strongest judge recovered human-assigned labels with Cohen’s κ of 0.76, and pairwise agreement among judges reached 0.84, indicating that the distinctions are reproducible when the definitions and evidence are held constant.
Why it matters
This research is highly relevant for Dutch AI researchers and developers building autonomous agents, as it provides a structured methodology for diagnosing and repairing complex AI systems. It aligns well with the EU's focus on AI robustness, transparency, and safety by offering a standardized way to trace and mitigate agent failures.
