SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents
06:00 · August 7, 2026 · arXiv cs.AI RSS
Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers. Diagnosing such failures is difficult, requiring the manual inspection of extremely long execution traces, which could be beyond human capacity. We therefore introduce SearchAuditBench, a benchmark that evaluates whether LLM auditors can localize, attribute, and repair these failures, thereby reducing the human burden. SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks, each expert-annotated with the critical error step, a search-specific root cause, and a reference repair with grading rubrics. We further propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures through evidence-grounded adjudication. Experimental results show that even the strongest baseline, when powered by a frontier model like GPT-5.5, attains only a 26.6% end-to-end pass rate. In contrast, our SearchAuditor consistently outperforms all baselines across different frontier models, achieving an end-to-end pass rate of 32.3%, and resuming failed runs with its repairs enables agents to better recover from errors.
Summary
Deep search agents pursue complex queries through extended sequences of web searches, page visits, and iterative reasoning. These trajectories are long and noisy, often exceeding 70 messages and 65 000 tokens, so minor early missteps can propagate silently into fluent but incorrect final answers. Manual diagnosis of such failures is impractical at scale, prompting the need for automated auditing tools that can localize the critical error, identify its root cause, and suggest a repair.
To support systematic evaluation, the authors created SearchAuditBench, a corpus of 1 243 failed trajectories drawn from eight open-weight models run on five established deep-search benchmarks. Each trajectory was annotated by experts with the location of the decisive error (including a tolerance span), a root cause drawn from a six-category taxonomy tailored to search agents, and a reference repair accompanied by atomic grading rubrics. Analysis of the corpus shows that nearly half of the critical errors occur in the final third of a trajectory and that candidate mismanagement is the most frequent root cause.
Building on this benchmark, the authors introduce SearchAuditor, a multi-perspective auditing framework that runs three complementary audit branches in parallel, adjudicates their outputs against a compact evidence view of the trajectory, and produces process-level repair directives. When evaluated on frontier models, SearchAuditor raises the end-to-end pass rate from 26.6 % achieved by the strongest baseline to 32.3 %. Applying its repairs to previously failed runs also improves downstream agent accuracy, for example lifting Kimi-K2.6 performance on one benchmark from 34.0 % to 45.1 %.
The work highlights both the diagnostic difficulty of long-horizon search agents and the measurable gains that structured, evidence-grounded auditing can deliver.
Why it matters
This research is highly relevant for Dutch AI researchers and developers building autonomous agents, as it provides a robust framework for auditing and debugging complex AI behaviors. Furthermore, its focus on transparency and error attribution aligns strongly with EU AI Act requirements for reliable and accountable AI systems.
