AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation
06:00 · July 9, 2026 · arXiv cs.AI RSS
We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at https://github.com/agent-lens/agent-lens-bench.
Summary
AgentLens is an open-source benchmark designed to evaluate interactive coding agents by examining their full execution trajectories rather than reducing performance to a binary pass-or-fail outcome. Conventional benchmarks typically register only whether a task completes successfully, yet real-world users observe every step: adherence to instructions, selection and application of tools, verification of intermediate results, recovery from errors, and the clarity of status messages or explanations provided along the way. AgentLens captures these dimensions so that developers can understand not merely whether an agent succeeded, but how it behaved throughout the process.
The benchmark integrates two complementary evaluation methods. Where objective criteria exist, formal verification supplies deterministic checks on task outcomes. For aspects that resist simple automation, large language models generate narrative reviews of the trajectory and produce side-by-side comparisons between runs. The resulting score is therefore accompanied by an explicit, human-readable rationale that links observed behavior to the final assessment.
This combination supports uses beyond model ranking. Teams can inspect individual trajectories to diagnose recurring failure modes, track behavioral changes across successive versions of an agent, and integrate the benchmark into automated pipelines that surface regressions before they reach production. The authors release the full benchmark and associated tooling under an open-source license to encourage reproducible evaluation practices in the development of coding agents.
Why it matters
AgentLens is highly relevant for Dutch AI researchers and developers as it provides a robust, open-source framework for evaluating the behavior and reliability of coding agents. Its focus on the entire trajectory rather than just the final output aligns well with the EU's emphasis on transparent, explainable, and trustworthy AI systems.
