Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading
06:00 · July 13, 2026 · arXiv cs.AI RSS
AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.
Summary
Long-Horizon-Terminal-Bench addresses a gap in current terminal-agent evaluation by providing 46 tasks that require sustained execution across many steps rather than short, self-contained actions. The benchmark covers nine categories such as experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task runs inside a containerized terminal environment with a natural-language goal, relevant files and tools, and a reference solution or simulator. Unlike prior terminal benchmarks that grade only the final state, Long-Horizon-Terminal-Bench decomposes every task into fine-grained subtasks that receive partial credit, producing dense reward signals throughout long trajectories.
Tasks are designed to last tens of minutes to hours and typically demand hundreds of episodes. Across evaluations of 15 frontier models, agents averaged 9.9 million tokens, 231 episodes, and 85.3 minutes of wall-clock time per task under a 90-minute timeout—roughly an order of magnitude more demanding than earlier suites such as Terminal-Bench 2. The strongest model reached a 15.2 percent pass rate at a 0.95 partial-reward threshold and 10.9 percent at a perfect 1.0 threshold; mean performance across all models was 4.3 percent and 1.7 percent under the same thresholds. These figures indicate that current agents struggle to maintain coherent plans, manage accumulating context, and recover from errors over extended horizons.
The authors also examine recurring failure patterns, including premature termination, weak self-verification, and inability to complete the final stages even after substantial intermediate progress. By releasing the benchmark and its evaluation harness, the work supplies a concrete testbed for measuring and improving the long-horizon reliability of terminal agents.
Why it matters
This research is highly relevant for Dutch AI researchers and developers focusing on autonomous agents and LLM evaluation. The introduction of dense reward grading provides a more nuanced framework for testing agent robustness, which is critical for deploying reliable AI systems in Dutch enterprises.
