Quantifying infrastructure noise in agentic coding evals
01:00 · February 5, 2026 · Antropic Engineering Blog
Infrastructure configuration can swing agentic coding benchmarks by several percentage points—sometimes more than the leaderboard gap between top models.\n\n
Summary
Anthropic’s engineering team has shown that infrastructure settings in agentic coding evaluations can shift reported scores by as much as six percentage points, a margin larger than the gaps separating leading models on current leaderboards. Benchmarks such as SWE-bench and Terminal-Bench 2.0 require models to operate inside live environments where they write code, install packages, spawn processes, and iterate across multiple turns. Because the runtime itself participates in problem solving, differences in CPU and memory allocation directly affect which solution paths remain feasible.
In controlled runs on Terminal-Bench 2.0, success rates rose steadily as resource headroom increased from a strict one-to-one mapping of guaranteed allocation to kill threshold up to fully uncapped containers. The largest gains appeared once allocations exceeded three times the per-task specification: infra-related failures fell from 5.8 % to 0.5 %, while overall success improved by nearly four additional points. The same monotonic trend, though smaller in magnitude, appeared on a 227-problem subset of SWE-bench when RAM was scaled to five times baseline. Tasks that succeeded only under generous limits typically involved large dependency installs or memory-intensive test suites, revealing that tight constraints inadvertently favor leaner but sometimes less effective strategies.
The experiments also clarified how container runtimes enforce limits. When the guaranteed allocation and the hard kill threshold are set to identical values, transient memory spikes trigger spurious OOM kills that would not occur with modest headroom. Terminal-Bench’s official leaderboard uses a more lenient sandbox that avoids immediate termination, producing higher and more stable scores than strict Kubernetes enforcement. The authors therefore recommend that benchmark maintainers publish two separate parameters per task—reserved resources and kill threshold—calibrated so that scores at the lower and upper bounds remain statistically indistinguishable while still preventing unbounded resource inflation.
These findings imply that small leaderboard differences on agentic evaluations should be interpreted with caution until resource configuration and enforcement method are documented and reproduced. Without such standardization, reported margins may reflect hardware generosity or sandbox policy as much as model capability.
Why it matters
This article is crucial for product teams and builders evaluating AI models, as it highlights how infrastructure choices can skew benchmark results. Dutch AI practitioners can apply these insights to build more rigorous, transparent evaluation pipelines, ensuring they select models based on true capabilities rather than hardware advantages.
