Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks
06:00 · August 3, 2026 · arXiv cs.AI RSS
Agent-safety benchmarks measure different behaviors, and their scores get quoted interchangeably as an agent's safety. We treat four of them (R-Judge, InjecAgent, AgentHarm, AgentDojo) as measurements to be validated, running each under its official implementation and author-provided scorer on up to 22 models, with MMLU and GPQA measured by us under one protocol as a capability composite. The metric is the first problem. On any binary trace-judgment benchmark scored by $F_1$, an ``always positive'' policy attains $F_1 = 2\pi/(1+\pi)$; on R-Judge that is $0.690$, above five of the 21 models that actually discriminate. The three broad-coverage benchmarks then rank the same 18 models differently, and the trade-off behind that disagreement is a small-panel artifact: R-Judge specificity against AgentHarm safety correlates $-0.64$ at $n{=}7$ and $+0.02$ at $n{=}18$, and a quarter of random size-7 subsets reach $|\rho| \geq 0.5$ around that near-zero value. Held-out validity turns on which outcome you pick. Capability predicts task success ($\rho{=}{+}0.60$) but correlates negatively with misalignment safety ($\rho{=}{-}0.44$, $n{=}21$). On their paired $n{=}20$ panel, the corresponding contrast is $\Delta{=}{-}1.00$ (95% CI $[-1.48, -0.49]$, $p<0.001$), and it survives leave-one-organization-out and organization-clustered bootstrap analyses. On an expanded 41-model panel, the misalignment correlation weakens to $-0.16$ (95% CI $[-0.54, +0.22]$) and jailbreak strengthens to $+0.34$, though neither change is significant. \mbox{AgentHarm} shows the strongest held-out association, $\rho{=}{+}0.72$ with three-template jailbreak safety after controlling capability. But both instruments score harmful compliance, so this is evidence of convergent validity rather than general safety. Naming the benchmark, metric, target behavior, and model panel is the minimum a safety claim needs.
Summary
The paper examines whether four prominent agent-safety benchmarks genuinely measure safety or largely reflect differences in model capability. Working with official implementations and scorers, the authors evaluate R-Judge, InjecAgent, AgentHarm and AgentDojo on panels of up to 22 models drawn from nine developers, while also collecting uniform MMLU and GPQA scores as a capability composite. The analysis separates questions of construct structure, metric soundness and criterion validity against held-out outcomes such as task success and jailbreak resistance.
A central finding concerns metric validity. On any binary trace-judgment benchmark scored by F1, an “always positive” policy yields a fixed score of 2π/(1+π). For R-Judge this baseline reaches 0.690, surpassing five of the 21 models that actually attempt to discriminate safe from unsafe traces. When the three broad-coverage benchmarks are compared on their common 18-model panel, they produce markedly different rankings. The apparent trade-off between R-Judge specificity and AgentHarm safety proves sensitive to panel size: the correlation shifts from −0.64 at n=7 to +0.02 at n=18, with roughly a quarter of random seven-model subsets still showing absolute correlations of 0.5 or higher.
Capability emerges as a strong predictor of task success (ρ=+0.60) yet correlates negatively with misalignment safety (ρ=−0.44 on the 21-model panel). On a paired 20-model set the contrast reaches Δ=−1.00 (95 % CI [−1.48, −0.49]). AgentHarm displays the clearest held-out association, correlating +0.72 with three-template jailbreak safety after capability is controlled; because both measures target harmful compliance, the result indicates convergent rather than general validity. On an expanded 41-model panel the misalignment correlation weakens and the jailbreak link strengthens, though neither shift reaches significance.
The authors conclude that any safety claim must specify the benchmark, metric, target behaviour and model panel. They release an API-only audit harness together with re-run artefacts to support such disciplined reporting.
Why it matters
Directly supports ethical, transparent AI development emphasized in Dutch/EU policy and the AI Act by validating safety measurements for LLM agents; Dutch practitioners can apply the released harness and findings to avoid over-reliance on unvalidated benchmarks in regulated deployments.
