InfraBench: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk
06:00 · August 13, 2026 · arXiv cs.AI RSS
Managing modern computing infrastructure has become a steadily harder problem due to the ever-increasing complexity. Recent advances in AI agents create a timely opportunity to automate infrastructure management tasks, but it remains unclear how well such agents can handle real-world infrastructure complexity. We present InfraBench, a benchmark suite for evaluating AI agents on realistic infrastructure tasks across the full system stack and full operational lifecycle with fine-grained risk assessment. Experiments with 15 agent-model configurations show that even the strongest agent cannot secure a full score across all tasks. Mean effective scores range from roughly 40% to 88% (with per-configuration standard errors of 6-12 points), repeating every task three times reveals that top configurations still pass only a fraction of their attempts, and per-check scoring exposes a general failure pattern: agents may routinely satisfy short-term objectives while leaving non-durable changes, broken distributed invariants, unsafe side effects, and uncleaned state behind. INFRABENCH, including its live leaderboard, tasks, and evaluation harness, is publicly available at infraben.ch.
Summary
InfraBench is a benchmark suite that measures how well AI agents perform realistic infrastructure management across hardware, operating systems, distributed systems, and user-facing services. It covers the entire operational lifecycle from initial deployment through runtime maintenance, restarts, and clean decommissioning, while tracking side effects that could affect production systems. The design draws on interviews with infrastructure operators, issue data from projects such as Slurm and Ceph, and commercial cloud documentation to produce twelve seed tasks that reflect actual deployment constraints and failure modes.
Evaluation proceeds through an executor that provisions bare-metal or virtual backends and a multi-stage checker that separates immediate task success from longer-term correctness. After an agent completes its operations, the harness runs live workload probes, restarts services to test durability, and attempts decommissioning to verify that resources are released without residue. A separate risk monitor classifies recorded command sequences against a taxonomy of unsafe actions, including privilege escalation, configuration drift, and interference with unrelated services, so that shortcuts that satisfy short-term checks are still penalized.
Experiments with fifteen agent–model combinations show that even the strongest configurations achieve mean effective scores between roughly 40 % and 88 %, with substantial variance across repeated trials. Per-check analysis reveals a recurring pattern: agents often meet the immediate objective yet leave non-durable state changes, broken distributed invariants, or uncleaned resources that surface only after restarts or decommissioning. These gaps persist across different model families and agent frameworks, indicating that current approaches still lack robust mechanisms for preserving long-term operational integrity.
The full benchmark, including tasks, evaluation harness, and live leaderboard, is released at infraben.ch to support further community development.
Why it matters
This research is highly relevant for Dutch AI researchers and MLOps practitioners developing autonomous agents, as it provides a rigorous, open-source framework for testing agent reliability and safety. Its focus on risk assessment and operational side-effects aligns strongly with the Netherlands' strategic emphasis on transparent, ethical, and secure AI deployments.
