HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents
06:00 · July 1, 2026 · arXiv cs.AI RSS
As AI agents become increasingly capable of complex, long-horizon reasoning, rigorous and holistic evaluation is essential for measuring progress toward real-world healthcare applications. We introduce HealthAgentBench, a suite of 54 agentic healthcare tasks across 7 categories each with its unique environment. The benchmark suite spans diverse workflows throughout the patient journey and a broad range of modalities. Each task is designed to replicate an end-to-end clinical workflow: given minimal instructions, an agent must explore raw healthcare data, operate within a complex environment, and execute multi-step solutions that go beyond naive prompting. A final task success rate is reported to provide a single, interpretable metric for HealthAgentBench overall performance for each agent. Evaluating frontier agents on HealthAgentBench, we find that overall task success rate remains low, underscoring the difficulty of the suite. The strongest and the most cost effective agent, Codex GPT-5.5, achieves only approximately 42% success rate. Beyond aggregate performance, HealthAgentBench reveals nuanced strengths and weaknesses across task categories. Frontier agents show promise in automatically developing research modeling pipelines over EHR data, but medical imaging remains especially challenging, particularly for Claude Code models, while Codex GPT-5.5 shows emerging capability. Tasks that combine large search spaces with compositional reasoning requirements remain difficult for all current agents. Together, these results suggest that HealthAgentBench provides a challenging and realistic benchmark with substantial room for future progress. We release our benchmark at https://github.com/microsoft/HealthAgentBench.
Summary
HealthAgentBench supplies a unified collection of 54 agentic tasks distributed across seven categories, each packaged in its own Docker environment. The tasks follow realistic clinical workflows that span the patient journey, from data staging and quality auditing through diagnostic imaging interpretation to clinical-trial matching and longitudinal EHR modeling. Every task supplies raw multimodal artefacts—longitudinal radiographs, 3D CT volumes, gigapixel pathology slides, tabular records, or free-text protocols—together with only minimal natural-language instructions, forcing an agent to explore files, install tools, issue terminal commands, and produce a final submission that is scored against hidden expert labels.
Evaluation uses a single binary success metric per task, enabling direct comparison of overall performance. When frontier agents are tested on the full suite, the strongest performer, Codex GPT-5.5, reaches only about 42 percent task success. Performance varies markedly by category: agents show early competence at constructing research pipelines over structured EHR data, yet medical imaging tasks remain difficult, especially for Claude-based agents, while Codex GPT-5.5 exhibits nascent capability. Tasks that combine large search spaces with compositional reasoning requirements continue to challenge every evaluated system.
The benchmark is released at https://github.com/microsoft/HealthAgentBench to support systematic measurement of progress toward reliable healthcare agents.
Why it matters
This benchmark is highly relevant for Dutch AI researchers and healthcare institutions developing or evaluating autonomous clinical agents. It provides a rigorous, open-source framework to test AI reliability and transparency, aligning with EU regulatory standards and the Netherlands' focus on ethical AI in healthcare.
