MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents
06:00 · July 7, 2026 · arXiv cs.AI RSS
Current benchmarks for evaluating large language models (LLMs) in medical calculation are largely based on simplified settings, where each patient case corresponds to a single calculator and the required tool is explicitly specified in the query. However, real clinical scenarios often require multiple calculators for joint evaluation, nested-scale calculation, and fuzzy queries that do not directly specify the target calculator. To this end, we propose a new medical calculation benchmark, MedCalc-Pro, which covers three progressively challenging task settings: single-calculator, multi-calculator, and nested-calculator calculation settings. MedCalc-Pro contains 2,268 real-world clinical cases, covering 77 medical calculators across 14 clinical departments. Meanwhile, to address the limited performance of existing frameworks and methods in complex clinical scenarios, we further propose a more generalizable agent framework that supports multi-tool selection and nested-tool calling, while suppressing parameter error propagation through structured validation and evidence review. We conduct systematic comparisons across open-source, closed-source, and medical-specialized LLMs, and the results show that our framework achieves the best performance across all three task settings. This work provides a new benchmark and method for evaluating and applying LLMs in challenging medical calculation scenarios.
Summary
Existing benchmarks for medical calculation with large language models typically assume simplified conditions in which each clinical case maps to a single, explicitly named calculator. Real patient records, however, frequently require several calculators to be applied together, sometimes with one calculator depending on the output of another, and clinicians rarely state the required tools in advance. MedCalc-Pro addresses these gaps by providing a benchmark that spans three graduated difficulty levels: single-calculator tasks, multi-calculator tasks that demand joint evaluation, and nested-calculator tasks that involve explicit tool dependencies.
The dataset comprises 2,268 real-world cases drawn from 14 clinical departments and covering 77 distinct medical calculators. Queries are formulated in a goal-driven style that withholds tool names, forcing models to infer the appropriate calculators from clinical intent. This design tests query understanding, tool selection, parameter extraction, and multi-step execution under conditions closer to actual practice than prior collections.
To improve performance on these harder settings, the authors introduce a generalizable agent framework organized into four stages: query rewriting to clarify clinical intent, retrieval and reranking to surface candidate calculators, tool selection that permits multiple or nested calls, and dependency-aware execution supported by structured validation and evidence review. The validation steps are intended to limit the propagation of parameter errors across sequential tool invocations.
Systematic experiments compare the framework against representative open-source, closed-source, and medically fine-tuned models. Results indicate that the proposed approach attains the strongest outcomes across all three task settings while exhibiting greater robustness when queries become less explicit or when calculator chains grow more complex.
Why it matters
This research is highly relevant for Dutch AI researchers and health-tech enterprises focusing on clinical decision support systems. The proposed benchmark and agent framework align with the Netherlands' strong emphasis on robust, validated, and ethical AI applications in healthcare.
