PHREEQC-MCQ-200: A Diagnostic Benchmark for Tool-Augmented Scientific Simulator Agents
06:00 · July 2, 2026 · arXiv cs.AI RSS
Large language model agents are increasingly connected to scientific software, yet it remains unclear when tool access makes scientific computation more reliable rather than merely more complex. We introduce PHREEQC-MCQ-200, a benchmark for evaluating tool-augmented agents on deterministic aqueous-geochemistry simulations. The benchmark contains 200 multiple-choice questions derived from 21 validated PHREEQC scenarios, requiring agents to construct simulator inputs, execute PHREEQC, inspect structured outputs, and commit to final answers. Across multiple frontier and mid-tier model families, simulator access substantially improves aggregate accuracy, confirming that grounded execution is necessary for many scientific-computation tasks. However, the gains are not monotonic: tool-augmented agents also lose items they answered correctly without tools, revealing regressions that average accuracy alone hides. We further show that output-access protocol matters. A table-of-contents interface can reduce token cost while preserving or improving accuracy for stronger models, but it degrades performance for mid-tier models that cannot reliably navigate structured simulator outputs. PHREEQC-MCQ-200 therefore frames scientific tool use as an end-to-end diagnostic problem rather than a simple tool-calling capability. We argue that evaluations of scientific agents should report not only accuracy, but also item-level retention, output-access sensitivity, trajectory failures, and where the computation chain breaks.
Summary
The article presents PHREEQC-MCQ-200, a benchmark of 200 multiple-choice questions drawn from 21 validated scenarios in the open-source PHREEQC simulator for aqueous geochemistry. Agents must generate valid input files, run the deterministic simulator, parse its structured output, and select the correct answer, providing a controlled setting in which tool use can be measured against reproducible ground truth rather than open-ended generation.
Experiments across frontier and mid-tier models show that simulator access raises aggregate accuracy by 15 to 41.5 percentage points compared with text-only baselines. The improvement confirms that many scientific-computation tasks require grounded execution rather than scaled chain-of-thought reasoning alone. At the same time, the gains are not monotonic: agents lose between 10 and 32 items they had previously solved without tools, yielding retention rates between 56 and 86 percent depending on the model.
The benchmark further isolates the effect of output-access protocols. A table-of-contents interface reduces input tokens by 11 to 57 percent while preserving or slightly improving accuracy for stronger models; the same interface degrades performance for mid-tier models that struggle to navigate the structured simulator output. These trade-offs cluster by capability tier across vendors rather than by model family.
The authors therefore treat scientific tool use as an end-to-end diagnostic problem. They recommend that evaluations of tool-augmented agents report item-level retention, output-access sensitivity, and the location of chain failures in addition to overall accuracy.
Why it matters
This research is highly relevant for Dutch AI researchers and enterprises developing agentic workflows for scientific discovery (AI4Science). Its focus on the reliability, transparency, and diagnostic evaluation of tool-augmented LLMs aligns well with the EU's push for trustworthy AI systems.
