MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning
06:00 · July 9, 2026 · arXiv cs.AI RSS
Mathematical reasoning benchmarks typically provide all facts needed to solve each problem, while interactive benchmarks often mix reasoning with tools, retrieval, and long-horizon dialogue. We introduce MIRA-Math, a benchmark for a narrower diagnostic capability: solving mathematical problems whose full latent state has a unique answer, but whose solver-facing view is missing exactly one necessary atomic fact. The solver must request the missing information in natural language under a strict budget and then integrate the returned fact into an exact final answer. A fixed constrained LLM responder sees only the dataset-provided atomic fact and must either offer the quoted fact when the request matches it, or decline otherwise. Thus, instance generation, typed hint specifications, validation, and final-answer verification are deterministic, while request metrics are measured under a fixed LLM-mediated responder channel. MIRA-Math contains 2{,}310 generated instances from 22 typed mathematical families spanning algebra, probability, linear systems, discrete structures, signal processing, Markov chains, circuits, interpolation, and numerical boundary-value problems. Experiments across frontier and small models show that request success and final-answer accuracy are separable: models may ask for the right fact yet fail the downstream computation, or fail before obtaining the canonical hint. We release generators, verifiers, prompts, run metadata, and dataset documentation to support reproducible evaluation of minimal information requesting in mathematical reasoning.
Summary
MIRA-Math targets a narrow but diagnostic capability in large language models: the ability to detect when a mathematical problem is missing exactly one atomic fact required for a unique solution, request that fact in natural language, and then incorporate the returned value into a correct final answer. Unlike conventional reasoning benchmarks that supply every necessary datum up front, the new suite deliberately withholds a single typed hint per instance while guaranteeing that the underlying latent state remains fully determined.
The benchmark comprises 2,310 generated problems drawn from 22 mathematical families that span algebra, probability, linear systems, discrete structures, signal processing, Markov chains, circuits, interpolation, and numerical boundary-value problems. Instance creation, hint specification, validation, and answer verification are performed deterministically through released generators and verifiers, so that differences in model performance can be attributed to information-requesting behavior rather than to variation in problem construction.
Evaluation uses a fixed, constrained responder model that returns the canonical fact only when a request matches it precisely and otherwise declines. This isolates the requesting step from downstream computation and enforces a strict interaction budget. Experiments across both frontier and smaller models reveal that request accuracy and final-answer correctness are separable: some systems obtain the needed fact yet still err on the subsequent calculation, while others fail to formulate an adequate request even when the missing datum is straightforward.
The authors release the full set of generators, verifiers, prompts, and run metadata to support reproducible assessment of minimal-information requesting within mathematical reasoning tasks.
Why it matters
This research provides a rigorous, reproducible benchmark for evaluating LLM reasoning and interactive capabilities, which is highly relevant for Dutch AI researchers developing reliable and transparent AI systems. It directly supports the advancement of agentic AI by testing a model's ability to recognize its own knowledge gaps.
