FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis
06:00 · July 22, 2026 · arXiv cs.AI RSS
We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis. Built from FindStat, it contains 2,329 tasks across 24 collections and 5.52M hidden instances, covering statistic synthesis, which maps objects to integers, and map synthesis, which maps objects to objects. Each task gives a mathematical description and at most five public input-output examples; a model must emit one Python solve function with no retrieval, tool use, execution feedback, voting, or reranking. Submissions are scored by exact sandboxed execution on held-out combinatorial objects. We evaluate eleven systems: four closed-source production models and seven open-weight models served through one inference provider. FindStatBench reveals three main patterns. First, the strongest open- and closed-source systems converge within 1 pp instance accuracy, and both an oracle over all systems and five-way sampling from one mid-tier model yield only limited task-accuracy gains. Second, examples can hurt: several classical bijections are solved perfectly with zero examples but fail under five-example prompts. Third, some failures reflect output-budget mechanics, as reasoning can exhaust the visible response before code is emitted. Overall, statistic synthesis is much easier than map synthesis, some collections remain near-zero, long prompts cause a sharp accuracy cliff, and exact symbolic rule induction remains brittle.
Summary
FindStatBench is an execution-based benchmark that tests how well large language models can synthesize Python functions for combinatorial problems drawn from the FindStat database. It comprises 2,329 tasks spread across 24 collections and 5.52 million hidden test instances. The tasks fall into two categories: statistic synthesis, which requires a function that maps combinatorial objects to integers, and map synthesis, which requires a function that maps objects to other objects. Each task supplies a mathematical description together with at most five public input-output examples; the model must return a single standalone solve function. No retrieval, tool calls, execution feedback, voting, or reranking are permitted. Correctness is measured by exact execution against held-out combinatorial objects inside a sandbox.
Eleven models were evaluated under identical conditions: four closed-source production systems and seven open-weight models accessed through a single inference provider. The results show that the strongest open-weight and closed-source systems now differ by less than one percentage point in instance-level accuracy. An oracle that could select the best answer across all models, as well as five-way sampling from a single mid-tier model, produced only marginal gains in task coverage. In several cases the provision of examples actually degraded performance; certain classical bijections were solved perfectly when no examples were given but failed when the prompt contained five examples. A further class of errors stems from output-length limits, where chain-of-thought reasoning consumes the visible response budget before the required code is emitted.
Overall, statistic-synthesis tasks proved substantially easier than map-synthesis tasks, while a number of collections remained near zero accuracy for every model tested. Accuracy also dropped sharply once prompt length exceeded a modest threshold. The benchmark therefore highlights both the convergence of current model families and the continued brittleness of exact symbolic rule induction under realistic constraints.
Why it matters
This research is highly relevant for AI researchers and developers in the Netherlands focused on LLM evaluation, code generation, and symbolic reasoning. Benchmarking tools like FindStatBench are essential for building robust, transparent AI systems that align with European standards for AI performance and reliability.
