Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics
06:00 · July 9, 2026 · arXiv cs.AI RSS
Recent advances in AI for Mathematics have focused largely on autoformalization and theorem proving, leaving the role of Computer Algebra Systems (CAS) in agentic LLM workflows underexplored. We propose a ReAct-style agentic setup that combines LLM reasoning with verifiable feedback from SageMath, together with Context7 for the up-to-date documentation. We evaluate this agentic setup across frontier models for solving research-level mathematical problems from the RealMath benchmark in a setting that emulates a computational-mathematics research loop. We also propose a refinement to the RealMath benchmark by introducing a multi-step post-processing procedure and a multi-stage validation pipeline, both of which improve the quality and reliability of the extracted problem set. Our experiments reveal substantial performance gains from SageMath access across all evaluated models on +9.7~pp on average, the gains range from 1.5~pp to 27.8~pp and narrow the gap between open-weight and closed models. Qwen~3.7-Max benefits from SageMath the most, while GPT-5.5 achieves the highest solve rate of $75.2\%$ and the lowest token usage among tool-enabled configurations. Our findings suggest that CAS-augmented agents represent a promising direction for assisting mathematicians in computational exploration, and we believe that this work is a step towards automated conjecture discovery. The project repository is available online.
Summary
Recent advances in AI for mathematics have emphasized autoformalization and theorem proving with formal proof assistants, while the integration of Computer Algebra Systems into agentic LLM workflows has received less attention. This paper addresses that gap by introducing a ReAct-style agent that pairs large language models with SageMath for verifiable symbolic computation and Context7 for retrieving current documentation. The setup is designed to emulate the iterative computational loop typical of research in areas such as algebraic combinatorics and finite group theory, where researchers use CAS tools to test examples, identify patterns, and refine conjectures.
The authors refine the RealMath benchmark through a multi-step post-processing procedure and a multi-stage validation pipeline that restrict problems to those with numerical or symbolic-expression answers, enrich statements with compressed context, and apply LLM-assisted normalization for executable verification. They then evaluate frontier models both with and without SageMath access, using a hybrid validation protocol that combines symbolic equivalence checking and LLM-as-a-judge assessment. This controlled ablation isolates the contribution of CAS feedback.
Across all tested models, SageMath integration yields an average performance increase of 9.7 percentage points, with gains ranging from 1.5 to 27.8 points and a noticeable narrowing of the performance difference between open-weight and closed models. Qwen 3.7-Max shows the largest improvement from tool access, while GPT-5.5 reaches the highest solve rate of 75.2 percent and the lowest token consumption among tool-enabled runs. The results indicate that CAS-augmented agents can meaningfully support computational exploration and move closer to automated conjecture discovery in mathematics.
Why it matters
This research is highly relevant for Dutch AI researchers and academic institutions focusing on agentic workflows and AI-assisted mathematics. The open-source nature of the project and its methodological advancements provide actionable insights for developing more reliable, tool-augmented LLM systems within the Netherlands' strong academic AI ecosystem.
