Reasoning Jury: Multi-Model Consensus for Evaluating Reasoning Traces
06:00 · August 15, 2026 · arXiv cs.AI RSS
Improving reasoning LLMs requires the ability to judge the quality of long reasoning traces for effective reasoning data curation, strong training signals during reinforcement learning, and an in-depth understanding of reasoning behaviors during model performance evaluation. Additionally, surfacing reasoning mistakes that the model makes would enable improving the model's performance at runtime through providing feedback. Due to the difficulty of this complex task on long reasoning traces, single-model judges (even frontier models) do not do well at identifying reasoning defects. Additionally, leveraging frontier models during online training of reasoning LLMs is generally prohibited due to guardrails in terms of use. In this work, we introduce Reasoning Jury, a system that replaces the single judge with a jury of LLMs and a moderated consensus mechanism, to improve the fidelity of judgments for identifying reasoning defects. In reasoning jury, defects of a reasoning trace and their severity are surfaced through a deliberation where a moderator conducts a discussion amongst the jury where the jurors critique each other's judgments and get to modify their initial votes. The moderator derives a consensus through deliberation amongst jurors or consolidation of judgements. We show that Reasoning Jury with a jury of open-weight models (e.g., gpt-oss-120b) is able to significantly outperform frontier models (opus-4.6, sonnet-4.6, and gemini-3.1-pro) at correctly identifying reasoning defects. Besides accuracy performance improvements, the aggregated cost of the jury (initial verdicts, deliberations, consolidation, etc.) is a fraction (8 to 15%) of the cost of running frontier models in LLM-as-a-judge setup. We also show how these judgements can be leveraged to understand failure modes of reasoning LLMs on benchmarks, which allows much deeper understanding of a model's performance.
Summary
Reasoning Jury addresses the challenge of evaluating long chain-of-thought traces produced by reasoning LLMs, where defects in individual steps can propagate and remain difficult for any single model to isolate. The system replaces a monolithic judge with a panel of open-weight models that first issue independent, step-grounded verdicts. Each verdict identifies specific defects, their locations via explicit [STEP-x] markers, severity levels, and supporting evidence from the trace.
A moderator then derives consensus through one of two modes. In deliberation mode the moderator orchestrates multi-turn discussion among jurors, allowing them to critique one another’s judgments before votes are finalized. In consolidation mode the moderator simply aggregates the initial verdicts into a unified report. Both approaches keep every defect traceable to particular reasoning steps, yielding outputs that are auditable by humans and directly usable for downstream tasks such as data filtering or runtime correction.
Evaluations on Hard2Verify show that a jury composed of three instances of gpt-oss-120b surpasses frontier models including Opus-4.6, Sonnet-4.6, and Gemini-3.1-Pro by as much as 12 Balanced F1 points. The same jury incurs only 8–16 % of the inference cost of the strongest single frontier model. The resulting defect distributions also expose systematic failure modes across benchmarks, supplying richer signals than aggregate accuracy scores alone for supervised fine-tuning, reinforcement learning, and inference-time refinement loops.
Why it matters
Directly actionable for Dutch researchers and advanced practitioners building or fine-tuning reasoning LLMs; leverages open models to bypass closed-model guardrails, supporting EU transparency and ethical-AI requirements; high technical depth and reproducibility make it suitable for Primary research stream readers.
