Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
06:00 · July 3, 2026 · arXiv cs.AI RSS
Training large language models (LLMs) with reinforcement learning (RL) has significantly advanced their performance on reasoning and question-answering tasks. However, prevailing RL reward designs typically prioritize response correctness, neglecting to incentivize models to express their confidence accurately. This leads to a critical problem: performance gains are often accompanied by poor calibration between confidence and accuracy, misleading models to overconfidently hallucinate when uncertain. To address this limitation, we propose $\textbf{C}$orrectness and $\textbf{C}$onfidence $\textbf{C}$alibration $\textbf{R}$einforcement $\textbf{L}$earning ($\textbf{C3RL}$), a novel RL algorithm integrating correctness, calibration and dataset-informed reference accuracy rewards together. Comprehensive evaluation across 8 text and multimodal datasets demonstrates that C3RL enhances calibration without sacrificing accuracy, outperforming the current state-of-the-art method in both performance and calibration metrics. Utilizing the well-calibrated verbalized confidence from C3RL, we further introduce $\textbf{C}$onfidence-based $\textbf{A}$daptive Test Time $\textbf{S}$caling ($\textbf{CAS}$), an adjustable inference-time strategy that allocates computational resources based on response confidence. Experiments show that CAS surpasses majority voting on both in-domain and out-of-domain datasets while reducing the inference budget by up to 12.33 times. We believe the synergy of C3RL and CAS paves the way for deploying more reliable and resource-efficient LLMs. The code, data and models will be released.
Summary
Training large language models with reinforcement learning has improved results on reasoning and question-answering tasks, yet standard reward designs emphasize correctness at the expense of accurate confidence reporting. This imbalance produces overconfident but incorrect outputs, commonly called hallucinations, especially when models encounter uncertainty. The authors address the issue by introducing Correctness and Confidence Calibration Reinforcement Learning (C3RL), which combines three reward components: one for response correctness, one that penalizes or rewards shifts in correctness relative to dataset-derived reference accuracy, and one that directly incentivizes alignment between verbalized confidence and observed accuracy.
Evaluations across eight text and multimodal datasets show that C3RL improves calibration metrics while preserving overall accuracy and outperforming prior state-of-the-art approaches on both dimensions. The resulting models produce verbalized that more faithfully reflects the likelihood of correctness, avoiding the common trade-off in which calibration gains come from deliberately low-confidence incorrect answers.
Building on these calibrated signals, the authors present Confidence-based Adaptive Test Time Scaling (CAS). Rather than running a fixed number of samples or relying on majority voting, CAS computes a stopping probability from the gap between the most confident answer and the second-most confident answer. When the top answer dominates, inference halts early; when is low, additional samples are generated. Experiments indicate that this adaptive allocation exceeds the accuracy of majority voting on both in-domain and out-of-domain data while cutting the inference budget by as much as 12.33 times. Together, C3RL and CAS form a pipeline that improves reliability and reduces computational cost without requiring changes to model architecture.
Why it matters
This research is highly relevant for Dutch AI researchers and enterprises focusing on trustworthy and resource-efficient AI. By improving LLM confidence calibration and reducing inference costs, it directly supports the Netherlands' strategic goals for ethical, transparent, and sustainable AI deployment.
