In LLM Reasoning, there is Irrationality on top of Value Misalignment
06:00 · June 23, 2026 · arXiv cs.AI RSS
Significant progress has been made in aligning LLMs with target value functions. We argue that, even when an LLM has been well aligned in (post-)training, it may still fail to maximise the aligned value in reasoning. We mathematically formalise this gap as rational value risk: the utility discrepancy between a model's deployed reasoning strategy and its rational counterpart, which is defined to be the responses that maximise expected utility in the steepest direction. The estimation error of rational value risk is further decomposed into three components from finite candidates, finite prompts, and imperfect verifiers. Extensive experiments are conducted, covering models Llama-3.1, Qwen-2.5, T{\"}ulu-3 families (7B-72B), GPT-5.2, GPT-5.5, and DeepSeek-V4, and benchmarks UltraFeedback, AlpacaEval, GSM8K, MATH, HumanEval, and MathArena. The results validate that (1) rational value risk is widespread; (2) value alignment can reduce, but cannot eliminate, it; (3) the risk is highly sensitive to inference-time reasoning strategy; and (4) longer reasoning improves rationality with diminishing returns. The code is at https://github.com/EVIEHub/LLM-Rationality.
Summary
Significant progress in aligning large language models with human preferences and task objectives has been achieved through techniques such as supervised fine-tuning, reinforcement learning from human feedback, and direct preference optimisation. These methods improve the degree to which model outputs reflect target value functions. Yet the paper demonstrates that alignment during training does not ensure that models will select responses that actually maximise those values when they generate reasoning at inference time.
The authors introduce the concept of rational value risk to quantify this gap. It measures the difference in expected utility between the reasoning strategy a model actually deploys and the rational counterpart that would select, among available responses, the one offering the steepest improvement in utility under the learned value function. This formulation isolates inference-time irrationality from any remaining misalignment in the value function itself. Because exact rational selection is intractable, the authors adopt a compute-bounded definition that treats the best response within a finite sample as the practical rational benchmark, then decompose estimation error into contributions from limited candidate sets, finite prompt samples, and imperfect verifiers.
Extensive tests across open-source families (Llama-3.1, Qwen-2.5, Tülu-3) and proprietary models (GPT-5.2, GPT-5.5, DeepSeek-V4) on both preference-based benchmarks such as UltraFeedback and AlpacaEval and verifiable tasks such as GSM8K, MATH, HumanEval, and MathArena confirm that rational value risk is widespread. Value-alignment procedures reduce the risk but leave a substantial residual. The magnitude of the risk varies sharply with inference-time choices such as sampling temperature and self-consistency, while extending reasoning length yields gains that diminish beyond a moderate compute budget.
Why it matters
The research provides deep technical insights into AI alignment and reasoning failures, which is crucial for Dutch AI researchers and enterprises focusing on ethical, transparent, and compliant AI deployment. The mathematical formalization of 'rational value risk' offers a novel framework for improving LLM reliability in high-stakes EU environments.
