Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models
06:00 · June 25, 2026 · arXiv cs.AI RSS
Quantization is widely used to reduce the inference cost of large language models, but its effect on reasoning models is not fully captured by final-answer accuracy or per-token latency. We show that low-bit post-training quantization can introduce a hidden test-time compute cost: quantized reasoning models often generate longer chains of thought even when they still answer correctly. Across mathematical reasoning, code generation, scientific question answering, and agentic tool-use benchmarks, we find that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage, offsetting the expected per-token speedup. To measure this effect, we introduce the CoT Token Inflation Ratio, which compares reasoning length between quantized and full-precision models averaged across all evaluation benchmarks. We further show that token inflation is accompanied by behavioral changes in the reasoning trace, including more intermediate steps and greater semantic repetition. These changes translate into measurable end-to-end real-world serving penalties. Finally, we evaluate mitigation strategies and find that prompting and decoding-time sampling offer inconsistent accuracy-length trade-offs, while quantization-aware training shows more promise in reducing both accuracy degradation and token inflation. Our results suggest that reasoning-token usage should be reported alongside accuracy when evaluating quantized reasoning models.
Summary
Post-training quantization to INT4 or INT3 precision is routinely applied to large language models to shrink memory footprint and accelerate token generation. When applied to models trained for explicit step-by-step reasoning, however, the same methods preserve final-answer accuracy on mathematical, coding, scientific, and agentic benchmarks while increasing the number of tokens produced in the chain-of-thought trace. The resulting growth in test-time compute can offset or even exceed the expected per-token latency gains, turning an ostensibly cheaper inference setting into a net slowdown under real serving conditions.
To quantify the effect, the authors define the CoT Token Inflation Ratio, which expresses the average relative increase in reasoning length between a quantized model and its full-precision counterpart across all evaluated tasks. The inflation is accompanied by measurable changes in reasoning behavior: quantized traces contain more intermediate verification steps and exhibit greater semantic repetition, revisiting earlier conclusions without improving correctness. These patterns were observed consistently across dense models ranging from 4 B to 14 B parameters and a 30 B mixture-of-experts model.
Standard mitigation approaches such as prompt engineering or adjusted decoding-time sampling produce inconsistent accuracy-length trade-offs. In contrast, quantization-aware training reduces both accuracy loss and token inflation, suggesting that preserving the model’s original reasoning trajectory during quantization is more effective than post-hoc length controls. The findings indicate that accuracy and per-token latency alone are insufficient metrics for quantized reasoning systems; reasoning-token usage must be reported alongside them when deployment cost is assessed.
Why it matters
Directly actionable for Dutch AI teams deploying quantized reasoning models; aligns with Netherlands/EU priorities on efficient, transparent AI infrastructure and evaluation standards.
