Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs
06:00 · July 24, 2026 · arXiv cs.AI RSS
When a language model gives different answers on repeated runs, does that variation reveal what it does not know? Self-consistency turns the variation into a per-question uncertainty estimate via majority voting. But does the same variation reveal cross-question structure -- related questions flipping together, the way a diverse ensemble does? We compare two regimes on the same questions: one model run $100$ times at $\tau=1$ versus an ensemble of $24$ LLMs run once each at $\tau=0$. A Marchenko--Pastur random-matrix test separates signal from sampling noise on both sides. Within any single model, at most one dimension rises above noise across five families and three benchmarks (MMLU, HellaSwag, GSM8K). Across the ensemble, four eigenvalues clear the noise edge, while a matched-difficulty Bernoulli null produces at most one in $500$ Monte Carlo draws. Self-consistency gives accurate per-question uncertainty but no detectable cross-question structure; only a diverse ensemble surfaces what a model does not know.
Summary
Temperature-based stochastic sampling within a single language model produces at most one dimension of structured epistemic uncertainty that exceeds sampling noise, according to a Marchenko–Pastur analysis of the run-by-question correctness matrix. When the same questions are presented to an ensemble of 24 diverse models, four eigenvalues rise above the noise threshold on MMLU, while matched-difficulty Bernoulli null matrices yield at most one such eigenvalue across 500 Monte Carlo trials. The gap persists across five model families and three benchmarks, including HellaSwag and GSM8K chain-of-thought.
Self-consistency methods convert repeated samples from one model into reliable per-question success probabilities through majority voting. Yet the same samples show near-zero correlations between questions, even within the same subject category, indicating that temperature-induced variation supplies independent noise rather than coordinated epistemic structure. In contrast, the across-model regime reveals correlated error patterns that survive the random-matrix null test, suggesting that model diversity captures shared limitations that single-model sampling does not.
The experimental design isolates this distinction by restricting analysis to borderline questions whose pass rates lie away from zero and one, then comparing the resulting correlation spectra against Tracy–Widom fluctuations around the Marchenko–Pastur edge. Within-model results remain consistent when temperature is varied between 0.5 and 1.5 and when the number of runs is reduced, confirming that the dimensionality ceiling is not an artifact of a single sampling regime. These findings imply that uncertainty estimates derived solely from intra-model stochasticity remain epistemically shallow relative to the richer signal obtained from heterogeneous model ensembles.
Why it matters
Directly actionable for Dutch AI teams building reliable LLM systems: ensembles outperform deeper sampling at far lower cost for selective prediction and uncertainty quantification, aligning with EU emphasis on transparent, trustworthy AI under the AI Act.
