LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation
06:00 · July 29, 2026 · arXiv cs.AI RSS
Evaluating frontier LLMs is challenging: static benchmarks suffer from contamination and saturation -- leaving users unable to distinguish top models and developers blind to specific failure modes -- while human preference is subjective. In this paper, our question is: \emph{Do LLMs know what other LLMs don't? And can we leverage this dynamic for evaluation?} We present \textbf{LivingArena}, an automated, contamination-resistant evaluation framework. In this framework, models take turns proposing questions, aiming to pose items that opponents cannot answer correctly. Questioners are encouraged to actively identify and exploit opponents' knowledge boundaries, receiving rewards when the answerer fails, while the answerer is rewarded otherwise. To ensure questions contain objectively verifiable answers, a judge panel of strong models validates them, penalizing questioners if validation fails. Evaluating ten frontier LLMs, LivingArena yields a stable Elo leaderboard. Our behavioral analyses show that models identify and exploit their peers' cognitive boundaries: self-play and tournament logs indicate that they localize and double down on opponents' weak dimensions. Beyond static knowledge recall, peer probing measures factual rigor and the higher-order ability to probe an opponent's weaknesses, correlating only weakly with human preference and offering a scalable, low-cost approach to continuous evaluation.
Summary
LivingArena addresses the core limitations of static benchmarks such as MMLU and MMLU-Pro, which suffer from data contamination through leakage into training sets, rapid saturation that collapses distinctions among leading models, and high expert authoring costs. Human-preference arenas avoid fixed test sets but rely on subjective ratings that do not isolate factual accuracy or reasoning failures. The framework instead lets frontier models probe one another directly by generating questions intended to expose verifiable gaps in an opponent’s knowledge or reasoning.
In each round, one model acts as questioner and must supply both a question and a correct reference answer; a panel of strong models then validates the item through majority consensus, rejecting any question that fails verification and penalizing the questioner. Validated questions proceed to the answerer, with scoring awarded only when the opponent fails. This zero-sum dynamic incentivizes models to localize and repeatedly target an opponent’s weaker dimensions, such as logic or abstract reasoning, rather than sampling uniformly across topics. The resulting tournament produces an Elo ranking from objective, binary outcomes instead of stylistic comparisons.
When ten models from OpenAI, Anthropic, Google, and DeepSeek competed in 360 matches, the system generated a stable leaderboard while revealing distinct strategies: some models aggressively test difficult items at the risk of self-harm penalties, while others remain more calibrated. Performance diverged most clearly in quantitative reasoning and code, with consistent weakness in logic. The approach correlates only weakly with human preference scores, indicating that peer probing captures a distinct dimension of factual rigor and opponent modeling. A full ten-model evaluation completes in roughly 100 minutes of wall-clock time and requires no static question bank or human annotation.
Why it matters
This research provides Dutch AI researchers and developers with a novel, open-source framework for dynamically evaluating LLMs, addressing critical challenges like benchmark saturation and data contamination. Its rigorous, automated testing methodology aligns well with the EU's growing emphasis on robust AI evaluation and compliance.
