RoPoLL: Robust Panel of LLM Judges
06:00 · July 1, 2026 · arXiv cs.AI RSS
The LLM Jury, a Panel of LLM Evaluators (PoLL) reporting consensus scores, has become a practical alternative to single-judge LLM evaluation, yet its statistical behavior remains poorly understood. We formalize the LLM Jury under the Huber contamination model and show that PoLL incurs unbounded bias under any positive contamination, regardless of jury size, whenever a single judge fails in a biased, LLM-typical way (mode collapse, sycophancy, safety refusal). Framing jury consensus as classical robust mean estimation, we propose RoPoLL (Robust Panel of LLM-as-Judge), which preserves the PoLL panel but replaces the aggregation function with a robust mean estimator, instantiated with the geometric median (GM): tuning-free, with the optimal finite-sample breakdown point 1/2. A finite-sample error bound and a matching information-theoretic minimax lower bound agree on the parametric rate sigma*sqrt(d/N) and differ on the breakdown floor by a factor of sqrt(d), a statistical-computational gap that polynomial-time RoPoLL pays relative to the intractable Tukey halfspace median. Across 13 open-weight judges (4B-675B), three reward-model benchmarks, and four corruption regimes at rates up to 50%, RoPoLL dominates PoLL on every biased corruption type: by about 19% on cross-dimensional attacks at matched compute, and by orders of magnitude on heavy-tailed Byzantine adversaries. A 3-judge RoPoLL committee at 38B beats Mistral-Large-3 (675B) by 1.31x on HelpSteer-2 under 30% bimodal-random corruption, an 18x parameter advantage at better accuracy; a Noisy-GT control confirms the premium is paid against biased contamination, not benign imprecision.
Summary
RoPoLL addresses a core weakness in panel-based LLM evaluation by treating the jury as a statistical estimation problem under the Huber contamination model. Standard PoLL approaches aggregate scores from multiple judges via arithmetic mean, which works when errors are light-tailed and symmetric but produces unbounded bias as soon as any judge exhibits the biased, LLM-typical failures common in practice: mode collapse, sycophancy, cross-attribute confusion, or heavy-tailed parser hallucinations. These failures map directly onto the contamination distribution in the Huber framework, and a direct calculation shows that the resulting bias grows linearly with the corruption shift regardless of jury size.
The method keeps the same diverse panel of smaller open-weight judges but replaces the mean with the geometric median as the aggregation step. This estimator is tuning-free, operates on the full score vector to preserve cross-attribute structure, and attains the optimal finite-sample breakdown point of one half. A finite-sample error bound of order sigma times square root of d over N is derived, together with a matching information-theoretic minimax lower bound that reveals a statistical-computational gap of order square root of d between the geometric median and the intractable Tukey halfspace median.
Extensive experiments across thirteen open-weight judges ranging from 4 B to 675 B parameters, three reward-model benchmarks, and four corruption regimes up to fifty percent contamination show that RoPoLL consistently outperforms arithmetic-mean aggregation on every biased attack. Gains reach roughly nineteen percent on cross-dimensional corruption at matched compute and orders of magnitude on heavy-tailed Byzantine adversaries. A three-judge RoPoLL committee with thirty-eight billion total parameters exceeds a 675-billion-parameter single judge on HelpSteer-2 under thirty percent bimodal-random corruption, confirming that the robustness premium arises specifically against structured contamination rather than benign noise. The authors also release the full judge-output corpus to support further calibration and aggregation research.
Why it matters
Directly actionable for Dutch research teams and SMEs building LLM evaluation pipelines; aligns with EU emphasis on reliable and transparent AI; high technical depth and novelty for advanced readers.
