Coresets Before Score Sets: Evaluation-Unsupervised Prompt Subset Selection for LLM Benchmarks
06:00 · July 14, 2026 · arXiv cs.AI RSS
We study LLM benchmark coreset selection: selecting a small subset of prompts over multiple benchmarks whose induced model scores and rankings approximate those obtained from the full benchmark suite. In evaluation-unsupervised benchmark coreset selection (our approach), the selection algorithm uses no model evaluation outcomes, and operates on a fine granularity by producing subsets of prompts over multiple benchmarks rather than producing a sub-collection of entire benchmarks. We use submodular subset selection, and we develop and evaluate many different submodular functions for this purpose, including determinantal point process (DPP) based approaches, submodular mutual information functions, and facility location-based functions. On a new large-scale suite of 35 heterogeneous benchmarks spanning five different capability categories, 18 frontier LLMs, and over 61K prompts, we find that the facility location (FL) function operating exclusively on inexpensive semantic prompt embeddings preserves LLM scores better than twelve separate score-based and diversity-based baselines, across a range of coreset budgets. Moreover, we show our proposed objective is not limited to the evaluation-unsupervised regime: in the setting where only a handful of whole benchmarks must be selected and a large amount of model scores are available, the same objective matches or outperforms state-of-the-art baselines on the MMLU and MTEB leaderboards, while being substantially cheaper to compute. Together, our results suggest that submodularity, in general, is a strong and reliable tool for benchmark compression.
Summary
This paper addresses the high computational cost of repeatedly evaluating large language models across expanding benchmark suites by developing methods to select compact coresets of individual prompts. The core contribution is an evaluation-unsupervised selection procedure that operates solely on the prompts themselves, without access to prior model scores, responses, or item-response statistics. Prompts from multiple benchmarks are first concatenated with their ground-truth fields and embedded into semantic vectors; a submodular objective is then maximized to choose a small subset whose induced scores and rankings closely match those of the full collection.
Among the submodular functions examined, facility location defined on a sparse similarity matrix of these embeddings yields the strongest results. It favors subsets that provide broad coverage across capability categories rather than concentrating on any single benchmark, avoiding the clustering bias observed in score-based and item-response-theory baselines. Experiments conducted on a newly assembled suite of 35 heterogeneous benchmarks spanning five capability areas, 18 frontier models, and more than 61 000 prompts show that facility-location coresets achieve lower normalized mean relative error in both overall and per-capability scores than twelve competing methods across budgets between 70 and 200 prompts.
The same objective remains effective when model scores are available and the task shifts to selecting entire benchmarks rather than individual prompts. In this regime it matches or exceeds prior state-of-the-art approaches on the MMLU and MTEB leaderboards while incurring substantially lower computational overhead. The findings indicate that submodular coverage criteria applied to inexpensive semantic representations offer a reliable, scalable route to benchmark compression that generalizes across both unsupervised and supervised settings.
Why it matters
This research is highly relevant for Dutch AI researchers and enterprises developing LLMs, as it offers a mathematically rigorous method to drastically reduce the computational cost and time required for model evaluation. This aligns with the European and Dutch focus on sustainable, resource-efficient AI development (Green AI).
