CoT-Core: Accelerating LLM Evaluation via CoT-Aware Coreset Selection
06:00 · August 4, 2026 · arXiv cs.AI RSS
Evaluating Large Language Models (LLMs) incurs prohibitive computational overhead during continuous development processes. While coreset selection accelerates evaluation, existing methods either suffer from a severe ``cold start'' bottleneck requiring massive historical logs (e.g., Item Response Theory) or exhibit a surface lexical bias that misses the underlying reasoning manifold of tasks. We propose CoT-Core, a novel training-free core question selection framework. Recognizing that lexically disparate questions can share equivalent underlying logic, CoT-Core prompts LLMs to unroll zero-shot Chain-of-Thought (CoT) reasoning trajectories. Projecting these paths into a latent space effectively clusters questions by intrinsic logical equivalence rather than superficial text similarity. Extensive experiments on GSM8K, MMLU, MMLU-Pro, and GPQA demonstrate that CoT-Core drastically reduces evaluation costs while maintaining high-fidelity score estimation, and delineate the boundary conditions of reasoning-aware pruning, revealing that its efficacy is intrinsically gated by task complexity.
Summary
CoT-Core addresses the high computational cost of repeatedly evaluating large language models on extensive benchmarks by constructing compact, representative coresets without requiring prior performance logs. Existing approaches either depend on large historical response matrices, as in Item Response Theory methods, or rely on shallow text embeddings that group questions by lexical overlap rather than shared reasoning structure. The latter approach often fails to identify questions that differ in wording yet demand equivalent logical steps, leading to suboptimal compression of evaluation suites.
The framework operates in three stages. It first elicits zero-shot Chain-of-Thought trajectories from an LLM for each benchmark item, then embeds these reasoning sequences into a latent space that reflects intrinsic logical equivalence. Standard k-means clustering is subsequently applied to select a minimal subset whose centroids preserve coverage of the underlying reasoning manifold. Because the selection step is decoupled from the final scoring procedure, the resulting coreset can be used with either simple average aggregation or more advanced probabilistic estimators such as GP-IRT.
Experiments on GSM8K, MMLU, MMLU-Pro and GPQA show that the method achieves substantial reductions in evaluation cost while producing score estimates that remain close to those obtained on the full datasets. Performance gains are most pronounced on tasks whose items exhibit diverse reasoning patterns; on simpler or more homogeneous benchmarks the advantage diminishes, indicating that the benefit of reasoning-aware pruning scales with task complexity. The approach remains entirely training-free and avoids the cold-start limitation of history-dependent techniques.
Why it matters
Directly actionable for Dutch AI researchers and advanced practitioners developing or continuously evaluating LLMs: reduces compute overhead while preserving ranking fidelity, aligns with EU emphasis on efficient and transparent AI, and requires no historical logs.
