Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
06:00 · July 3, 2026 · arXiv cs.AI RSS
Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set toward experts favored by dominant calibration patterns. We propose \textbf{Generic TB-Coverage}, a coverage-aware expert pruning method that uses only generic text corpora (WikiText2 and C4) for calibration. Instead of collapsing expert utility into one score, our method profiles per-expert utility separately on each corpus and enforces a fixed-budget coverage rule that preserves high-utility experts from each corpus before constructing the final pruning mask. Across Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base at 25\%, 50\%, and 75\% retention budgets, our method improves average accuracy on six common zero-shot benchmarks over random pruning, REAP, and ExpertSparsity, while also reducing perplexity degradation on WikiText2 and C4. The gains are largest under aggressive pruning (25\% and 50\% retain), suggesting that preserving cross-corpus expert coverage is an effective generic-data prior for MoE pruning. Our improvements hold with fixed pruning budgets and no downstream calibration data.
Summary
Sparsely activated Mixture-of-Experts language models contain substantial structured redundancy among their routed experts, yet selecting which experts to retain without access to downstream task data remains difficult. Conventional pruning approaches typically collapse expert utility into a single aggregated importance score derived from routing frequency or reconstruction error. When calibration data contain heterogeneous patterns, this scalar ranking tends to favor experts that dominate the average signal and to discard specialists that are critical for particular generic language distributions.
Generic TB-Coverage addresses the limitation by profiling each expert independently on two generic corpora, WikiText2 and C4. For every MoE layer it computes a REAP-style utility score that multiplies router probability by expert output norm, conditioned on tokens that actually route to the expert. Rather than averaging these scores, the method constructs separate per-corpus rankings and applies a round-robin coverage rule that protects a fixed number of high-utility experts from each corpus before the final fixed-budget mask is assembled.
The resulting pruning masks were evaluated on Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base at retention ratios of 25 percent, 50 percent and 75 percent. Across six standard zero-shot benchmarks the coverage-aware masks consistently outperformed random pruning, REAP and ExpertSparsity, with the largest gains observed under the most aggressive budgets. Perplexity degradation on the calibration corpora themselves was also reduced, indicating that preserving cross-corpus expert coverage functions as an effective generic-data prior for maintaining broad language-model behavior without task-specific calibration.
Why it matters
This research is highly relevant for Dutch AI researchers and practitioners focused on optimizing large language models for cost-effective and sustainable deployment. Efficient MoE pruning aligns with the EU's push for Green AI and enables local SMEs to leverage advanced models with lower computational overhead.
