Depth-Aware Sensitivity Analysis of Mixture-of-Experts Models via Magnitude-Based Expert Masking
06:00 · August 17, 2026 · arXiv cs.AI RSS
Mixture-of-Experts (MoE) architectures scale large language models (LLMs) while preserving computational efficiency through sparse activation. Despite their widespread adoption, the relative importance of individual MoE layers remains insufficiently characterized, particularly for model compression. This paper presents a systematic layer-wise sensitivity analysis of the Qwen3.6-35B-A3B model (40 MoE layers, 256 experts per layer, top-8 routing) using magnitude-based expert masking on the XLCoST cross-lingual code translation benchmark. We conduct a multi-phase study spanning 100, 300, and 500 prompt evaluation scales across three H100 GPU servers. Our central finding is that layer sensitivity is strongly depth-dependent: early layers (0-9) and middle layers (10-29) are highly fragile to expert masking, while late layers (30-39), and especially very-late layers (35-39), tolerate aggressive masking of low-magnitude experts. Flat all-layer masking at 30% retains only 150/300 Good+Similar outputs at 300-prompt scale, whereas late-focused policies retain 249-255/300 while masking 640-1,145 experts. On a later 500-prompt held-out validation slice, the narrow very-late policy (layers 35-39 @ 50%) achieves the strongest quality/masked-expert tradeoff among tested candidates, retaining 419/500 Good+Similar outputs while masking only 640 of 10,240 total experts. We additionally characterize top-k routing width reduction from 8 to 6 active experts per token, which shows a large observed wall-clock reduction on a 100-prompt probe with no Good+Similar loss, though it does not yet compose cleanly with aggressive expert masking. These findings provide an empirical foundation for depth-aware MoE expert masking and establish a practical path toward physical weight surgery, activation-based expert scoring, and training-based recovery.
Summary
Mixture-of-Experts models achieve computational efficiency by activating only a subset of experts per token, yet the contribution of individual layers to overall model behavior has remained poorly quantified. This work examines the Qwen3.6-35B-A3B architecture, which contains 40 MoE layers with 256 experts each and routes the top-8 experts per token. Using magnitude-based masking of low-magnitude experts, the authors evaluate sensitivity on the XLCoST cross-lingual code translation benchmark across evaluation scales of 100, 300, and 500 prompts.
The central observation is that layer sensitivity varies sharply with depth. Early layers (0–9) and middle layers (10–29) suffer rapid degradation when experts are masked, whereas late layers (30–39), and particularly the final five layers (35–39), maintain output quality under substantially more aggressive masking. Uniform 30 % masking across all layers retains only 150 of 300 Good+Similar outputs at the 300-prompt scale, while policies that concentrate masking on late layers preserve 249–255 outputs while removing between 640 and 1 145 experts. On a held-out 500-prompt validation set, restricting 50 % masking to layers 35–39 yields 419 retained outputs while masking only 640 of the total 10 240 experts.
The study also records that narrowing the routing width from eight to six active experts produces a clear reduction in wall-clock time on a 100-prompt probe without measurable quality loss, although this change does not combine cleanly with the expert-masking regimes. The results supply an empirical basis for depth-aware compression strategies and indicate concrete next steps toward weight-level surgery and activation-based expert scoring.
Why it matters
This research is highly relevant for Dutch AI researchers and engineers focused on optimizing Large Language Models for efficient deployment. By providing a method to compress MoE models without sacrificing performance, it supports the Netherlands' push for sustainable, cost-effective AI solutions that lower the barrier to entry for SMEs.
