Making Knowledge Distillation Cheap Enough to Run at Scale
12:05 · August 10, 2026 · Hugging Face Blog
Summary
Knowledge distillation has become a standard route to compress large open-source language models into deployable sizes, yet the training step itself has remained prohibitively expensive. The conventional online approach keeps both teacher and student in memory while computing a full-vocabulary probability distribution at every step, quickly exhausting hundreds of gigabytes of VRAM once sequence lengths reach tens of thousands of tokens. For a 120-billion-parameter model with a 201 k vocabulary and a 32 k context, the probability tensor alone can consume roughly 50 GB in bfloat16 before gradients, activations, and optimizer states are added, pushing peak usage near 250 GB.
Two targeted systems changes remove this barrier. First, the teacher is run only once; its top-100 logits per token position are cached offline, eliminating the need to reload the teacher or recompute its outputs during student training. Second, the Kullback-Leibler divergence loss is reformulated as a fused, chunked kernel that never materializes the full vocabulary-by-sequence matrix. Instead of allocating the entire comparison grid, the kernel processes and discards successive slices, keeping memory bounded by a single chunk. On an H200, this combination reduces peak memory from 250 GB to roughly 128 GB at 8 k context and delivers a 15.6× reduction at 32 k context, where the dense formulation fails outright.
Empirical results confirm that accuracy is preserved. Training-loss curves for the offline top-K method overlap with those of full online distillation, and downstream scores on BoolQ, HellaSwag, and MMLU remain within a few points of the teacher despite halving the parameter count. Throughput improves markedly as well: at 32 k context the same workload moves from four GPU nodes to a single H200, cutting step time by a factor of five and raising per-GPU utilization from 74 to 346 TFLOP/s. The chunked loss implementation has been released publicly, enabling teams to run large-scale distillation campaigns and long-context recovery on modest hardware.
Why it matters
Directly addresses ML Engineer pain points in LLM distillation: VRAM constraints, distributed vs single-GPU setups, and loss implementation efficiency with concrete metrics and code. Actionable for Dutch teams iterating on model compression under compute limits.
