Serving Masked Diffusion LLMs: Characterization and Design Principles from Real Hardware
06:00 · August 26, 2026 · arXiv cs.AI RSS
Masked diffusion language models (dLLMs) can in principle generate text faster than autoregressive (AR) models, since they denoise many tokens at once. Recent systems have begun building serving infrastructure for dLLMs, but none first measure how these models behave under real, concurrent serving load. Serving systems built without this grounding risk carrying over assumptions from AR serving that may not hold for dLLMs. We characterize dLLM serving to close this gap, using LLaDA-8B-Instruct with a D2F (Discrete Diffusion Forcing) LoRA adapter on a single NVIDIA H200 GPU, evaluated on GSM8K and HumanEval. We report three findings. First, request difficulty, the number of denoising steps a request needs, is discrete rather than continuous: requests fall into 11 fixed step-count levels (178 + 29k), and no signal we test predicts the level before generation starts (best R2 = 0.150). Second, benchmarks with short generation budgets below 320 tokens understate serving variance, since requests are cut off before the latency spread appears. Third, only 24% of single-request wall-clock time is GPU computation; the rest is CPU-side dispatch overhead. Batching mainly helps by amortizing this overhead: sharing one forward pass per denoising step improves throughput by 16.0x at batch size 16 over a per-request-dispatch baseline. We also argue structurally that output quality should not degrade with batch size, stating three assumptions this rests on; we measure 74 to 76% GSM8K accuracy at single-request scale. Finally, we derive a batch-timeout rule for fixed-fill synchronized batching under Poisson arrivals. Together, these results show that serving diffusion language models needs parallelism at the level of each denoising step, which differs from AR serving in how admission and eviction interact with an already shared forward pass.
Summary
Masked diffusion language models generate text by iteratively denoising an initially masked sequence rather than producing tokens sequentially. This block-parallel approach can in principle raise throughput compared with autoregressive decoding, yet serving systems built for the latter risk carrying over assumptions that do not hold when many requests share a GPU. The present study therefore measures dLLM behavior under realistic concurrent load on a single NVIDIA H200, using LLaDA-8B-Instruct equipped with a Discrete Diffusion Forcing LoRA adapter and evaluating on GSM8K and HumanEval.
Three empirical results stand out. First, the number of denoising steps required by any request falls into one of eleven discrete levels rather than varying continuously, and no pre-generation signal predicts the level with useful accuracy (maximum R² of 0.15). Second, generation-length budgets below roughly 320 tokens truncate requests before the full latency distribution appears, thereby understating serving variance. Third, GPU computation accounts for only 24 percent of single-request wall-clock time; the remaining 76 percent is CPU-side dispatch overhead. When requests share a forward pass at each denoising step, this overhead is amortized, producing a 16-fold throughput gain at batch size 16 relative to a per-request dispatch baseline.
The measurements also indicate that output quality remains stable across batch sizes provided three structural assumptions hold; single-request GSM8K accuracy stays between 74 and 76 percent. Because all requests in a synchronized batch advance together through the same denoising step, admission and eviction policies must be reconsidered relative to continuous batching used for autoregressive models. From these observations the authors derive a batch-timeout stability rule for fixed-fill synchronized batching under Poisson arrivals, supplying a concrete operating point (around 70 percent utilization) for facility-scale deployments.
Why it matters
This research is highly relevant for Dutch AI infrastructure researchers and HPC operators looking to optimize the serving of emerging diffusion LLMs. The findings on CPU bottlenecks and step-level parallelism provide actionable design principles for building efficient, scalable, and cost-effective AI inference systems in the Netherlands.
