DiffusionGemma: 4x faster text generation

Summary

DiffusionGemma is an experimental 26B-parameter Mixture of Experts model that replaces the left-to-right, token-by-token decoding of conventional autoregressive language models with a diffusion process. Instead of predicting one token at a time, the model begins with a canvas of random placeholder tokens and performs multiple refinement passes over a fixed block of 256 tokens, using bi-directional attention so that every position can condition on all others. This parallel generation shifts the inference bottleneck from memory bandwidth to compute, yielding measured throughputs above 1000 tokens per second on an NVIDIA H100 and above 700 tokens per second on an RTX 5090.

The model activates only 3.8 billion parameters per forward pass and, when quantized, runs within an 18 GB VRAM budget, making it practical for local GPU deployments. Because the entire block is visible during each iteration, the approach supports tasks that benefit from global context, such as closing complex markup, performing code infilling, or solving constraint problems like Sudoku after task-specific fine-tuning. Iterative self-correction is inherent: early passes lock in high-confidence tokens that then guide later refinements, reducing the need for external post-processing.

These speed gains come with explicit quality trade-offs. DiffusionGemma’s outputs are generally lower in coherence and factual precision than those of the autoregressive Gemma 4 models from which it derives, so the authors recommend the latter for production workloads that prioritize maximum accuracy. The parallel-decoding advantage is also workload-dependent: it is most pronounced at low-to-medium batch sizes on a single accelerator and diminishes in high-concurrency cloud serving where autoregressive models can already saturate hardware through request batching. On architectures with lower compute-to-memory bandwidth ratios, such as Apple Silicon, the reported acceleration is smaller.

The release includes Apache 2.0 weights, integration examples for vLLM and MLX serving stacks, JAX-based fine-tuning recipes, and NVIDIA-optimized kernels that exploit NVFP4 quantization on both consumer and datacenter GPUs. The design therefore targets researchers and developers who need low-latency, interactive text generation on dedicated local hardware rather than maximum output quality or high-throughput cloud inference.

Why it matters

Directly addresses production latency, VRAM constraints, and parallel decoding for ML engineers building interactive local applications; provides quantitative benchmarks and tooling guidance applicable to Dutch SME and research deployments.