Bringing Nunchaku 4-bit Diffusion Inference to Diffusers

Summary

Nunchaku's SVDQuant method quantizes both weights and activations to 4 bits in diffusion transformers, unlike typical weight-only schemes that store low-precision parameters but dequantize them to higher precision during computation. This W4A4 approach reduces memory footprint while also accelerating the denoising loop by keeping the bulk of operations in low precision. The technique isolates large activation outliers through a low-rank branch before quantizing the residual, then relies on fused kernels to execute the 4-bit path efficiently.

Native support in Diffusers now allows these checkpoints to load through the standard from_pretrained interface, without requiring a separate inference engine or local CUDA compilation. The kernels package fetches the necessary runtime components from the Hub on first use. Nunchaku Lite achieves this by patching nn.Linear modules in a stock Diffusers model with SVDQ or AWQ layers before loading, preserving the original module structure so that schedulers, LoRA adapters, offloading utilities, and torch.compile continue to function unchanged.

On Blackwell GPUs the integration delivers roughly 50 percent lower peak VRAM and end-to-end speedups between 1.3x and 1.8x compared with BF16 baselines, with further gains when torch.compile is applied or when text encoders are also quantized via bitsandbytes NF4. Earlier GPU generations use INT4 kernels instead. The same compatibility extends to sequential and model CPU offloading for memory-constrained setups.

A companion diffuse-compressor toolkit lets practitioners inspect a model, calibrate and quantize the transformer blocks, package the result as a regular Diffusers repository, and publish it to the Hub. For architectures that benefit from fused projections, model-specific target configurations and small runtime adapters handle the necessary structural rewrites while the generic path covers the majority of transformer blocks without modification.

The result is a practical route for ML engineers to run diffusion pipelines at lower memory cost and higher throughput while retaining image quality close to the original precision.

Why it matters

Directly addresses production challenges of VRAM and latency for diffusion models with quantitative benchmarks and actionable Diffusers workflows that Dutch ML teams can apply immediately.