Up to 3.2x Faster Inference with LFM2.5-DSpark
18:52 · August 20, 2026 · Hugging Face Blog
Summary
Speculative decoding mitigates the memory-bound character of the decode phase in large language model inference, where most latency stems from repeated weight transfers between DRAM and SRAM. DSpark implements this by pairing a target model with a compact draft model that proposes blocks of candidate tokens; the target then verifies the entire block in one forward pass, amortizing the cost of loading its parameters.
The draft models released for the LFM2.5 series follow the DSpark architecture but use a simplified attention-only design with five layers and a block size of nine. Each draft contains roughly 300 million parameters and was trained for up to fifteen epochs on a broad mixture of supervised fine-tuning, chat, code, and function-calling data. The epoch with the highest token acceptance rate was retained for each model rather than the one with lowest training loss.
Because acceptance is determined by exact match with the target model’s output distribution under greedy decoding, rejected tokens are replaced by the target’s own prediction. The final sequence is therefore identical to standard greedy generation, preserving benchmark metrics such as pass@1 or exact match without modification.
Integration support is provided for both llama.cpp with experimental Metal kernels on Apple silicon and the official SGLang implementation. Throughput measurements on an M4 Max MacBook Pro (FP16 GGUF) and a single H100 80 GB (BF16) show consistent gains across five evaluation sets, reaching a peak of 3.2 times higher tokens per second. Latency reductions average 57 percent for the 2.6 B model in multi-turn tool-use scenarios, while the 1.2 B and 8 B mixture-of-experts variants exhibit dataset-dependent improvements limited in the latter case by current MoE kernel behavior.
Setup requires the corresponding patched builds (llama.cpp PR 27383, SGLang PR 31041) together with the draft checkpoints published on Hugging Face in both Safetensors and GGUF formats. The block size is read automatically from the draft configuration, and the same command without the speculative flags serves as the baseline.
Why it matters
Directly addresses production inference challenges like memory-bound decode latency and GPU/edge deployment for ML Engineers, with quantitative benchmarks and open implementations applicable in Dutch AI workflows.
