LFM2.5 Q4\_0 Checkpoints from Quantization-Aware Distillation

Summary

Quantization-Aware Distillation produces Q4_0 checkpoints for the LFM2.5 family that close much of the accuracy gap left by conventional post-training quantization. On four model sizes ranging from 230 M to 2.6 B parameters, the resulting GGUF files retain between 96.5 % and 97.4 % of the corresponding BF16 baseline when evaluated on GPQA Diamond, MMLU-Pro, IFEval, IFBench, Multi-IF, BFCLv4 and a scale-appropriate math suite (GSM8K or AIME25). All scores are reported as means over five runs, with the BF16 GGUF serving as the in-format performance ceiling.

Decode throughput was measured on four representative edge platforms. MacBook Pro and NucBox EVO-X2 runs used GPU inference, while Samsung Galaxy S26 Ultra and Raspberry Pi 5 runs used Arm CPU inference. Under these conditions the 230 M and 350 M QAD checkpoints match the quality of Q5_K_M files while delivering 4–33 % higher tokens per second. The 1.2 B and 2.6 B checkpoints reach parity with Q4_K_M at a 3–14 % throughput advantage and also match the external Unsloth UD-Q4_K_XL reference where available.

The QAD artifacts are released as standard GGUF Q4_0 files and can be loaded directly by llama.cpp or any compatible runtime. They are hosted on Hugging Face under the LFM2.5-230M, LFM2.5-350M, LFM2.5-1.2B-Instruct and LFM2.5-2.6B repositories.

Why it matters

Directly addresses production quantization, throughput optimization, and benchmark-driven evaluation for efficient inference, enabling Dutch ML engineers to deploy high-quality small models under VRAM and latency constraints.