LFM2.5-Encoders for Fast Long-Context Inference on CPU

Summary

LFM2.5-Encoders are bidirectional models released in two sizes, 230 million and 350 million parameters, that were derived from the LFM2.5 decoder family. The conversion starts from the original causal LFM2 backbones and applies targeted architectural adjustments to enable full bidirectional attention, followed by a two-stage pre-training process that uses a masked-language modeling objective. The resulting encoders support an 8,192-token context and are intended for high-volume understanding workloads such as classification, intent routing, safety filtering, and token-level extraction that typically run on CPU.

On standard GLUE and SuperGLUE benchmarks plus multilingual classification tasks, the 350M model places fourth among fourteen evaluated systems, ahead of several larger models including one nearly ten times its size. The 230M variant outperforms ModernBERT-base and all reported EuroBERT models while remaining smaller than most of them. Both encoders also exceed the performance of the earlier LFM2.5-Retrievers on the same suite. Results are reported as means over five held-out random seeds, with the full evaluation framework released openly.

Inference measurements highlight the largest gains on CPU. At the full 8,192-token length, LFM2.5-Encoder-230M completes a forward pass in roughly 28 seconds compared with more than 90 seconds for ModernBERT-base, a 3.7× improvement. Throughput for the new encoders remains higher across the entire length range, with the advantage widening as sequence length increases. On GPU the margin narrows, yet the LFM2.5 models still lead once inputs exceed approximately 2,000 tokens. These characteristics make the encoders practical for scanning long documents such as contracts or transcripts on ordinary laptop hardware.

The models are distributed as open weights on Hugging Face and integrate directly with the transformers library. Users can load them for masked-token prediction or attach task-specific heads for classification, regression, or retrieval. A separate fine-tuning guide demonstrates adaptation to long-context legal documents at 8k tokens. The release positions the encoders as a cost-effective alternative to generative models for production NLP pipelines that must remain inexpensive to run continuously on existing CPU infrastructure.

Why it matters

Strong match for ML Engineers: delivers concrete implementation details, latency benchmarks, CPU memory advantages, and actionable fine-tuning guidance for long-context encoders. Directly addresses accuracy-vs-cost trade-offs in high-volume inference workloads.