Native-speed vLLM transformers modeling backend

Summary

The transformers library serves as the reference implementation for more than 450 model architectures, with an emphasis on self-contained code that remains readable for contributors who later port models to inference engines. Last year’s integration of transformers as a modeling backend inside vLLM already let users run any compliant model without writing a separate vLLM port; vLLM supplied continuous batching and attention kernels while transformers supplied the forward pass.

A recent update extends that backend so that it now matches, and sometimes exceeds, the throughput of hand-written vLLM implementations. The improvement rests on two runtime techniques. First, torch.fx traces the model into a static graph and identifies patterns that can be fused for inference. Second, the detected patterns are rewritten in place by manipulating the Python abstract syntax tree, automatically inserting fused operators such as MergedColumnParallelLinear, QKVParallelLinear, and expert-parallel kernels for Mixture-of-Experts layers.

Benchmarks on the Qwen3 family (4 B, 32 B, and 235 B-FP8) show parity or better performance under single-GPU, tensor-parallel, and combined data-plus-expert-parallel regimes on eight H100 GPUs. The same flag that activates the backend, --model-impl transformers, works unchanged with torch.compile and CUDA graphs, preserving both training-time usability and the ability to serve models that were never manually ported. Linear-attention models and repository-hosted custom code remain unsupported for now.

Why it matters

Directly addresses production inference challenges (latency, VRAM, distributed parallelism, fused kernels) with quantitative benchmarks and immediately actionable commands for ML engineers serving LLMs at scale.