Smaller, faster, safer: running Kimi and GLM at scale
15:00 · August 3, 2026 · Cloudflare AI Blog
Serving frontier models like Kimi and GLM means fighting for GPU memory. Here's how we quantize KV caches, compress model weights, and add integrity checks to serve them faster, cheaper, and safely.
Summary
Cloudflare’s Workers AI platform runs inference for demanding open models such as Moonshot’s Kimi K-series and Z.ai’s GLM on GPUs located in its edge data centers. These large, long-context mixture-of-experts models place heavy demands on memory, with the KV cache often becoming the limiting factor before model weights themselves. To increase capacity and throughput without accuracy loss, the team applies three targeted optimizations on top of an existing separation of prefill and decode phases, all implemented and benchmarked with the SGLang inference framework.
The first technique replaces the default BF16 KV cache with FP8 (e4m3) storage. On Kimi K2.6 this change doubles the number of tokens that fit in memory, from roughly 686,000 to 1.37 million. Although the narrower format adds a small conversion cost per token, the net effect at high concurrency is substantial: a disaggregated H200 deployment sustains 64 concurrent requests and reaches 2,192 tokens per second, roughly 41 percent above the BF16 peak before out-of-memory errors occur. Prefill remains in BF16 because it is compute-bound rather than memory-bound.
For GLM 5.2 the weights are further compressed from FP8 to INT4, shrinking the checkpoint from 705 GB to 421 GB and reducing per-GPU memory from about 88 GB to 52 GB in an eight-way tensor-parallel setup. This leaves room for approximately 1.18 million tokens of KV cache on the same hardware. Decode throughput improves markedly at low to moderate concurrency because less data must cross the memory bus; gains range from 55 percent at one request to 16 percent at 64 requests. Prefill stays in FP8 to avoid the overhead of expanding INT4 weights during compute-bound work. Across standard benchmarks the INT4 and FP8 configurations remain statistically indistinguishable.
Because both forms of compression allow far more requests to share the same physical KV cache, the team added an integrity-checking layer. Each physical cache page receives a tag that changes on reallocation; before any decode step reads from the cache, the server verifies that the pages and tags match the requesting session. Mismatches cause the affected request to abort. The check adds less than one percent overhead to throughput and tail latency and can be disabled entirely for deployments that do not require it. Together these measures let Cloudflare serve the models to more users at lower cost while preserving output quality and protecting against cache misuse in multi-tenant environments.
Why it matters
Provides actionable security measures (integrity checks) and efficiency techniques applicable to Dutch AI teams running inference workloads, with direct relevance to secure multi-user GPU serving.
