SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation
06:00 · July 21, 2026 · arXiv cs.AI RSS
Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottleneck. Recent compression methods mitigate this cost via token merging; however, these approaches often rely on indiscriminate aggregation, which degrades representations and introduces attention sag, a mismatch where merged tokens receive the same softmax mass as individual tokens despite encoding multiple inputs. We propose a training-free, dual-component framework for KV cache compression that addresses these limitations. First, a soft cosine gate adaptively modulates merging decisions based on value-vector similarity, suppressing or discarding dissimilar tokens to preserve semantic fidelity. Second, we introduce an attention-ratio compensation mechanism that applies a decoding-time logit bias derived from prefill attention statistics, correcting the softmax imbalance induced by merging. Evaluated on LongBench (16 English datasets) while retaining only 25% of the KV cache, our framework achieves strong compressed performance against representative one-shot baselines. It is especially robust on the evaluated grouped-query attention (GQA) models, maintaining nearlossless generation quality. Furthermore, the method outperforms the full-cache baseline on complex multi-document QA tasks and delivers a 3.3x decoding speedup at 100k tokens.
Summary
SelKV addresses the memory bottleneck created by the growing key-value cache in autoregressive large language models. The cache stores past key and value vectors for attention computation and scales linearly with sequence length, quickly consuming gigabytes even at moderate context sizes. While recent token-merging approaches reduce this footprint by consolidating less important entries into retained ones, they typically apply a uniform merge-or-drop rule to all evicted tokens and leave merged positions under-attended during softmax normalization, an effect known as attention sag.
The method introduces two training-free components that can be inserted into existing merging pipelines. A soft cosine gate computes the cosine similarity between each evicted token’s value vector and its chosen merge target, then uses this value to scale merge intensity continuously between zero and one. Tokens that are highly similar merge fully, dissimilar ones are discarded, and intermediate cases receive partial merging, avoiding the representation corruption that occurs when dissimilar vectors are averaged unconditionally. An attention-ratio compensation step then adds a decoding-time logit bias derived from prefill attention statistics, restoring the attention mass that merged positions would otherwise lose relative to the number of original tokens they represent.
Evaluated on the sixteen English tasks of LongBench at a 25 percent KV-cache retention budget, SelKV maintains near-lossless performance on grouped-query attention models and surpasses the full-cache baseline on several multi-document question-answering settings. At one hundred thousand tokens the compressed cache yields a 3.3 times decoding speedup. The paper supplies the full algorithms, ablation studies, and comparative results against representative one-shot eviction and merging baselines, making the work directly usable for researchers working on long-context inference efficiency.
Why it matters
High technical depth and novelty in LLM inference optimization make it directly actionable for Dutch researchers and advanced practitioners developing efficient AI infrastructure.
