Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory
06:00 · August 24, 2026 · arXiv cs.AI RSS
Agentic large language models (LLMs) on the Model Context Protocol (MCP) re-encode verbose tool schemas every turn, so prefill - quadratic in sequence length - dominates time-to-first-token (TTFT) as the tool registry grows. Nexus's primary lever is to decouple routing from the schema-prefill cost: an INT8 semantic lookaside buffer (SLB) with a calibrated cross-encoder margin gate selects tools by retrieval, and arguments are generated over a compressed textual signature (median 19 tokens) rather than over spliced key/value (KV) cache. This path is depth-independent: routing accuracy stays near 89% as the registry scales to 250 tools - where a concatenate-all-schemas baseline overflows the context window entirely - and it reaches a first-argument token 1.66x sooner than a full-schema re-prefill at a ~80% main-context token saving. As a secondary, bounded lever we transplant a compiled schema KV block directly into the live context. This is fundamentally limited by rotary position embedding (RoPE) phase drift: an anchored splice is output-exact, but off-anchor placement corrupts attention, so beyond a threshold P=256 Nexus repairs the seam with a depth-adaptive suffix redecode that escalates to a full re-prefill. The resulting never-regress property is a guarantee on output fidelity (top-1 agreement, D_KL approx. 0) - not on latency, which can dip to 0.98x before converging to parity - alongside a 1.1-1.7x TTFT speedup at moderate depth that narrows to parity at deep context. Two negative results bound the design: the off-anchor RoPE fidelity boundary, and the failure of a reference-free drift gate to predict drift (Spearman rho = 0.193). All measurements are from one model tuple (Qwen2.5-14B-Instruct Q4_K_M) on Apple-silicon unified memory; the qualitative boundaries generalize, while the quantitative envelope is tuple-specific.
Summary
Nexus addresses the growing prefill bottleneck in agentic LLM systems that follow the Model Context Protocol. As tool registries expand, verbose JSON schemas must be re-encoded each turn, and the quadratic cost of self-attention quickly dominates time-to-first-token. The system’s main contribution is to separate tool routing from full schema re-prefill. An INT8 semantic lookaside buffer combined with a calibrated cross-encoder margin gate performs retrieval-based selection, after which arguments are produced from a compact textual signature whose median length is 19 tokens. This routing path remains accurate at roughly 89 percent even when the registry reaches 250 tools and yields a 1.66-times reduction in first-argument latency together with an approximately 80 percent saving in main-context tokens.
A secondary mechanism transplants already-compiled schema KV blocks directly into the live context. Because rotary position embeddings are position-dependent, such transplantation is reliable only when the block is placed at its original anchor position. Beyond a 256-token offset, attention corruption appears, so Nexus applies a depth-adaptive suffix redecode whose recompute fraction grows with context depth and eventually falls back to a full prefill. The design guarantees output fidelity—top-1 agreement and near-zero KL divergence—while delivering TTFT speedups between 1.1 and 1.7 times at moderate depths that converge to parity at deeper contexts.
All measurements were obtained on a single model and hardware tuple, Qwen2.5-14B-Instruct quantized to Q4_K_M running on Apple M4 Max unified memory. Two negative findings delimit the approach: an off-anchor RoPE fidelity boundary that cannot be crossed without correction, and the inability of a reference-free drift detector to predict the onset of corruption. The retrieval-decoupled path is depth-independent and therefore the more transferable result; the KV-splice path remains a bounded, hardware-specific optimization whose quantitative envelope is specific to the evaluated configuration.
Why it matters
This research is highly relevant for AI infrastructure engineers and researchers optimizing LLM inference and agentic workflows. The techniques for reducing compute overhead and latency are directly applicable to Dutch AI enterprises and SMEs looking to deploy efficient, scalable AI agents on unified memory hardware.
