Akashic: A Low-Overhead LLM Inference Service with MemAttention
06:00 · July 8, 2026 · arXiv cs.AI RSS
Recent LLM-based agent systems continuously accumulate context across multi-turn interactions, tool invocations, and cross-session workflows. Replaying the full history for every request quickly becomes impractical: long contexts increase prefill cost, may exceed context limits, and often bury task-relevant evidence in irrelevant content, degrading both serving efficiency and output quality. We propose Akashic, a low-overhead memory system built around MemAttention, which organizes context into bounded chunks and models semantic relationships across chunks, preserving cross-chunk evidence without repeatedly rewriting the full history. Akashic further applies hardware-software co-designed memory placement to co-locate likely co-retrieved chunks, reducing retrieval fragmentation and I/O overhead. Across four representative workloads and three model sizes, Akashic improves task accuracy by up to 10.2 points, throughput by up to 1.21x, and sustainable request rate by up to 1.88x over strong prior memory baselines.
Summary
Akashic addresses the growing strain on LLM inference services caused by agent systems that accumulate context across multi-turn interactions, tool calls, and cross-session workflows. Repeatedly replaying full histories drives up prefill costs, risks exceeding context windows, and often buries task-relevant evidence amid noise, degrading both serving throughput and output quality. The system introduces a memory layer centered on MemAttention that maintains context without forcing complete history rewrites on every request.
MemAttention partitions accumulated context into bounded chunks and performs cross-chunk inference to reconcile each new chunk with a small set of semantically related prior chunks. This approach preserves evidence that spans chunk boundaries while keeping per-update overhead bounded, in contrast to whole-context summarization methods that mix unrelated topics or segment-level methods that fragment dependent information. The design therefore avoids the granularity trade-off that arises when fixed global compression policies encounter non-uniform information density across workloads.
Akashic complements this mechanism with a hardware–software co-designed memory manager that places chunks likely to be retrieved together in contiguous storage. By reducing physical fragmentation and read amplification during concurrent serving, the manager lowers I/O overhead and contention without altering the semantic retrieval logic. Evaluations across four workloads—LoCoMo, SWE-bench, BrowseComp, and WebArena—and three model sizes show consistent gains, with task accuracy improving by up to 10.2 points, throughput by up to 1.21×, and sustainable request rate by up to 1.88× relative to strong prior memory baselines.
Why it matters
This research is highly relevant for Dutch AI researchers and infrastructure engineers focusing on efficient and scalable LLM deployment. The proposed MemAttention mechanism offers actionable insights for reducing computational overhead and improving the sustainability of AI services, aligning with the Netherlands' push for cost-effective and green AI solutions.
