Eashan Dash

Author directory

2026

When serving LLM agents in production, the KV cache grows over turns, and mainstream serving engines (e.g., vLLM) rely on least recently used (LRU) eviction to manage memory pressure. While LRU is well-understood for OS pages and single-turn LLM serving, its behavior on agent workloads remains underexplored. We present an inverted age-importance phenomenon in agent workflows, where the oldest KV blocks (system prompts, tool outputs) are the most persistently attended, while the .newest blocks (chain-of-thought reasoning) decay fastest. As a result, recency-based eviction systematically discards the wrong blocks in agent workloads, producing two distinct overheads: (1) wasted prefill work on preempt-discard engines, and (2) wasted PCIe bandwidth on CPU-offload engines. We propose RoleKV, a training-free, lossless eviction policy that evicts KV blocks based on their semantic role in agent workflows. A lightweight Role Tagger assigns each block to a semantic role, and RoleKV then evicts by role-aware decay rather than recency. Unlike model-level KV compression methods that incur quality loss, RoleKV is a system-level policy that decides where blocks reside (GPU vs. CPU/disk), without affecting model outputs. RoleKV cuts preemptions by up to 64.3%, PCIe traffic by up to 63.8%, and improves end-to-end latency by up to 1.41× on 8B-70B models, with gains that grow on longer agent traces and persist on multi-GPU setups. This recasts KV eviction in production LLM serving from a recency problem to a role-aware retention problem.