Duc Duong
Author directory2026
Forget Without Compromise: Nexus Sampling for Streaming KV-Cache Eviction Under Fixed Budgets
Duc Duong | Hoang Anh Duy Le | Jianwen Xie | Anshumali Shrivastava | Zhaozhuo Xu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Duc Duong | Hoang Anh Duy Le | Jianwen Xie | Anshumali Shrivastava | Zhaozhuo Xu
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Long-context and agentic LLM workloads push the KV cache past any fixed memory budget, forcing the inference stack to permanently *evict* tokens at every step of a continuous-inference stream. Existing methods all share the same template: a per-step direct-attention score followed by *deterministic top-K* selection, which converts a single below-cutoff step into an irreversible verdict and permanently erases any subtly important token that direct attention cannot single out from noise. To address this challenge, we propose **Nexus Sampling**, a training-free eviction method that pairs *Nexus scoring*, an iterative walk over direct attention that surfaces bridge tokens, with *weighted reservoir sampling*, which retains tokens with inclusion probability in place of deterministic top-K. Theoretically, we show that Nexus Sampling dominates deterministic top-K in long-run survival of subtly important tokens. Empirically, at 80% KV cache eviction, Nexus Sampling matches dense attention within ∼1 point on LongBench while outperforming top-K baselines on retrieval-heavy tasks, with up to 10× smaller per-sequence cache memory. Our code is available at <https://github.com/duongduc388222/NexusSampling>.