Pi-Cheng Hsiu
Author directory2026
KV Admission: Learning What to Write for Efficient Long-Context LLM Inference
Yen-Chieh Huang | Pi-Cheng Hsiu | Rui Fang | Ming-syan Chen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yen-Chieh Huang | Pi-Cheng Hsiu | Rui Fang | Ming-syan Chen
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Long-context LLM inference is bottlenecked by the quadratic attention complexity and linear Key-Value (KV) cache growth. Prior approaches mitigate this via post-hoc selection or eviction but overlook the root inefficiency: *indiscriminate token admission*. In this paper, we formalize KV management as a causal system of three primitives: KV Admission, Selection, and Eviction. We instantiate KV Admission via Write-Gated KV (WG-KV), a lightweight mechanism that learns to predict token utility *before* cache entry. By filtering out redundant states early to maintain a compact global cache alongside a sliding local cache, WG-KV significantly reduces memory usage and accelerates both prefill and decode phases. Our results demonstrate that *learning what to write* is a principled and practical recipe for efficient long-context inference. Code is available at https://github.com/EMCLab-Sinica/WG-KV.