Farid Talibli

Author directory

2026

KV cache compression is essential for long-context LLM inference. Most token-level KV compressors reduce cache length by retaining selected token KV entries and evicting the rest. Once evicted, those spans no longer receive attention, even though some may still carry information useful enough to preserve in compact form. We propose SurrogateKV, a training-free, runtime-only method that reframes KV compression as region-level representation allocation. Under the same KV-slot budget, SurrogateKV assigns historical regions to raw KV, surrogate KV, or drop. It does not train a compressor, modify model weights, or introduce a new attention operator. After prefill, SurrogateKV constructs lightweight surrogate entries as ordinary KV pairs, keeping their information available during standard attention decoding. This representation decision is orthogonal to layer- and head-wise budget allocation. We evaluate SurrogateKV-Snap, SurrogateKV-Dynamic, and SurrogateKV-Ada, obtained by applying SurrogateKV to SnapKV, DynamicKV, and Ada-KV, respectively. As representative results, SurrogateKV-Snap raises Needle-in-a-Haystack (NIAH) retrieval accuracy from 87.51 to 98.84 at 128 slots per layer. SurrogateKV-Ada raises the LongBench average score from 40.77 to 41.26 at 512 slots per layer while preserving 98.4% of full-cache performance with only 10.5% of the prefill KV cache. Code is available at https://github.com/kimjunewon/SurrogateKV.