KV cache compression is critical for long-context LLM inference, yet existing training-free methods fall into a dilemma: attention-based eviction clusters retained tokens around semantically “hot” regions, creating context holes that degrade long-range reasoning, while diversity-based selection retains directionally broad but semantically irrelevant tokens. We identify the root cause: at prefill time, future decoding queries are unknown, making query-conditioned signals unreliable as a sole selection criterion. The retained KV set must instead provide broad directional coverage of the context—so that arbitrary future queries can find supporting keys—while filtering out tokens that expand the coverage span without contributing to generation. We propose AnchorKV, which realizes this principle through block-wise partitioning with uniform per-block budgets and anchor-driven cosine dissimilarity scoring for linear-time diversity estimation. A compression-rate adaptive hybrid score dynamically reweights diversity and importance. Experiments on LongBench across three model families show AnchorKV consistently outperforms existing training-free methods, with the largest margins at low retention ratios. A query-subsampled variant reduces prefill latency by up to 2.7× with minimal accuracy loss. We provide theoretical guarantees on approximation error and variance reduction.

Large Language Models (LLMs) face significant memory and latency overheads during long-context inference due to the growing KV cache, especially in Knowledge Base Question Answering (KBQA) settings that require support for multiple downstream queries. Query-aware eviction methods do not generalize across queries, while existing query-agnostic approaches rely on a single proxy query, leading to fragile eviction decisions under high eviction ratios. We propose ContrastKV, a robust query-agnostic KV cache eviction algorithm for multi-query generalization. ContrastKV introduces a contrastive signal fusion mechanism that jointly exploits complementary semantic and non-semantic signals. By contrasting semantic consistency with structural robustness, the method constructs a more reliable eviction criterion that alleviates the blind spots of single-query proxies. The framework integrates efficient signal generation, parallel importance scoring, and multi-level fusion across heads and layers. Experiments show that ContrastKV outperforms state-of-the-art methods, retaining up to 92% accuracy with only 20% of the KV cache budget, while reducing decoding latency by approximately 50% and significantly lowering GPU memory usage.