Long-context inference is increasingly limited by the memory and access cost of the key-value (KV) cache. Existing KV cache compression methods face a fundamental trade-off between efficiency and information preservation. Eviction-based methods are efficient but may discard critical tokens, whereas low-rank-based methods preserve all tokens at the cost of substantial computational overhead. We propose HachiWare, a dual-branch framework that applies one-shot selective SVD to semantic-critical cache states while handling less critical states with lightweight eviction. It stores these states in a Semantic-Critical Branch (SCB) using low-rank factors, while assigning the remaining states to an Eviction Branch (EB) for cheaper eviction. During decoding, bounded active-cache refresh selectively reconstructs only active SCB entries, avoiding reconstruction of the full SCB cache. On the RULER benchmark with Llama-3.1-8B-Instruct, HachiWare incurs only a 3% accuracy drop compared with the full KV cache baseline at a KV cache compression ratio of up to 16x, while improving accuracy by 5% over existing compression methods. For decoding efficiency, HachiWare achieves up to 9.44x and 16.50x throughput speedups over Full KV at 64K and 128K context lengths. These results highlight selective allocation of reconstructability as an effective way to improve the accuracy-efficiency trade-off of KV cache compression.

Although large language models (LLM) have achieved remarkable performance, their enormous parameter counts hinder deployment on resource-constrained hardware. Low-rank compression can reduce both memory usage and computational demand, but applying a uniform compression ratio across all layers often leads to significant performance degradation, and previous methods perform poorly during decoding. To address these issues, we propose the Fine-grained Low-Rank Compressor (FLRC), which efficiently determines an optimal rank allocation for each layer, and incorporates progressive low-rank decoding to maintain text generation quality. Comprehensive experiments on diverse benchmarks demonstrate the superiority of FLRC, achieving up to a 17% improvement in ROUGE-L on summarization tasks compared to state-of-the-art low-rank compression methods, establishing a more robust and efficient framework to improve LLM inference.