KV caching is essential for efficient autoregressive decoding, but its memory footprint grows linearly with context length and quickly becomes a bottleneck for long-context large language models (LLMs). We study whether feature-dimension compression can serve as an orthogonal second stage for KV-cache reduction, and introduce AURA-KV, an adaptive-rank subspace approximation method for KV caches. AURA-KV is training-free: it decomposes post-RoPE key and value caches into compact bases and coefficients, supports incremental decoding, and allocates rank adaptively across layers under a global memory budget. Because it compresses the feature dimension rather than selecting tokens or lowering precision, it can be combined with token eviction and quantization methods. Across LongBench, RULER, and InfiniteBench, AURA-KV preserves long-context performance on Llama-3-8B, Mistral-7B, and LongChat-7B under moderate compression and degrades gracefully under more aggressive budgets. In latency measurements, AURA-KV improves per-layer decoding speed by up to 2.75 ×, and by up to 3.23 × when combined with token-level cache reduction, with a small one-time prefill overhead.