Long-form audio makes the Key-Value (KV) cache the dominant bottleneck in speech LLMs, yet existing KV compression methods commit to an irreversible retention set at prefill. However, we reveal a severe attention mismatch in this static paradigm particularly for long-form audio: prefill attention concentrates at the audio’s start, whereas decode-time attention distributes broadly. This weak overlap between their top-attended positions leads to performance degradation, with generation collapsing into repetition at extreme compression rates. To bridge this gap, we propose WnW (Waxing-and-Waning KV cache), which makes eviction reversible by partitioning KV-heads into three roles via offline calibration. Anchor heads keep all audio KV on GPU and provide a per-step signal of which audio region the current token needs; tidal heads, guided by that signal, recall evicted KV from CPU on demand; fixed heads discard theirs permanently. Evaluations on LibriSpeech-Long Benchmark across two 3B backbones (Voxtral-Mini and Qwen2.5-Omni) demonstrate that WnW preserves near-full-cache accuracy while maintaining only 20% of audio tokens on GPU—a regime where irreversible methods fail to terminate. WnW exhibits strong generalization across languages, tasks, and domains, with CPU-GPU chunk swapping introducing negligible decoding overhead. More broadly, WnW transforms KV compression into a reversible memory hierarchy in which a small set of calibrated observer heads governs the placement of the remaining cache.