As LLMs process increasingly long contexts, the KV cache has become a major source of memory and bandwidth overhead during inference. Existing methods compress the KV cache by preserving attention sinks, recent tokens, or layer-local important tokens, but they rarely exploit how token importance is shared and specialized across layers. In this paper, we introduce ConsistKV, an adaptive KV cache compression method that exploits consensus in token importance across layers. For each layer, ConsistKV aggregates importance scores from neighboring layers to identify shared contextual anchors, while using residual local importance to preserve tokens that are salient to the current layer. It further estimates the cache demand of each layer from disagreement between layers and importance dispersion, enabling adaptive budget allocation under a fixed global cache budget. Experiments on LongBench show that ConsistKV achieves the highest average performance among evaluated KV cache compression baselines under the same cache budget.