Large language models (LLMs) have made significant progress in various NLP tasks, especially with the rise of long-context models that handle extended inputs. However, the growing size of the key-value (KV) cache in Transformer architectures imposes heavy memory demands during decoding, becoming a major bottleneck. Existing sparse attention methods aim to address this issue but face two main limitations: (1) they introduce high overhead when retrieving important tokens for attention, and (2) they often fail to significantly reduce memory usage in practice. This paper presents Seer, a simple yet effective system for fast and accurate LLM decoding on long-context sequences. Seer leverages semantic space representations to manage the KV cache more efficiently. It dynamically estimates cluster-level importance to reduce both computation and memory access, while preserving output quality. Through offline training, Seer learns semantic centroids to enable fast token importance estimation. To further improve efficiency, it integrates product quantization with sparse attention for KV cache compression and uses an asymmetric distance lookup table to reduce computational cost. Evaluation on a diverse set of LLMs and tasks shows that Seer reduces theoretical attention bandwidth by 40× with minimal accuracy loss. Seer achieves a 5.57× attention speedup and a 2× End-To-End speedup over full attention at 32K sequence length. Our code is available at https://github.com/Acaciasama/Seer.