Given a Video Large Language Model (VLLM) and a pre-recorded video, how can we minimize the cumulative inference latency over future questions? Addressing this is crucial since VLLMs not only serve a massive volume of questions over the video’s lifespan but also incur substantial latency for each question. Existing methods compress visual tokens generated from a video to accelerate inference for each question. However, under aggressive compression ratios, prior approaches suffer from severe accuracy degradation, still resulting in high inference latency on each question. In this paper, we propose Dive, a visual KV cache compression method that preserves accuracy even under aggressive compression ratios. To maintain accuracy, Dive conserves the temporal context of the input video by evicting KV pairs after the prefill stage, rather than filtering out visual tokens before the VLLM processes the tokens through attention mechanisms. Dive significantly reduces per-question decoding latency by carefully evicting KV pairs, thereby achieving low cumulative latency over a massive volume of questions. Experiments show that Dive retains 98.2% of accuracy, using only 5% of the KV cache in Qwen2.5-VL.