Given a Video Large Language Model (VLLM) and a pre-recorded video, how can we minimize the cumulative inference latency over future questions? Addressing this is crucial since VLLMs not only serve a massive volume of questions over the video’s lifespan but also incur substantial latency for each question. Existing methods compress visual tokens generated from a video to accelerate inference for each question. However, under aggressive compression ratios, prior approaches suffer from severe accuracy degradation, still resulting in high inference latency on each question. In this paper, we propose Dive, a visual KV cache compression method that preserves accuracy even under aggressive compression ratios. To maintain accuracy, Dive conserves the temporal context of the input video by evicting KV pairs after the prefill stage, rather than filtering out visual tokens before the VLLM processes the tokens through attention mechanisms. Dive significantly reduces per-question decoding latency by carefully evicting KV pairs, thereby achieving low cumulative latency over a massive volume of questions. Experiments show that Dive retains 98.2% of accuracy, using only 5% of the KV cache in Qwen2.5-VL.

How can we share parameters within large language models to significantly reduce memory costs while preserving accuracy? While parameter sharing is a promising solution to the memory overhead of large language models, existing methods rely on naive grouping and fail to correct sharing-induced discrepancies. We propose an accurate and efficient parameter sharing framework, SharVeT (Similarity-aware sharing with Vector-based Tuning), which performs similarity-based grouping to ensure accurate sharing, allocates parameters adaptively to preserve diversity within each group, and applies lightweight refinement with knowledge distillation to correct sharing-induced discrepancies. Experiments show that SharVeT outperforms existing sharing methods, achieving up to 32.1% lower perplexity and 23.3% higher few-shot reasoning accuracy.