Visual tokens dominate the inference cost of multimodal large language models (MLLMs), but aggressive token reduction often degrades reasoning. We argue that the fragility comes from a saliency-ranking view, i.e., existing training-free methods typically rank tokens by a scalar proxy, such as cross-attention, [CLS] attention, or feature similarity, and then prune or merge tokens independently. However, visual evidence is set-valued and structured: salient patches can be redundant, low-saliency tokens can be answer-critical, and feature-similar patches can be unsafe to merge when they are spatially incompatible. We propose CondenseVLM, a simple framework that formulates visual token compression as a budgeted evidence-preservation question. The formulation jointly optimizes salient evidence coverage, contextual coverage, and spatially constrained merge fidelity. It yields a monotone submodular coverage term, a marginal-gain interpretation of diversity-aware anchor selection, and a local distortion bound for spatial-semantic merging. Comprehensive experiments across six MLLM families and diverse general-purpose, grounding, text-dense, and video benchmarks demonstrate that improves the accuracy-efficiency trade-off, and achieves the SOTA performance.