Knowledge-Based Visual Question Answering (KB-VQA) requires retrieving external textual knowledge to answer visual queries. However, current methods often suffer from two critical misalignments during retrieval: distributional misalignment, driven by the severe stylistic shift from descriptive pre-training image captions to formal knowledge texts, and granularity misalignment, where lengthy knowledge documents contain excessive noise that drowns out key information, making it difficult to accurately match the visual query. To resolve these challenges, we propose RGCC, a Retriever-Guided Context Compression framework. By mining alignment-friendly data and further training a context compression model, RGCC condenses lengthy documents into compact, retriever-preferred representations that significantly improve cross-modal retrieval. Extensive experiments on the InfoSeek and E-VQA benchmarks demonstrate that RGCC achieves state-of-the-art retrieval performance and boosts downstream question-answering accuracy without additional online inference overhead.