Visual instruction tuning adapts pre-trained Multimodal Large Language Models (MLLMs) to follow human instructions for real-world applications. However, rapidly growing instruction datasets introduce substantial redundancy and computational costs. Existing data selection methods aim to prune this redundancy, but predominantly rely on computationally demanding proxy-based inference or training-based metrics, often exacerbating the efficiency bottlenecks they are intended to resolve. To address this challenge, we identify a critical yet overlooked factor: anisotropy in visual feature distributions. We find that this anisotropy induces a Global Semantic Drift that distorts similarity estimates and limits efficient data selection. Motivated by this insight, we devise PRISM, a training-free framework for efficient visual instruction selection. PRISM mitigates the dominant global drift through implicit re-centering, allowing intrinsic visual semantics to provide a reliable redundancy signal without proxy models or additional training. Empirically, PRISM reduces end-to-end GPU time by about 70% while simultaneously enhancing performance, surpassing models fine-tuned on the full dataset across multimodal and language understanding benchmarks and reaching 101.7% relative performance against the full-data baseline.

Multimodal Large Language Models (MLLMs) enhance visual tasks by integrating visual representations into large language models (LLMs). The textual modality, inherited from LLMs, enables instruction following and in-context learning, while the visual modality boosts downstream task performance through rich semantic content, spatial information, and grounding capabilities. These modalities work synergistically across various visual tasks. Our research reveals a persistent imbalance between these modalities, with text often dominating output generation during visual instruction tuning, regardless of using full or parameter-efficient fine-tuning (PEFT). We found that re-balancing these modalities can significantly reduce trainable parameters, inspiring further optimization of visual instruction tuning. To this end, we introduce Modality Linear Representation-Steering (MoReS), which re-balances intrinsic modalities by steering visual representations through linear transformations in the visual subspace across each model layer. We validated our approach by developing LLaVA Steering, a suite of models using MoReS. Results show that LLaVA Steering requires, on average, 500 times fewer trainable parameters than LoRA while maintaining comparable performance across three visual benchmarks and eight visual question-answering tasks. Finally, we introduce the LLaVA Steering Factory, a platform that enables rapid customization of MLLMs with a component-based architecture, seamlessly integrating state-of-the-art models and evaluating intrinsic modality imbalance. This open-source project facilitates a deeper understanding of MLLMs within the research community.