The success of vision-language models is primarily attributed to effective cross-modal alignment between vision and language. However, modality gaps persist even in well-aligned models and may be necessary for human perception, as evidenced by modality-specific phenomena such as visual texture and linguistic tone. These observations motivate us to computationally measure and leverage modality gaps to explore their utility in downstream applications. In this paper, we introduce the Modality Dominance Score (MDS), which attributes multimodal features to specific modalities by categorizing them as vision-dominant, language-dominant, or cross-modal. We then propose automatic interpretability metrics to evaluate these modality-specific features in a scalable manner. Finally, we demonstrate how the identified modality-specific features enable training-free probing and editing methods for understanding model perception across genders, generating adversarial examples, and controlling text-to-image generation. Combined with task-agnostic interpretability tools, our work provides a systematic framework for analyzing and efficiently controlling multimodal models.

Regret in large language models (LLMs) refers to their explicit expression of regret when confronted with evidence contradicting previously generated misinformation. Understanding the neural encoding of regret is crucial for advancing artificial metacognition and model reliability. However, three key gaps remain: the absence of specialized datasets for regret expressions, the lack of understanding of how early regret signals emerge in hidden states, and the unknown neuron-level architecture underlying regret encoding. To address these gaps, we make three contributions: (1) we propose a multi-stage workflow for constructing the first regret dataset and train probes achieving near-perfect accuracy across LLaMA-2-7B, Mistral-7B, and Vicuna-13B. (2) we discover that regret signals begin emerging in preceding hidden states before the regret token appears, and that probes generalize to semantically related keywords such as “incorrect” and “apologize.” (3) at the neuron level, we introduce the Regret Dominance Score (RDS) and Group Impact Coefficient (GIC) to reveal a compositional regret architecture where cooperative neuron groups collectively encode regret, with larger models (70B) exhibiting more pronounced compositional effects. Targeted interventions on these neurons successfully suppress regret expression in generated text, validating our mechanistic understanding. Our code and data are available at https://github.com/AbnerAI/LLMs-Regret-Before-They-Say-It.