Cross-Lingual Activation Steering for Visual-Prior Conflicts

Yufei Wang, Xiang Lorraine Li, Adriana Kovashka


Abstract
Multimodal large language models (MLLMs) must often arbitrate between visual evidence and world-knowledge priors when an image conflicts with commonsense expectations. Existing work has demonstrated that activation steering vectors can bias model outputs toward either source of information, but it remains unclear whether this arbitration mechanism generalizes across languages. To investigate this, we construct a multilingual extension of the Visual CounterFact color split and derive activation steering directions that contrast image-grounded answers with prior-based answers. Across 12 languages, we find that English-derived and shared multilingual steering directions consistently shift model outputs on non-English prompts, suggesting that visual-prior arbitration has a substantial cross-lingual component. At the same time, we find that performance gains on common knowledge questions can come at the cost of image-grounded accuracy, a trade-off that is particularly pronounced for lower-resource languages. Code and data are available at https://github.com/elffiona/cross-lingual-visual-prior-steering.
Anthology ID:
2026.findings-emnlp.1280
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
27124–27135
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.1280/
DOI:
Bibkey:
Cite (ACL):
Yufei Wang, Xiang Lorraine Li, and Adriana Kovashka. 2026. Cross-Lingual Activation Steering for Visual-Prior Conflicts. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 27124–27135, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Cross-Lingual Activation Steering for Visual-Prior Conflicts (Wang et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.1280.pdf