Runze Xia
Author directory2026
See Beyond Words: Joint Visual–Language Representations Improve Brain Encoding of Language-Only Stimuli
Yongpeng Zhang | Congchi Yin | Runze Xia | Xuyun Wen | Piji Li
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Yongpeng Zhang | Congchi Yin | Runze Xia | Xuyun Wen | Piji Li
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Human language processing in the brain involves the coordinated engagement of multiple perceptual pathways. In parallel, recent multimodal large language models (MLLMs) integrate visual and linguistic information into language generation. However, it remains unclear whether and how the injection of visual information alters language representations, and how such changes in turn affect neural predictions during language processing. To address this question, we analyze joint vision–language representations extracted from MLLMs using brain encoding models. We visually augment textual stimuli via image generation and use the resulting representations to predict fMRI responses recorded while participants process language-only stimuli. We find that joint vision–language representations substantially improve neural prediction performance, with pronounced gains in higher-level visual regions. Further analyses indicate that these improvements reflect both the engagement of visual pathways and the enrichment of linguistic representations through visually grounded information. Together, our results provide evidence for multimodal pathway involvement in human language processing and offer insights into the interpretability of multimodal models.
DUD: Decoupled Update Dynamics for Reliable Uncertainty Quantification in Large Language Models
Yixin Bu | Runze Xia | Guanyun Zou | Yupeng Ji | Hongliang Dai | Haodong Liu | Piji Li
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Yixin Bu | Runze Xia | Guanyun Zou | Yupeng Ji | Hongliang Dai | Haodong Liu | Piji Li
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Accurate Uncertainty Quantification (UQ) is critical for reliable deployment of Large Language Models (LLMs), yet traditional probability-based metrics often fail to capture the model’s true epistemic state. While recent mechanistic approaches leverage hidden state dynamics, they typically aggregate residual stream updates, conflating the distinct roles of parametric memory (Feed-Forward Networks) and contextual processing (Attention). We argue that this aggregation obscures fine-grained mechanistic conflicts, such as memory-context misalignment, that are fundamental indicators of uncertainty. To address this, we introduce Decoupled Update Dynamics (DUD), a framework that explicitly decouples FFN and Attention contributions via noise-induced causal interventions. By quantifying the independent restoration capabilities of each module, we construct a dual-stream dynamic profile that captures the model’s internal fragility. Extensive experiments demonstrate that DUD significantly outperforms state-of-the-art baselines in both uncertainty estimation and calibration, while exhibiting superior cross-dataset generalization, validating decoupled dynamics as a robust proxy for model faithfulness.