Human language processing in the brain involves the coordinated engagement of multiple perceptual pathways. In parallel, recent multimodal large language models (MLLMs) integrate visual and linguistic information into language generation. However, it remains unclear whether and how the injection of visual information alters language representations, and how such changes in turn affect neural predictions during language processing. To address this question, we analyze joint vision–language representations extracted from MLLMs using brain encoding models. We visually augment textual stimuli via image generation and use the resulting representations to predict fMRI responses recorded while participants process language-only stimuli. We find that joint vision–language representations substantially improve neural prediction performance, with pronounced gains in higher-level visual regions. Further analyses indicate that these improvements reflect both the engagement of visual pathways and the enrichment of linguistic representations through visually grounded information. Together, our results provide evidence for multimodal pathway involvement in human language processing and offer insights into the interpretability of multimodal models.

Associative memory engages in the integration of relevant information for comprehension in the human cognition system. In this work, we seek to improve alignment between language models and human brain while processing speech information by integrating associative memory. After verifying the alignment between language model and brain by mapping language model activations to brain activity, the original text stimuli expanded with simulated associative memory are regarded as input to computational language models. We find the alignment between language model and brain is improved in brain regions closely related to associative memory processing. We also demonstrate large language models after specific supervised fine-tuning better align with brain response, by building the Association dataset containing 1000 samples of stories, with instructions encouraging associative memory as input and associated content as output.