Hierarchical Processing of Visual and Language Information in the Brain

Haruka Kawasaki, Satoshi Nishida, Ichiro Kobayashi


Abstract
In recent years, many studies using deep learning have been conducted to elucidate the mechanism of information representation in the brain under stimuli evoked by various modalities. On the other hand, it has not yet been clarified how we humans link information of different modalities in the brain. In this study, to elucidate the relationship between visual and language information in the brain, we constructed encoding models that predict brain activity based on features extracted from the hidden layers of VGG16 for visual information and BERT for language information. We investigated the hierarchical characteristics of cortical localization and representational content of visual and semantic information in the cortex based on the brain activity predicted by the encoding model. The results showed that the cortical localization modeled by VGG16 is getting close to that of BERT as VGG16 moves to higher layers, while the representational contents differ significantly between the two modalities.
Anthology ID:
2022.findings-aacl.38
Volume:
Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022
Month:
November
Year:
2022
Address:
Online only
Editors:
Yulan He, Heng Ji, Sujian Li, Yang Liu, Chua-Hui Chang
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
405–410
Language:
URL:
https://aclanthology.org/2022.findings-aacl.38
DOI:
Bibkey:
Cite (ACL):
Haruka Kawasaki, Satoshi Nishida, and Ichiro Kobayashi. 2022. Hierarchical Processing of Visual and Language Information in the Brain. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 405–410, Online only. Association for Computational Linguistics.
Cite (Informal):
Hierarchical Processing of Visual and Language Information in the Brain (Kawasaki et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-aacl.38.pdf