Which Sense Dominates Multisensory Semantic Understanding? A Brain Decoding Study

Dandan Huang, Lu Cao, Zhenting Li, Yue Zhang


Abstract
Decoding semantic meanings from brain activity has attracted increasing attention. Neurolinguists have found that semantic perception is open to multisensory stimulation, as word meanings can be delivered by both auditory and visual inputs. Prior work which decodes semantic meanings from neuroimaging data largely exploits brain activation patterns triggered by stimulation in cross-modality (i.e. text-audio pairs, text-picture pairs). Their goal is to develop a more sophisticated computational model to probing what information from the act of language understanding is represented in human brain. While how the brain receiving such information influences decoding performance is underestimated. This study dissociates multisensory integration of word understanding into written text, spoken text and image perception respectively, exploring the decoding efficiency and reliability of unisensory information in the brain representation. The findings suggest that, in terms of unisensory, decoding is most successful when semantics is represented in pictures, but the effect disappears in the case of congeneric words which share a related meaning. These results reveal the modality dependence and multisensory enhancement in the brain decoding methodology.
Anthology ID:
2024.lrec-main.1527
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
17557–17563
Language:
URL:
https://aclanthology.org/2024.lrec-main.1527
DOI:
Bibkey:
Cite (ACL):
Dandan Huang, Lu Cao, Zhenting Li, and Yue Zhang. 2024. Which Sense Dominates Multisensory Semantic Understanding? A Brain Decoding Study. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 17557–17563, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Which Sense Dominates Multisensory Semantic Understanding? A Brain Decoding Study (Huang et al., LREC-COLING 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.lrec-main.1527.pdf