Multimodal Grounding for Language Processing

Lisa Beinborn, Teresa Botschen, Iryna Gurevych


Abstract
This survey discusses how recent developments in multimodal processing facilitate conceptual grounding of language. We categorize the information flow in multimodal processing with respect to cognitive models of human information processing and analyze different methods for combining multimodal representations. Based on this methodological inventory, we discuss the benefit of multimodal grounding for a variety of language processing tasks and the challenges that arise. We particularly focus on multimodal grounding of verbs which play a crucial role for the compositional power of language.
Anthology ID:
C18-1197
Volume:
Proceedings of the 27th International Conference on Computational Linguistics
Month:
August
Year:
2018
Address:
Santa Fe, New Mexico, USA
Venue:
COLING
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2325–2339
Language:
URL:
https://aclanthology.org/C18-1197
DOI:
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/C18-1197.pdf
Code
 UKPLab/coling18-multimodalSurvey