Grounded Word Sense Translation

Chiraag Lala, Pranava Madhyastha, Lucia Specia


Abstract
Recent work on visually grounded language learning has focused on broader applications of grounded representations, such as visual question answering and multimodal machine translation. In this paper we consider grounded word sense translation, i.e. the task of correctly translating an ambiguous source word given the corresponding textual and visual context. Our main objective is to investigate the extent to which images help improve word-level (lexical) translation quality. We do so by first studying the dataset for this task to understand the scope and challenges of the task. We then explore different data settings, image features, and ways of grounding to investigate the gain from using images in each of the combinations. We find that grounding on the image is specially beneficial in weaker unidirectional recurrent translation models. We observe that adding structured image information leads to stronger gains in lexical translation accuracy.
Anthology ID:
W19-1808
Volume:
Proceedings of the Second Workshop on Shortcomings in Vision and Language
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Editors:
Raffaella Bernardi, Raquel Fernandez, Spandana Gella, Kushal Kafle, Christopher Kanan, Stefan Lee, Moin Nabi
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
78–85
Language:
URL:
https://aclanthology.org/W19-1808
DOI:
10.18653/v1/W19-1808
Bibkey:
Cite (ACL):
Chiraag Lala, Pranava Madhyastha, and Lucia Specia. 2019. Grounded Word Sense Translation. In Proceedings of the Second Workshop on Shortcomings in Vision and Language, pages 78–85, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
Grounded Word Sense Translation (Lala et al., NAACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-1808.pdf