A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions

Takuma Udagawa, Takato Yamazaki, Akiko Aizawa


Abstract
Recent models achieve promising results in visually grounded dialogues. However, existing datasets often contain undesirable biases and lack sophisticated linguistic analyses, which make it difficult to understand how well current models recognize their precise linguistic structures. To address this problem, we make two design choices: first, we focus on OneCommon Corpus (CITATION), a simple yet challenging common grounding dataset which contains minimal bias by design. Second, we analyze their linguistic structures based on spatial expressions and provide comprehensive and reliable annotation for 600 dialogues. We show that our annotation captures important linguistic structures including predicate-argument structure, modification and ellipsis. In our experiments, we assess the model’s understanding of these structures through reference resolution. We demonstrate that our annotation can reveal both the strengths and weaknesses of baseline models in essential levels of detail. Overall, we propose a novel framework and resource for investigating fine-grained language understanding in visually grounded dialogues.
Anthology ID:
2020.findings-emnlp.67
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2020
Month:
November
Year:
2020
Address:
Online
Editors:
Trevor Cohn, Yulan He, Yang Liu
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
750–765
Language:
URL:
https://aclanthology.org/2020.findings-emnlp.67
DOI:
10.18653/v1/2020.findings-emnlp.67
Bibkey:
Cite (ACL):
Takuma Udagawa, Takato Yamazaki, and Akiko Aizawa. 2020. A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 750–765, Online. Association for Computational Linguistics.
Cite (Informal):
A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions (Udagawa et al., Findings 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.findings-emnlp.67.pdf
Video:
 https://slideslive.com/38940097
Code
 Alab-NII/onecommon