Investigating Robustness of Dialog Models to Popular Figurative Language Constructs

Harsh Jhamtani, Varun Gangal, Eduard Hovy, Taylor Berg-Kirkpatrick


Abstract
Humans often employ figurative language use in communication, including during interactions with dialog systems. Thus, it is important for real-world dialog systems to be able to handle popular figurative language constructs like metaphor and simile. In this work, we analyze the performance of existing dialog models in situations where the input dialog context exhibits use of figurative language. We observe large gaps in handling of figurative language when evaluating the models on two open domain dialog datasets. When faced with dialog contexts consisting of figurative language, some models show very large drops in performance compared to contexts without figurative language. We encourage future research in dialog modeling to separately analyze and report results on figurative language in order to better test model capabilities relevant to real-world use. Finally, we propose lightweight solutions to help existing models become more robust to figurative language by simply using an external resource to translate figurative language to literal (non-figurative) forms while preserving the meaning to the best extent possible.
Anthology ID:
2021.emnlp-main.592
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7476–7485
Language:
URL:
https://aclanthology.org/2021.emnlp-main.592
DOI:
10.18653/v1/2021.emnlp-main.592
Bibkey:
Cite (ACL):
Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Taylor Berg-Kirkpatrick. 2021. Investigating Robustness of Dialog Models to Popular Figurative Language Constructs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7476–7485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Investigating Robustness of Dialog Models to Popular Figurative Language Constructs (Jhamtani et al., EMNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.emnlp-main.592.pdf
Video:
 https://aclanthology.org/2021.emnlp-main.592.mp4
Code
 vgtomahawk/dialog-fig-speech-robust
Data
DailyDialog