I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling

Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, Jason Weston


Abstract
To quantify how well natural language understanding models can capture consistency in a general conversation, we introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues. We show that: (i) our newly collected dataset is notably more effective at providing supervision for the dialogue contradiction detection task than existing NLI data including those aimed to cover the dialogue domain; (ii) Transformer models that explicitly hinge on utterance structures for dialogue contradiction detection are more robust and generalize well on both analysis and out-of-distribution dialogues than standard (unstructured) Transformers. We also show that our best contradiction detection model correlates well with human judgments and further provide evidence for its usage in both automatically evaluating and improving the consistency of state-of-the-art generative chatbots.
Anthology ID:
2021.acl-long.134
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1699–1713
Language:
URL:
https://aclanthology.org/2021.acl-long.134
DOI:
10.18653/v1/2021.acl-long.134
Bibkey:
Cite (ACL):
Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, and Jason Weston. 2021. I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1699–1713, Online. Association for Computational Linguistics.
Cite (Informal):
I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling (Nie et al., ACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-long.134.pdf
Optional supplementary material:
 2021.acl-long.134.OptionalSupplementaryMaterial.zip
Video:
 https://aclanthology.org/2021.acl-long.134.mp4
Data
ANLIBlended Skill TalkConvAI2SNLIWizard of Wikipedia