Inconsistent dialogue responses and how to recover from them

Mian Zhang, Lifeng Jin, Linfeng Song, Haitao Mi, Dong Yu


Abstract
One critical issue for chat systems is to stay consistent about preferences, opinions, beliefs and facts of itself, which has been shown a difficult problem. In this work, we study methods to assess and bolster utterance consistency of chat systems. A dataset is first developed for studying the inconsistencies, where inconsistent dialogue responses, explanations of the inconsistencies, and recovery utterances are authored by annotators. This covers the life span of inconsistencies, namely introduction, understanding, and resolution. Building on this, we introduce a set of tasks centered on dialogue consistency, specifically focused on its detection and resolution. Our experimental findings indicate that our dataset significantly helps the progress in identifying and resolving conversational inconsistencies, and current popular large language models like ChatGPT which are good at resolving inconsistencies however still struggle with detection.
Anthology ID:
2024.findings-eacl.16
Volume:
Findings of the Association for Computational Linguistics: EACL 2024
Month:
March
Year:
2024
Address:
St. Julian’s, Malta
Editors:
Yvette Graham, Matthew Purver
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
220–230
Language:
URL:
https://aclanthology.org/2024.findings-eacl.16
DOI:
Bibkey:
Cite (ACL):
Mian Zhang, Lifeng Jin, Linfeng Song, Haitao Mi, and Dong Yu. 2024. Inconsistent dialogue responses and how to recover from them. In Findings of the Association for Computational Linguistics: EACL 2024, pages 220–230, St. Julian’s, Malta. Association for Computational Linguistics.
Cite (Informal):
Inconsistent dialogue responses and how to recover from them (Zhang et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-eacl.16.pdf