What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts

James D. Finch, Sarah E. Finch, Jinho D. Choi


Abstract
Improving user experience of a dialogue system often requires intensive developer effort to read conversation logs, run statistical analyses, and intuit the relative importance of system shortcomings. This paper presents a novel approach to automated analysis of conversation logs that learns the relationship between user-system interactions and overall dialogue quality. Unlike prior work on utterance-level quality prediction, our approach learns the impact of each interaction from the overall user rating without utterance-level annotation, allowing resultant model conclusions to be derived on the basis of empirical evidence and at low cost. Our model identifies interactions that have a strong correlation with the overall dialogue quality in a chatbot setting. Experiments show that the automated analysis from our model agrees with expert judgments, making this work the first to show that such weakly-supervised learning of utterance-level quality prediction is highly achievable.
Anthology ID:
2021.nlp4convai-1.9
Volume:
Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI
Month:
November
Year:
2021
Address:
Online
Venues:
EMNLP | NLP4ConvAI
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
93–101
Language:
URL:
https://aclanthology.org/2021.nlp4convai-1.9
DOI:
10.18653/v1/2021.nlp4convai-1.9
Bibkey:
Cite (ACL):
James D. Finch, Sarah E. Finch, and Jinho D. Choi. 2021. What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 93–101, Online. Association for Computational Linguistics.
Cite (Informal):
What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts (Finch et al., NLP4ConvAI 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.nlp4convai-1.9.pdf
Data
Topical-Chat