HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations

Weixin Liang, Kai-Hui Liang, Zhou Yu


Abstract
Open-domain dialog systems have a user-centric goal: to provide humans with an engaging conversation experience. User engagement is one of the most important metrics for evaluating open-domain dialog systems, and could also be used as real-time feedback to benefit dialog policy learning. Existing work on detecting user disengagement typically requires hand-labeling many dialog samples. We propose HERALD, an efficient annotation framework that reframes the training data annotation process as a denoising problem. Specifically, instead of manually labeling training samples, we first use a set of labeling heuristics to label training samples automatically. We then denoise the weakly labeled data using the Shapley algorithm. Finally, we use the denoised data to train a user engagement detector. Our experiments show that HERALD improves annotation efficiency significantly and achieves 86% user disengagement detection accuracy in two dialog corpora.
Anthology ID:
2021.acl-long.283
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Editors:
Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3652–3665
Language:
URL:
https://aclanthology.org/2021.acl-long.283
DOI:
10.18653/v1/2021.acl-long.283
Bibkey:
Cite (ACL):
Weixin Liang, Kai-Hui Liang, and Zhou Yu. 2021. HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3652–3665, Online. Association for Computational Linguistics.
Cite (Informal):
HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations (Liang et al., ACL-IJCNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-long.283.pdf
Video:
 https://aclanthology.org/2021.acl-long.283.mp4
Code
 Weixin-Liang/HERALD
Data
ConvAI2