A View From the Crowd: Evaluation Challenges for Time-Offset Interaction Applications

Alberto Chierici, Nizar Habash


Abstract
Dialogue systems like chatbots, and tasks like question-answering (QA) have gained traction in recent years; yet evaluating such systems remains difficult. Reasons include the great variety in contexts and use cases for these systems as well as the high cost of human evaluation. In this paper, we focus on a specific type of dialogue systems: Time-Offset Interaction Applications (TOIAs) are intelligent, conversational software that simulates face-to-face conversations between humans and pre-recorded human avatars. Under the constraint that a TOIA is a single output system interacting with users with different expectations, we identify two challenges: first, how do we define a ‘good’ answer? and second, what’s an appropriate metric to use? We explore both challenges through the creation of a novel dataset that identifies multiple good answers to specific TOIA questions through the help of Amazon Mechanical Turk workers. This ‘view from the crowd’ allows us to study the variations of how TOIA interrogators perceive its answers. Our contributions include the annotated dataset that we make publicly available and the proposal of Success Rate @k as an evaluation metric that is more appropriate than the traditional QA’s and information retrieval’s metrics.
Anthology ID:
2021.humeval-1.9
Volume:
Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)
Month:
April
Year:
2021
Address:
Online
Editors:
Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, Anastasia Shimorina
Venue:
HumEval
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
75–85
Language:
URL:
https://aclanthology.org/2021.humeval-1.9
DOI:
Bibkey:
Cite (ACL):
Alberto Chierici and Nizar Habash. 2021. A View From the Crowd: Evaluation Challenges for Time-Offset Interaction Applications. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 75–85, Online. Association for Computational Linguistics.
Cite (Informal):
A View From the Crowd: Evaluation Challenges for Time-Offset Interaction Applications (Chierici & Habash, HumEval 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.humeval-1.9.pdf
Data
CoQADouban