DialCrowd: A toolkit for easy dialog system assessment

Kyusong Lee, Tiancheng Zhao, Alan W. Black, Maxine Eskenazi


Abstract
When creating a dialog system, developers need to test each version to ensure that it is performing correctly. Recently the trend has been to test on large datasets or to ask many users to try out a system. Crowdsourcing has solved the issue of finding users, but it presents new challenges such as how to use a crowdsourcing platform and what type of test is appropriate. DialCrowd has been designed to make system assessment easier and to ensure the quality of the result. This paper describes DialCrowd, what specific needs it fulfills and how it works. It then relates a test of DialCrowd by a group of dialog system developer.
Anthology ID:
W18-5028
Volume:
Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Kazunori Komatani, Diane Litman, Kai Yu, Alex Papangelis, Lawrence Cavedon, Mikio Nakano
Venue:
SIGDIAL
SIG:
SIGDIAL
Publisher:
Association for Computational Linguistics
Note:
Pages:
245–248
Language:
URL:
https://aclanthology.org/W18-5028
DOI:
10.18653/v1/W18-5028
Bibkey:
Cite (ACL):
Kyusong Lee, Tiancheng Zhao, Alan W. Black, and Maxine Eskenazi. 2018. DialCrowd: A toolkit for easy dialog system assessment. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 245–248, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
DialCrowd: A toolkit for easy dialog system assessment (Lee et al., SIGDIAL 2018)
Copy Citation:
PDF:
https://aclanthology.org/W18-5028.pdf