Wizard of Tasks: A Novel Conversational Dataset for Solving Real-World Tasks in Conversational Settings

Jason Ingyu Choi, Saar Kuzi, Nikhita Vedula, Jie Zhao, Giuseppe Castellucci, Marcus Collins, Shervin Malmasi, Oleg Rokhlenko, Eugene Agichtein


Abstract
Conversational Task Assistants (CTAs) are conversational agents whose goal is to help humans perform real-world tasks. CTAs can help in exploring available tasks, answering task-specific questions and guiding users through step-by-step instructions. In this work, we present Wizard of Tasks, the first corpus of such conversations in two domains: Cooking and Home Improvement. We crowd-sourced a total of 549 conversations (18,077 utterances) with an asynchronous Wizard-of-Oz setup, relying on recipes from WholeFoods Market for the cooking domain, and WikiHow articles for the home improvement domain. We present a detailed data analysis and show that the collected data can be a valuable and challenging resource for CTAs in two tasks: Intent Classification (IC) and Abstractive Question Answering (AQA). While on IC we acquired a high performing model (>85% F1), on AQA the performance is far from being satisfactory (~27% BertScore-F1), suggesting that more work is needed to solve the task of low-resource AQA.
Anthology ID:
2022.coling-1.310
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
3514–3529
Language:
URL:
https://aclanthology.org/2022.coling-1.310
DOI:
Bibkey:
Cite (ACL):
Jason Ingyu Choi, Saar Kuzi, Nikhita Vedula, Jie Zhao, Giuseppe Castellucci, Marcus Collins, Shervin Malmasi, Oleg Rokhlenko, and Eugene Agichtein. 2022. Wizard of Tasks: A Novel Conversational Dataset for Solving Real-World Tasks in Conversational Settings. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3514–3529, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Wizard of Tasks: A Novel Conversational Dataset for Solving Real-World Tasks in Conversational Settings (Choi et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.310.pdf