Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models

Elizabeth Clark, Noah A. Smith


Abstract
Story generation is an open-ended and subjective task, which poses a challenge for evaluating story generation models. We present Choose Your Own Adventure, a collaborative writing setup for pairwise model evaluation. Two models generate suggestions to people as they write a short story; we ask writers to choose one of the two suggestions, and we observe which model’s suggestions they prefer. The setup also allows further analysis based on the revisions people make to the suggestions. We show that these measures, combined with automatic metrics, provide an informative picture of the models’ performance, both in cases where the differences in generation methods are small (nucleus vs. top-k sampling) and large (GPT2 vs. Fusion models).
Anthology ID:
2021.naacl-main.279
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3566–3575
Language:
URL:
https://aclanthology.org/2021.naacl-main.279
DOI:
10.18653/v1/2021.naacl-main.279
Bibkey:
Cite (ACL):
Elizabeth Clark and Noah A. Smith. 2021. Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3566–3575, Online. Association for Computational Linguistics.
Cite (Informal):
Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models (Clark & Smith, NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.279.pdf
Video:
 https://aclanthology.org/2021.naacl-main.279.mp4
Data
WritingPrompts