The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank

Daphne Ippolito, Liam Dugan, Emily Reif, Ann Yuan, Andy Coenen, Chris Callison-Burch


Abstract
The task of inserting text into a specified position in a passage, known as fill in the blank (FitB), is useful for a variety of applications where writers interact with a natural language generation (NLG) system to craft text. While previous work has tackled this problem with models trained specifically to do fill in the blank, a more useful model is one that can effectively perform _both_ FitB and continuation tasks. In this work, we evaluate the feasibility of using a single model to do both tasks. We show that models pre-trained with a FitB-style objective are capable of both tasks, while models pre-trained for continuation are not. Finally, we show how these models can be easily finetuned to allow for fine-grained control over the length and word choice of the generation.
Anthology ID:
2022.findings-naacl.185
Volume:
Findings of the Association for Computational Linguistics: NAACL 2022
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2421–2432
Language:
URL:
https://aclanthology.org/2022.findings-naacl.185
DOI:
10.18653/v1/2022.findings-naacl.185
Bibkey:
Cite (ACL):
Daphne Ippolito, Liam Dugan, Emily Reif, Ann Yuan, Andy Coenen, and Chris Callison-Burch. 2022. The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2421–2432, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
The Case for a Single Model that can Both Generate Continuations and Fill-in-the-Blank (Ippolito et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-naacl.185.pdf
Video:
 https://aclanthology.org/2022.findings-naacl.185.mp4
Data
WritingPrompts