Coordination Generation via Synchronized Text-Infilling

Hiroki Teranishi, Yuji Matsumoto


Abstract
Generating synthetic data for supervised learning from large-scale pre-trained language models has enhanced performances across several NLP tasks, especially in low-resource scenarios. In particular, many studies of data augmentation employ masked language models to replace words with other words in a sentence. However, most of them are evaluated on sentence classification tasks and cannot immediately be applied to tasks related to the sentence structure. In this paper, we propose a simple yet effective approach to generating sentences with a coordinate structure in which the boundaries of its conjuncts are explicitly specified. For a given span in a sentence, our method embeds a mask with a coordinating conjunction in two ways (”X and [mask]”, ”[mask] and X”) and forces masked language models to fill the two blanks with an identical text. To achieve this, we introduce decoding methods for BERT and T5 models with the constraint that predictions for different masks are synchronized. Furthermore, we develop a training framework that effectively selects synthetic examples for the supervised coordination disambiguation task. We demonstrate that our method produces promising coordination instances that provide gains for the task in low-resource settings.
Anthology ID:
2022.coling-1.517
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
5914–5924
Language:
URL:
https://aclanthology.org/2022.coling-1.517
DOI:
Bibkey:
Cite (ACL):
Hiroki Teranishi and Yuji Matsumoto. 2022. Coordination Generation via Synchronized Text-Infilling. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5914–5924, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Coordination Generation via Synchronized Text-Infilling (Teranishi & Matsumoto, COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.517.pdf