On the Role of Pre-trained Language Models in Word Ordering: A Case Study with BART

Zebin Ou, Meishan Zhang, Yue Zhang


Abstract
Word ordering is a constrained language generation task taking unordered words as input. Existing work uses linear models and neural networks for the task, yet pre-trained language models have not been studied in word ordering, let alone why they help. We use BART as an instance and show its effectiveness in the task. To explain why BART helps word ordering, we extend analysis with probing and empirically identify that syntactic dependency knowledge in BART is a reliable explanation. We also report performance gains with BART in the related partial tree linearization task, which readily extends our analysis.
Anthology ID:
2022.coling-1.567
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
6516–6529
Language:
URL:
https://aclanthology.org/2022.coling-1.567
DOI:
Bibkey:
Cite (ACL):
Zebin Ou, Meishan Zhang, and Yue Zhang. 2022. On the Role of Pre-trained Language Models in Word Ordering: A Case Study with BART. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6516–6529, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
On the Role of Pre-trained Language Models in Word Ordering: A Case Study with BART (Ou et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.567.pdf
Code
 simtony/bart-word-orderer
Data
CommonGen