FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT

Abhisek Chakrabarty, Raj Dabre, Chenchen Ding, Hideki Tanaka, Masao Utiyama, Eiichiro Sumita


Abstract
In this paper we present FeatureBART, a linguistically motivated sequence-to-sequence monolingual pre-training strategy in which syntactic features such as lemma, part-of-speech and dependency labels are incorporated into the span prediction based pre-training framework (BART). These automatically extracted features are incorporated via approaches such as concatenation and relevance mechanisms, among which the latter is known to be better than the former. When used for low-resource NMT as a downstream task, we show that these feature based models give large improvements in bilingual settings and modest ones in multilingual settings over their counterparts that do not use features.
Anthology ID:
2022.coling-1.443
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
5014–5020
Language:
URL:
https://aclanthology.org/2022.coling-1.443
DOI:
Bibkey:
Cite (ACL):
Abhisek Chakrabarty, Raj Dabre, Chenchen Ding, Hideki Tanaka, Masao Utiyama, and Eiichiro Sumita. 2022. FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5014–5020, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
FeatureBART: Feature Based Sequence-to-Sequence Pre-Training for Low-Resource NMT (Chakrabarty et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.443.pdf