DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks

Ziyang Luo, Yadong Xi, Jing Ma, Zhiwei Yang, Xiaoxi Mao, Changjie Fan, Rongsheng Zhang


Abstract
Since 2017, the Transformer-based models play critical roles in various downstream Natural Language Processing tasks. However, a common limitation of the attention mechanism utilized in Transformer Encoder is that it cannot automatically capture the information of word order, so explicit position embeddings are generally required to be fed into the target model. In contrast, Transformer Decoder with the causal attention masks is naturally sensitive to the word order. In this work, we focus on improving the position encoding ability of BERT with the causal attention masks. Furthermore, we propose a new pre-trained language model DecBERT and evaluate it on the GLUE benchmark. Experimental results show that (1) the causal attention mask is effective for BERT on the language understanding tasks; (2) our DecBERT model without position embeddings achieve comparable performance on the GLUE benchmark; and (3) our modification accelerates the pre-training process and DecBERT w/ PE achieves better overall performance than the baseline systems when pre-training with the same amount of computational resources.
Anthology ID:
2022.findings-naacl.89
Volume:
Findings of the Association for Computational Linguistics: NAACL 2022
Month:
July
Year:
2022
Address:
Seattle, United States
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1185–1197
Language:
URL:
https://aclanthology.org/2022.findings-naacl.89
DOI:
10.18653/v1/2022.findings-naacl.89
Bibkey:
Cite (ACL):
Ziyang Luo, Yadong Xi, Jing Ma, Zhiwei Yang, Xiaoxi Mao, Changjie Fan, and Rongsheng Zhang. 2022. DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1185–1197, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
DecBERT: Enhancing the Language Understanding of BERT with Causal Attention Masks (Luo et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-naacl.89.pdf
Data
CoLAGLUEMRPCQNLISSTSuperGLUEWikiText-103WikiText-2