Better Pre-Training by Reducing Representation Confusion

Haojie Zhang, Mingfei Liang, Ruobing Xie, Zhenlong Sun, Bo Zhang, Leyu Lin


Abstract
In this work, we revisit the Transformer-based pre-trained language models and identify two different types of information confusion in position encoding and model representations, respectively. Firstly, we show that in the relative position encoding, the joint modeling about relative distances and directions brings confusion between two heterogeneous information. It may make the model unable to capture the associative semantics of the same distance and the opposite directions, which in turn affects the performance of downstream tasks. Secondly, we notice the BERT with Mask Language Modeling (MLM) pre-training objective outputs similar token representations (last hidden states of different tokens) and head representations (attention weightsof different heads), which may make the diversity of information expressed by different tokens and heads limited. Motivated by the above investigation, we propose two novel techniques to improve pre-trained language models: Decoupled Directional Relative Position (DDRP) encoding and MTH pre-training objective. DDRP decouples the relative distance features and the directional features in classical relative position encoding. MTH applies two novel auxiliary regularizers besides MLM to enlarge the dissimilarities between (a) last hidden states of different tokens, and (b) attention weights of different heads. These designs allow the model to capture different categories of information more clearly, as a way to alleviate information confusion in representation learning for better optimization. Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness of our proposed methods.
Anthology ID:
2023.findings-eacl.176
Volume:
Findings of the Association for Computational Linguistics: EACL 2023
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2325–2336
Language:
URL:
https://aclanthology.org/2023.findings-eacl.176
DOI:
10.18653/v1/2023.findings-eacl.176
Bibkey:
Cite (ACL):
Haojie Zhang, Mingfei Liang, Ruobing Xie, Zhenlong Sun, Bo Zhang, and Leyu Lin. 2023. Better Pre-Training by Reducing Representation Confusion. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2325–2336, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
Better Pre-Training by Reducing Representation Confusion (Zhang et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-eacl.176.pdf
Video:
 https://aclanthology.org/2023.findings-eacl.176.mp4