CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations

Borun Chen, Hongyin Tang, Jiahao Bu, Kai Zhang, Jingang Wang, Qifan Wang, Hai-Tao Zheng, Wei Wu, Liqian Yu


Abstract
Pre-trained Language Models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding. Various Chinese PLMs have been successively proposed for learning better Chinese language representation. However, most current models use Chinese characters as inputs and are not able to encode semantic information contained in Chinese words. While recent pre-trained models incorporate both words and characters simultaneously, they usually suffer from deficient semantic interactions and fail to capture the semantic relation between words and characters. To address the above issues, we propose a simple yet effective PLM CLOWER, which adopts the Contrastive Learning Over Word and charactER representations. In particular, CLOWER implicitly encodes the coarse-grained information (i.e., words) into the fine-grained representations (i.e., characters) through contrastive learning on multi-grained information. CLOWER is of great value in realistic scenarios since it can be easily incorporated into any existing fine-grained based PLMs without modifying the production pipelines. Extensive experiments conducted on a range of downstream tasks demonstrate the superior performance of CLOWER over several state-of-the-art baselines.
Anthology ID:
2022.coling-1.274
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
3098–3108
Language:
URL:
https://aclanthology.org/2022.coling-1.274
DOI:
Bibkey:
Cite (ACL):
Borun Chen, Hongyin Tang, Jiahao Bu, Kai Zhang, Jingang Wang, Qifan Wang, Hai-Tao Zheng, Wei Wu, and Liqian Yu. 2022. CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3098–3108, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations (Chen et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.274.pdf
Data
CMRCCMRC 2018DRCDOCNLI