CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding

Dong Wang, Ning Ding, Piji Li, Haitao Zheng


Abstract
Despite pre-trained language models have proven useful for learning high-quality semantic representations, these models are still vulnerable to simple perturbations. Recent works aimed to improve the robustness of pre-trained models mainly focus on adversarial training from perturbed examples with similar semantics, neglecting the utilization of different or even opposite semantics. Different from the image processing field, the text is discrete and few word substitutions can cause significant semantic changes. To study the impact of semantics caused by small perturbations, we conduct a series of pilot experiments and surprisingly find that adversarial training is useless or even harmful for the model to detect these semantic changes. To address this problem, we propose Contrastive Learning with semantIc Negative Examples (CLINE), which constructs semantic negative examples unsupervised to improve the robustness under semantically adversarial attacking. By comparing with similar and opposite semantic examples, the model can effectively perceive the semantic changes caused by small perturbations. Empirical results show that our approach yields substantial improvements on a range of sentiment analysis, reasoning, and reading comprehension tasks. And CLINE also ensures the compactness within the same semantics and separability across different semantics in sentence-level.
Anthology ID:
2021.acl-long.181
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2332–2342
Language:
URL:
https://aclanthology.org/2021.acl-long.181
DOI:
10.18653/v1/2021.acl-long.181
Bibkey:
Cite (ACL):
Dong Wang, Ning Ding, Piji Li, and Haitao Zheng. 2021. CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2332–2342, Online. Association for Computational Linguistics.
Cite (Informal):
CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding (Wang et al., ACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-long.181.pdf
Video:
 https://aclanthology.org/2021.acl-long.181.mp4
Code
 kandorm/CLINE
Data
BoolQIMDb Movie ReviewsPerspectrumSNLI