Nihal Jain
2023
ContraCLM: Contrastive Learning For Causal Language Model
Nihal Jain
|
Dejiao Zhang
|
Wasi Uddin Ahmad
|
Zijian Wang
|
Feng Nan
|
Xiaopeng Li
|
Ming Tan
|
Ramesh Nallapati
|
Baishakhi Ray
|
Parminder Bhatia
|
Xiaofei Ma
|
Bing Xiang
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Despite exciting progress in causal language models, the expressiveness of their representations is largely limited due to poor discrimination ability. To remedy this issue, we present CONTRACLM, a novel contrastive learning framework at both the token-level and the sequence-level. We assess CONTRACLM on a variety of downstream tasks. We show that CONTRACLM enhances the discrimination of representations and bridges the gap with encoder-only models, which makes causal language models better suited for tasks beyond language generation. Specifically, we attain 44% relative improvement on the Semantic Textual Similarity tasks and 34% on Code-to-Code Search tasks. Furthermore, by improving the expressiveness of representations, CONTRACLM also boosts the source code generation capability with 9% relative improvement on execution accuracy on the HumanEval benchmark.
Search
Co-authors
- Dejiao Zhang 1
- Wasi Ahmad 1
- Zijian Wang 1
- Feng Nan 1
- Xiaopeng Li 1
- show all...
Venues
- acl1