Contrastive Preference Learning for Neural Machine Translation

Jianfei He, Shichao Sun, Sen Peng, Jie Xu, Xiaohua Jia, Wenjie Li


Abstract
There exists a discrepancy between the token-level objective during training and the overall sequence-level quality that is expected from the model. This discrepancy leads to issues like exposure bias.To align the model with human expectations, sequence-level objectives are often used to fine-tune pre-trained models.In this paper, we introduce a contrastive preference model that enhances the traditional Plackett-Luce model by incorporating an indicator function. Building upon this novel preference model, we propose Contrastive Preference Learning (CPL), which uses offline samples with list-wise preferences to fine-tune a pre-trained model in Neural Machine Translation. Our experiments, conducted on three language pairs, demonstrate that CPL outperforms not only the vanilla Transformer model but also other token-level and sequence-level baselines. Furthermore, the ablation study highlights the essential role of the proposed indicator function in achieving this improvement.
Anthology ID:
2024.findings-naacl.174
Volume:
Findings of the Association for Computational Linguistics: NAACL 2024
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2723–2735
Language:
URL:
https://aclanthology.org/2024.findings-naacl.174
DOI:
Bibkey:
Cite (ACL):
Jianfei He, Shichao Sun, Sen Peng, Jie Xu, Xiaohua Jia, and Wenjie Li. 2024. Contrastive Preference Learning for Neural Machine Translation. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2723–2735, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Contrastive Preference Learning for Neural Machine Translation (He et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-naacl.174.pdf
Copyright:
 2024.findings-naacl.174.copyright.pdf