CoLo: A Contrastive Learning Based Re-ranking Framework for One-Stage Summarization

Chenxin An, Ming Zhong, Zhiyong Wu, Qin Zhu, Xuanjing Huang, Xipeng Qiu


Abstract
Traditional training paradigms for extractive and abstractive summarization systems always only use token-level or sentence-level training objectives. However, the output summary is always evaluated from summary-level which leads to the inconsistency in training and evaluation. In this paper, we propose a Contrastive Learning based re-ranking framework for one-stage summarization called CoLo. By modeling a contrastive objective, we show that the summarization model is able to directly generate summaries according to the summary-level score without additional modules and parameters. Extensive experiments demonstrate that CoLo boosts the extractive and abstractive results of one-stage systems on CNN/DailyMail benchmark to 44.58 and 46.33 ROUGE-1 score while preserving the parameter efficiency and inference efficiency. Compared with state-of-the-art multi-stage systems, we save more than 100 GPU training hours and obtaining 3x 8x speed-up ratio during inference while maintaining comparable results.
Anthology ID:
2022.coling-1.508
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
5783–5793
Language:
URL:
https://aclanthology.org/2022.coling-1.508
DOI:
Bibkey:
Cite (ACL):
Chenxin An, Ming Zhong, Zhiyong Wu, Qin Zhu, Xuanjing Huang, and Xipeng Qiu. 2022. CoLo: A Contrastive Learning Based Re-ranking Framework for One-Stage Summarization. In Proceedings of the 29th International Conference on Computational Linguistics, pages 5783–5793, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
CoLo: A Contrastive Learning Based Re-ranking Framework for One-Stage Summarization (An et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.508.pdf
Code
 chenxinan-fdu/colo
Data
CNN/Daily MailSSN