Multi-View Source Ablation for Faithful Summarization

Shuyang Cao, Liang Ma, Di Lu, Robert L Logan IV, Joel Tetreault, Alejandro Jaimes


Abstract
In this paper, we present MuFaSSa (Multi-view Faithfulness Scoring via Source Ablation), a metric for evaluating faithfulness of abstractive summaries, and for guiding training of more faithful summarizers. For evaluation, MuFaSSa employs different strategies (e.g., masking entity mentions) to first remove information from the source document to form multiple ablated views. Then, the faithfulness level of each token in a generated summary is measured by the difference between the token generation probabilities when given the original document and the ablated document as inputs to trained summarizers. For training, MuFaSSa uses a novel word truncation objective that drops unfaithful tokens located by MuFaSSa in both the decoder input and output. Alignments with human-annotated faithfulness labels on AggreFact show that MuFaSSa is comparable to or better than existing metrics built on classifiers or QA models pre-trained on other tasks. In experiments on summarization with XSum and CNN/DailyMail, models trained with word truncation using MuFaSSa outperform competitive methods according to both automatic faithfulness metrics and human assessments.
Anthology ID:
2023.findings-eacl.151
Volume:
Findings of the Association for Computational Linguistics: EACL 2023
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2029–2047
Language:
URL:
https://aclanthology.org/2023.findings-eacl.151
DOI:
10.18653/v1/2023.findings-eacl.151
Bibkey:
Cite (ACL):
Shuyang Cao, Liang Ma, Di Lu, Robert L Logan IV, Joel Tetreault, and Alejandro Jaimes. 2023. Multi-View Source Ablation for Faithful Summarization. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2029–2047, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
Multi-View Source Ablation for Faithful Summarization (Cao et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-eacl.151.pdf
Video:
 https://aclanthology.org/2023.findings-eacl.151.mp4