QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization

Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, Caiming Xiong


Abstract
Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.
Anthology ID:
2022.naacl-main.187
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2587–2601
Language:
URL:
https://aclanthology.org/2022.naacl-main.187
DOI:
10.18653/v1/2022.naacl-main.187
Bibkey:
Cite (ACL):
Alexander Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2022. QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2587–2601, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization (Fabbri et al., NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.187.pdf
Code
 salesforce/qafacteval
Data
ANLICNN/Daily MailMultiNLIQA2DSQuAD