%0 Conference Proceedings %T QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization %A Fabbri, Alexander %A Wu, Chien-Sheng %A Liu, Wenhao %A Xiong, Caiming %Y Carpuat, Marine %Y de Marneffe, Marie-Catherine %Y Meza Ruiz, Ivan Vladimir %S Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies %D 2022 %8 July %I Association for Computational Linguistics %C Seattle, United States %F fabbri-etal-2022-qafacteval %X Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost. %R 10.18653/v1/2022.naacl-main.187 %U https://aclanthology.org/2022.naacl-main.187 %U https://doi.org/10.18653/v1/2022.naacl-main.187 %P 2587-2601