Phrase-Level Localization of Inconsistency Errors in Summarization by Weak Supervision

Masato Takatsuka, Tetsunori Kobayashi, Yoshihiko Hayashi


Abstract
Although the fluency of automatically generated abstractive summaries has improved significantly with advanced methods, the inconsistency that remains in summarization is recognized as an issue to be addressed. In this study, we propose a methodology for localizing inconsistency errors in summarization. A synthetic dataset that contains a variety of factual errors likely to be produced by a common summarizer is created by applying sentence fusion, compression, and paraphrasing operations. In creating the dataset, we automatically label erroneous phrases and the dependency relations between them as “inconsistent,” which can contribute to detecting errors more adequately than existing models that rely only on dependency arc-level labels. Subsequently, this synthetic dataset is employed as weak supervision to train a model called SumPhrase, which jointly localizes errors in a summary and their corresponding sentences in the source document. The empirical results demonstrate that our SumPhrase model can detect factual errors in summarization more effectively than existing weakly supervised methods owing to the phrase-level labeling. Moreover, the joint identification of error-corresponding original sentences is proven to be effective in improving error detection accuracy.
Anthology ID:
2022.coling-1.537
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Editors:
Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, Seung-Hoon Na
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
6151–6164
Language:
URL:
https://aclanthology.org/2022.coling-1.537
DOI:
Bibkey:
Cite (ACL):
Masato Takatsuka, Tetsunori Kobayashi, and Yoshihiko Hayashi. 2022. Phrase-Level Localization of Inconsistency Errors in Summarization by Weak Supervision. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6151–6164, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Phrase-Level Localization of Inconsistency Errors in Summarization by Weak Supervision (Takatsuka et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.537.pdf
Data
CNN/Daily Mail