Semantic Overlap Summarization among Multiple Alternative Narratives: An Exploratory Study

Naman Bansal, Mousumi Akter, Shubhra Kanti Karmaker Santu


Abstract
In this paper, we introduce an important yet relatively unexplored NLP task called Semantic Overlap Summarization (SOS), which entails generating a single summary from multiple alternative narratives which can convey the common information provided by those narratives. As no benchmark dataset is readily available for this task, we created one by collecting 2,925 alternative narrative pairs from the web and then, went through the tedious process of manually creating 411 different reference summaries by engaging human annotators. As a way to evaluate this novel task, we first conducted a systematic study by borrowing the popular ROUGE metric from text-summarization literature and discovered that ROUGE is not suitable for our task. Subsequently, we conducted further human annotations to create 200 document-level and 1,518 sentence-level ground-truth overlap labels. Our experiments show that the sentence-wise annotation technique with three overlap labels, i.e., Absent (A), Partially-Present (PP), and Present (P), yields a higher correlation with human judgment and higher inter-rater agreement compared to the ROUGE metric.
Anthology ID:
2022.coling-1.541
Volume:
Proceedings of the 29th International Conference on Computational Linguistics
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
6195–6207
Language:
URL:
https://aclanthology.org/2022.coling-1.541
DOI:
Bibkey:
Cite (ACL):
Naman Bansal, Mousumi Akter, and Shubhra Kanti Karmaker Santu. 2022. Semantic Overlap Summarization among Multiple Alternative Narratives: An Exploratory Study. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6195–6207, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Cite (Informal):
Semantic Overlap Summarization among Multiple Alternative Narratives: An Exploratory Study (Bansal et al., COLING 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.coling-1.541.pdf