Perceived Text Quality and Readability in Extractive and Abstractive Summaries

Julius Monsen, Evelina Rennes


Abstract
We present results from a study investigating how users perceive text quality and readability in extractive and abstractive summaries. We trained two summarisation models on Swedish news data and used these to produce summaries of articles. With the produced summaries, we conducted an online survey in which the extractive summaries were compared to the abstractive summaries in terms of fluency, adequacy and simplicity. We found statistically significant differences in perceived fluency and adequacy between abstractive and extractive summaries but no statistically significant difference in simplicity. Extractive summaries were preferred in most cases, possibly due to the types of errors the summaries tend to have.
Anthology ID:
2022.lrec-1.32
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
305–312
Language:
URL:
https://aclanthology.org/2022.lrec-1.32
DOI:
Bibkey:
Cite (ACL):
Julius Monsen and Evelina Rennes. 2022. Perceived Text Quality and Readability in Extractive and Abstractive Summaries. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 305–312, Marseille, France. European Language Resources Association.
Cite (Informal):
Perceived Text Quality and Readability in Extractive and Abstractive Summaries (Monsen & Rennes, LREC 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.lrec-1.32.pdf
Data
CNN/Daily Mail