Evaluating Factuality in Text Simplification

Ashwin Devaraj, William Sheffield, Byron Wallace, Junyi Jessy Li


Abstract
Automated simplification models aim to make input texts more readable. Such methods have the potential to make complex information accessible to a wider audience, e.g., providing access to recent medical literature which might otherwise be impenetrable for a lay reader. However, such models risk introducing errors into automatically simplified texts, for instance by inserting statements unsupported by the corresponding original text, or by omitting key information. Providing more readable but inaccurate versions of texts may in many cases be worse than providing no such access at all. The problem of factual accuracy (and the lack thereof) has received heightened attention in the context of summarization models, but the factuality of automatically simplified texts has not been investigated. We introduce a taxonomy of errors that we use to analyze both references drawn from standard simplification datasets and state-of-the-art model outputs. We find that errors often appear in both that are not captured by existing evaluation metrics, motivating a need for research into ensuring the factual accuracy of automated simplification models.
Anthology ID:
2022.acl-long.506
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7331–7345
Language:
URL:
https://aclanthology.org/2022.acl-long.506
DOI:
10.18653/v1/2022.acl-long.506
Award:
 Outstanding Paper
Bibkey:
Cite (ACL):
Ashwin Devaraj, William Sheffield, Byron Wallace, and Junyi Jessy Li. 2022. Evaluating Factuality in Text Simplification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7331–7345, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Evaluating Factuality in Text Simplification (Devaraj et al., ACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.acl-long.506.pdf
Code
 ashologn/evaluating-factuality-in-text-simplification
Data
NewselaWikiLarge