Evaluating Document Simplification: On the Importance of Separately Assessing Simplicity and Meaning Preservation

Liam Cripwell, Joël Legrand, Claire Gardent


Abstract
Text simplification intends to make a text easier to read while preserving its core meaning. Intuitively and as shown in previous works, these two dimensions (simplification and meaning preservation) are often-times inversely correlated. An overly conservative text will fail to simplify sufficiently, whereas extreme simplification will degrade meaning preservation. Yet, popular evaluation metrics either aggregate meaning preservation and simplification into a single score (SARI, LENS), or target meaning preservation alone (BERTScore, QuestEval). Moreover, these metrics usually require a set of references and most previous work has only focused on sentence-level simplification. In this paper, we focus on the evaluation of document-level text simplification and compare existing models using distinct metrics for meaning preservation and simplification. We leverage existing metrics from similar tasks and introduce a reference-less metric variant for simplicity, showing that models are mostly biased towards either simplification or meaning preservation, seldom performing well on both dimensions. Making use of the fact that the metrics we use are all reference-less, we also investigate the performance of existing models when applied to unseen data (where reference simplifications are unavailable).
Anthology ID:
2024.readi-1.1
Volume:
Proceedings of the 3rd Workshop on Tools and Resources for People with REAding DIfficulties (READI) @ LREC-COLING 2024
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Rodrigo Wilkens, Rémi Cardon, Amalia Todirascu, Núria Gala
Venues:
READI | WS
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
1–14
Language:
URL:
https://aclanthology.org/2024.readi-1.1
DOI:
Bibkey:
Cite (ACL):
Liam Cripwell, Joël Legrand, and Claire Gardent. 2024. Evaluating Document Simplification: On the Importance of Separately Assessing Simplicity and Meaning Preservation. In Proceedings of the 3rd Workshop on Tools and Resources for People with REAding DIfficulties (READI) @ LREC-COLING 2024, pages 1–14, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Evaluating Document Simplification: On the Importance of Separately Assessing Simplicity and Meaning Preservation (Cripwell et al., READI-WS 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.readi-1.1.pdf