Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

Juri Opitz, Anette Frank


Abstract
Systems that generate natural language text from abstract meaning representations such as AMR are typically evaluated using automatic surface matching metrics that compare the generated texts to reference texts from which the input meaning representations were constructed. We show that besides well-known issues from which such metrics suffer, an additional problem arises when applying these metrics for AMR-to-text evaluation, since an abstract meaning representation allows for numerous surface realizations. In this work we aim to alleviate these issues by proposing ℳℱ𝛽, a decomposable metric that builds on two pillars. The first is the principle of meaning preservation : it measures to what extent a given AMR can be reconstructed from the generated sentence using SOTA AMR parsers and applying (fine-grained) AMR evaluation metrics to measure the distance between the original and the reconstructed AMR. The second pillar builds on a principle of (grammatical) form that measures the linguistic quality of the generated text, which we implement using SOTA language models. In two extensive pilot studies we show that fulfillment of both principles offers benefits for AMR-to-text evaluation, including explainability of scores. Since ℳℱ𝛽 does not necessarily rely on gold AMRs, it may extend to other text generation tasks.
Anthology ID:
2021.eacl-main.129
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
Month:
April
Year:
2021
Address:
Online
Editors:
Paola Merlo, Jorg Tiedemann, Reut Tsarfaty
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1504–1518
Language:
URL:
https://aclanthology.org/2021.eacl-main.129
DOI:
10.18653/v1/2021.eacl-main.129
Bibkey:
Cite (ACL):
Juri Opitz and Anette Frank. 2021. Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1504–1518, Online. Association for Computational Linguistics.
Cite (Informal):
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (Opitz & Frank, EACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eacl-main.129.pdf
Code
 Heidelberg-NLP/MFscore