SMATCH++: Standardized and Extended Evaluation of Semantic Graphs

Juri Opitz


Abstract
The Smatch metric is a popular method for evaluating graph distances, as is necessary, for instance, to assess the performance of semantic graph parsing systems. However, we observe some issues in the metric that jeopardize meaningful evaluation. E.g., opaque pre-processing choices can affect results, and current graph-alignment solvers do not provide us with upper-bounds. Without upper-bounds, however, fair evaluation is not guaranteed. Furthermore, adaptions of Smatch for extended tasks (e.g., fine-grained semantic similarity) are spread out, and lack a unifying framework. For better inspection, we divide the metric into three modules: pre-processing, alignment, and scoring. Examining each module, we specify its goals and diagnose potential issues, for which we discuss and test mitigation strategies. For pre-processing, we show how to fully conform to annotation guidelines that allow structurally deviating but valid graphs. For safer and enhanced alignment, we show the feasibility of optimal alignment in a standard evaluation setup, and develop a lossless graph compression method that shrinks the search space and significantly increases efficiency. For improved scoring, we propose standardized and extended metric calculation of fine-grained sub-graph meaning aspects. Our code is available at https://github.com/flipz357/smatchpp
Anthology ID:
2023.findings-eacl.118
Volume:
Findings of the Association for Computational Linguistics: EACL 2023
Month:
May
Year:
2023
Address:
Dubrovnik, Croatia
Editors:
Andreas Vlachos, Isabelle Augenstein
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1595–1607
Language:
URL:
https://aclanthology.org/2023.findings-eacl.118
DOI:
10.18653/v1/2023.findings-eacl.118
Bibkey:
Cite (ACL):
Juri Opitz. 2023. SMATCH++: Standardized and Extended Evaluation of Semantic Graphs. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1595–1607, Dubrovnik, Croatia. Association for Computational Linguistics.
Cite (Informal):
SMATCH++: Standardized and Extended Evaluation of Semantic Graphs (Opitz, Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-eacl.118.pdf
Video:
 https://aclanthology.org/2023.findings-eacl.118.mp4