<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="opitz-2023-smatch">
    <titleInfo>
        <title>SMATCH++: Standardized and Extended Evaluation of Semantic Graphs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Juri</namePart>
        <namePart type="family">Opitz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EACL 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Andreas</namePart>
            <namePart type="family">Vlachos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Isabelle</namePart>
            <namePart type="family">Augenstein</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dubrovnik, Croatia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The Smatch metric is a popular method for evaluating graph distances, as is necessary, for instance, to assess the performance of semantic graph parsing systems. However, we observe some issues in the metric that jeopardize meaningful evaluation. E.g., opaque pre-processing choices can affect results, and current graph-alignment solvers do not provide us with upper-bounds. Without upper-bounds, however, fair evaluation is not guaranteed. Furthermore, adaptions of Smatch for extended tasks (e.g., fine-grained semantic similarity) are spread out, and lack a unifying framework. For better inspection, we divide the metric into three modules: pre-processing, alignment, and scoring. Examining each module, we specify its goals and diagnose potential issues, for which we discuss and test mitigation strategies. For pre-processing, we show how to fully conform to annotation guidelines that allow structurally deviating but valid graphs. For safer and enhanced alignment, we show the feasibility of optimal alignment in a standard evaluation setup, and develop a lossless graph compression method that shrinks the search space and significantly increases efficiency. For improved scoring, we propose standardized and extended metric calculation of fine-grained sub-graph meaning aspects. Our code is available at https://github.com/flipz357/smatchpp</abstract>
    <identifier type="citekey">opitz-2023-smatch</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-eacl.118</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-eacl.118</url>
    </location>
    <part>
        <date>2023-05</date>
        <extent unit="page">
            <start>1595</start>
            <end>1607</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T SMATCH++: Standardized and Extended Evaluation of Semantic Graphs
%A Opitz, Juri
%Y Vlachos, Andreas
%Y Augenstein, Isabelle
%S Findings of the Association for Computational Linguistics: EACL 2023
%D 2023
%8 May
%I Association for Computational Linguistics
%C Dubrovnik, Croatia
%F opitz-2023-smatch
%X The Smatch metric is a popular method for evaluating graph distances, as is necessary, for instance, to assess the performance of semantic graph parsing systems. However, we observe some issues in the metric that jeopardize meaningful evaluation. E.g., opaque pre-processing choices can affect results, and current graph-alignment solvers do not provide us with upper-bounds. Without upper-bounds, however, fair evaluation is not guaranteed. Furthermore, adaptions of Smatch for extended tasks (e.g., fine-grained semantic similarity) are spread out, and lack a unifying framework. For better inspection, we divide the metric into three modules: pre-processing, alignment, and scoring. Examining each module, we specify its goals and diagnose potential issues, for which we discuss and test mitigation strategies. For pre-processing, we show how to fully conform to annotation guidelines that allow structurally deviating but valid graphs. For safer and enhanced alignment, we show the feasibility of optimal alignment in a standard evaluation setup, and develop a lossless graph compression method that shrinks the search space and significantly increases efficiency. For improved scoring, we propose standardized and extended metric calculation of fine-grained sub-graph meaning aspects. Our code is available at https://github.com/flipz357/smatchpp
%R 10.18653/v1/2023.findings-eacl.118
%U https://aclanthology.org/2023.findings-eacl.118
%U https://doi.org/10.18653/v1/2023.findings-eacl.118
%P 1595-1607

Markdown (Informal)

[SMATCH++: Standardized and Extended Evaluation of Semantic Graphs](https://aclanthology.org/2023.findings-eacl.118) (Opitz, Findings 2023)

• SMATCH++: Standardized and Extended Evaluation of Semantic Graphs (Opitz, Findings 2023)

ACL

• Juri Opitz. 2023. SMATCH++: Standardized and Extended Evaluation of Semantic Graphs. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1595–1607, Dubrovnik, Croatia. Association for Computational Linguistics.