<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="risch-etal-2021-semantic">
    <titleInfo>
        <title>Semantic Answer Similarity for Evaluating Question Answering Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Julian</namePart>
        <namePart type="family">Risch</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Timo</namePart>
        <namePart type="family">Möller</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Julian</namePart>
        <namePart type="family">Gutsch</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Malte</namePart>
        <namePart type="family">Pietsch</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 3rd Workshop on Machine Reading for Question Answering</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Adam</namePart>
            <namePart type="family">Fisch</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alon</namePart>
            <namePart type="family">Talmor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Danqi</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Eunsol</namePart>
            <namePart type="family">Choi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Minjoon</namePart>
            <namePart type="family">Seo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Patrick</namePart>
            <namePart type="family">Lewis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Robin</namePart>
            <namePart type="family">Jia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sewon</namePart>
            <namePart type="family">Min</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The evaluation of question answering models compares ground-truth annotations with model predictions. However, as of today, this comparison is mostly lexical-based and therefore misses out on answers that have no lexical overlap but are still semantically similar, thus treating correct answers as false. This underestimation of the true performance of models hinders user acceptance in applications and complicates a fair comparison of different models. Therefore, there is a need for an evaluation metric that is based on semantics instead of pure string similarity. In this short paper, we present SAS, a cross-encoder-based metric for the estimation of semantic answer similarity, and compare it to seven existing metrics. To this end, we create an English and a German three-way annotated evaluation dataset containing pairs of answers along with human judgment of their semantic similarity, which we release along with an implementation of the SAS metric and the experiments. We find that semantic similarity metrics based on recent transformer models correlate much better with human judgment than traditional lexical similarity metrics on our two newly created datasets and one dataset from related work.</abstract>
    <identifier type="citekey">risch-etal-2021-semantic</identifier>
    <identifier type="doi">10.18653/v1/2021.mrqa-1.15</identifier>
    <location>
        <url>https://aclanthology.org/2021.mrqa-1.15</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>149</start>
            <end>157</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Semantic Answer Similarity for Evaluating Question Answering Models
%A Risch, Julian
%A Möller, Timo
%A Gutsch, Julian
%A Pietsch, Malte
%Y Fisch, Adam
%Y Talmor, Alon
%Y Chen, Danqi
%Y Choi, Eunsol
%Y Seo, Minjoon
%Y Lewis, Patrick
%Y Jia, Robin
%Y Min, Sewon
%S Proceedings of the 3rd Workshop on Machine Reading for Question Answering
%D 2021
%8 November
%I Association for Computational Linguistics
%C Punta Cana, Dominican Republic
%F risch-etal-2021-semantic
%X The evaluation of question answering models compares ground-truth annotations with model predictions. However, as of today, this comparison is mostly lexical-based and therefore misses out on answers that have no lexical overlap but are still semantically similar, thus treating correct answers as false. This underestimation of the true performance of models hinders user acceptance in applications and complicates a fair comparison of different models. Therefore, there is a need for an evaluation metric that is based on semantics instead of pure string similarity. In this short paper, we present SAS, a cross-encoder-based metric for the estimation of semantic answer similarity, and compare it to seven existing metrics. To this end, we create an English and a German three-way annotated evaluation dataset containing pairs of answers along with human judgment of their semantic similarity, which we release along with an implementation of the SAS metric and the experiments. We find that semantic similarity metrics based on recent transformer models correlate much better with human judgment than traditional lexical similarity metrics on our two newly created datasets and one dataset from related work.
%R 10.18653/v1/2021.mrqa-1.15
%U https://aclanthology.org/2021.mrqa-1.15
%U https://doi.org/10.18653/v1/2021.mrqa-1.15
%P 149-157

Markdown (Informal)

[Semantic Answer Similarity for Evaluating Question Answering Models](https://aclanthology.org/2021.mrqa-1.15) (Risch et al., MRQA 2021)

• Semantic Answer Similarity for Evaluating Question Answering Models (Risch et al., MRQA 2021)

ACL

• Julian Risch, Timo Möller, Julian Gutsch, and Malte Pietsch. 2021. Semantic Answer Similarity for Evaluating Question Answering Models. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 149–157, Punta Cana, Dominican Republic. Association for Computational Linguistics.