<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lo-etal-2023-beyond">
    <titleInfo>
        <title>Beyond Correlation: Making Sense of the Score Differences of New MT Evaluation Metrics</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Chi-kiu</namePart>
        <namePart type="family">Lo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rebecca</namePart>
        <namePart type="family">Knowles</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Cyril</namePart>
        <namePart type="family">Goutte</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Masao</namePart>
            <namePart type="family">Utiyama</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Rui</namePart>
            <namePart type="family">Wang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Asia-Pacific Association for Machine Translation</publisher>
            <place>
                <placeTerm type="text">Macau SAR, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>While many new automatic metrics for machine translation evaluation have been proposed in recent years, BLEU scores are still used as the primary metric in the vast majority of MT research papers. There are many reasons that researchers may be reluctant to switch to new metrics, from external pressures (reviewers, prior work) to the ease of use of metric toolkits. Another reason is a lack of intuition about the meaning of novel metric scores. In this work, we examine “rules of thumb” about metric score differences and how they do (and do not) correspond to human judgments of statistically significant differences between systems. In particular, we show that common rules of thumb about BLEU score differences do not in fact guarantee that human annotators will find significant differences between systems. We also show ways in which these rules of thumb fail to generalize across translation directions or domains.</abstract>
    <identifier type="citekey">lo-etal-2023-beyond</identifier>
    <location>
        <url>https://aclanthology.org/2023.mtsummit-research.16</url>
    </location>
    <part>
        <date>2023-09</date>
        <extent unit="page">
            <start>186</start>
            <end>199</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Beyond Correlation: Making Sense of the Score Differences of New MT Evaluation Metrics
%A Lo, Chi-kiu
%A Knowles, Rebecca
%A Goutte, Cyril
%Y Utiyama, Masao
%Y Wang, Rui
%S Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track
%D 2023
%8 September
%I Asia-Pacific Association for Machine Translation
%C Macau SAR, China
%F lo-etal-2023-beyond
%X While many new automatic metrics for machine translation evaluation have been proposed in recent years, BLEU scores are still used as the primary metric in the vast majority of MT research papers. There are many reasons that researchers may be reluctant to switch to new metrics, from external pressures (reviewers, prior work) to the ease of use of metric toolkits. Another reason is a lack of intuition about the meaning of novel metric scores. In this work, we examine “rules of thumb” about metric score differences and how they do (and do not) correspond to human judgments of statistically significant differences between systems. In particular, we show that common rules of thumb about BLEU score differences do not in fact guarantee that human annotators will find significant differences between systems. We also show ways in which these rules of thumb fail to generalize across translation directions or domains.
%U https://aclanthology.org/2023.mtsummit-research.16
%P 186-199

Markdown (Informal)

[Beyond Correlation: Making Sense of the Score Differences of New MT Evaluation Metrics](https://aclanthology.org/2023.mtsummit-research.16) (Lo et al., MTSummit 2023)

• Beyond Correlation: Making Sense of the Score Differences of New MT Evaluation Metrics (Lo et al., MTSummit 2023)

ACL

• Chi-kiu Lo, Rebecca Knowles, and Cyril Goutte. 2023. Beyond Correlation: Making Sense of the Score Differences of New MT Evaluation Metrics. In Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 186–199, Macau SAR, China. Asia-Pacific Association for Machine Translation.