<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="graham-etal-2020-assessing">
    <titleInfo>
        <title>Assessing Human-Parity in Machine Translation on the Segment Level</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yvette</namePart>
        <namePart type="family">Graham</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Christian</namePart>
        <namePart type="family">Federmann</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Maria</namePart>
        <namePart type="family">Eskevich</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Barry</namePart>
        <namePart type="family">Haddow</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2020</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Trevor</namePart>
            <namePart type="family">Cohn</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yulan</namePart>
            <namePart type="family">He</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yang</namePart>
            <namePart type="family">Liu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent machine translation shared tasks have shown top-performing systems to tie or in some cases even outperform human translation. Such conclusions about system and human performance are, however, based on estimates aggregated from scores collected over large test sets of translations and unfortunately leave some remaining questions unanswered. For instance, simply because a system significantly outperforms the human translator on average may not necessarily mean that it has done so for every translation in the test set. Firstly, are there remaining source segments present in evaluation test sets that cause significant challenges for top-performing systems and can such challenging segments go unnoticed due to the opacity of current human evaluation procedures? To provide insight into these issues we carefully inspect the outputs of top-performing systems in the most recent WMT-19 news translation shared task for all language pairs in which a system either tied or outperformed human translation. Our analysis provides a new method of identifying the remaining segments for which either machine or human perform poorly. For example, in our close inspection of WMT-19 English to German and German to English we discover the segments that disjointly proved a challenge for human and machine. For English to Russian, there were no segments included in our sample of translations that caused a significant challenge for the human translator, while we again identify the set of segments that caused issues for the top-performing system.</abstract>
    <identifier type="citekey">graham-etal-2020-assessing</identifier>
    <identifier type="doi">10.18653/v1/2020.findings-emnlp.375</identifier>
    <location>
        <url>https://aclanthology.org/2020.findings-emnlp.375</url>
    </location>
    <part>
        <date>2020-11</date>
        <extent unit="page">
            <start>4199</start>
            <end>4207</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Assessing Human-Parity in Machine Translation on the Segment Level
%A Graham, Yvette
%A Federmann, Christian
%A Eskevich, Maria
%A Haddow, Barry
%Y Cohn, Trevor
%Y He, Yulan
%Y Liu, Yang
%S Findings of the Association for Computational Linguistics: EMNLP 2020
%D 2020
%8 November
%I Association for Computational Linguistics
%C Online
%F graham-etal-2020-assessing
%X Recent machine translation shared tasks have shown top-performing systems to tie or in some cases even outperform human translation. Such conclusions about system and human performance are, however, based on estimates aggregated from scores collected over large test sets of translations and unfortunately leave some remaining questions unanswered. For instance, simply because a system significantly outperforms the human translator on average may not necessarily mean that it has done so for every translation in the test set. Firstly, are there remaining source segments present in evaluation test sets that cause significant challenges for top-performing systems and can such challenging segments go unnoticed due to the opacity of current human evaluation procedures? To provide insight into these issues we carefully inspect the outputs of top-performing systems in the most recent WMT-19 news translation shared task for all language pairs in which a system either tied or outperformed human translation. Our analysis provides a new method of identifying the remaining segments for which either machine or human perform poorly. For example, in our close inspection of WMT-19 English to German and German to English we discover the segments that disjointly proved a challenge for human and machine. For English to Russian, there were no segments included in our sample of translations that caused a significant challenge for the human translator, while we again identify the set of segments that caused issues for the top-performing system.
%R 10.18653/v1/2020.findings-emnlp.375
%U https://aclanthology.org/2020.findings-emnlp.375
%U https://doi.org/10.18653/v1/2020.findings-emnlp.375
%P 4199-4207

Markdown (Informal)

[Assessing Human-Parity in Machine Translation on the Segment Level](https://aclanthology.org/2020.findings-emnlp.375) (Graham et al., Findings 2020)

• Assessing Human-Parity in Machine Translation on the Segment Level (Graham et al., Findings 2020)

ACL

• Yvette Graham, Christian Federmann, Maria Eskevich, and Barry Haddow. 2020. Assessing Human-Parity in Machine Translation on the Segment Level. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4199–4207, Online. Association for Computational Linguistics.