<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="grundkiewicz-etal-2021-user">
    <titleInfo>
        <title>On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Roman</namePart>
        <namePart type="family">Grundkiewicz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marcin</namePart>
        <namePart type="family">Junczys-Dowmunt</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Christian</namePart>
        <namePart type="family">Federmann</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tom</namePart>
        <namePart type="family">Kocmi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-04</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anya</namePart>
            <namePart type="family">Belz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Shubham</namePart>
            <namePart type="family">Agarwal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yvette</namePart>
            <namePart type="family">Graham</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ehud</namePart>
            <namePart type="family">Reiter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Anastasia</namePart>
            <namePart type="family">Shimorina</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent studies emphasize the need of document context in human evaluation of machine translations, but little research has been done on the impact of user interfaces on annotator productivity and the reliability of assessments. In this work, we compare human assessment data from the last two WMT evaluation campaigns collected via two different methods for document-level evaluation. Our analysis shows that a document-centric approach to evaluation where the annotator is presented with the entire document context on a screen leads to higher quality segment and document level assessments. It improves the correlation between segment and document scores and increases inter-annotator agreement for document scores but is considerably more time consuming for annotators.</abstract>
    <identifier type="citekey">grundkiewicz-etal-2021-user</identifier>
    <location>
        <url>https://aclanthology.org/2021.humeval-1.11</url>
    </location>
    <part>
        <date>2021-04</date>
        <extent unit="page">
            <start>97</start>
            <end>106</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputs
%A Grundkiewicz, Roman
%A Junczys-Dowmunt, Marcin
%A Federmann, Christian
%A Kocmi, Tom
%Y Belz, Anya
%Y Agarwal, Shubham
%Y Graham, Yvette
%Y Reiter, Ehud
%Y Shimorina, Anastasia
%S Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)
%D 2021
%8 April
%I Association for Computational Linguistics
%C Online
%F grundkiewicz-etal-2021-user
%X Recent studies emphasize the need of document context in human evaluation of machine translations, but little research has been done on the impact of user interfaces on annotator productivity and the reliability of assessments. In this work, we compare human assessment data from the last two WMT evaluation campaigns collected via two different methods for document-level evaluation. Our analysis shows that a document-centric approach to evaluation where the annotator is presented with the entire document context on a screen leads to higher quality segment and document level assessments. It improves the correlation between segment and document scores and increases inter-annotator agreement for document scores but is considerably more time consuming for annotators.
%U https://aclanthology.org/2021.humeval-1.11
%P 97-106

Markdown (Informal)

[On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputs](https://aclanthology.org/2021.humeval-1.11) (Grundkiewicz et al., HumEval 2021)

• On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputs (Grundkiewicz et al., HumEval 2021)

ACL

• Roman Grundkiewicz, Marcin Junczys-Dowmunt, Christian Federmann, and Tom Kocmi. 2021. On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputs. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 97–106, Online. Association for Computational Linguistics.