<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="pal-heafield-2023-cheating">
    <titleInfo>
        <title>Cheating to Identify Hard Problems for Neural Machine Translation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Proyag</namePart>
        <namePart type="family">Pal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kenneth</namePart>
        <namePart type="family">Heafield</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EACL 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Andreas</namePart>
            <namePart type="family">Vlachos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Isabelle</namePart>
            <namePart type="family">Augenstein</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dubrovnik, Croatia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We identify hard problems for neural machine translation models by analyzing progressively higher-scoring translations generated by letting models cheat to various degrees. If a system cheats and still gets something wrong, that suggests it is a hard problem. We experiment with two forms of cheating: providing the model a compressed representation of the target as an additional input, and fine-tuning on the test set. Contrary to popular belief, we find that the most frequent tokens are not necessarily the most accurately translated due to these often being function words and punctuation that can be used more flexibly in translation, or content words which can easily be paraphrased. We systematically analyze system outputs to identify categories of tokens which are particularly hard for the model to translate, and find that this includes certain types of named entities, subordinating conjunctions, and unknown and foreign words. We also encounter a phenomenon where words, often names, which were not infrequent in the training data are still repeatedly mistranslated by the models — we dub this the Fleetwood Mac problem.</abstract>
    <identifier type="citekey">pal-heafield-2023-cheating</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-eacl.120</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-eacl.120</url>
    </location>
    <part>
        <date>2023-05</date>
        <extent unit="page">
            <start>1620</start>
            <end>1631</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Cheating to Identify Hard Problems for Neural Machine Translation
%A Pal, Proyag
%A Heafield, Kenneth
%Y Vlachos, Andreas
%Y Augenstein, Isabelle
%S Findings of the Association for Computational Linguistics: EACL 2023
%D 2023
%8 May
%I Association for Computational Linguistics
%C Dubrovnik, Croatia
%F pal-heafield-2023-cheating
%X We identify hard problems for neural machine translation models by analyzing progressively higher-scoring translations generated by letting models cheat to various degrees. If a system cheats and still gets something wrong, that suggests it is a hard problem. We experiment with two forms of cheating: providing the model a compressed representation of the target as an additional input, and fine-tuning on the test set. Contrary to popular belief, we find that the most frequent tokens are not necessarily the most accurately translated due to these often being function words and punctuation that can be used more flexibly in translation, or content words which can easily be paraphrased. We systematically analyze system outputs to identify categories of tokens which are particularly hard for the model to translate, and find that this includes certain types of named entities, subordinating conjunctions, and unknown and foreign words. We also encounter a phenomenon where words, often names, which were not infrequent in the training data are still repeatedly mistranslated by the models — we dub this the Fleetwood Mac problem.
%R 10.18653/v1/2023.findings-eacl.120
%U https://aclanthology.org/2023.findings-eacl.120
%U https://doi.org/10.18653/v1/2023.findings-eacl.120
%P 1620-1631

Markdown (Informal)

[Cheating to Identify Hard Problems for Neural Machine Translation](https://aclanthology.org/2023.findings-eacl.120) (Pal & Heafield, Findings 2023)

• Cheating to Identify Hard Problems for Neural Machine Translation (Pal & Heafield, Findings 2023)

ACL

• Proyag Pal and Kenneth Heafield. 2023. Cheating to Identify Hard Problems for Neural Machine Translation. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1620–1631, Dubrovnik, Croatia. Association for Computational Linguistics.