<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="thompson-2026-amanamt">
    <titleInfo>
        <title>AmanaMT: Translation Direction Predicts Automatic Metric Reliability</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Isaac</namePart>
        <namePart type="family">Thompson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Eleventh Conference on Machine Translation</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Barry</namePart>
            <namePart type="family">Haddow</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tom</namePart>
            <namePart type="family">Kocmi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Philipp</namePart>
            <namePart type="family">Koehn</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Christof</namePart>
            <namePart type="family">Monz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-474-3</identifier>
    </relatedItem>
    <abstract>When machine translation (MT) metrics fail in low-resource settings, the field typically blames data scarcity. We show this diagnosis is often wrong using AmanaMT, a unified benchmark spanning 730,748 segments, 33 languages, three annotation tiers, and 10 metrics: the dominant predictor of metric unreliability is not resource level but domain mismatch and translation direction. Direction is the single largest predictor of between-language variance in metric reliability (\ensuremathη2dir = 0.309, nearly a third of between-language variance); morphological type is a substantially smaller main effect (\ensuremathη2morph = 0.073), though the joint direction×morphology cells account for \ensuremathη2cell = 0.618, capturing direction-specific morphological variation. Ukrainian’s near-zero COMET correlation is fully explained by a domain label (other, not news): a genre confound, not a language problem. Russian’s weak aggregate dissolves into clean per-year signals once stratified by WMT year, a textbook Simpson’s Paradox from pooling heterogeneous system generations. Neural metrics (MetricX-24, xCOMET, COMET) substantially outperform n-gram baselines at every tier; the gap is largest in morphologically complex and non-news settings where surface overlap is an especially poor proxy for adequacy.</abstract>
    <identifier type="citekey">thompson-2026-amanamt</identifier>
    <location>
        <url>https://aclanthology.org/2026.wmt-main.42/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>747</start>
            <end>764</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T AmanaMT: Translation Direction Predicts Automatic Metric Reliability
%A Thompson, Isaac
%Y Haddow, Barry
%Y Kocmi, Tom
%Y Koehn, Philipp
%Y Monz, Christof
%S Proceedings of the Eleventh Conference on Machine Translation
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-474-3
%F thompson-2026-amanamt
%X When machine translation (MT) metrics fail in low-resource settings, the field typically blames data scarcity. We show this diagnosis is often wrong using AmanaMT, a unified benchmark spanning 730,748 segments, 33 languages, three annotation tiers, and 10 metrics: the dominant predictor of metric unreliability is not resource level but domain mismatch and translation direction. Direction is the single largest predictor of between-language variance in metric reliability (\ensuremathη2dir = 0.309, nearly a third of between-language variance); morphological type is a substantially smaller main effect (\ensuremathη2morph = 0.073), though the joint direction×morphology cells account for \ensuremathη2cell = 0.618, capturing direction-specific morphological variation. Ukrainian’s near-zero COMET correlation is fully explained by a domain label (other, not news): a genre confound, not a language problem. Russian’s weak aggregate dissolves into clean per-year signals once stratified by WMT year, a textbook Simpson’s Paradox from pooling heterogeneous system generations. Neural metrics (MetricX-24, xCOMET, COMET) substantially outperform n-gram baselines at every tier; the gap is largest in morphologically complex and non-news settings where surface overlap is an especially poor proxy for adequacy.
%U https://aclanthology.org/2026.wmt-main.42/
%P 747-764

Markdown (Informal)

[AmanaMT: Translation Direction Predicts Automatic Metric Reliability](https://aclanthology.org/2026.wmt-main.42/) (Thompson, WMT 2026)

• AmanaMT: Translation Direction Predicts Automatic Metric Reliability (Thompson, WMT 2026)

ACL

• Isaac Thompson. 2026. AmanaMT: Translation Direction Predicts Automatic Metric Reliability. In Proceedings of the Eleventh Conference on Machine Translation, pages 747–764, Budapest, Hungary. Association for Computational Linguistics.