<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zerva-martins-2024-conformalizing">
    <titleInfo>
        <title>Conformalizing Machine Translation Evaluation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Chrysoula</namePart>
        <namePart type="family">Zerva</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">André</namePart>
        <namePart type="given">F</namePart>
        <namePart type="given">T</namePart>
        <namePart type="family">Martins</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Several uncertainty estimation methods have been recently proposed for machine translation evaluation. While these methods can provide a useful indication of when not to trust model predictions, we show in this paper that the majority of them tend to underestimate model uncertainty, and as a result, they often produce misleading confidence intervals that do not cover the ground truth. We propose as an alternative the use of conformal prediction, a distribution-free method to obtain confidence intervals with a theoretically established guarantee on coverage. First, we demonstrate that split conformal prediction can “correct” the confidence intervals of previous methods to yield a desired coverage level, and we demonstrate these findings across multiple machine translation evaluation metrics and uncertainty quantification methods. Further, we highlight biases in estimated confidence intervals, reflected in imbalanced coverage for different attributes, such as the language and the quality of translations. We address this by applying conditional conformal prediction techniques to obtain calibration subsets for each data subgroup, leading to equalized coverage. Overall, we show that, provided access to a calibration set, conformal prediction can help identify the most suitable uncertainty quantification methods and adapt the predicted confidence intervals to ensure fairness with respect to different attributes.1</abstract>
    <identifier type="citekey">zerva-martins-2024-conformalizing</identifier>
    <identifier type="doi">10.1162/tacl_a_00711</identifier>
    <location>
        <url>https://aclanthology.org/2024.tacl-1.80/</url>
    </location>
    <part>
        <date>2024</date>
        <detail type="volume"><number>12</number></detail>
        <extent unit="page">
            <start>1460</start>
            <end>1478</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Conformalizing Machine Translation Evaluation
%A Zerva, Chrysoula
%A Martins, André F. T.
%J Transactions of the Association for Computational Linguistics
%D 2024
%V 12
%I MIT Press
%C Cambridge, MA
%F zerva-martins-2024-conformalizing
%X Several uncertainty estimation methods have been recently proposed for machine translation evaluation. While these methods can provide a useful indication of when not to trust model predictions, we show in this paper that the majority of them tend to underestimate model uncertainty, and as a result, they often produce misleading confidence intervals that do not cover the ground truth. We propose as an alternative the use of conformal prediction, a distribution-free method to obtain confidence intervals with a theoretically established guarantee on coverage. First, we demonstrate that split conformal prediction can “correct” the confidence intervals of previous methods to yield a desired coverage level, and we demonstrate these findings across multiple machine translation evaluation metrics and uncertainty quantification methods. Further, we highlight biases in estimated confidence intervals, reflected in imbalanced coverage for different attributes, such as the language and the quality of translations. We address this by applying conditional conformal prediction techniques to obtain calibration subsets for each data subgroup, leading to equalized coverage. Overall, we show that, provided access to a calibration set, conformal prediction can help identify the most suitable uncertainty quantification methods and adapt the predicted confidence intervals to ensure fairness with respect to different attributes.1
%R 10.1162/tacl_a_00711
%U https://aclanthology.org/2024.tacl-1.80/
%U https://doi.org/10.1162/tacl_a_00711
%P 1460-1478

Markdown (Informal)

[Conformalizing Machine Translation Evaluation](https://aclanthology.org/2024.tacl-1.80/) (Zerva & Martins, TACL 2024)

• Conformalizing Machine Translation Evaluation (Zerva & Martins, TACL 2024)

ACL

• Chrysoula Zerva and André F. T. Martins. 2024. Conformalizing Machine Translation Evaluation. Transactions of the Association for Computational Linguistics, 12:1460–1478.