<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zarzu-zouhar-2026-hurdles">
    <titleInfo>
        <title>Hurdles of Automatic Metric for Speech Translation Evaluation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Victor</namePart>
        <namePart type="given">Eugen</namePart>
        <namePart type="family">Zarzu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vilem</namePart>
        <namePart type="family">Zouhar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Elizabeth</namePart>
            <namePart type="family">Salesky</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Antonios</namePart>
            <namePart type="family">Anastasopoulos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Matteo</namePart>
            <namePart type="family">Negri</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marcello</namePart>
            <namePart type="family">Federico</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">San Diego, USA (in-person and online)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-411-8</identifier>
    </relatedItem>
    <abstract>Automatic evaluation of speech translation has so far relied on text-only automated metrics that ignore speech phenomena. One would expect that incorporating the source audio modality would improve the performance of automatic metrics. We implement two standard metric paradigms: a COMET-audio regression model using audio and text encoders, and one based on prompting a speech large language model. Surprisingly, both audio-infused models fail to reliably surpass text-only baselines. We attribute this failure to the noise pollution and audio-transcript mismatches present in the audio signal, which makes the modality unreliable from the metric’s perspective. Furthermore, we argue that current human-annotated evaluation datasets for automated metrics predominantly feature technical content or short texts where paralinguistic features like prosody lack importance, rendering the extra audio information unhelpful for quality estimation (QE).</abstract>
    <identifier type="citekey">zarzu-zouhar-2026-hurdles</identifier>
    <location>
        <url>https://aclanthology.org/2026.iwslt-1.34/</url>
    </location>
    <part>
        <date>2026-07</date>
        <extent unit="page">
            <start>305</start>
            <end>315</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Hurdles of Automatic Metric for Speech Translation Evaluation
%A Zarzu, Victor Eugen
%A Zouhar, Vilem
%Y Salesky, Elizabeth
%Y Anastasopoulos, Antonios
%Y Negri, Matteo
%Y Federico, Marcello
%S Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)
%D 2026
%8 July
%I Association for Computational Linguistics
%C San Diego, USA (in-person and online)
%@ 979-8-89176-411-8
%F zarzu-zouhar-2026-hurdles
%X Automatic evaluation of speech translation has so far relied on text-only automated metrics that ignore speech phenomena. One would expect that incorporating the source audio modality would improve the performance of automatic metrics. We implement two standard metric paradigms: a COMET-audio regression model using audio and text encoders, and one based on prompting a speech large language model. Surprisingly, both audio-infused models fail to reliably surpass text-only baselines. We attribute this failure to the noise pollution and audio-transcript mismatches present in the audio signal, which makes the modality unreliable from the metric’s perspective. Furthermore, we argue that current human-annotated evaluation datasets for automated metrics predominantly feature technical content or short texts where paralinguistic features like prosody lack importance, rendering the extra audio information unhelpful for quality estimation (QE).
%U https://aclanthology.org/2026.iwslt-1.34/
%P 305-315

Markdown (Informal)

[Hurdles of Automatic Metric for Speech Translation Evaluation](https://aclanthology.org/2026.iwslt-1.34/) (Zarzu & Zouhar, IWSLT 2026)

• Hurdles of Automatic Metric for Speech Translation Evaluation (Zarzu & Zouhar, IWSLT 2026)

ACL

• Victor Eugen Zarzu and Vilem Zouhar. 2026. Hurdles of Automatic Metric for Speech Translation Evaluation. In Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026), pages 305–315, San Diego, USA (in-person and online). Association for Computational Linguistics.