<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="manohar-pillai-2024-lost">
    <titleInfo>
        <title>What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Kavya</namePart>
        <namePart type="family">Manohar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Leena</namePart>
        <namePart type="family">Pillai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yaser</namePart>
            <namePart type="family">Al-Onaizan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>This paper explores the pitfalls in evaluating multilingual automatic speech recognition (ASR) models, with a particular focus on Indic language scripts. We investigate the text normalization routine employed by leading ASR models, including OpenAI Whisper, Meta’s MMS, Seamless, and Assembly AI’s Conformer, and their unintended consequences on performance metrics. Our research reveals that current text normalization practices, while aiming to standardize ASR outputs for fair comparison, by removing inconsistencies such as variations in spelling, punctuation, and special characters, are fundamentally flawed when applied to Indic scripts. Through empirical analysis using text similarity scores and in-depth linguistic examination, we demonstrate that these flaws lead to artificially improved performance metrics for Indic languages. We conclude by proposing a shift towards developing text normalization routines that leverage native linguistic expertise, ensuring more robust and accurate evaluations of multilingual ASR models.</abstract>
    <identifier type="citekey">manohar-pillai-2024-lost</identifier>
    <location>
        <url>https://aclanthology.org/2024.emnlp-main.607</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>10864</start>
            <end>10869</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations
%A Manohar, Kavya
%A Pillai, Leena
%Y Al-Onaizan, Yaser
%Y Bansal, Mohit
%Y Chen, Yun-Nung
%S Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F manohar-pillai-2024-lost
%X This paper explores the pitfalls in evaluating multilingual automatic speech recognition (ASR) models, with a particular focus on Indic language scripts. We investigate the text normalization routine employed by leading ASR models, including OpenAI Whisper, Meta’s MMS, Seamless, and Assembly AI’s Conformer, and their unintended consequences on performance metrics. Our research reveals that current text normalization practices, while aiming to standardize ASR outputs for fair comparison, by removing inconsistencies such as variations in spelling, punctuation, and special characters, are fundamentally flawed when applied to Indic scripts. Through empirical analysis using text similarity scores and in-depth linguistic examination, we demonstrate that these flaws lead to artificially improved performance metrics for Indic languages. We conclude by proposing a shift towards developing text normalization routines that leverage native linguistic expertise, ensuring more robust and accurate evaluations of multilingual ASR models.
%U https://aclanthology.org/2024.emnlp-main.607
%P 10864-10869

Markdown (Informal)

[What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations](https://aclanthology.org/2024.emnlp-main.607) (Manohar & Pillai, EMNLP 2024)

• What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations (Manohar & Pillai, EMNLP 2024)

ACL

• Kavya Manohar and Leena Pillai. 2024. What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10864–10869, Miami, Florida, USA. Association for Computational Linguistics.