<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="alastruey-etal-2024-unveiling">
    <titleInfo>
        <title>Unveiling the Role of Pretraining in Direct Speech Translation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Belen</namePart>
        <namePart type="family">Alastruey</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gerard</namePart>
        <namePart type="family">Gállego</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marta</namePart>
        <namePart type="family">Costa-jussà</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yaser</namePart>
            <namePart type="family">Al-Onaizan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Direct speech-to-text translation systems encounter an important drawback in data scarcity. A common solution consists on pretraining the encoder on automatic speech recognition, hence losing efficiency in the training process. In this study, we compare the training dynamics of a system using a pretrained encoder, the conventional approach, and one trained from scratch. We observe that, throughout the training, the randomly initialized model struggles to incorporate information from the speech inputs for its predictions. Hence, we hypothesize that this issue stems from the difficulty of effectively training an encoder for direct speech translation. While a model trained from scratch needs to learn acoustic and semantic modeling simultaneously, a pretrained one can just focus on the latter. Based on these findings, we propose a subtle change in the decoder cross-attention to integrate source information from earlier steps in training. We show that with this change, the model trained from scratch can achieve comparable performance to the pretrained one, while reducing the training time.</abstract>
    <identifier type="citekey">alastruey-etal-2024-unveiling</identifier>
    <location>
        <url>https://aclanthology.org/2024.emnlp-main.630</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>11259</start>
            <end>11265</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Unveiling the Role of Pretraining in Direct Speech Translation
%A Alastruey, Belen
%A Gállego, Gerard
%A Costa-jussà, Marta
%Y Al-Onaizan, Yaser
%Y Bansal, Mohit
%Y Chen, Yun-Nung
%S Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F alastruey-etal-2024-unveiling
%X Direct speech-to-text translation systems encounter an important drawback in data scarcity. A common solution consists on pretraining the encoder on automatic speech recognition, hence losing efficiency in the training process. In this study, we compare the training dynamics of a system using a pretrained encoder, the conventional approach, and one trained from scratch. We observe that, throughout the training, the randomly initialized model struggles to incorporate information from the speech inputs for its predictions. Hence, we hypothesize that this issue stems from the difficulty of effectively training an encoder for direct speech translation. While a model trained from scratch needs to learn acoustic and semantic modeling simultaneously, a pretrained one can just focus on the latter. Based on these findings, we propose a subtle change in the decoder cross-attention to integrate source information from earlier steps in training. We show that with this change, the model trained from scratch can achieve comparable performance to the pretrained one, while reducing the training time.
%U https://aclanthology.org/2024.emnlp-main.630
%P 11259-11265

Markdown (Informal)

[Unveiling the Role of Pretraining in Direct Speech Translation](https://aclanthology.org/2024.emnlp-main.630) (Alastruey et al., EMNLP 2024)

• Unveiling the Role of Pretraining in Direct Speech Translation (Alastruey et al., EMNLP 2024)

ACL

• Belen Alastruey, Gerard Gállego, and Marta Costa-jussà. 2024. Unveiling the Role of Pretraining in Direct Speech Translation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11259–11265, Miami, Florida, USA. Association for Computational Linguistics.