<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zufle-niehues-2025-contrastive">
    <titleInfo>
        <title>Contrastive Learning for Task-Independent SpeechLLM-Pretraining</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Maike</namePart>
        <namePart type="family">Züfle</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jan</namePart>
        <namePart type="family">Niehues</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2025</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Wanxiang</namePart>
            <namePart type="family">Che</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joyce</namePart>
            <namePart type="family">Nabende</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ekaterina</namePart>
            <namePart type="family">Shutova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohammad</namePart>
            <namePart type="given">Taher</namePart>
            <namePart type="family">Pilehvar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Vienna, Austria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-256-5</identifier>
    </relatedItem>
    <abstract>Large language models (LLMs) excel in natural language processing but adapting these LLMs to speech processing tasks efficiently is not straightforward. Direct task-specific fine-tuning is limited by overfitting risks, data requirements, and computational costs. To address these challenges, we propose a scalable, two-stage training approach: (1) A task-independent speech pretraining stage using contrastive learning to align text and speech representations over all layers, followed by (2) a task-specific fine-tuning stage requiring minimal data. This approach outperforms traditional ASR pretraining and enables the model to surpass models specialized on speech translation and question answering while being trained on only 10% of the task-specific data.</abstract>
    <identifier type="citekey">zufle-niehues-2025-contrastive</identifier>
    <identifier type="doi">10.18653/v1/2025.findings-acl.445</identifier>
    <location>
        <url>https://aclanthology.org/2025.findings-acl.445/</url>
    </location>
    <part>
        <date>2025-07</date>
        <extent unit="page">
            <start>8469</start>
            <end>8490</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Contrastive Learning for Task-Independent SpeechLLM-Pretraining
%A Züfle, Maike
%A Niehues, Jan
%Y Che, Wanxiang
%Y Nabende, Joyce
%Y Shutova, Ekaterina
%Y Pilehvar, Mohammad Taher
%S Findings of the Association for Computational Linguistics: ACL 2025
%D 2025
%8 July
%I Association for Computational Linguistics
%C Vienna, Austria
%@ 979-8-89176-256-5
%F zufle-niehues-2025-contrastive
%X Large language models (LLMs) excel in natural language processing but adapting these LLMs to speech processing tasks efficiently is not straightforward. Direct task-specific fine-tuning is limited by overfitting risks, data requirements, and computational costs. To address these challenges, we propose a scalable, two-stage training approach: (1) A task-independent speech pretraining stage using contrastive learning to align text and speech representations over all layers, followed by (2) a task-specific fine-tuning stage requiring minimal data. This approach outperforms traditional ASR pretraining and enables the model to surpass models specialized on speech translation and question answering while being trained on only 10% of the task-specific data.
%R 10.18653/v1/2025.findings-acl.445
%U https://aclanthology.org/2025.findings-acl.445/
%U https://doi.org/10.18653/v1/2025.findings-acl.445
%P 8469-8490

Markdown (Informal)

[Contrastive Learning for Task-Independent SpeechLLM-Pretraining](https://aclanthology.org/2025.findings-acl.445/) (Züfle & Niehues, Findings 2025)

• Contrastive Learning for Task-Independent SpeechLLM-Pretraining (Züfle & Niehues, Findings 2025)

ACL

• Maike Züfle and Jan Niehues. 2025. Contrastive Learning for Task-Independent SpeechLLM-Pretraining. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8469–8490, Vienna, Austria. Association for Computational Linguistics.