<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="pasad-etal-2024-self">
    <titleInfo>
        <title>What Do Self-Supervised Speech Models Know About Words?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ankita</namePart>
        <namePart type="family">Pasad</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Chung-Ming</namePart>
        <namePart type="family">Chien</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shane</namePart>
        <namePart type="family">Settle</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Karen</namePart>
        <namePart type="family">Livescu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Many self-supervised speech models (S3Ms) have been introduced over the last few years, improving performance and data efficiency on various speech tasks. However, these empirical successes alone do not give a complete picture of what is learned during pre-training. Recent work has begun analyzing how S3Ms encode certain properties, such as phonetic and speaker information, but we still lack a proper understanding of knowledge encoded at the word level and beyond. In this work, we use lightweight analysis methods to study segment-level linguistic properties—word identity, boundaries, pronunciation, syntactic features, and semantic features—encoded in S3Ms. We present a comparative study of layer-wise representations from ten S3Ms and find that (i) the frame-level representations within each word segment are not all equally informative, and (ii) the pre-training objective and model size heavily influence the accessibility and distribution of linguistic information across layers. We also find that on several tasks—word discrimination, word segmentation, and semantic sentence similarity—S3Ms trained with visual grounding outperform their speech-only counterparts. Finally, our task-based analyses demonstrate improved performance on word segmentation and acoustic word discrimination while using simpler methods than prior work.1</abstract>
    <identifier type="citekey">pasad-etal-2024-self</identifier>
    <identifier type="doi">10.1162/tacl_a_00656</identifier>
    <location>
        <url>https://aclanthology.org/2024.tacl-1.21</url>
    </location>
    <part>
        <date>2024</date>
        <detail type="volume"><number>12</number></detail>
        <extent unit="page">
            <start>372</start>
            <end>391</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T What Do Self-Supervised Speech Models Know About Words?
%A Pasad, Ankita
%A Chien, Chung-Ming
%A Settle, Shane
%A Livescu, Karen
%J Transactions of the Association for Computational Linguistics
%D 2024
%V 12
%I MIT Press
%C Cambridge, MA
%F pasad-etal-2024-self
%X Many self-supervised speech models (S3Ms) have been introduced over the last few years, improving performance and data efficiency on various speech tasks. However, these empirical successes alone do not give a complete picture of what is learned during pre-training. Recent work has begun analyzing how S3Ms encode certain properties, such as phonetic and speaker information, but we still lack a proper understanding of knowledge encoded at the word level and beyond. In this work, we use lightweight analysis methods to study segment-level linguistic properties—word identity, boundaries, pronunciation, syntactic features, and semantic features—encoded in S3Ms. We present a comparative study of layer-wise representations from ten S3Ms and find that (i) the frame-level representations within each word segment are not all equally informative, and (ii) the pre-training objective and model size heavily influence the accessibility and distribution of linguistic information across layers. We also find that on several tasks—word discrimination, word segmentation, and semantic sentence similarity—S3Ms trained with visual grounding outperform their speech-only counterparts. Finally, our task-based analyses demonstrate improved performance on word segmentation and acoustic word discrimination while using simpler methods than prior work.1
%R 10.1162/tacl_a_00656
%U https://aclanthology.org/2024.tacl-1.21
%U https://doi.org/10.1162/tacl_a_00656
%P 372-391

Markdown (Informal)

[What Do Self-Supervised Speech Models Know About Words?](https://aclanthology.org/2024.tacl-1.21) (Pasad et al., TACL 2024)

• What Do Self-Supervised Speech Models Know About Words? (Pasad et al., TACL 2024)

ACL

• Ankita Pasad, Chung-Ming Chien, Shane Settle, and Karen Livescu. 2024. What Do Self-Supervised Speech Models Know About Words?. Transactions of the Association for Computational Linguistics, 12:372–391.