<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sinclair-etal-2022-structural">
    <titleInfo>
        <title>Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Arabella</namePart>
        <namePart type="family">Sinclair</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jaap</namePart>
        <namePart type="family">Jumelet</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Willem</namePart>
        <namePart type="family">Zuidema</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Raquel</namePart>
        <namePart type="family">Fernández</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>We investigate the extent to which modern neural language models are susceptible to structural priming, the phenomenon whereby the structure of a sentence makes the same structure more probable in a follow-up sentence. We explore how priming can be used to study the potential of these models to learn abstract structural information, which is a prerequisite for good performance on tasks that require natural language understanding skills. We introduce a novel metric and release Prime-LM, a large corpus where we control for various linguistic factors that interact with priming strength. We find that Transformer models indeed show evidence of structural priming, but also that the generalizations they learned are to some extent modulated by semantic information. Our experiments also show that the representations acquired by the models may not only encode abstract sequential structure but involve certain level of hierarchical syntactic information. More generally, our study shows that the priming paradigm is a useful, additional tool for gaining insights into the capacities of language models and opens the door to future priming-based investigations that probe the model’s internal states.1</abstract>
    <identifier type="citekey">sinclair-etal-2022-structural</identifier>
    <identifier type="doi">10.1162/tacl_a_00504</identifier>
    <location>
        <url>https://aclanthology.org/2022.tacl-1.60</url>
    </location>
    <part>
        <date>2022</date>
        <detail type="volume"><number>10</number></detail>
        <extent unit="page">
            <start>1031</start>
            <end>1050</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations
%A Sinclair, Arabella
%A Jumelet, Jaap
%A Zuidema, Willem
%A Fernández, Raquel
%J Transactions of the Association for Computational Linguistics
%D 2022
%V 10
%I MIT Press
%C Cambridge, MA
%F sinclair-etal-2022-structural
%X We investigate the extent to which modern neural language models are susceptible to structural priming, the phenomenon whereby the structure of a sentence makes the same structure more probable in a follow-up sentence. We explore how priming can be used to study the potential of these models to learn abstract structural information, which is a prerequisite for good performance on tasks that require natural language understanding skills. We introduce a novel metric and release Prime-LM, a large corpus where we control for various linguistic factors that interact with priming strength. We find that Transformer models indeed show evidence of structural priming, but also that the generalizations they learned are to some extent modulated by semantic information. Our experiments also show that the representations acquired by the models may not only encode abstract sequential structure but involve certain level of hierarchical syntactic information. More generally, our study shows that the priming paradigm is a useful, additional tool for gaining insights into the capacities of language models and opens the door to future priming-based investigations that probe the model’s internal states.1
%R 10.1162/tacl_a_00504
%U https://aclanthology.org/2022.tacl-1.60
%U https://doi.org/10.1162/tacl_a_00504
%P 1031-1050

Markdown (Informal)

[Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations](https://aclanthology.org/2022.tacl-1.60) (Sinclair et al., TACL 2022)

• Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations (Sinclair et al., TACL 2022)

ACL

• Arabella Sinclair, Jaap Jumelet, Willem Zuidema, and Raquel Fernández. 2022. Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations. Transactions of the Association for Computational Linguistics, 10:1031–1050.