<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="timkey-linzen-2023-language">
    <titleInfo>
        <title>A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">William</namePart>
        <namePart type="family">Timkey</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tal</namePart>
        <namePart type="family">Linzen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Two of the central factors believed to underpin human sentence processing difficulty are expectations and retrieval from working memory. A recent attempt to create a unified cognitive model integrating these two factors have relied on the parallels between the self-attention mechanism of transformer language models and cue-based retrieval theories of working memory in human sentence processing (Ryu and Lewis 2021). While the authors show that attention patterns in specialized attention heads of GPT-2 are consistent with a key prediction of cue-based retrieval models, similarity-based interference effects, their method requires the identification of syntactically specialized attention heads, and makes an cognitively implausible implicit assumption that hundreds of memory retrieval operations take place in parallel. In the present work, we develop a recurrent neural language model with a single self-attention head, which more closely parallels the memory system assumed by cognitive theories. We show that our model’s single attention head can capture semantic and syntactic interference effects observed in human experiments.</abstract>
    <identifier type="citekey">timkey-linzen-2023-language</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.582</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.582</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>8705</start>
            <end>8720</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing
%A Timkey, William
%A Linzen, Tal
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F timkey-linzen-2023-language
%X Two of the central factors believed to underpin human sentence processing difficulty are expectations and retrieval from working memory. A recent attempt to create a unified cognitive model integrating these two factors have relied on the parallels between the self-attention mechanism of transformer language models and cue-based retrieval theories of working memory in human sentence processing (Ryu and Lewis 2021). While the authors show that attention patterns in specialized attention heads of GPT-2 are consistent with a key prediction of cue-based retrieval models, similarity-based interference effects, their method requires the identification of syntactically specialized attention heads, and makes an cognitively implausible implicit assumption that hundreds of memory retrieval operations take place in parallel. In the present work, we develop a recurrent neural language model with a single self-attention head, which more closely parallels the memory system assumed by cognitive theories. We show that our model’s single attention head can capture semantic and syntactic interference effects observed in human experiments.
%R 10.18653/v1/2023.findings-emnlp.582
%U https://aclanthology.org/2023.findings-emnlp.582
%U https://doi.org/10.18653/v1/2023.findings-emnlp.582
%P 8705-8720

Markdown (Informal)

[A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing](https://aclanthology.org/2023.findings-emnlp.582) (Timkey & Linzen, Findings 2023)

• A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing (Timkey & Linzen, Findings 2023)

ACL

• William Timkey and Tal Linzen. 2023. A Language Model with Limited Memory Capacity Captures Interference in Human Sentence Processing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8705–8720, Singapore. Association for Computational Linguistics.