<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="armeni-etal-2024-transformer">
    <titleInfo>
        <title>Transformer verbatim in-context retrieval across time and scale</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Kristijan</namePart>
        <namePart type="family">Armeni</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marko</namePart>
        <namePart type="family">Pranjić</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Senja</namePart>
        <namePart type="family">Pollak</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 28th Conference on Computational Natural Language Learning</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Libby</namePart>
            <namePart type="family">Barak</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Malihe</namePart>
            <namePart type="family">Alikhani</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, FL, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>To predict upcoming text, language models must in some cases retrieve in-context information verbatim. In this report, we investigated how the ability of language models to retrieve arbitrary in-context nouns developed during training (across time) and as language models trained on the same dataset increase in size (across scale). We then asked whether learning of in-context retrieval correlates with learning of more challenging zero-shot benchmarks. Furthermore, inspired by semantic effects in human short-term memory, we evaluated the retrieval with respect to a major semantic component of target nouns, namely whether they denote a concrete or abstract entity, as rated by humans. We show that verbatim in-context retrieval developed in a sudden transition early in the training process, after about 1% of the training tokens. This was observed across model sizes (from 14M and up to 12B parameters), and the transition occurred slightly later for the two smallest models. We further found that the development of verbatim in-context retrieval is positively correlated with the learning of zero-shot benchmarks. Around the transition point, all models showed the advantage of retrieving concrete nouns as opposed to abstract nouns. In all but two smallest models, the advantage dissipated away toward the end of training.</abstract>
    <identifier type="citekey">armeni-etal-2024-transformer</identifier>
    <location>
        <url>https://aclanthology.org/2024.conll-1.6</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>56</start>
            <end>68</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Transformer verbatim in-context retrieval across time and scale
%A Armeni, Kristijan
%A Pranjić, Marko
%A Pollak, Senja
%Y Barak, Libby
%Y Alikhani, Malihe
%S Proceedings of the 28th Conference on Computational Natural Language Learning
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, FL, USA
%F armeni-etal-2024-transformer
%X To predict upcoming text, language models must in some cases retrieve in-context information verbatim. In this report, we investigated how the ability of language models to retrieve arbitrary in-context nouns developed during training (across time) and as language models trained on the same dataset increase in size (across scale). We then asked whether learning of in-context retrieval correlates with learning of more challenging zero-shot benchmarks. Furthermore, inspired by semantic effects in human short-term memory, we evaluated the retrieval with respect to a major semantic component of target nouns, namely whether they denote a concrete or abstract entity, as rated by humans. We show that verbatim in-context retrieval developed in a sudden transition early in the training process, after about 1% of the training tokens. This was observed across model sizes (from 14M and up to 12B parameters), and the transition occurred slightly later for the two smallest models. We further found that the development of verbatim in-context retrieval is positively correlated with the learning of zero-shot benchmarks. Around the transition point, all models showed the advantage of retrieving concrete nouns as opposed to abstract nouns. In all but two smallest models, the advantage dissipated away toward the end of training.
%U https://aclanthology.org/2024.conll-1.6
%P 56-68

Markdown (Informal)

[Transformer verbatim in-context retrieval across time and scale](https://aclanthology.org/2024.conll-1.6) (Armeni et al., CoNLL 2024)

• Transformer verbatim in-context retrieval across time and scale (Armeni et al., CoNLL 2024)

ACL

• Kristijan Armeni, Marko Pranjić, and Senja Pollak. 2024. Transformer verbatim in-context retrieval across time and scale. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 56–68, Miami, FL, USA. Association for Computational Linguistics.