<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="huang-hollenstein-2023-long">
    <titleInfo>
        <title>Long-Range Language Modeling with Selective Cache</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Xinting</namePart>
        <namePart type="family">Huang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nora</namePart>
        <namePart type="family">Hollenstein</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The computational cost of transformer-based language models grows quadratically with the sequence length. In this paper, we introduce the selective cache, which stores the selected key-value pairs from the previous context. By selecting important key-value pairs the model makes better use of the cache so that in limited cache size, a longer context history can be stored. We design three kinds of selection methods. The first is based on human language processing. The key-value pairs are selected if they correspond to tokens that are fixated longer, as recorded in eye-tracking-while-reading experiments. We also incorporate the cognitively-inspired selection process into the language model as a trainable process, resulting in two additional methods with improved performance. The selection task is converted into a pruning task so they can be trained with differentiable masks. We demonstrate that the proposed selective cache improves the language modeling performance across different datasets. With the same number of stored key-value pairs (cache size), our selective cache outperforms XL cache and compressive cache by considerable margins.</abstract>
    <identifier type="citekey">huang-hollenstein-2023-long</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.321</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.321</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>4838</start>
            <end>4858</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Long-Range Language Modeling with Selective Cache
%A Huang, Xinting
%A Hollenstein, Nora
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F huang-hollenstein-2023-long
%X The computational cost of transformer-based language models grows quadratically with the sequence length. In this paper, we introduce the selective cache, which stores the selected key-value pairs from the previous context. By selecting important key-value pairs the model makes better use of the cache so that in limited cache size, a longer context history can be stored. We design three kinds of selection methods. The first is based on human language processing. The key-value pairs are selected if they correspond to tokens that are fixated longer, as recorded in eye-tracking-while-reading experiments. We also incorporate the cognitively-inspired selection process into the language model as a trainable process, resulting in two additional methods with improved performance. The selection task is converted into a pruning task so they can be trained with differentiable masks. We demonstrate that the proposed selective cache improves the language modeling performance across different datasets. With the same number of stored key-value pairs (cache size), our selective cache outperforms XL cache and compressive cache by considerable margins.
%R 10.18653/v1/2023.findings-emnlp.321
%U https://aclanthology.org/2023.findings-emnlp.321
%U https://doi.org/10.18653/v1/2023.findings-emnlp.321
%P 4838-4858

Markdown (Informal)

[Long-Range Language Modeling with Selective Cache](https://aclanthology.org/2023.findings-emnlp.321) (Huang & Hollenstein, Findings 2023)

• Long-Range Language Modeling with Selective Cache (Huang & Hollenstein, Findings 2023)

ACL

• Xinting Huang and Nora Hollenstein. 2023. Long-Range Language Modeling with Selective Cache. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4838–4858, Singapore. Association for Computational Linguistics.