<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="corallo-papotti-2024-finch">
    <titleInfo>
        <title>FINCH: Prompt-guided Key-Value Cache Compression for Large Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Giulio</namePart>
        <namePart type="family">Corallo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Paolo</namePart>
        <namePart type="family">Papotti</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Recent large language model applications, such as Retrieval-Augmented Generation and chatbots, have led to an increased need to process longer input contexts. However, this requirement is hampered by inherent limitations. Architecturally, models are constrained by a context window defined during training. Additionally, processing extensive texts requires substantial GPU memory. We propose a novel approach, Finch, to compress the input context by leveraging the pre-trained model weights of the self-attention. Given a prompt and a long text, Finch iteratively identifies the most relevant Key (K) and Value (V) pairs over chunks of the text conditioned on the prompt. Only such pairs are stored in the KV cache, which, within the space constrained by the context window, ultimately contains a compressed version of the long text. Our proposal enables models to consume large inputs even with high compression (up to 93x) while preserving semantic integrity without the need for fine-tuning.</abstract>
    <identifier type="citekey">corallo-papotti-2024-finch</identifier>
    <identifier type="doi">10.1162/tacl_a_00716</identifier>
    <location>
        <url>https://aclanthology.org/2024.tacl-1.83/</url>
    </location>
    <part>
        <date>2024</date>
        <detail type="volume"><number>12</number></detail>
        <extent unit="page">
            <start>1517</start>
            <end>1532</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T FINCH: Prompt-guided Key-Value Cache Compression for Large Language Models
%A Corallo, Giulio
%A Papotti, Paolo
%J Transactions of the Association for Computational Linguistics
%D 2024
%V 12
%I MIT Press
%C Cambridge, MA
%F corallo-papotti-2024-finch
%X Recent large language model applications, such as Retrieval-Augmented Generation and chatbots, have led to an increased need to process longer input contexts. However, this requirement is hampered by inherent limitations. Architecturally, models are constrained by a context window defined during training. Additionally, processing extensive texts requires substantial GPU memory. We propose a novel approach, Finch, to compress the input context by leveraging the pre-trained model weights of the self-attention. Given a prompt and a long text, Finch iteratively identifies the most relevant Key (K) and Value (V) pairs over chunks of the text conditioned on the prompt. Only such pairs are stored in the KV cache, which, within the space constrained by the context window, ultimately contains a compressed version of the long text. Our proposal enables models to consume large inputs even with high compression (up to 93x) while preserving semantic integrity without the need for fine-tuning.
%R 10.1162/tacl_a_00716
%U https://aclanthology.org/2024.tacl-1.83/
%U https://doi.org/10.1162/tacl_a_00716
%P 1517-1532

Markdown (Informal)

[FINCH: Prompt-guided Key-Value Cache Compression for Large Language Models](https://aclanthology.org/2024.tacl-1.83/) (Corallo & Papotti, TACL 2024)

• FINCH: Prompt-guided Key-Value Cache Compression for Large Language Models (Corallo & Papotti, TACL 2024)

ACL

• Giulio Corallo and Paolo Papotti. 2024. FINCH: Prompt-guided Key-Value Cache Compression for Large Language Models. Transactions of the Association for Computational Linguistics, 12:1517–1532.