<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="he-etal-2026-value">
    <titleInfo>
        <title>Do Value Vectors in Deep Layers Need Context from the Residual Stream?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Muyu</namePart>
        <namePart type="family">He</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yuchen</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Qingya</namePart>
        <namePart type="family">Huang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Li</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>The success of the transformer architecture is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn a context-free value vector that preserves the original token information, without drawing on any context information from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component of the value vector provides no gain. Such context-free value vectors can be stored as sparse model parameters, thereby eliminating the need for recomputation or caching. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. BoV consistently improves upon the standard attention mechanism in both validation loss and downstream benchmark performance across two model sizes, and surpasses competing attention architectures while using a fraction of the compute and memory.</abstract>
    <identifier type="citekey">he-etal-2026-value</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.1784/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>40637</start>
            <end>40650</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Do Value Vectors in Deep Layers Need Context from the Residual Stream?
%A He, Muyu
%A Liu, Yuchen
%A Huang, Qingya
%A Zhang, Li
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F he-etal-2026-value
%X The success of the transformer architecture is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn a context-free value vector that preserves the original token information, without drawing on any context information from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component of the value vector provides no gain. Such context-free value vectors can be stored as sparse model parameters, thereby eliminating the need for recomputation or caching. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. BoV consistently improves upon the standard attention mechanism in both validation loss and downstream benchmark performance across two model sizes, and surpasses competing attention architectures while using a fraction of the compute and memory.
%U https://aclanthology.org/2026.emnlp-main.1784/
%P 40637-40650

Markdown (Informal)

[Do Value Vectors in Deep Layers Need Context from the Residual Stream?](https://aclanthology.org/2026.emnlp-main.1784/) (He et al., EMNLP 2026)

• Do Value Vectors in Deep Layers Need Context from the Residual Stream? (He et al., EMNLP 2026)

ACL

• Muyu He, Yuchen Liu, Qingya Huang, and Li Zhang. 2026. Do Value Vectors in Deep Layers Need Context from the Residual Stream?. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 40637–40650, Budapest, Hungary. Association for Computational Linguistics.