The success of the transformer architecture is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layers learn a context-free value vector that preserves the original token information, without drawing on any context information from the residual stream. When the model has access to this context-free value vector, adding back the context-dependent component of the value vector provides no gain. Such context-free value vectors can be stored as sparse model parameters, thereby eliminating the need for recomputation or caching. Through systematic ablations on the key design choices for such context-free value vectors, we propose Bank of Values (BoV), a new way of computing value vectors in attention by learning a lookup table of token-specific value vectors for each of the last third of layers. BoV consistently improves upon the standard attention mechanism in both validation loss and downstream benchmark performance across two model sizes, and surpasses competing attention architectures while using a fraction of the compute and memory.