Response-Aware User Memory Selection for LLM Personalization

Jillian Fisher, Jennifer Neville, Chan Young Park


Abstract
A common approach to personalization in large language models (LLMs) is to incorporate a subset of the user memory into the prompt at inference time to guide the model’s generation. Existing methods select these subsets primarily using similarity between user memory items and input queries, ignoring how features actually affect the model’s response distribution. We propose **R**esponse-**U**tility optimization for **M**emory **S**election (RUMS), a novel method that selects user memory items by measuring the mutual information between a subset of memory and the model’s outputs, identifying items that reduce response uncertainty and sharpen predictions beyond semantic similarity. We demonstrate that this information-theoretic foundation enables more principled user memory selection that aligns more closely with human selection compared to state-of-the-art methods, and models that are 400 × larger. Additionally, we show that memory items selected using RUMS result in better response quality compared to existing approaches, while having up to 95% reduction in computational cost.
Anthology ID:
2026.emnlp-main.185
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
4215–4267
Language:
URL:
https://aclanthology.org/2026.emnlp-main.185/
DOI:
Bibkey:
Cite (ACL):
Jillian Fisher, Jennifer Neville, and Chan Young Park. 2026. Response-Aware User Memory Selection for LLM Personalization. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 4215–4267, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Response-Aware User Memory Selection for LLM Personalization (Fisher et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.185.pdf