Calibrating Hierarchical Memory Utilization for Long-Term Memory QA

Hang Lv, Xuyang Zhi, Hongchao Gu, Hao Wang


Abstract
Long-term memory systems increasingly organize interaction histories into hierarchical contexts, where coarse summaries, mid-level events, and fine-grained evidence provide complementary signals for future queries. Yet after retrieval, these signals are often collapsed into a single generation context and used implicitly. We ask how retrieved hierarchical memory should shape generation after access, framing it as a post-retrieval calibration problem. We propose HiR, a training-free inference-time framework that contrasts full-hierarchy decoding with a coarse-only reference view, preserving the complete hierarchy while tempering over-reliance on broad abstractions. Extensive experiments and analyses show that HiR enables more effective and robust use of hierarchical memory without changing memory construction or retrieval; more broadly, our results suggest that memory research should consider not only what to store or retrieve, but also how retrieved information shapes generation.
Anthology ID:
2026.findings-emnlp.1927
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
41072–41081
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.1927/
DOI:
Bibkey:
Cite (ACL):
Hang Lv, Xuyang Zhi, Hongchao Gu, and Hao Wang. 2026. Calibrating Hierarchical Memory Utilization for Long-Term Memory QA. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 41072–41081, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Calibrating Hierarchical Memory Utilization for Long-Term Memory QA (Lv et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.1927.pdf