<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sumida-etal-2026-memuse">
    <titleInfo>
        <title>MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ryuichi</namePart>
        <namePart type="family">Sumida</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Koji</namePart>
        <namePart type="family">Inoue</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tatsuya</namePart>
        <namePart type="family">Kawahara</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>Memory systems for conversational LLMs are conventionally evaluated by direct, fact-seeking questions about prior dialogue (Direct QA): can the model recall fact X from a prior conversation? We tested whether higher Direct QA accuracy correlates with higher user satisfaction in a four-month deployment involving 40 users, 1,872 sessions, and seven memory conditions. Existing-benchmark Direct QA accuracy varies from 19.7% to 70.1% across the seven conditions, but satisfaction does not change. We hypothesize that existing benchmarks and user satisfaction measure different capabilities: benchmarks measure elicited retrieval—recall when explicitly asked—whereas natural conversation requires detecting relevance and appropriately integrating prior context into a response. To examine this distinction, we introduce MemUse, a benchmark of real user-cued memory moments drawn from the deployment and evaluated using integration-aware judgments of natural conversational responses. Holding the model and context fixed, the same system that scores 78.8% on Direct QA references only 7.9% of those facts in conversation, revealing a 71-point retrieval–integration gap. Within a small observational subset of naturally occurring memory moments, successful Natural Integration is positively associated with user satisfaction, whereas Direct QA performance is not. We release the deployment corpus and MemUse together with the associated judgments and scoring prompts.</abstract>
    <identifier type="citekey">sumida-etal-2026-memuse</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.804/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>18719</start>
            <end>18749</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation
%A Sumida, Ryuichi
%A Inoue, Koji
%A Kawahara, Tatsuya
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F sumida-etal-2026-memuse
%X Memory systems for conversational LLMs are conventionally evaluated by direct, fact-seeking questions about prior dialogue (Direct QA): can the model recall fact X from a prior conversation? We tested whether higher Direct QA accuracy correlates with higher user satisfaction in a four-month deployment involving 40 users, 1,872 sessions, and seven memory conditions. Existing-benchmark Direct QA accuracy varies from 19.7% to 70.1% across the seven conditions, but satisfaction does not change. We hypothesize that existing benchmarks and user satisfaction measure different capabilities: benchmarks measure elicited retrieval—recall when explicitly asked—whereas natural conversation requires detecting relevance and appropriately integrating prior context into a response. To examine this distinction, we introduce MemUse, a benchmark of real user-cued memory moments drawn from the deployment and evaluated using integration-aware judgments of natural conversational responses. Holding the model and context fixed, the same system that scores 78.8% on Direct QA references only 7.9% of those facts in conversation, revealing a 71-point retrieval–integration gap. Within a small observational subset of naturally occurring memory moments, successful Natural Integration is positively associated with user satisfaction, whereas Direct QA performance is not. We release the deployment corpus and MemUse together with the associated judgments and scoring prompts.
%U https://aclanthology.org/2026.emnlp-main.804/
%P 18719-18749

Markdown (Informal)

[MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation](https://aclanthology.org/2026.emnlp-main.804/) (Sumida et al., EMNLP 2026)

• MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation (Sumida et al., EMNLP 2026)

ACL

• Ryuichi Sumida, Koji Inoue, and Tatsuya Kawahara. 2026. MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 18719–18749, Budapest, Hungary. Association for Computational Linguistics.