Tracing False Speaker Links Through Retrieval and Answering

Dev D. Goyal


Abstract
Memory systems for large language model (LLM) agents can organize past conversations by person, for example in a speaker index or a knowledge graph. To answer “Who promised the budget?”, such a system searches the relevant people’s messages and passes a few to the LLM. Building the index requires entity resolution, which can wrongly link two people. A false link can change what the LLM reads and make it name the wrong person, but final-answer scores may not show this. We add false links at known rates to a retrieval pipeline, keep every message and name unchanged, and follow 280 questions from two synthetic benchmarks from search to answer. With one false link per 40 people, almost one in four workplace searches reaches a wrong person. At the highest rate, gpt-oss-120b reads a different context in 81.0% of workplace runs, but its accuracy falls by 1.75 percentage points, because improved and worsened answers offset each other. For gpt-oss on the social benchmark, one question that loses its only labeled supporting message accounts for 79% of history-weighted loss. Accuracy alone can therefore hide identity errors. We provide a tool and a memory error card that measure each step.
Anthology ID:
2026.nora-1.10
Volume:
Proceedings of the 3rd Workshop on KNOwledge GRaphs Agentic Systems Interplay (NORA 2026)
Month:
November
Year:
2026
Address:
Hengqin, China
Editors:
Btissam Er-Rahmadi, Sébastien Montella, Damien Graux, André Melo, Hajira Jabeen
Venues:
NORA | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
109–130
Language:
URL:
https://aclanthology.org/2026.nora-1.10/
DOI:
Bibkey:
Cite (ACL):
Dev D. Goyal. 2026. Tracing False Speaker Links Through Retrieval and Answering. In Proceedings of the 3rd Workshop on KNOwledge GRaphs Agentic Systems Interplay (NORA 2026), pages 109–130, Hengqin, China. Association for Computational Linguistics.
Cite (Informal):
Tracing False Speaker Links Through Retrieval and Answering (Goyal, NORA 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.nora-1.10.pdf
Supplementary material:
 2026.nora-1.10.supplementary_material.pdf
Source files:
 2026.nora-1.10.source_files.zip