@inproceedings{su-etal-2026-s3mem,
title = "{S}3{M}em: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering",
author = "Su, Encheng and
Wu, Jianyu and
Zhang, Jinouwen and
Yu, Qiucheng and
Tang, Chen and
Li, Pengze and
Wang, Lintao and
Wang, Aoran and
Ma, Xinzhu and
Tang, Shixiang and
Wang, Yizhou and
Li, Houqiang",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.992/",
pages = "22835--22852",
ISBN = "979-8-89176-459-0",
abstract = "Long-horizon memory question answering often requires sparse evidence from heterogeneous histories, including events, object states, visual observations, temporal relations, and causal steps. Existing memory interfaces expand reader context, retrieve semantically related chunks, or expose graph neighborhoods, but they are not explicitly designed to select compact evidence for a fixed reader. We propose Structured Spatiotemporal Scene-Event Memory (S3Mem), a query-time memory interface that writes textual, visual, and agent-use histories into structured scene-event units and routes compact evidence packs to the reader. Its router scores candidate units, query anchors, and anchor-support links, enabling both single-hop selection and short multi-hop evidence chains without reader fine-tuning or test-time training. Across LoCoMo, EMemBench Visual Games, and AMA-Bench, S3Mem provides a strong score-token trade-off, with the clearest gains on localized event, state, temporal, causal, or provenance evidence. On LoCoMo, S3Mem reaches 0.48 F1 and 0.40 BLEU with 1,073 evidence tokens per question, about 15.8 times fewer than the LoCoMo reference. On EMemBench Visual Games, it obtains the best F1 and second-best accuracy with only 189 tokens. On AMA-Bench, it is not the highest-scoring method, but remains competitive while using the fewest reader-visible evidence tokens."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="su-etal-2026-s3mem">
<titleInfo>
<title>S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering</title>
</titleInfo>
<name type="personal">
<namePart type="given">Encheng</namePart>
<namePart type="family">Su</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jianyu</namePart>
<namePart type="family">Wu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jinouwen</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Qiucheng</namePart>
<namePart type="family">Yu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Chen</namePart>
<namePart type="family">Tang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Pengze</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Lintao</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Aoran</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Xinzhu</namePart>
<namePart type="family">Ma</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Shixiang</namePart>
<namePart type="family">Tang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yizhou</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Houqiang</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-459-0</identifier>
</relatedItem>
<abstract>Long-horizon memory question answering often requires sparse evidence from heterogeneous histories, including events, object states, visual observations, temporal relations, and causal steps. Existing memory interfaces expand reader context, retrieve semantically related chunks, or expose graph neighborhoods, but they are not explicitly designed to select compact evidence for a fixed reader. We propose Structured Spatiotemporal Scene-Event Memory (S3Mem), a query-time memory interface that writes textual, visual, and agent-use histories into structured scene-event units and routes compact evidence packs to the reader. Its router scores candidate units, query anchors, and anchor-support links, enabling both single-hop selection and short multi-hop evidence chains without reader fine-tuning or test-time training. Across LoCoMo, EMemBench Visual Games, and AMA-Bench, S3Mem provides a strong score-token trade-off, with the clearest gains on localized event, state, temporal, causal, or provenance evidence. On LoCoMo, S3Mem reaches 0.48 F1 and 0.40 BLEU with 1,073 evidence tokens per question, about 15.8 times fewer than the LoCoMo reference. On EMemBench Visual Games, it obtains the best F1 and second-best accuracy with only 189 tokens. On AMA-Bench, it is not the highest-scoring method, but remains competitive while using the fewest reader-visible evidence tokens.</abstract>
<identifier type="citekey">su-etal-2026-s3mem</identifier>
<location>
<url>https://aclanthology.org/2026.emnlp-main.992/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>22835</start>
<end>22852</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering
%A Su, Encheng
%A Wu, Jianyu
%A Zhang, Jinouwen
%A Yu, Qiucheng
%A Tang, Chen
%A Li, Pengze
%A Wang, Lintao
%A Wang, Aoran
%A Ma, Xinzhu
%A Tang, Shixiang
%A Wang, Yizhou
%A Li, Houqiang
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F su-etal-2026-s3mem
%X Long-horizon memory question answering often requires sparse evidence from heterogeneous histories, including events, object states, visual observations, temporal relations, and causal steps. Existing memory interfaces expand reader context, retrieve semantically related chunks, or expose graph neighborhoods, but they are not explicitly designed to select compact evidence for a fixed reader. We propose Structured Spatiotemporal Scene-Event Memory (S3Mem), a query-time memory interface that writes textual, visual, and agent-use histories into structured scene-event units and routes compact evidence packs to the reader. Its router scores candidate units, query anchors, and anchor-support links, enabling both single-hop selection and short multi-hop evidence chains without reader fine-tuning or test-time training. Across LoCoMo, EMemBench Visual Games, and AMA-Bench, S3Mem provides a strong score-token trade-off, with the clearest gains on localized event, state, temporal, causal, or provenance evidence. On LoCoMo, S3Mem reaches 0.48 F1 and 0.40 BLEU with 1,073 evidence tokens per question, about 15.8 times fewer than the LoCoMo reference. On EMemBench Visual Games, it obtains the best F1 and second-best accuracy with only 189 tokens. On AMA-Bench, it is not the highest-scoring method, but remains competitive while using the fewest reader-visible evidence tokens.
%U https://aclanthology.org/2026.emnlp-main.992/
%P 22835-22852
Markdown (Informal)
[S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering](https://aclanthology.org/2026.emnlp-main.992/) (Su et al., EMNLP 2026)
ACL
- Encheng Su, Jianyu Wu, Jinouwen Zhang, Qiucheng Yu, Chen Tang, Pengze Li, Lintao Wang, Aoran Wang, Xinzhu Ma, Shixiang Tang, Yizhou Wang, and Houqiang Li. 2026. S3Mem: Structured Spatiotemporal Scene-Event Memory for Long-Horizon Interactive Question Answering. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 22835–22852, Budapest, Hungary. Association for Computational Linguistics.