SynapseHSG: One-Shot Hyper-Semantic Grounding for Long-Context Multimodal Question Answering

Sherricane Jiayu Xu, Zhijie Shen, Yongqiang Liu, Xiaojun Jing, Shuangyin Li


Abstract
Answering complex queries over long-context multimodal streams requires dynamically synthesizing asynchronous and heterogeneous evidence across extended temporal and logical spans. Existing approaches often rely on iterative Multimodal Retrieval-Augmented Generation (RAG), facing structural challenges from semantic and modality gaps as well as context bloat induced by multi-step retrieval. To address these challenges, we propose SynapseHSG, a decoupled one-shot grounding framework for long-context multimodal Question Answering. SynapseHSG introduces a hyper-semantic evidence organization method (named M2HSG) that constructs an offline graph index to align cross-modal evidence. During inference, we design a Semantic Decomposer to convert natural-language queries into graph-operable constraints. Guided by these constraints, SynapseHSG retrieves a compact query-anchored subgraph in one shot and linearizes it as structured evidence for a frozen generator. This avoids repeated LLM-driven retrieval while restricting trainable components to graph construction and query parsing. To support systematic evaluation, we introduce MM-ConfQA, a new long-context multimodal QA benchmark built from academic conference presentations. Experimental results on MM-ConfQA show that SynapseHSG improves grounded answer generation over strong multimodal and retrieval baselines, especially on dispersed-evidence questions, highlighting the value of pre-organized multimodal evidence over repeated search in raw heterogeneous contexts. We release the code and data to public.
Anthology ID:
2026.findings-emnlp.206
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
4326–4344
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.206/
DOI:
Bibkey:
Cite (ACL):
Sherricane Jiayu Xu, Zhijie Shen, Yongqiang Liu, Xiaojun Jing, and Shuangyin Li. 2026. SynapseHSG: One-Shot Hyper-Semantic Grounding for Long-Context Multimodal Question Answering. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 4326–4344, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
SynapseHSG: One-Shot Hyper-Semantic Grounding for Long-Context Multimodal Question Answering (Xu et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.206.pdf