Uncertainty-Aware Hybrid Retrieval for Long-Document RAG

Hoin Jung, Xiaoqian Wang


Abstract
Retrieval-augmented generation (RAG) depends critically on the quality and granularity of retrieved evidence. Large retrieval units preserve context but often introduce irrelevant content, which can dilute answer-bearing evidence and worsen long context utilization. Fine-grained units are more compact, but they may be difficult to retrieve reliably because short chunks can lack semantic, lexical, or bridging cues needed to match the query. We propose **Uncertainty-aware Multi-Granularity RAG (UMG-RAG)**, a training-free hybrid retrieval framework that adapts retrieval across chunk granularities based on query-specific reliability estimates. Instead of training a new retriever or modifying the generator, UMG-RAG uses existing dense and sparse retrievers as complementary experts across multiple chunk granularities. For each query, it converts each expert-granularity score list into an evidence distribution, estimates reliability from distribution entropy, and fuses candidates according to query-specific semantic, lexical, and granularity confidence. We further introduce **UMGP-RAG**, a parent promotion variant that uses fine-grained hits to locate relevant evidence while returning broader non-redundant parent chunks for local coherence. Experiments on question answering benchmarks show that uncertainty-aware fusion and parent promotion improve generation quality in a training-free, plug-and-play retrieval pipeline.
Anthology ID:
2026.emnlp-main.2104
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
47893–47911
Language:
URL:
https://aclanthology.org/2026.emnlp-main.2104/
DOI:
Bibkey:
Cite (ACL):
Hoin Jung and Xiaoqian Wang. 2026. Uncertainty-Aware Hybrid Retrieval for Long-Document RAG. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 47893–47911, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Uncertainty-Aware Hybrid Retrieval for Long-Document RAG (Jung & Wang, EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-main.2104.pdf