Dongju Seo
Author directory2026
When Retrieval Similarity Hurts: A Diagnostic Framework for Document Weighting in Black-Box RAG
Dongju Seo | Beakcheol Jang
Findings of the Association for Computational Linguistics: EMNLP 2026
Dongju Seo | Beakcheol Jang
Findings of the Association for Computational Linguistics: EMNLP 2026
Document weighting is an underspecified bottleneck in black-box retrieval-augmented generation. Training-free systems inherit answer-aggregation weights from retrieval similarity, yet this proxy has not been systematically compared with evidence-based alternatives. We propose a unified two-parameter framework that generalizes REPLUG with an evidence term; it is training-free, black-box, and needs no large language model (LLM) calls beyond voting. Similarity-only (λ=0) and evidence-only (λ→∞) weighting are its limiting cases, so one interpolation parameter separates the retrieval prior from the evidence signal. We evaluate across 3 LLMs (7B–70B), 3 open-domain question-answering datasets (31,861 queries), and 3 off-the-shelf evidence types. Cross-encoder and embedding-stability evidence improve exact match in all 9 LLM–dataset cells, whereas natural language inference (NLI) fails as a weighting signal, to our knowledge the first systematic evidence of this failure mode. The similarity prior is largely redundant on NQ-Open and PopQA but helps on TriviaQA, where retriever scores track gold documents better; the pattern replicates under the E5 retriever, with smaller gains concentrated on long-tail PopQA. One default transfers across datasets and scales with minimal loss; calibration depends on signal type, not scale. Practically: prefer cross-encoder evidence, avoid NLI, and use vote mass for selective prediction; weighting itself adds sub-millisecond overhead.