<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="wang-etal-2026-e2e">
    <titleInfo>
        <title>E2E-VisRAG: End-to-End Visual Retrieval-Augmented Generation with Differentiable Retrieval</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Zehua</namePart>
        <namePart type="family">Wang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Zhaojin</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xiaolong</namePart>
        <namePart type="family">Weng</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yanzhi</namePart>
        <namePart type="family">Chen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ying</namePart>
        <namePart type="family">Xiong</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Buzhou</namePart>
        <namePart type="family">Tang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>Retrieval-Augmented Generation (RAG) grounds large language models (LLMs) in external evidence, but existing RAG systems remain poorly matched to visually rich documents. OCR-based methods discard layout and non-textual visual cues, and although vision-language models (VLMs) can better understand native visual information, existing visual retrieval approaches often compress entire pages into global embedding vectors, obscuring fine-grained spatial information such as table cells and local annotations. Meanwhile, discrete Top-K selection in cascaded architectures blocks gradient flow between retrieval and generation, preventing joint optimization.In this work, we propose E2E-VisRAG, an end-to-end visual RAG framework that operates directly on native visual patches. Building upon patch-level visual encoding, we introduce a differentiable Top-K selection mechanism to enable joint training of retrieval and generation, and employ a gated residual fusion module to inject retrieval-enhanced visual features while suppressing noise. This design abandons the traditional paradigms of relying on OCR, global embeddings, and retrieval–generation separation, maximizing the preservation of original document visual semantics. Experiments on Vidore and MMLongBench-Doc show that, under matched 3B generator capacity, E2E-VisRAG outperforms non-end-to-end visual retrieval baselines, improving both Recall@1 and Accuracy@1 over OCR-based, global-embedding, and patch-level retrieval methods.</abstract>
    <identifier type="citekey">wang-etal-2026-e2e</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.1030/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>23670</start>
            <end>23683</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T E2E-VisRAG: End-to-End Visual Retrieval-Augmented Generation with Differentiable Retrieval
%A Wang, Zehua
%A Zhang, Zhaojin
%A Weng, Xiaolong
%A Chen, Yanzhi
%A Xiong, Ying
%A Tang, Buzhou
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F wang-etal-2026-e2e
%X Retrieval-Augmented Generation (RAG) grounds large language models (LLMs) in external evidence, but existing RAG systems remain poorly matched to visually rich documents. OCR-based methods discard layout and non-textual visual cues, and although vision-language models (VLMs) can better understand native visual information, existing visual retrieval approaches often compress entire pages into global embedding vectors, obscuring fine-grained spatial information such as table cells and local annotations. Meanwhile, discrete Top-K selection in cascaded architectures blocks gradient flow between retrieval and generation, preventing joint optimization.In this work, we propose E2E-VisRAG, an end-to-end visual RAG framework that operates directly on native visual patches. Building upon patch-level visual encoding, we introduce a differentiable Top-K selection mechanism to enable joint training of retrieval and generation, and employ a gated residual fusion module to inject retrieval-enhanced visual features while suppressing noise. This design abandons the traditional paradigms of relying on OCR, global embeddings, and retrieval–generation separation, maximizing the preservation of original document visual semantics. Experiments on Vidore and MMLongBench-Doc show that, under matched 3B generator capacity, E2E-VisRAG outperforms non-end-to-end visual retrieval baselines, improving both Recall@1 and Accuracy@1 over OCR-based, global-embedding, and patch-level retrieval methods.
%U https://aclanthology.org/2026.emnlp-main.1030/
%P 23670-23683

Markdown (Informal)

[E2E-VisRAG: End-to-End Visual Retrieval-Augmented Generation with Differentiable Retrieval](https://aclanthology.org/2026.emnlp-main.1030/) (Wang et al., EMNLP 2026)

• E2E-VisRAG: End-to-End Visual Retrieval-Augmented Generation with Differentiable Retrieval (Wang et al., EMNLP 2026)

ACL

• Zehua Wang, Zhaojin Zhang, Xiaolong Weng, Yanzhi Chen, Ying Xiong, and Buzhou Tang. 2026. E2E-VisRAG: End-to-End Visual Retrieval-Augmented Generation with Differentiable Retrieval. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 23670–23683, Budapest, Hungary. Association for Computational Linguistics.