<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="joshi-etal-2026-cap">
    <titleInfo>
        <title>Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Aviral</namePart>
        <namePart type="family">Joshi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hanoz</namePart>
        <namePart type="family">Bhathena</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Max</namePart>
        <namePart type="family">Nelson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Saket</namePart>
        <namePart type="family">Sharma</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Pepa</namePart>
            <namePart type="family">Atanasova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gholamreza</namePart>
            <namePart type="family">Haffari</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Minghan</namePart>
            <namePart type="family">Wang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-460-6</identifier>
    </relatedItem>
    <abstract>Retrieval-augmented generation (RAG) is hard to monitor in production: exhaustive relevance labels do not exist for non-stationary multi-million-passage corpora that re-index in real time. As a result, retrieval quality is generally understudied and often deprioritised in favour of generation-oriented metrics. In this work, we propose auditing retrieval coverage by probing for evidence of missing documents rather than enumerating every relevant one. Our method Re:CAP (REtrieval Coverage Audit by iterative Probing) is a reference-free audit loop applied to a deployed RAG pipeline’s initial answer and retrieved context: it identifies the topics already covered, generates probing questions for plausibly missing topics, retrieves candidate documents, and applies an LLM-as-judge to retain only those that introduce previously-unretrieved information. On four public benchmarks, Re:CAP recovers 9–29% of gold labels that flat BM25 top-500 cannot reach, rising to 48% on TREC-COVID. On MuSiQue Re:CAP beats flat hybrid top-500 by +12.9 pp on recall at less than half the document budget. An ensemble BM25, dense, and hybrid baseline (top-500 each) still leaves out 21.2% of gold docs on TREC-COVID that Re:CAP recovers; human annotators judge that 78.9% of those structurally distinct documents add new information to the baseline answer (Fleiss \ensuremathąppa = 0.79, n = 123), and 73.9% on live production traffic (n = 180). End-to-end recall is reproducible to within \ensuremath\pm1% across three independent runs, making Re:CAP a stable instrument for periodic retrieval audits.</abstract>
    <identifier type="citekey">joshi-etal-2026-cap</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-industry.150/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>2532</start>
            <end>2565</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines
%A Joshi, Aviral
%A Bhathena, Hanoz
%A Nelson, Max
%A Sharma, Saket
%Y Atanasova, Pepa
%Y Haffari, Gholamreza
%Y Wang, Minghan
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-460-6
%F joshi-etal-2026-cap
%X Retrieval-augmented generation (RAG) is hard to monitor in production: exhaustive relevance labels do not exist for non-stationary multi-million-passage corpora that re-index in real time. As a result, retrieval quality is generally understudied and often deprioritised in favour of generation-oriented metrics. In this work, we propose auditing retrieval coverage by probing for evidence of missing documents rather than enumerating every relevant one. Our method Re:CAP (REtrieval Coverage Audit by iterative Probing) is a reference-free audit loop applied to a deployed RAG pipeline’s initial answer and retrieved context: it identifies the topics already covered, generates probing questions for plausibly missing topics, retrieves candidate documents, and applies an LLM-as-judge to retain only those that introduce previously-unretrieved information. On four public benchmarks, Re:CAP recovers 9–29% of gold labels that flat BM25 top-500 cannot reach, rising to 48% on TREC-COVID. On MuSiQue Re:CAP beats flat hybrid top-500 by +12.9 pp on recall at less than half the document budget. An ensemble BM25, dense, and hybrid baseline (top-500 each) still leaves out 21.2% of gold docs on TREC-COVID that Re:CAP recovers; human annotators judge that 78.9% of those structurally distinct documents add new information to the baseline answer (Fleiss \ensuremathąppa = 0.79, n = 123), and 73.9% on live production traffic (n = 180). End-to-end recall is reproducible to within \ensuremath\pm1% across three independent runs, making Re:CAP a stable instrument for periodic retrieval audits.
%U https://aclanthology.org/2026.emnlp-industry.150/
%P 2532-2565

Markdown (Informal)

[Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines](https://aclanthology.org/2026.emnlp-industry.150/) (Joshi et al., EMNLP 2026)

• Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines (Joshi et al., EMNLP 2026)

ACL

• Aviral Joshi, Hanoz Bhathena, Max Nelson, and Saket Sharma. 2026. Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 2532–2565, Budapest, Hungary. Association for Computational Linguistics.