<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lee-etal-2025-finding">
    <titleInfo>
        <title>Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yohan</namePart>
        <namePart type="family">Lee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yongwoo</namePart>
        <namePart type="family">Song</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sangyeop</namePart>
        <namePart type="family">Kim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Saloni</namePart>
            <namePart type="family">Potdar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lina</namePart>
            <namePart type="family">Rojas-Barahona</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sebastien</namePart>
            <namePart type="family">Montella</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Suzhou (China)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-333-3</identifier>
    </relatedItem>
    <abstract>We present the Conversational Data Retrieval (CDR) benchmark, the first comprehensive test set for evaluating systems that retrieve conversation data for product insights. With 1.6k queries across five analytical tasks and 9.1k conversations, our benchmark provides a reliable standard for measuring conversational data retrieval performance. Our evaluation of 16 popular embedding models shows that even the best models reach only around NDCG@10 of 0.51, revealing a substantial gap between document and conversational data retrieval capabilities. Our work identifies unique challenges in conversational data retrieval (implicit state recognition, turn dynamics, contextual references) while providing practical query templates and detailed error analysis across different task categories. The benchmark dataset and code are available at https://github.com/l-yohai/CDR-Benchmark.</abstract>
    <identifier type="citekey">lee-etal-2025-finding</identifier>
    <location>
        <url>https://aclanthology.org/2025.emnlp-industry.162/</url>
    </location>
    <part>
        <date>2025-11</date>
        <extent unit="page">
            <start>2343</start>
            <end>2366</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval
%A Lee, Yohan
%A Song, Yongwoo
%A Kim, Sangyeop
%Y Potdar, Saloni
%Y Rojas-Barahona, Lina
%Y Montella, Sebastien
%S Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track
%D 2025
%8 November
%I Association for Computational Linguistics
%C Suzhou (China)
%@ 979-8-89176-333-3
%F lee-etal-2025-finding
%X We present the Conversational Data Retrieval (CDR) benchmark, the first comprehensive test set for evaluating systems that retrieve conversation data for product insights. With 1.6k queries across five analytical tasks and 9.1k conversations, our benchmark provides a reliable standard for measuring conversational data retrieval performance. Our evaluation of 16 popular embedding models shows that even the best models reach only around NDCG@10 of 0.51, revealing a substantial gap between document and conversational data retrieval capabilities. Our work identifies unique challenges in conversational data retrieval (implicit state recognition, turn dynamics, contextual references) while providing practical query templates and detailed error analysis across different task categories. The benchmark dataset and code are available at https://github.com/l-yohai/CDR-Benchmark.
%U https://aclanthology.org/2025.emnlp-industry.162/
%P 2343-2366

Markdown (Informal)

[Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval](https://aclanthology.org/2025.emnlp-industry.162/) (Lee et al., EMNLP 2025)

• Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval (Lee et al., EMNLP 2025)

ACL

• Yohan Lee, Yongwoo Song, and Sangyeop Kim. 2025. Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 2343–2366, Suzhou (China). Association for Computational Linguistics.