<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="xia-etal-2025-beyond-haystack">
    <titleInfo>
        <title>Beyond the Haystack: Sensitivity to Context in Legal Reference Recall</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Eric</namePart>
        <namePart type="family">Xia</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Karthik</namePart>
        <namePart type="family">Srikumar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Keshav</namePart>
        <namePart type="family">Karthik</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Advaith</namePart>
        <namePart type="family">Renjith</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ashwinee</namePart>
        <namePart type="family">Panda</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Natural Legal Language Processing Workshop 2025</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Nikolaos</namePart>
            <namePart type="family">Aletras</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ilias</namePart>
            <namePart type="family">Chalkidis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Leslie</namePart>
            <namePart type="family">Barrett</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Cătălina</namePart>
            <namePart type="family">Goanță</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Daniel</namePart>
            <namePart type="family">Preoțiuc-Pietro</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gerasimos</namePart>
            <namePart type="family">Spanakis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Suzhou, China</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-338-8</identifier>
    </relatedItem>
    <abstract>Reference retrieval is critical for many applications in the legal domain, for instance in determining which case texts support a particular claim. However, existing benchmarking methods do not rigorously enable evaluation of recall capabilities in previously unseen contexts. We develop an evaluation framework from U.S. court opinions which ensures models have no prior knowledge of case results or context. Applying our framework, we identify an consistent gap across models and tasks between traditional needle-in-a-haystack retrieval and actual performance in legal recall. Our work shows that standard needle-in-a-haystack benchmarks consistently overestimate recall performance in the legal domain. By isolating the causes of performance degradation to contextual informativity rather than distributional differences, our findings highlight the need for specialized testing in reference-critical applications, and establish an evaluation framework for improving retrieval across informativity levels.</abstract>
    <identifier type="citekey">xia-etal-2025-beyond-haystack</identifier>
    <location>
        <url>https://aclanthology.org/2025.nllp-1.5/</url>
    </location>
    <part>
        <date>2025-11</date>
        <extent unit="page">
            <start>48</start>
            <end>53</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Beyond the Haystack: Sensitivity to Context in Legal Reference Recall
%A Xia, Eric
%A Srikumar, Karthik
%A Karthik, Keshav
%A Renjith, Advaith
%A Panda, Ashwinee
%Y Aletras, Nikolaos
%Y Chalkidis, Ilias
%Y Barrett, Leslie
%Y Goanță, Cătălina
%Y Preoțiuc-Pietro, Daniel
%Y Spanakis, Gerasimos
%S Proceedings of the Natural Legal Language Processing Workshop 2025
%D 2025
%8 November
%I Association for Computational Linguistics
%C Suzhou, China
%@ 979-8-89176-338-8
%F xia-etal-2025-beyond-haystack
%X Reference retrieval is critical for many applications in the legal domain, for instance in determining which case texts support a particular claim. However, existing benchmarking methods do not rigorously enable evaluation of recall capabilities in previously unseen contexts. We develop an evaluation framework from U.S. court opinions which ensures models have no prior knowledge of case results or context. Applying our framework, we identify an consistent gap across models and tasks between traditional needle-in-a-haystack retrieval and actual performance in legal recall. Our work shows that standard needle-in-a-haystack benchmarks consistently overestimate recall performance in the legal domain. By isolating the causes of performance degradation to contextual informativity rather than distributional differences, our findings highlight the need for specialized testing in reference-critical applications, and establish an evaluation framework for improving retrieval across informativity levels.
%U https://aclanthology.org/2025.nllp-1.5/
%P 48-53

Markdown (Informal)

[Beyond the Haystack: Sensitivity to Context in Legal Reference Recall](https://aclanthology.org/2025.nllp-1.5/) (Xia et al., NLLP 2025)

• Beyond the Haystack: Sensitivity to Context in Legal Reference Recall (Xia et al., NLLP 2025)

ACL

• Eric Xia, Karthik Srikumar, Keshav Karthik, Advaith Renjith, and Ashwinee Panda. 2025. Beyond the Haystack: Sensitivity to Context in Legal Reference Recall. In Proceedings of the Natural Legal Language Processing Workshop 2025, pages 48–53, Suzhou, China. Association for Computational Linguistics.