<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="diddee-etal-2026-benchbrowser">
    <titleInfo>
        <title>BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Harshita</namePart>
        <namePart type="family">Diddee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gregory</namePart>
        <namePart type="family">Yauney</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Swabha</namePart>
        <namePart type="family">Swayamdipta</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Daphne</namePart>
        <namePart type="family">Ippolito</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-459-0</identifier>
    </relatedItem>
    <abstract>Do language model benchmarks actually measure what practitioners intend them to? Benchmark metadata often obscures how benchmarks operationalize a capability: a “poetry” benchmark may omit haikus, while “instruction-following” benchmarks may test for some arbitrary set of skills. This coarseness prevents practitioners from inspecting how capabilities are operationalized at the item level, leading to unclear estimates of model competence. We introduce BenchBrowser, an auditing infrastructure that retrieves use-case-relevant items from 20+ benchmarks, letting practitioners inspect the evidence behind benchmark claims. Through a human evaluation with ten NLP practitioners, we show that the retrieved items are relevant and help support two meta-evaluations: surfacing content validity threats, where advertised capabilities are unevenly represented across facets, and convergent validity threats, where nominally similar benchmarks induce unstable model rankings. By tying validity concerns to concrete examples, BenchBrowser provides interpretable evidence to calibrate interpretation of benchmark scores and improve future iterations of benchmarks.</abstract>
    <identifier type="citekey">diddee-etal-2026-benchbrowser</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-main.1755/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>40024</start>
            <end>40053</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity
%A Diddee, Harshita
%A Yauney, Gregory
%A Swayamdipta, Swabha
%A Ippolito, Daphne
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F diddee-etal-2026-benchbrowser
%X Do language model benchmarks actually measure what practitioners intend them to? Benchmark metadata often obscures how benchmarks operationalize a capability: a “poetry” benchmark may omit haikus, while “instruction-following” benchmarks may test for some arbitrary set of skills. This coarseness prevents practitioners from inspecting how capabilities are operationalized at the item level, leading to unclear estimates of model competence. We introduce BenchBrowser, an auditing infrastructure that retrieves use-case-relevant items from 20+ benchmarks, letting practitioners inspect the evidence behind benchmark claims. Through a human evaluation with ten NLP practitioners, we show that the retrieved items are relevant and help support two meta-evaluations: surfacing content validity threats, where advertised capabilities are unevenly represented across facets, and convergent validity threats, where nominally similar benchmarks induce unstable model rankings. By tying validity concerns to concrete examples, BenchBrowser provides interpretable evidence to calibrate interpretation of benchmark scores and improve future iterations of benchmarks.
%U https://aclanthology.org/2026.emnlp-main.1755/
%P 40024-40053

Markdown (Informal)

[BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity](https://aclanthology.org/2026.emnlp-main.1755/) (Diddee et al., EMNLP 2026)

• BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity (Diddee et al., EMNLP 2026)

ACL

• Harshita Diddee, Gregory Yauney, Swabha Swayamdipta, and Daphne Ippolito. 2026. BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 40024–40053, Budapest, Hungary. Association for Computational Linguistics.