<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="madhyastha-2025-task">
    <titleInfo>
        <title>Task-Aware Evaluation and Error-Overlap Analysis for Large Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Pranava</namePart>
        <namePart type="family">Madhyastha</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settings (CHOMPS 2025)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Aman</namePart>
            <namePart type="family">Sinha</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Raúl</namePart>
            <namePart type="family">Vázquez</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Timothee</namePart>
            <namePart type="family">Mickus</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Rohit</namePart>
            <namePart type="family">Agarwal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ioana</namePart>
            <namePart type="family">Buhnila</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Patrícia</namePart>
            <namePart type="family">Schmidtová</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Federica</namePart>
            <namePart type="family">Gamba</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Dilip</namePart>
            <namePart type="given">K</namePart>
            <namePart type="family">Prasad</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jörg</namePart>
            <namePart type="family">Tiedemann</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Mumbai, India</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-308-1</identifier>
    </relatedItem>
    <abstract>Public leaderboards for large language models often rely on aggregate scores that conceal critical information about model behavior. In this paper, we present a methodology for task-aware evaluation that combines (i) correctness metrics aligned with task semantics compliance checks for instruction-following and numeric equivalence for mathematics with (ii) pairwise error-overlap analysis to identify complementary model pairs. We apply this methodology to 17 outputs of recent state of the art and frontier LLMs across multiple-choice QA, instruction-following, and mathematical reasoning tasks. We observe that task-aware metrics can reorder model rankings relative to generic lexical metrics, and that error-overlap patterns vary substantially across model pairs and scenarios. We finally conclude by discussing implications for model selection, routing strategies, and LLM-as-judge calibration, and release our analysis pipeline to support further investigation.</abstract>
    <identifier type="citekey">madhyastha-2025-task</identifier>
    <location>
        <url>https://aclanthology.org/2025.chomps-main.1/</url>
    </location>
    <part>
        <date>2025-12</date>
        <extent unit="page">
            <start>1</start>
            <end>10</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Task-Aware Evaluation and Error-Overlap Analysis for Large Language Models
%A Madhyastha, Pranava
%Y Sinha, Aman
%Y Vázquez, Raúl
%Y Mickus, Timothee
%Y Agarwal, Rohit
%Y Buhnila, Ioana
%Y Schmidtová, Patrícia
%Y Gamba, Federica
%Y Prasad, Dilip K.
%Y Tiedemann, Jörg
%S Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settings (CHOMPS 2025)
%D 2025
%8 December
%I Association for Computational Linguistics
%C Mumbai, India
%@ 979-8-89176-308-1
%F madhyastha-2025-task
%X Public leaderboards for large language models often rely on aggregate scores that conceal critical information about model behavior. In this paper, we present a methodology for task-aware evaluation that combines (i) correctness metrics aligned with task semantics compliance checks for instruction-following and numeric equivalence for mathematics with (ii) pairwise error-overlap analysis to identify complementary model pairs. We apply this methodology to 17 outputs of recent state of the art and frontier LLMs across multiple-choice QA, instruction-following, and mathematical reasoning tasks. We observe that task-aware metrics can reorder model rankings relative to generic lexical metrics, and that error-overlap patterns vary substantially across model pairs and scenarios. We finally conclude by discussing implications for model selection, routing strategies, and LLM-as-judge calibration, and release our analysis pipeline to support further investigation.
%U https://aclanthology.org/2025.chomps-main.1/
%P 1-10

Markdown (Informal)

[Task-Aware Evaluation and Error-Overlap Analysis for Large Language Models](https://aclanthology.org/2025.chomps-main.1/) (Madhyastha, CHOMPS 2025)

• Task-Aware Evaluation and Error-Overlap Analysis for Large Language Models (Madhyastha, CHOMPS 2025)

ACL

• Pranava Madhyastha. 2025. Task-Aware Evaluation and Error-Overlap Analysis for Large Language Models. In Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settings (CHOMPS 2025), pages 1–10, Mumbai, India. Association for Computational Linguistics.