<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sadjoli-etal-2025-optimization">
    <titleInfo>
        <title>Optimization before Evaluation: Evaluation with Unoptimized Prompts Can be Misleading</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nicholas</namePart>
        <namePart type="family">Sadjoli</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tim</namePart>
        <namePart type="family">Siefken</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Atin</namePart>
        <namePart type="family">Ghosh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yifan</namePart>
        <namePart type="family">Mai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Daniel</namePart>
        <namePart type="family">Dahlmeier</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Georg</namePart>
            <namePart type="family">Rehm</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yunyao</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Vienna, Austria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-288-6</identifier>
    </relatedItem>
    <abstract>Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task.</abstract>
    <identifier type="citekey">sadjoli-etal-2025-optimization</identifier>
    <identifier type="doi">10.18653/v1/2025.acl-industry.44</identifier>
    <location>
        <url>https://aclanthology.org/2025.acl-industry.44/</url>
    </location>
    <part>
        <date>2025-07</date>
        <extent unit="page">
            <start>619</start>
            <end>638</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Optimization before Evaluation: Evaluation with Unoptimized Prompts Can be Misleading
%A Sadjoli, Nicholas
%A Siefken, Tim
%A Ghosh, Atin
%A Mai, Yifan
%A Dahlmeier, Daniel
%Y Rehm, Georg
%Y Li, Yunyao
%S Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)
%D 2025
%8 July
%I Association for Computational Linguistics
%C Vienna, Austria
%@ 979-8-89176-288-6
%F sadjoli-etal-2025-optimization
%X Current Large Language Model (LLM) evaluation frameworks utilize the same static prompt template across all models under evaluation. This differs from the common industry practice of using prompt optimization (PO) techniques to optimize the prompt for each model to maximize application performance. In this paper, we investigate the effect of PO towards LLM evaluations. Our results on public academic and internal industry benchmarks show that PO greatly affects the final ranking of models. This highlights the importance of practitioners performing PO per model when conducting evaluations to choose the best model for a given task.
%R 10.18653/v1/2025.acl-industry.44
%U https://aclanthology.org/2025.acl-industry.44/
%U https://doi.org/10.18653/v1/2025.acl-industry.44
%P 619-638

Markdown (Informal)

[Optimization before Evaluation: Evaluation with Unoptimized Prompts Can be Misleading](https://aclanthology.org/2025.acl-industry.44/) (Sadjoli et al., ACL 2025)

• Optimization before Evaluation: Evaluation with Unoptimized Prompts Can be Misleading (Sadjoli et al., ACL 2025)

ACL

• Nicholas Sadjoli, Tim Siefken, Atin Ghosh, Yifan Mai, and Daniel Dahlmeier. 2025. Optimization before Evaluation: Evaluation with Unoptimized Prompts Can be Misleading. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track), pages 619–638, Vienna, Austria. Association for Computational Linguistics.