<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="oh-etal-2024-generative">
    <titleInfo>
        <title>The Generative AI Paradox in Evaluation: “What It Can Solve, It May Not Evaluate”</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Juhyun</namePart>
        <namePart type="family">Oh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Eunsu</namePart>
        <namePart type="family">Kim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Inha</namePart>
        <namePart type="family">Cha</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alice</namePart>
        <namePart type="family">Oh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Neele</namePart>
            <namePart type="family">Falk</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sara</namePart>
            <namePart type="family">Papi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mike</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">St. Julian’s, Malta</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>This paper explores the assumption that Large Language Models (LLMs) skilled in generation tasks are equally adept as evaluators. We assess the performance of three LLMs and one open-source LM in Question-Answering (QA) and evaluation tasks using the TriviaQA (Joshi et al., 2017) dataset. Results indicate a significant disparity, with LLMs exhibiting lower performance in evaluation tasks compared to generation tasks. Intriguingly, we discover instances of unfaithful evaluation where models accurately evaluate answers in areas where they lack competence, underscoring the need to examine the faithfulness and trustworthiness of LLMs as evaluators. This study contributes to the understanding of “the Generative AI Paradox” (West et al., 2023), highlighting a need to explore the correlation between generative excellence and evaluation proficiency, and the necessity to scrutinize the faithfulness aspect in model evaluations.</abstract>
    <identifier type="citekey">oh-etal-2024-generative</identifier>
    <location>
        <url>https://aclanthology.org/2024.eacl-srw.19</url>
    </location>
    <part>
        <date>2024-03</date>
        <extent unit="page">
            <start>248</start>
            <end>257</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T The Generative AI Paradox in Evaluation: “What It Can Solve, It May Not Evaluate”
%A Oh, Juhyun
%A Kim, Eunsu
%A Cha, Inha
%A Oh, Alice
%Y Falk, Neele
%Y Papi, Sara
%Y Zhang, Mike
%S Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop
%D 2024
%8 March
%I Association for Computational Linguistics
%C St. Julian’s, Malta
%F oh-etal-2024-generative
%X This paper explores the assumption that Large Language Models (LLMs) skilled in generation tasks are equally adept as evaluators. We assess the performance of three LLMs and one open-source LM in Question-Answering (QA) and evaluation tasks using the TriviaQA (Joshi et al., 2017) dataset. Results indicate a significant disparity, with LLMs exhibiting lower performance in evaluation tasks compared to generation tasks. Intriguingly, we discover instances of unfaithful evaluation where models accurately evaluate answers in areas where they lack competence, underscoring the need to examine the faithfulness and trustworthiness of LLMs as evaluators. This study contributes to the understanding of “the Generative AI Paradox” (West et al., 2023), highlighting a need to explore the correlation between generative excellence and evaluation proficiency, and the necessity to scrutinize the faithfulness aspect in model evaluations.
%U https://aclanthology.org/2024.eacl-srw.19
%P 248-257

Markdown (Informal)

[The Generative AI Paradox in Evaluation: “What It Can Solve, It May Not Evaluate”](https://aclanthology.org/2024.eacl-srw.19) (Oh et al., EACL 2024)

• The Generative AI Paradox in Evaluation: “What It Can Solve, It May Not Evaluate” (Oh et al., EACL 2024)

ACL

• Juhyun Oh, Eunsu Kim, Inha Cha, and Alice Oh. 2024. The Generative AI Paradox in Evaluation: “What It Can Solve, It May Not Evaluate”. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 248–257, St. Julian’s, Malta. Association for Computational Linguistics.