<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ishii-etal-2024-analysis">
    <titleInfo>
        <title>Analysis of LLM’s “Spurious” Correct Answers Using Evidence Information of Multi-hop QA Datasets</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ai</namePart>
        <namePart type="family">Ishii</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Naoya</namePart>
        <namePart type="family">Inoue</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hisami</namePart>
        <namePart type="family">Suzuki</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Satoshi</namePart>
        <namePart type="family">Sekine</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Russa</namePart>
            <namePart type="family">Biswas</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lucie-Aimée</namePart>
            <namePart type="family">Kaffee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Oshin</namePart>
            <namePart type="family">Agarwal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Pasquale</namePart>
            <namePart type="family">Minervini</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sameer</namePart>
            <namePart type="family">Singh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gerard</namePart>
            <namePart type="family">de Melo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent LLMs show an impressive accuracy on one of the hallmark tasks of language understanding, namely Question Answering (QA). However, it is not clear if the correct answers provided by LLMs are actually grounded on the correct knowledge related to the question. In this paper, we use multi-hop QA datasets to evaluate the accuracy of the knowledge LLMs use to answer questions, and show that as much as 31% of the correct answers by the LLMs are in fact spurious, i.e., the knowledge LLMs used to ground the answer is wrong while the answer is correct. We present an analysis of these spurious correct answers by GPT-4 using three datasets in two languages, while suggesting future pathways to correct the grounding information using existing external knowledge bases.</abstract>
    <identifier type="citekey">ishii-etal-2024-analysis</identifier>
    <location>
        <url>https://aclanthology.org/2024.kallm-1.3</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>24</start>
            <end>34</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Analysis of LLM’s “Spurious” Correct Answers Using Evidence Information of Multi-hop QA Datasets
%A Ishii, Ai
%A Inoue, Naoya
%A Suzuki, Hisami
%A Sekine, Satoshi
%Y Biswas, Russa
%Y Kaffee, Lucie-Aimée
%Y Agarwal, Oshin
%Y Minervini, Pasquale
%Y Singh, Sameer
%Y de Melo, Gerard
%S Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024)
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand
%F ishii-etal-2024-analysis
%X Recent LLMs show an impressive accuracy on one of the hallmark tasks of language understanding, namely Question Answering (QA). However, it is not clear if the correct answers provided by LLMs are actually grounded on the correct knowledge related to the question. In this paper, we use multi-hop QA datasets to evaluate the accuracy of the knowledge LLMs use to answer questions, and show that as much as 31% of the correct answers by the LLMs are in fact spurious, i.e., the knowledge LLMs used to ground the answer is wrong while the answer is correct. We present an analysis of these spurious correct answers by GPT-4 using three datasets in two languages, while suggesting future pathways to correct the grounding information using existing external knowledge bases.
%U https://aclanthology.org/2024.kallm-1.3
%P 24-34

Markdown (Informal)

[Analysis of LLM’s “Spurious” Correct Answers Using Evidence Information of Multi-hop QA Datasets](https://aclanthology.org/2024.kallm-1.3) (Ishii et al., KaLLM-WS 2024)

• Analysis of LLM’s “Spurious” Correct Answers Using Evidence Information of Multi-hop QA Datasets (Ishii et al., KaLLM-WS 2024)

ACL

• Ai Ishii, Naoya Inoue, Hisami Suzuki, and Satoshi Sekine. 2024. Analysis of LLM’s “Spurious” Correct Answers Using Evidence Information of Multi-hop QA Datasets. In Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024), pages 24–34, Bangkok, Thailand. Association for Computational Linguistics.