<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="vladika-etal-2024-medreqal">
    <titleInfo>
        <title>MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Juraj</namePart>
        <namePart type="family">Vladika</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Phillip</namePart>
        <namePart type="family">Schneider</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Florian</namePart>
        <namePart type="family">Matthes</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics ACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lun-Wei</namePart>
            <namePart type="family">Ku</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Andre</namePart>
            <namePart type="family">Martins</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivek</namePart>
            <namePart type="family">Srikumar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand and virtual meeting</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In recent years, Large Language Models (LLMs) have demonstrated an impressive ability to encode knowledge during pre-training on large text corpora. They can leverage this knowledge for downstream tasks like question answering (QA), even in complex areas involving health topics. Considering their high potential for facilitating clinical work in the future, understanding the quality of encoded medical knowledge and its recall in LLMs is an important step forward. In this study, we examine the capability of LLMs to exhibit medical knowledge recall by constructing a novel dataset derived from systematic reviews – studies synthesizing evidence-based answers for specific medical questions. Through experiments on the new MedREQAL dataset, comprising question-answer pairs extracted from rigorous systematic reviews, we assess six LLMs, such as GPT and Mixtral, analyzing their classification and generation performance. Our experimental insights into LLM performance on the novel biomedical QA dataset reveal the still challenging nature of this task.</abstract>
    <identifier type="citekey">vladika-etal-2024-medreqal</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-acl.860</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>14459</start>
            <end>14469</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering
%A Vladika, Juraj
%A Schneider, Phillip
%A Matthes, Florian
%Y Ku, Lun-Wei
%Y Martins, Andre
%Y Srikumar, Vivek
%S Findings of the Association for Computational Linguistics ACL 2024
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand and virtual meeting
%F vladika-etal-2024-medreqal
%X In recent years, Large Language Models (LLMs) have demonstrated an impressive ability to encode knowledge during pre-training on large text corpora. They can leverage this knowledge for downstream tasks like question answering (QA), even in complex areas involving health topics. Considering their high potential for facilitating clinical work in the future, understanding the quality of encoded medical knowledge and its recall in LLMs is an important step forward. In this study, we examine the capability of LLMs to exhibit medical knowledge recall by constructing a novel dataset derived from systematic reviews – studies synthesizing evidence-based answers for specific medical questions. Through experiments on the new MedREQAL dataset, comprising question-answer pairs extracted from rigorous systematic reviews, we assess six LLMs, such as GPT and Mixtral, analyzing their classification and generation performance. Our experimental insights into LLM performance on the novel biomedical QA dataset reveal the still challenging nature of this task.
%U https://aclanthology.org/2024.findings-acl.860
%P 14459-14469

Markdown (Informal)

[MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering](https://aclanthology.org/2024.findings-acl.860) (Vladika et al., Findings 2024)

• MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering (Vladika et al., Findings 2024)

ACL

• Juraj Vladika, Phillip Schneider, and Florian Matthes. 2024. MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering. In Findings of the Association for Computational Linguistics ACL 2024, pages 14459–14469, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.