<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lu-etal-2026-beyond-scalar">
    <titleInfo>
        <title>Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Qingyu</namePart>
        <namePart type="family">Lu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ruochen</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Liang</namePart>
        <namePart type="family">Ding</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yufei</namePart>
        <namePart type="family">Xia</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Youxiang</namePart>
        <namePart type="family">Zhu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dacheng</namePart>
        <namePart type="family">Tao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-462-0</identifier>
    </relatedItem>
    <abstract>Automated evaluation of radiology reports is essential for clinical trust, yet clinically significant errors (e.g., missed findings or incorrect descriptions) can critically mislead diagnosis. Traditional metrics fail to distinguish between clinically meaningful and harmless textual variations, collapsing report quality into a single scalar score. Although Large Language Models (LLMs) embody richer medical knowledge, such metrics still lack an explicit boundary of clinical significance. To bridge this gap, we adopt the ReEvalMed benchmark as testbed and evaluate metric-level clinical significance from detecting true clinical errors (”Discrimination”) and tolerating insignificant variations (”Robustness”). We design an evaluation framework with one-pass and two-pass settings, and compare 8 LLM evaluators against score-based baselines. Results reveal a discrimination bias: LLMs readily detect true clinical errors but struggle to tolerate harmless variations. To mitigate this, we synthesize 4k report pairs and train lightweight interpretable metrics on Qwen3-8B and MedGemma-4B. Our trained metric sharpens the clinical significance boundary, surpassing 32B-scale medical LLMs and remaining competitive with proprietary models. Notably, two-pass inference does not eliminate errors, but redistributes them between discrimination and robustness. We will release the dataset and metric to advance future research.</abstract>
    <identifier type="citekey">lu-etal-2026-beyond-scalar</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-emnlp.1788/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>38237</start>
            <end>38264</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports
%A Lu, Qingyu
%A Li, Ruochen
%A Ding, Liang
%A Xia, Yufei
%A Zhu, Youxiang
%A Tao, Dacheng
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F lu-etal-2026-beyond-scalar
%X Automated evaluation of radiology reports is essential for clinical trust, yet clinically significant errors (e.g., missed findings or incorrect descriptions) can critically mislead diagnosis. Traditional metrics fail to distinguish between clinically meaningful and harmless textual variations, collapsing report quality into a single scalar score. Although Large Language Models (LLMs) embody richer medical knowledge, such metrics still lack an explicit boundary of clinical significance. To bridge this gap, we adopt the ReEvalMed benchmark as testbed and evaluate metric-level clinical significance from detecting true clinical errors (”Discrimination”) and tolerating insignificant variations (”Robustness”). We design an evaluation framework with one-pass and two-pass settings, and compare 8 LLM evaluators against score-based baselines. Results reveal a discrimination bias: LLMs readily detect true clinical errors but struggle to tolerate harmless variations. To mitigate this, we synthesize 4k report pairs and train lightweight interpretable metrics on Qwen3-8B and MedGemma-4B. Our trained metric sharpens the clinical significance boundary, surpassing 32B-scale medical LLMs and remaining competitive with proprietary models. Notably, two-pass inference does not eliminate errors, but redistributes them between discrimination and robustness. We will release the dataset and metric to advance future research.
%U https://aclanthology.org/2026.findings-emnlp.1788/
%P 38237-38264

Markdown (Informal)

[Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports](https://aclanthology.org/2026.findings-emnlp.1788/) (Lu et al., Findings 2026)

• Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports (Lu et al., Findings 2026)

ACL

• Qingyu Lu, Ruochen Li, Liang Ding, Yufei Xia, Youxiang Zhu, and Dacheng Tao. 2026. Beyond Scalar Scores: Exploring LLM-based Metrics for Clinical Significance Evaluation in Radiology Reports. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 38237–38264, Budapest, Hungary. Association for Computational Linguistics.