<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lee-etal-2026-useful">
    <titleInfo>
        <title>Useful to Whom? A Persona-Driven Evaluation of Knowledge-Adapted Health Question Reformulation via LLM Simulation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jooyeon</namePart>
        <namePart type="family">Lee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Luan</namePart>
        <namePart type="given">Huy</namePart>
        <namePart type="family">Pham</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Özlem</namePart>
        <namePart type="family">Uzuner</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Deepak</namePart>
            <namePart type="family">Gupta</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Paul</namePart>
            <namePart type="family">Thompson</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sophia</namePart>
            <namePart type="family">Ananiadou</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Dina</namePart>
            <namePart type="family">Demner-Fushman</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>ELRA Language Resources Association (ELRA)</publisher>
            <place>
                <placeTerm type="text">Palma, Mallorca (Spain)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Automatic metrics such as F1 and BERTScore are often insufficient for evaluating user-centric generative tasks like Consumer Health Question (CHQ) reformulation. A high F1-score may not correlate with user satisfaction, especially when the user’s knowledge level (UKL) dictates their needs. We propose a robust, Persona-Driven Evaluation Framework (PDEF), grounded in cognitive science and health literacy literature, to measure persona-specific utility. This framework assesses reformulations from the perspectives of a ‘Layperson’ (requiring foundational context) and an ‘Expert’ (requiring efficient, precise answers). We apply this framework to a set of reformulated questions generated by LLMs, and test the robustness of our evaluation by using three state-of-the-art LLMs (GPT-4o, Llama 3.3, and Mistral Large) as the evaluators. Our results reveal a significant disconnect between automatic metrics and user-perceived quality: the model with the highest F1-score (0.6134) was consistently outperformed in user preference by a Pipelined model, with experts preferring the latter by a statistically significant margin (p \ensuremath&lt; 0.001). Furthermore, our persona-driven ablation analysis provides robust evidence that specific architectural components, specifically UKL inference and Entailment logic, are linked to significant gains in persona-driven utility for Layperson cohorts. This work demonstrates the critical need for user-centric evaluation and shows that its findings are generalizable across different LLM architectures.</abstract>
    <identifier type="citekey">lee-etal-2026-useful</identifier>
    <identifier type="doi">10.63317/2gh4zdzvdgft</identifier>
    <location>
        <url>https://aclanthology.org/2026.cl4health-1.25/</url>
    </location>
    <part>
        <date>2026-05</date>
        <extent unit="page">
            <start>281</start>
            <end>295</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Useful to Whom? A Persona-Driven Evaluation of Knowledge-Adapted Health Question Reformulation via LLM Simulation
%A Lee, Jooyeon
%A Pham, Luan Huy
%A Uzuner, Özlem
%Y Gupta, Deepak
%Y Thompson, Paul
%Y Ananiadou, Sophia
%Y Demner-Fushman, Dina
%S Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026
%D 2026
%8 May
%I ELRA Language Resources Association (ELRA)
%C Palma, Mallorca (Spain)
%F lee-etal-2026-useful
%X Automatic metrics such as F1 and BERTScore are often insufficient for evaluating user-centric generative tasks like Consumer Health Question (CHQ) reformulation. A high F1-score may not correlate with user satisfaction, especially when the user’s knowledge level (UKL) dictates their needs. We propose a robust, Persona-Driven Evaluation Framework (PDEF), grounded in cognitive science and health literacy literature, to measure persona-specific utility. This framework assesses reformulations from the perspectives of a ‘Layperson’ (requiring foundational context) and an ‘Expert’ (requiring efficient, precise answers). We apply this framework to a set of reformulated questions generated by LLMs, and test the robustness of our evaluation by using three state-of-the-art LLMs (GPT-4o, Llama 3.3, and Mistral Large) as the evaluators. Our results reveal a significant disconnect between automatic metrics and user-perceived quality: the model with the highest F1-score (0.6134) was consistently outperformed in user preference by a Pipelined model, with experts preferring the latter by a statistically significant margin (p \ensuremath< 0.001). Furthermore, our persona-driven ablation analysis provides robust evidence that specific architectural components, specifically UKL inference and Entailment logic, are linked to significant gains in persona-driven utility for Layperson cohorts. This work demonstrates the critical need for user-centric evaluation and shows that its findings are generalizable across different LLM architectures.
%R 10.63317/2gh4zdzvdgft
%U https://aclanthology.org/2026.cl4health-1.25/
%U https://doi.org/10.63317/2gh4zdzvdgft
%P 281-295

Markdown (Informal)

[Useful to Whom? A Persona-Driven Evaluation of Knowledge-Adapted Health Question Reformulation via LLM Simulation](https://aclanthology.org/2026.cl4health-1.25/) (Lee et al., CL4Health 2026)

• Useful to Whom? A Persona-Driven Evaluation of Knowledge-Adapted Health Question Reformulation via LLM Simulation (Lee et al., CL4Health 2026)

ACL

• Jooyeon Lee, Luan Huy Pham, and Özlem Uzuner. 2026. Useful to Whom? A Persona-Driven Evaluation of Knowledge-Adapted Health Question Reformulation via LLM Simulation. In Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026, pages 281–295, Palma, Mallorca (Spain). ELRA Language Resources Association (ELRA).