<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="limpijankit-etal-2025-counterfactual">
    <titleInfo>
        <title>Counterfactual Simulatability of LLM Explanations for Generation Tasks</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Marvin</namePart>
        <namePart type="family">Limpijankit</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yanda</namePart>
        <namePart type="family">Chen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Melanie</namePart>
        <namePart type="family">Subbiah</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nicholas</namePart>
        <namePart type="family">Deas</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kathleen</namePart>
        <namePart type="family">McKeown</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 18th International Natural Language Generation Conference</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lucie</namePart>
            <namePart type="family">Flek</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Shashi</namePart>
            <namePart type="family">Narayan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lê</namePart>
            <namePart type="given">Hồng</namePart>
            <namePart type="family">Phương</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jiahuan</namePart>
            <namePart type="family">Pei</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Hanoi, Vietnam</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>LLMs can be unpredictable, as even slight alterations to the prompt can cause the output to change in unexpected ways. Thus, the ability of models to accurately explain their behavior is critical, especially in high-stakes settings. Counterfactual simulatability measures how well an explanation allows users to infer the model’s output on related counterfactuals and has been previously studied for yes/no question answering. We provide a general framework for extending this method to generation tasks, using news summarization and medical suggestion as example use cases. We find that while LLM explanations do enable users to better predict their outputs on counterfactuals in the summarization setting, there is significant room for improvement for medical suggestion. Furthermore, our results suggest that evaluating counterfactual simulatability may be more appropriate for skill-based tasks as opposed to knowledge-based tasks.</abstract>
    <identifier type="citekey">limpijankit-etal-2025-counterfactual</identifier>
    <location>
        <url>https://aclanthology.org/2025.inlg-main.38/</url>
    </location>
    <part>
        <date>2025-10</date>
        <extent unit="page">
            <start>659</start>
            <end>683</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Counterfactual Simulatability of LLM Explanations for Generation Tasks
%A Limpijankit, Marvin
%A Chen, Yanda
%A Subbiah, Melanie
%A Deas, Nicholas
%A McKeown, Kathleen
%Y Flek, Lucie
%Y Narayan, Shashi
%Y Phương, Lê Hồng
%Y Pei, Jiahuan
%S Proceedings of the 18th International Natural Language Generation Conference
%D 2025
%8 October
%I Association for Computational Linguistics
%C Hanoi, Vietnam
%F limpijankit-etal-2025-counterfactual
%X LLMs can be unpredictable, as even slight alterations to the prompt can cause the output to change in unexpected ways. Thus, the ability of models to accurately explain their behavior is critical, especially in high-stakes settings. Counterfactual simulatability measures how well an explanation allows users to infer the model’s output on related counterfactuals and has been previously studied for yes/no question answering. We provide a general framework for extending this method to generation tasks, using news summarization and medical suggestion as example use cases. We find that while LLM explanations do enable users to better predict their outputs on counterfactuals in the summarization setting, there is significant room for improvement for medical suggestion. Furthermore, our results suggest that evaluating counterfactual simulatability may be more appropriate for skill-based tasks as opposed to knowledge-based tasks.
%U https://aclanthology.org/2025.inlg-main.38/
%P 659-683

Markdown (Informal)

[Counterfactual Simulatability of LLM Explanations for Generation Tasks](https://aclanthology.org/2025.inlg-main.38/) (Limpijankit et al., INLG 2025)

• Counterfactual Simulatability of LLM Explanations for Generation Tasks (Limpijankit et al., INLG 2025)

ACL

• Marvin Limpijankit, Yanda Chen, Melanie Subbiah, Nicholas Deas, and Kathleen McKeown. 2025. Counterfactual Simulatability of LLM Explanations for Generation Tasks. In Proceedings of the 18th International Natural Language Generation Conference, pages 659–683, Hanoi, Vietnam. Association for Computational Linguistics.