<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="braun-kunz-2024-hypothesis">
    <titleInfo>
        <title>A Hypothesis-Driven Framework for the Analysis of Self-Rationalising Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Marc</namePart>
        <namePart type="family">Braun</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jenny</namePart>
        <namePart type="family">Kunz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Neele</namePart>
            <namePart type="family">Falk</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sara</namePart>
            <namePart type="family">Papi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mike</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">St. Julian’s, Malta</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The self-rationalising capabilities of LLMs are appealing because the generated explanations can give insights into the plausibility of the predictions. However, how faithful the explanations are to the predictions is questionable, raising the need to explore the patterns behind them further.To this end, we propose a hypothesis-driven statistical framework. We use a Bayesian network to implement a hypothesis about how a task (in our example, natural language inference) is solved, and its internal states are translated into natural language with templates. Those explanations are then compared to LLM-generated free-text explanations using automatic and human evaluations. This allows us to judge how similar the LLM’s and the Bayesian network’s decision processes are. We demonstrate the usage of our framework with an example hypothesis and two realisations in Bayesian networks. The resulting models do not exhibit a strong similarity to GPT-3.5. We discuss the implications of this as well as the framework’s potential to approximate LLM decisions better in future work.</abstract>
    <identifier type="citekey">braun-kunz-2024-hypothesis</identifier>
    <location>
        <url>https://aclanthology.org/2024.eacl-srw.11</url>
    </location>
    <part>
        <date>2024-03</date>
        <extent unit="page">
            <start>148</start>
            <end>161</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A Hypothesis-Driven Framework for the Analysis of Self-Rationalising Models
%A Braun, Marc
%A Kunz, Jenny
%Y Falk, Neele
%Y Papi, Sara
%Y Zhang, Mike
%S Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop
%D 2024
%8 March
%I Association for Computational Linguistics
%C St. Julian’s, Malta
%F braun-kunz-2024-hypothesis
%X The self-rationalising capabilities of LLMs are appealing because the generated explanations can give insights into the plausibility of the predictions. However, how faithful the explanations are to the predictions is questionable, raising the need to explore the patterns behind them further.To this end, we propose a hypothesis-driven statistical framework. We use a Bayesian network to implement a hypothesis about how a task (in our example, natural language inference) is solved, and its internal states are translated into natural language with templates. Those explanations are then compared to LLM-generated free-text explanations using automatic and human evaluations. This allows us to judge how similar the LLM’s and the Bayesian network’s decision processes are. We demonstrate the usage of our framework with an example hypothesis and two realisations in Bayesian networks. The resulting models do not exhibit a strong similarity to GPT-3.5. We discuss the implications of this as well as the framework’s potential to approximate LLM decisions better in future work.
%U https://aclanthology.org/2024.eacl-srw.11
%P 148-161

Markdown (Informal)

[A Hypothesis-Driven Framework for the Analysis of Self-Rationalising Models](https://aclanthology.org/2024.eacl-srw.11) (Braun & Kunz, EACL 2024)

• A Hypothesis-Driven Framework for the Analysis of Self-Rationalising Models (Braun & Kunz, EACL 2024)

ACL

• Marc Braun and Jenny Kunz. 2024. A Hypothesis-Driven Framework for the Analysis of Self-Rationalising Models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 148–161, St. Julian’s, Malta. Association for Computational Linguistics.