<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="clinciu-etal-2021-study">
    <titleInfo>
        <title>A Study of Automatic Metrics for the Evaluation of Natural Language Explanations</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Miruna-Adriana</namePart>
        <namePart type="family">Clinciu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Arash</namePart>
        <namePart type="family">Eshghi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Helen</namePart>
        <namePart type="family">Hastie</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-04</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Paola</namePart>
            <namePart type="family">Merlo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jorg</namePart>
            <namePart type="family">Tiedemann</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Reut</namePart>
            <namePart type="family">Tsarfaty</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>As transparency becomes key for robotics and AI, it will be necessary to evaluate the methods through which transparency is provided, including automatically generated natural language (NL) explanations. Here, we explore parallels between the generation of such explanations and the much-studied field of evaluation of Natural Language Generation (NLG). Specifically, we investigate which of the NLG evaluation measures map well to explanations. We present the ExBAN corpus: a crowd-sourced corpus of NL explanations for Bayesian Networks. We run correlations comparing human subjective ratings with NLG automatic measures. We find that embedding-based automatic NLG evaluation methods, such as BERTScore and BLEURT, have a higher correlation with human ratings, compared to word-overlap metrics, such as BLEU and ROUGE. This work has implications for Explainable AI and transparent robotic and autonomous systems.</abstract>
    <identifier type="citekey">clinciu-etal-2021-study</identifier>
    <identifier type="doi">10.18653/v1/2021.eacl-main.202</identifier>
    <location>
        <url>https://aclanthology.org/2021.eacl-main.202</url>
    </location>
    <part>
        <date>2021-04</date>
        <extent unit="page">
            <start>2376</start>
            <end>2387</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A Study of Automatic Metrics for the Evaluation of Natural Language Explanations
%A Clinciu, Miruna-Adriana
%A Eshghi, Arash
%A Hastie, Helen
%Y Merlo, Paola
%Y Tiedemann, Jorg
%Y Tsarfaty, Reut
%S Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
%D 2021
%8 April
%I Association for Computational Linguistics
%C Online
%F clinciu-etal-2021-study
%X As transparency becomes key for robotics and AI, it will be necessary to evaluate the methods through which transparency is provided, including automatically generated natural language (NL) explanations. Here, we explore parallels between the generation of such explanations and the much-studied field of evaluation of Natural Language Generation (NLG). Specifically, we investigate which of the NLG evaluation measures map well to explanations. We present the ExBAN corpus: a crowd-sourced corpus of NL explanations for Bayesian Networks. We run correlations comparing human subjective ratings with NLG automatic measures. We find that embedding-based automatic NLG evaluation methods, such as BERTScore and BLEURT, have a higher correlation with human ratings, compared to word-overlap metrics, such as BLEU and ROUGE. This work has implications for Explainable AI and transparent robotic and autonomous systems.
%R 10.18653/v1/2021.eacl-main.202
%U https://aclanthology.org/2021.eacl-main.202
%U https://doi.org/10.18653/v1/2021.eacl-main.202
%P 2376-2387

Markdown (Informal)

[A Study of Automatic Metrics for the Evaluation of Natural Language Explanations](https://aclanthology.org/2021.eacl-main.202) (Clinciu et al., EACL 2021)

• A Study of Automatic Metrics for the Evaluation of Natural Language Explanations (Clinciu et al., EACL 2021)

ACL

• Miruna-Adriana Clinciu, Arash Eshghi, and Helen Hastie. 2021. A Study of Automatic Metrics for the Evaluation of Natural Language Explanations. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2376–2387, Online. Association for Computational Linguistics.