<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="nguyen-etal-2024-ceval-benchmark">
    <titleInfo>
        <title>CEval: A Benchmark for Evaluating Counterfactual Text Generation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Van</namePart>
        <namePart type="given">Bach</namePart>
        <namePart type="family">Nguyen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Christin</namePart>
        <namePart type="family">Seifert</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jörg</namePart>
        <namePart type="family">Schlötterer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 17th International Natural Language Generation Conference</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Saad</namePart>
            <namePart type="family">Mahamood</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nguyen</namePart>
            <namePart type="given">Le</namePart>
            <namePart type="family">Minh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Daphne</namePart>
            <namePart type="family">Ippolito</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Tokyo, Japan</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Counterfactual text generation aims to minimally change a text, such that it is classified differently. Assessing progress in method development for counterfactual text generation is hindered by a non-uniform usage of data sets and metrics in related work. We propose CEval, a benchmark for comparing counterfactual text generation methods. CEval unifies counterfactual and text quality metrics, includes common counterfactual datasets with human annotations, standard baselines (MICE, GDBA, CREST) and the open-source language model LLAMA-2. Our experiments found no perfect method for generating counterfactual text. Methods that excel at counterfactual metrics often produce lower-quality text while LLMs with simple prompts generate high-quality text but struggle with counterfactual criteria. By making CEval available as an open-source Python library, we encourage the community to contribute additional methods and maintain consistent evaluation in future work.</abstract>
    <identifier type="citekey">nguyen-etal-2024-ceval-benchmark</identifier>
    <location>
        <url>https://aclanthology.org/2024.inlg-main.6</url>
    </location>
    <part>
        <date>2024-09</date>
        <extent unit="page">
            <start>55</start>
            <end>69</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T CEval: A Benchmark for Evaluating Counterfactual Text Generation
%A Nguyen, Van Bach
%A Seifert, Christin
%A Schlötterer, Jörg
%Y Mahamood, Saad
%Y Minh, Nguyen Le
%Y Ippolito, Daphne
%S Proceedings of the 17th International Natural Language Generation Conference
%D 2024
%8 September
%I Association for Computational Linguistics
%C Tokyo, Japan
%F nguyen-etal-2024-ceval-benchmark
%X Counterfactual text generation aims to minimally change a text, such that it is classified differently. Assessing progress in method development for counterfactual text generation is hindered by a non-uniform usage of data sets and metrics in related work. We propose CEval, a benchmark for comparing counterfactual text generation methods. CEval unifies counterfactual and text quality metrics, includes common counterfactual datasets with human annotations, standard baselines (MICE, GDBA, CREST) and the open-source language model LLAMA-2. Our experiments found no perfect method for generating counterfactual text. Methods that excel at counterfactual metrics often produce lower-quality text while LLMs with simple prompts generate high-quality text but struggle with counterfactual criteria. By making CEval available as an open-source Python library, we encourage the community to contribute additional methods and maintain consistent evaluation in future work.
%U https://aclanthology.org/2024.inlg-main.6
%P 55-69

Markdown (Informal)

[CEval: A Benchmark for Evaluating Counterfactual Text Generation](https://aclanthology.org/2024.inlg-main.6) (Nguyen et al., INLG 2024)

• CEval: A Benchmark for Evaluating Counterfactual Text Generation (Nguyen et al., INLG 2024)

ACL

• Van Bach Nguyen, Christin Seifert, and Jörg Schlötterer. 2024. CEval: A Benchmark for Evaluating Counterfactual Text Generation. In Proceedings of the 17th International Natural Language Generation Conference, pages 55–69, Tokyo, Japan. Association for Computational Linguistics.