<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="jia-etal-2023-zero">
    <titleInfo>
        <title>Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Qi</namePart>
        <namePart type="family">Jia</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Siyu</namePart>
        <namePart type="family">Ren</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yizhu</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kenny</namePart>
        <namePart type="family">Zhu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Despite tremendous improvements in natural language generation, summarization models still suffer from the unfaithfulness issue. Previous work evaluates faithfulness either using models trained on the other tasks or in-domain synthetic data, or prompting a large model such as ChatGPT. This paper proposes to do zero-shot faithfulness evaluation simply with a moderately-sized foundation language model. We introduce a new metric FFLM, which is a combination of probability changes based on the intuition that prefixing a piece of text that is consistent with the output will increase the probability of predicting the output. Experiments show that FFLM performs competitively with or even outperforms ChatGPT on both inconsistency detection and faithfulness rating with 24x fewer parameters. FFLM also achieves improvements over other strong baselines.</abstract>
    <identifier type="citekey">jia-etal-2023-zero</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.679</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.679</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>11017</start>
            <end>11031</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model
%A Jia, Qi
%A Ren, Siyu
%A Liu, Yizhu
%A Zhu, Kenny
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F jia-etal-2023-zero
%X Despite tremendous improvements in natural language generation, summarization models still suffer from the unfaithfulness issue. Previous work evaluates faithfulness either using models trained on the other tasks or in-domain synthetic data, or prompting a large model such as ChatGPT. This paper proposes to do zero-shot faithfulness evaluation simply with a moderately-sized foundation language model. We introduce a new metric FFLM, which is a combination of probability changes based on the intuition that prefixing a piece of text that is consistent with the output will increase the probability of predicting the output. Experiments show that FFLM performs competitively with or even outperforms ChatGPT on both inconsistency detection and faithfulness rating with 24x fewer parameters. FFLM also achieves improvements over other strong baselines.
%R 10.18653/v1/2023.emnlp-main.679
%U https://aclanthology.org/2023.emnlp-main.679
%U https://doi.org/10.18653/v1/2023.emnlp-main.679
%P 11017-11031

Markdown (Informal)

[Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model](https://aclanthology.org/2023.emnlp-main.679) (Jia et al., EMNLP 2023)

• Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model (Jia et al., EMNLP 2023)

ACL

• Qi Jia, Siyu Ren, Yizhu Liu, and Kenny Zhu. 2023. Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11017–11031, Singapore. Association for Computational Linguistics.