<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="flores-cohan-2024-benefits">
    <titleInfo>
        <title>On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Lorenzo</namePart>
        <namePart type="given">Jaime</namePart>
        <namePart type="family">Flores</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Arman</namePart>
        <namePart type="family">Cohan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yvette</namePart>
            <namePart type="family">Graham</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Matthew</namePart>
            <namePart type="family">Purver</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">St. Julian’s, Malta</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Text summarization and simplification are among the most widely used applications of AI. However, such models are often prone to hallucination, which can result from training models on unaligned data. One efficient approach to address this issue is Loss Truncation (Kang and Hashimoto, 2020), an approach to modify the standard log loss to adaptively remove noisy examples during training. However, we find that LT alone yields a considerable number of hallucinated entities on various datasets. We study the behavior of the underlying losses between factual and non-factual examples, to understand and refine the performance of LT. We demonstrate that LT’s performance is limited when the underlying assumption that noisy targets have higher NLL loss is not satisfied, and find that word-level NLL among entities provides better signal for distinguishing factuality. We then leverage this to propose a fine-grained NLL loss and fine-grained data cleaning strategies, and observe improvements in hallucination reduction across some datasets. Our work is available at https://github.com/yale-nlp/Simplification-Projects.</abstract>
    <identifier type="citekey">flores-cohan-2024-benefits</identifier>
    <location>
        <url>https://aclanthology.org/2024.eacl-short.13</url>
    </location>
    <part>
        <date>2024-03</date>
        <extent unit="page">
            <start>138</start>
            <end>150</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization
%A Flores, Lorenzo Jaime
%A Cohan, Arman
%Y Graham, Yvette
%Y Purver, Matthew
%S Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
%D 2024
%8 March
%I Association for Computational Linguistics
%C St. Julian’s, Malta
%F flores-cohan-2024-benefits
%X Text summarization and simplification are among the most widely used applications of AI. However, such models are often prone to hallucination, which can result from training models on unaligned data. One efficient approach to address this issue is Loss Truncation (Kang and Hashimoto, 2020), an approach to modify the standard log loss to adaptively remove noisy examples during training. However, we find that LT alone yields a considerable number of hallucinated entities on various datasets. We study the behavior of the underlying losses between factual and non-factual examples, to understand and refine the performance of LT. We demonstrate that LT’s performance is limited when the underlying assumption that noisy targets have higher NLL loss is not satisfied, and find that word-level NLL among entities provides better signal for distinguishing factuality. We then leverage this to propose a fine-grained NLL loss and fine-grained data cleaning strategies, and observe improvements in hallucination reduction across some datasets. Our work is available at https://github.com/yale-nlp/Simplification-Projects.
%U https://aclanthology.org/2024.eacl-short.13
%P 138-150

Markdown (Informal)

[On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization](https://aclanthology.org/2024.eacl-short.13) (Flores & Cohan, EACL 2024)

• On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization (Flores & Cohan, EACL 2024)

ACL

• Lorenzo Jaime Flores and Arman Cohan. 2024. On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarization. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 138–150, St. Julian’s, Malta. Association for Computational Linguistics.