<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="fan-etal-2023-evaluating">
    <titleInfo>
        <title>Evaluating Factual Consistency of Texts with Semantic Role Labeling</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jing</namePart>
        <namePart type="family">Fan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dennis</namePart>
        <namePart type="family">Aumiller</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Michael</namePart>
        <namePart type="family">Gertz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Alexis</namePart>
            <namePart type="family">Palmer</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jose</namePart>
            <namePart type="family">Camacho-collados</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Automated evaluation of text generation systems has recently seen increasing attention, particularly checking whether generated text stays truthful to input sources. Existing methods frequently rely on an evaluation using task-specific language models, which in turn allows for little interpretability of generated scores. We introduce SRLScore, a reference-free evaluation metric designed with text summarization in mind. Our approach generates fact tuples constructed from Semantic Role Labels, applied to both input and summary texts.A final factuality score is computed by an adjustable scoring mechanism, which allows for easy adaption of the method across domains. Correlation with human judgments on English summarization datasets shows that SRLScore is competitive with state-of-the-art methods and exhibits stable generalization across datasets without requiring further training or hyperparameter tuning. We experiment with an optional co-reference resolution step, but find that the performance boost is mostly outweighed by the additional compute required. Our metric is available online at: https://github.com/heyjing/SRLScore</abstract>
    <identifier type="citekey">fan-etal-2023-evaluating</identifier>
    <identifier type="doi">10.18653/v1/2023.starsem-1.9</identifier>
    <location>
        <url>https://aclanthology.org/2023.starsem-1.9</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>89</start>
            <end>100</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Evaluating Factual Consistency of Texts with Semantic Role Labeling
%A Fan, Jing
%A Aumiller, Dennis
%A Gertz, Michael
%Y Palmer, Alexis
%Y Camacho-collados, Jose
%S Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023)
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F fan-etal-2023-evaluating
%X Automated evaluation of text generation systems has recently seen increasing attention, particularly checking whether generated text stays truthful to input sources. Existing methods frequently rely on an evaluation using task-specific language models, which in turn allows for little interpretability of generated scores. We introduce SRLScore, a reference-free evaluation metric designed with text summarization in mind. Our approach generates fact tuples constructed from Semantic Role Labels, applied to both input and summary texts.A final factuality score is computed by an adjustable scoring mechanism, which allows for easy adaption of the method across domains. Correlation with human judgments on English summarization datasets shows that SRLScore is competitive with state-of-the-art methods and exhibits stable generalization across datasets without requiring further training or hyperparameter tuning. We experiment with an optional co-reference resolution step, but find that the performance boost is mostly outweighed by the additional compute required. Our metric is available online at: https://github.com/heyjing/SRLScore
%R 10.18653/v1/2023.starsem-1.9
%U https://aclanthology.org/2023.starsem-1.9
%U https://doi.org/10.18653/v1/2023.starsem-1.9
%P 89-100

Markdown (Informal)

[Evaluating Factual Consistency of Texts with Semantic Role Labeling](https://aclanthology.org/2023.starsem-1.9) (Fan et al., *SEM 2023)

• Evaluating Factual Consistency of Texts with Semantic Role Labeling (Fan et al., *SEM 2023)

ACL

• Jing Fan, Dennis Aumiller, and Michael Gertz. 2023. Evaluating Factual Consistency of Texts with Semantic Role Labeling. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 89–100, Toronto, Canada. Association for Computational Linguistics.