<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="popovic-etal-2022-reproducing">
    <titleInfo>
        <title>Reproducing a Manual Evaluation of the Simplicity of Text Simplification System Outputs</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Maja</namePart>
        <namePart type="family">Popović</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sheila</namePart>
        <namePart type="family">Castilho</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rudali</namePart>
        <namePart type="family">Huidrom</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Anya</namePart>
        <namePart type="family">Belz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Samira</namePart>
            <namePart type="family">Shaikh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Thiago</namePart>
            <namePart type="family">Ferreira</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Amanda</namePart>
            <namePart type="family">Stent</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Waterville, Maine, USA and virtual meeting</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In this paper we describe our reproduction study of the human evaluation of text simplic- ity reported by Nisioi et al. (2017). The work was carried out as part of the ReproGen Shared Task 2022 on Reproducibility of Evaluations in NLG. Our aim was to repeat the evaluation of simplicity for nine automatic text simplification systems with a different set of evaluators. We describe our experimental design together with the known aspects of the original experimental design and present the results from both studies. Pearson correlation between the original and reproduction scores is moderate to high (0.776). Inter-annotator agreement in the reproduction study is lower (0.40) than in the original study (0.66). We discuss challenges arising from the unavailability of certain aspects of the origi- nal set-up, and make several suggestions as to how reproduction of similar evaluations can be made easier in future.</abstract>
    <identifier type="citekey">popovic-etal-2022-reproducing</identifier>
    <location>
        <url>https://aclanthology.org/2022.inlg-genchal.12</url>
    </location>
    <part>
        <date>2022-07</date>
        <extent unit="page">
            <start>80</start>
            <end>85</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Reproducing a Manual Evaluation of the Simplicity of Text Simplification System Outputs
%A Popović, Maja
%A Castilho, Sheila
%A Huidrom, Rudali
%A Belz, Anya
%Y Shaikh, Samira
%Y Ferreira, Thiago
%Y Stent, Amanda
%S Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges
%D 2022
%8 July
%I Association for Computational Linguistics
%C Waterville, Maine, USA and virtual meeting
%F popovic-etal-2022-reproducing
%X In this paper we describe our reproduction study of the human evaluation of text simplic- ity reported by Nisioi et al. (2017). The work was carried out as part of the ReproGen Shared Task 2022 on Reproducibility of Evaluations in NLG. Our aim was to repeat the evaluation of simplicity for nine automatic text simplification systems with a different set of evaluators. We describe our experimental design together with the known aspects of the original experimental design and present the results from both studies. Pearson correlation between the original and reproduction scores is moderate to high (0.776). Inter-annotator agreement in the reproduction study is lower (0.40) than in the original study (0.66). We discuss challenges arising from the unavailability of certain aspects of the origi- nal set-up, and make several suggestions as to how reproduction of similar evaluations can be made easier in future.
%U https://aclanthology.org/2022.inlg-genchal.12
%P 80-85

Markdown (Informal)

[Reproducing a Manual Evaluation of the Simplicity of Text Simplification System Outputs](https://aclanthology.org/2022.inlg-genchal.12) (Popović et al., INLG 2022)

• Reproducing a Manual Evaluation of the Simplicity of Text Simplification System Outputs (Popović et al., INLG 2022)

ACL

• Maja Popović, Sheila Castilho, Rudali Huidrom, and Anya Belz. 2022. Reproducing a Manual Evaluation of the Simplicity of Text Simplification System Outputs. In Proceedings of the 15th International Conference on Natural Language Generation: Generation Challenges, pages 80–85, Waterville, Maine, USA and virtual meeting. Association for Computational Linguistics.