<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="belz-etal-2020-reprogen">
    <titleInfo>
        <title>ReproGen: Proposal for a Shared Task on Reproducibility of Human Evaluations in NLG</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Anya</namePart>
        <namePart type="family">Belz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shubham</namePart>
        <namePart type="family">Agarwal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Anastasia</namePart>
        <namePart type="family">Shimorina</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ehud</namePart>
        <namePart type="family">Reiter</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 13th International Conference on Natural Language Generation</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Brian</namePart>
            <namePart type="family">Davis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yvette</namePart>
            <namePart type="family">Graham</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">John</namePart>
            <namePart type="family">Kelleher</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yaji</namePart>
            <namePart type="family">Sripada</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dublin, Ireland</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Across NLP, a growing body of work is looking at the issue of reproducibility. However, replicability of human evaluation experiments and reproducibility of their results is currently under-addressed, and this is of particular concern for NLG where human evaluations are the norm. This paper outlines our ideas for a shared task on reproducibility of human evaluations in NLG which aims (i) to shed light on the extent to which past NLG evaluations are replicable and reproducible, and (ii) to draw conclusions regarding how evaluations can be designed and reported to increase replicability and reproducibility. If the task is run over several years, we hope to be able to document an overall increase in levels of replicability and reproducibility over time.</abstract>
    <identifier type="citekey">belz-etal-2020-reprogen</identifier>
    <identifier type="doi">10.18653/v1/2020.inlg-1.29</identifier>
    <location>
        <url>https://aclanthology.org/2020.inlg-1.29</url>
    </location>
    <part>
        <date>2020-12</date>
        <extent unit="page">
            <start>232</start>
            <end>236</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T ReproGen: Proposal for a Shared Task on Reproducibility of Human Evaluations in NLG
%A Belz, Anya
%A Agarwal, Shubham
%A Shimorina, Anastasia
%A Reiter, Ehud
%Y Davis, Brian
%Y Graham, Yvette
%Y Kelleher, John
%Y Sripada, Yaji
%S Proceedings of the 13th International Conference on Natural Language Generation
%D 2020
%8 December
%I Association for Computational Linguistics
%C Dublin, Ireland
%F belz-etal-2020-reprogen
%X Across NLP, a growing body of work is looking at the issue of reproducibility. However, replicability of human evaluation experiments and reproducibility of their results is currently under-addressed, and this is of particular concern for NLG where human evaluations are the norm. This paper outlines our ideas for a shared task on reproducibility of human evaluations in NLG which aims (i) to shed light on the extent to which past NLG evaluations are replicable and reproducible, and (ii) to draw conclusions regarding how evaluations can be designed and reported to increase replicability and reproducibility. If the task is run over several years, we hope to be able to document an overall increase in levels of replicability and reproducibility over time.
%R 10.18653/v1/2020.inlg-1.29
%U https://aclanthology.org/2020.inlg-1.29
%U https://doi.org/10.18653/v1/2020.inlg-1.29
%P 232-236

Markdown (Informal)

[ReproGen: Proposal for a Shared Task on Reproducibility of Human Evaluations in NLG](https://aclanthology.org/2020.inlg-1.29) (Belz et al., INLG 2020)

• ReproGen: Proposal for a Shared Task on Reproducibility of Human Evaluations in NLG (Belz et al., INLG 2020)

ACL

• Anya Belz, Shubham Agarwal, Anastasia Shimorina, and Ehud Reiter. 2020. ReproGen: Proposal for a Shared Task on Reproducibility of Human Evaluations in NLG. In Proceedings of the 13th International Conference on Natural Language Generation, pages 232–236, Dublin, Ireland. Association for Computational Linguistics.