<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="mieskes-2022-replicability">
    <titleInfo>
        <title>Replicability under Near-Perfect Conditions – A Case-Study from Automatic Summarization</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Margot</namePart>
        <namePart type="family">Mieskes</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Third Workshop on Insights from Negative Results in NLP</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Shabnam</namePart>
            <namePart type="family">Tafreshi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">João</namePart>
            <namePart type="family">Sedoc</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aleksandr</namePart>
            <namePart type="family">Drozd</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rumshisky</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Arjun</namePart>
            <namePart type="family">Akula</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dublin, Ireland</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Replication of research results has become more and more important in Natural Language Processing. Nevertheless, we still rely on results reported in the literature for comparison. Additionally, elements of an experimental setup are not always completely reported. This includes, but is not limited to reporting specific parameters used or omitting an implementational detail. In our experiment based on two frequently used data sets from the domain of automatic summarization and the seemingly full disclosure of research artefacts, we examine how well results reported are replicable and what elements influence the success or failure of replication. Our results indicate that publishing research artifacts is far from sufficient, that that publishing all relevant parameters in all possible detail is cruicial.</abstract>
    <identifier type="citekey">mieskes-2022-replicability</identifier>
    <identifier type="doi">10.18653/v1/2022.insights-1.23</identifier>
    <location>
        <url>https://aclanthology.org/2022.insights-1.23</url>
    </location>
    <part>
        <date>2022-05</date>
        <extent unit="page">
            <start>165</start>
            <end>171</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Replicability under Near-Perfect Conditions – A Case-Study from Automatic Summarization
%A Mieskes, Margot
%Y Tafreshi, Shabnam
%Y Sedoc, João
%Y Rogers, Anna
%Y Drozd, Aleksandr
%Y Rumshisky, Anna
%Y Akula, Arjun
%S Proceedings of the Third Workshop on Insights from Negative Results in NLP
%D 2022
%8 May
%I Association for Computational Linguistics
%C Dublin, Ireland
%F mieskes-2022-replicability
%X Replication of research results has become more and more important in Natural Language Processing. Nevertheless, we still rely on results reported in the literature for comparison. Additionally, elements of an experimental setup are not always completely reported. This includes, but is not limited to reporting specific parameters used or omitting an implementational detail. In our experiment based on two frequently used data sets from the domain of automatic summarization and the seemingly full disclosure of research artefacts, we examine how well results reported are replicable and what elements influence the success or failure of replication. Our results indicate that publishing research artifacts is far from sufficient, that that publishing all relevant parameters in all possible detail is cruicial.
%R 10.18653/v1/2022.insights-1.23
%U https://aclanthology.org/2022.insights-1.23
%U https://doi.org/10.18653/v1/2022.insights-1.23
%P 165-171

Markdown (Informal)

[Replicability under Near-Perfect Conditions – A Case-Study from Automatic Summarization](https://aclanthology.org/2022.insights-1.23) (Mieskes, insights 2022)

• Replicability under Near-Perfect Conditions – A Case-Study from Automatic Summarization (Mieskes, insights 2022)

ACL

• Margot Mieskes. 2022. Replicability under Near-Perfect Conditions – A Case-Study from Automatic Summarization. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 165–171, Dublin, Ireland. Association for Computational Linguistics.