The Shared Task on Reproducibility of Evaluations in NLP (ReproNLP) 2026: Overview and Results

Anja Belz; Craig Thomson; Javier González Corbelle

The Shared Task on Reproducibility of Evaluations in NLP (ReproNLP) 2026: Overview and Results

Anya Belz, Craig Thomson, Javier González Corbelle

Correct Metadata for

Use this form to create a GitHub issue with structured data describing the correction. You will need a GitHub account. Once you create that issue, the correction will be reviewed by a staff member.

⚠️ Mobile Users: Submitting this form to create a new issue will only work with github.com, not the GitHub Mobile app.

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>. You may use <b>...</b> for bold, <i>...</i> for italic, and <url>...</url> for URLs.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

We present the 2026 Shared Task on Reproducibility of Evaluations in NLP (ReproNLP’26) which followed on from five predecessor shared tasks on reproducibility of evaluations, ReproNLP’25, ReproNLP’24, ReproNLP’23, ReproGen’22 and ReproGen’21.This shared task series forms part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP and machine learning, against a backdrop of increasing recognition of the importance of the topic across the two fields. We describe the ReproNLP’26 shared task, summarise results from the reproduction studies submitted, and provide additional comparative analysis of their results.

Anthology ID:: 2026.gem-main.83
Volume:: Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)
Month:: July
Year:: 2026
Address:: San Diego, California, USA
Editors:: Simon Mille, Sebastian Gehrmann, Patrícia Schmidtová, Ondřej Dušek, Marzieh Fadaee, Kyle Lo, Enrico Santus, Gabriel Stanovsky
Venues:: GEM | WS
SIG:
Publisher:: Association for Computational Linguistics
Note:
Pages:: 1055–1070
Language:
URL:: https://aclanthology.org/2026.gem-main.83/
DOI:
Bibkey:
Cite (ACL):: Anya Belz, Craig Thomson, and Javier González Corbelle. 2026. The Shared Task on Reproducibility of Evaluations in NLP (ReproNLP) 2026: Overview and Results. In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), pages 1055–1070, San Diego, California, USA. Association for Computational Linguistics.
Cite (Informal):: The Shared Task on Reproducibility of Evaluations in NLP (ReproNLP) 2026: Overview and Results (Belz et al., GEM 2026)
Copy Citation:
PDF:: https://aclanthology.org/2026.gem-main.83.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{belz-etal-2026-shared,
    title = "The Shared Task on Reproducibility of Evaluations in {NLP} ({R}epro{NLP}) 2026: Overview and Results",
    author = "Belz, Anya  and
      Thomson, Craig  and
      Gonz{\'a}lez Corbelle, Javier",
    editor = "Mille, Simon  and
      Gehrmann, Sebastian  and
      Schmidtov{\'a}, Patr{\'i}cia  and
      Du{\v{s}}ek, Ond{\v{r}}ej  and
      Fadaee, Marzieh  and
      Lo, Kyle  and
      Santus, Enrico  and
      Stanovsky, Gabriel",
    booktitle = "Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics ({GEM})",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.gem-main.83/",
    pages = "1055--1070",
    ISBN = "979-8-89176-423-1",
    abstract = "We present the 2026 Shared Task on Reproducibility of Evaluations in NLP (ReproNLP{'}26) which followed on from five predecessor shared tasks on reproducibility of evaluations, ReproNLP{'}25, ReproNLP{'}24, ReproNLP{'}23, ReproGen{'}22 and ReproGen{'}21.This shared task series forms part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP and machine learning, against a backdrop of increasing recognition of the importance of the topic across the two fields. We describe the ReproNLP{'}26 shared task, summarise results from the reproduction studies submitted, and provide additional comparative analysis of their results."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="belz-etal-2026-shared">
    <titleInfo>
        <title>The Shared Task on Reproducibility of Evaluations in NLP (ReproNLP) 2026: Overview and Results</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Anya</namePart>
        <namePart type="family">Belz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Craig</namePart>
        <namePart type="family">Thomson</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Javier</namePart>
        <namePart type="family">González Corbelle</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Simon</namePart>
            <namePart type="family">Mille</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sebastian</namePart>
            <namePart type="family">Gehrmann</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Patrícia</namePart>
            <namePart type="family">Schmidtová</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ondřej</namePart>
            <namePart type="family">Dušek</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marzieh</namePart>
            <namePart type="family">Fadaee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kyle</namePart>
            <namePart type="family">Lo</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Enrico</namePart>
            <namePart type="family">Santus</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gabriel</namePart>
            <namePart type="family">Stanovsky</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">San Diego, California, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-423-1</identifier>
    </relatedItem>
    <abstract>We present the 2026 Shared Task on Reproducibility of Evaluations in NLP (ReproNLP’26) which followed on from five predecessor shared tasks on reproducibility of evaluations, ReproNLP’25, ReproNLP’24, ReproNLP’23, ReproGen’22 and ReproGen’21.This shared task series forms part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP and machine learning, against a backdrop of increasing recognition of the importance of the topic across the two fields. We describe the ReproNLP’26 shared task, summarise results from the reproduction studies submitted, and provide additional comparative analysis of their results.</abstract>
    <identifier type="citekey">belz-etal-2026-shared</identifier>
    <location>
        <url>https://aclanthology.org/2026.gem-main.83/</url>
    </location>
    <part>
        <date>2026-07</date>
        <extent unit="page">
            <start>1055</start>
            <end>1070</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T The Shared Task on Reproducibility of Evaluations in NLP (ReproNLP) 2026: Overview and Results
%A Belz, Anya
%A Thomson, Craig
%A González Corbelle, Javier
%Y Mille, Simon
%Y Gehrmann, Sebastian
%Y Schmidtová, Patrícia
%Y Dušek, Ondřej
%Y Fadaee, Marzieh
%Y Lo, Kyle
%Y Santus, Enrico
%Y Stanovsky, Gabriel
%S Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)
%D 2026
%8 July
%I Association for Computational Linguistics
%C San Diego, California, USA
%@ 979-8-89176-423-1
%F belz-etal-2026-shared
%X We present the 2026 Shared Task on Reproducibility of Evaluations in NLP (ReproNLP’26) which followed on from five predecessor shared tasks on reproducibility of evaluations, ReproNLP’25, ReproNLP’24, ReproNLP’23, ReproGen’22 and ReproGen’21.This shared task series forms part of an ongoing research programme designed to develop theory and practice of reproducibility assessment in NLP and machine learning, against a backdrop of increasing recognition of the importance of the topic across the two fields. We describe the ReproNLP’26 shared task, summarise results from the reproduction studies submitted, and provide additional comparative analysis of their results.
%U https://aclanthology.org/2026.gem-main.83/
%P 1055-1070

Download as File

Markdown (Informal)

[The Shared Task on Reproducibility of Evaluations in NLP (ReproNLP) 2026: Overview and Results](https://aclanthology.org/2026.gem-main.83/) (Belz et al., GEM 2026)

The Shared Task on Reproducibility of Evaluations in NLP (ReproNLP) 2026: Overview and Results (Belz et al., GEM 2026)

ACL

Anya Belz, Craig Thomson, and Javier González Corbelle. 2026. The Shared Task on Reproducibility of Evaluations in NLP (ReproNLP) 2026: Overview and Results. In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), pages 1055–1070, San Diego, California, USA. Association for Computational Linguistics.