<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="chollampatt-ng-2018-reassessment">
    <titleInfo>
        <title>A Reassessment of Reference-Based Grammatical Error Correction Metrics</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Shamil</namePart>
        <namePart type="family">Chollampatt</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hwee</namePart>
        <namePart type="given">Tou</namePart>
        <namePart type="family">Ng</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2018-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 27th International Conference on Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Emily</namePart>
            <namePart type="given">M</namePart>
            <namePart type="family">Bender</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Leon</namePart>
            <namePart type="family">Derczynski</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Pierre</namePart>
            <namePart type="family">Isabelle</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Santa Fe, New Mexico, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Several metrics have been proposed for evaluating grammatical error correction (GEC) systems based on grammaticality, fluency, and adequacy of the output sentences. Previous studies of the correlation of these metrics with human quality judgments were inconclusive, due to the lack of appropriate significance tests, discrepancies in the methods, and choice of datasets used. In this paper, we re-evaluate reference-based GEC metrics by measuring the system-level correlations with humans on a large dataset of human judgments of GEC outputs, and by properly conducting statistical significance tests. Our results show no significant advantage of GLEU over MaxMatch (M2), contradicting previous studies that claim GLEU to be superior. For a finer-grained analysis, we additionally evaluate these metrics for their agreement with human judgments at the sentence level. Our sentence-level analysis indicates that comparing GLEU and M2, one metric may be more useful than the other depending on the scenario. We further qualitatively analyze these metrics and our findings show that apart from being less interpretable and non-deterministic, GLEU also produces counter-intuitive scores in commonly occurring test examples.</abstract>
    <identifier type="citekey">chollampatt-ng-2018-reassessment</identifier>
    <location>
        <url>https://aclanthology.org/C18-1231</url>
    </location>
    <part>
        <date>2018-08</date>
        <extent unit="page">
            <start>2730</start>
            <end>2741</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A Reassessment of Reference-Based Grammatical Error Correction Metrics
%A Chollampatt, Shamil
%A Ng, Hwee Tou
%Y Bender, Emily M.
%Y Derczynski, Leon
%Y Isabelle, Pierre
%S Proceedings of the 27th International Conference on Computational Linguistics
%D 2018
%8 August
%I Association for Computational Linguistics
%C Santa Fe, New Mexico, USA
%F chollampatt-ng-2018-reassessment
%X Several metrics have been proposed for evaluating grammatical error correction (GEC) systems based on grammaticality, fluency, and adequacy of the output sentences. Previous studies of the correlation of these metrics with human quality judgments were inconclusive, due to the lack of appropriate significance tests, discrepancies in the methods, and choice of datasets used. In this paper, we re-evaluate reference-based GEC metrics by measuring the system-level correlations with humans on a large dataset of human judgments of GEC outputs, and by properly conducting statistical significance tests. Our results show no significant advantage of GLEU over MaxMatch (M2), contradicting previous studies that claim GLEU to be superior. For a finer-grained analysis, we additionally evaluate these metrics for their agreement with human judgments at the sentence level. Our sentence-level analysis indicates that comparing GLEU and M2, one metric may be more useful than the other depending on the scenario. We further qualitatively analyze these metrics and our findings show that apart from being less interpretable and non-deterministic, GLEU also produces counter-intuitive scores in commonly occurring test examples.
%U https://aclanthology.org/C18-1231
%P 2730-2741

Markdown (Informal)

[A Reassessment of Reference-Based Grammatical Error Correction Metrics](https://aclanthology.org/C18-1231) (Chollampatt & Ng, COLING 2018)

• A Reassessment of Reference-Based Grammatical Error Correction Metrics (Chollampatt & Ng, COLING 2018)

ACL

• Shamil Chollampatt and Hwee Tou Ng. 2018. A Reassessment of Reference-Based Grammatical Error Correction Metrics. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2730–2741, Santa Fe, New Mexico, USA. Association for Computational Linguistics.