<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="davis-etal-2022-probing">
    <titleInfo>
        <title>Probing for targeted syntactic knowledge through grammatical error detection</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Christopher</namePart>
        <namePart type="family">Davis</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Christopher</namePart>
        <namePart type="family">Bryant</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Andrew</namePart>
        <namePart type="family">Caines</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marek</namePart>
        <namePart type="family">Rei</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Paula</namePart>
        <namePart type="family">Buttery</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Antske</namePart>
            <namePart type="family">Fokkens</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivek</namePart>
            <namePart type="family">Srikumar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, United Arab Emirates (Hybrid)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Targeted studies testing knowledge of subject-verb agreement (SVA) indicate that pre-trained language models encode syntactic information. We assert that if models robustly encode subject-verb agreement, they should be able to identify when agreement is correct and when it is incorrect. To that end, we propose grammatical error detection as a diagnostic probe to evaluate token-level contextual representations for their knowledge of SVA. We evaluate contextual representations at each layer from five pre-trained English language models: BERT, XLNet, GPT-2, RoBERTa and ELECTRA. We leverage public annotated training data from both English second language learners and Wikipedia edits, and report results on manually crafted stimuli for subject-verb agreement. We find that masked language models linearly encode information relevant to the detection of SVA errors, while the autoregressive models perform on par with our baseline. However, we also observe a divergence in performance when probes are trained on different training sets, and when they are evaluated on different syntactic constructions, suggesting the information pertaining to SVA error detection is not robustly encoded.</abstract>
    <identifier type="citekey">davis-etal-2022-probing</identifier>
    <identifier type="doi">10.18653/v1/2022.conll-1.25</identifier>
    <location>
        <url>https://aclanthology.org/2022.conll-1.25</url>
    </location>
    <part>
        <date>2022-12</date>
        <extent unit="page">
            <start>360</start>
            <end>373</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Probing for targeted syntactic knowledge through grammatical error detection
%A Davis, Christopher
%A Bryant, Christopher
%A Caines, Andrew
%A Rei, Marek
%A Buttery, Paula
%Y Fokkens, Antske
%Y Srikumar, Vivek
%S Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)
%D 2022
%8 December
%I Association for Computational Linguistics
%C Abu Dhabi, United Arab Emirates (Hybrid)
%F davis-etal-2022-probing
%X Targeted studies testing knowledge of subject-verb agreement (SVA) indicate that pre-trained language models encode syntactic information. We assert that if models robustly encode subject-verb agreement, they should be able to identify when agreement is correct and when it is incorrect. To that end, we propose grammatical error detection as a diagnostic probe to evaluate token-level contextual representations for their knowledge of SVA. We evaluate contextual representations at each layer from five pre-trained English language models: BERT, XLNet, GPT-2, RoBERTa and ELECTRA. We leverage public annotated training data from both English second language learners and Wikipedia edits, and report results on manually crafted stimuli for subject-verb agreement. We find that masked language models linearly encode information relevant to the detection of SVA errors, while the autoregressive models perform on par with our baseline. However, we also observe a divergence in performance when probes are trained on different training sets, and when they are evaluated on different syntactic constructions, suggesting the information pertaining to SVA error detection is not robustly encoded.
%R 10.18653/v1/2022.conll-1.25
%U https://aclanthology.org/2022.conll-1.25
%U https://doi.org/10.18653/v1/2022.conll-1.25
%P 360-373

Markdown (Informal)

[Probing for targeted syntactic knowledge through grammatical error detection](https://aclanthology.org/2022.conll-1.25) (Davis et al., CoNLL 2022)

• Probing for targeted syntactic knowledge through grammatical error detection (Davis et al., CoNLL 2022)

ACL

• Christopher Davis, Christopher Bryant, Andrew Caines, Marek Rei, and Paula Buttery. 2022. Probing for targeted syntactic knowledge through grammatical error detection. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 360–373, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.