<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ettinger-2020-bert">
    <titleInfo>
        <title>What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Allyson</namePart>
        <namePart type="family">Ettinger</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role reversal, albeit with less sensitivity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inference and role-based event prediction— and, in particular, it shows clear insensitivity to the contextual impacts of negation.</abstract>
    <identifier type="citekey">ettinger-2020-bert</identifier>
    <identifier type="doi">10.1162/tacl_a_00298</identifier>
    <location>
        <url>https://aclanthology.org/2020.tacl-1.3</url>
    </location>
    <part>
        <date>2020</date>
        <detail type="volume"><number>8</number></detail>
        <extent unit="page">
            <start>34</start>
            <end>48</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models
%A Ettinger, Allyson
%J Transactions of the Association for Computational Linguistics
%D 2020
%V 8
%I MIT Press
%C Cambridge, MA
%F ettinger-2020-bert
%X Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role reversal, albeit with less sensitivity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inference and role-based event prediction— and, in particular, it shows clear insensitivity to the contextual impacts of negation.
%R 10.1162/tacl_a_00298
%U https://aclanthology.org/2020.tacl-1.3
%U https://doi.org/10.1162/tacl_a_00298
%P 34-48

Markdown (Informal)

[What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models](https://aclanthology.org/2020.tacl-1.3) (Ettinger, TACL 2020)

• What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (Ettinger, TACL 2020)

ACL

• Allyson Ettinger. 2020. What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models. Transactions of the Association for Computational Linguistics, 8:34–48.