<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="linzen-2020-accelerate">
    <titleInfo>
        <title>How Can We Accelerate Progress Towards Human-like Linguistic Generalization?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tal</namePart>
        <namePart type="family">Linzen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>This position paper describes and critiques the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm, which has become a central tool for measuring progress in natural language understanding. This paradigm consists of three stages: (1) pre-training of a word prediction model on a corpus of arbitrary size; (2) fine-tuning (transfer learning) on a training set representing a classification task; (3) evaluation on a test set drawn from the same distribution as that training set. This paradigm favors simple, low-bias architectures, which, first, can be scaled to process vast amounts of data, and second, can capture the fine-grained statistical properties of a particular data set, regardless of whether those properties are likely to generalize to examples of the task outside the data set. This contrasts with humans, who learn language from several orders of magnitude less data than the systems favored by this evaluation paradigm, and generalize to new tasks in a consistent way. We advocate for supplementing or replacing PAID with paradigms that reward architectures that generalize as quickly and robustly as humans.</abstract>
    <identifier type="citekey">linzen-2020-accelerate</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-main.465</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-main.465</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>5210</start>
            <end>5217</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T How Can We Accelerate Progress Towards Human-like Linguistic Generalization?
%A Linzen, Tal
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
%D 2020
%8 July
%I Association for Computational Linguistics
%C Online
%F linzen-2020-accelerate
%X This position paper describes and critiques the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm, which has become a central tool for measuring progress in natural language understanding. This paradigm consists of three stages: (1) pre-training of a word prediction model on a corpus of arbitrary size; (2) fine-tuning (transfer learning) on a training set representing a classification task; (3) evaluation on a test set drawn from the same distribution as that training set. This paradigm favors simple, low-bias architectures, which, first, can be scaled to process vast amounts of data, and second, can capture the fine-grained statistical properties of a particular data set, regardless of whether those properties are likely to generalize to examples of the task outside the data set. This contrasts with humans, who learn language from several orders of magnitude less data than the systems favored by this evaluation paradigm, and generalize to new tasks in a consistent way. We advocate for supplementing or replacing PAID with paradigms that reward architectures that generalize as quickly and robustly as humans.
%R 10.18653/v1/2020.acl-main.465
%U https://aclanthology.org/2020.acl-main.465
%U https://doi.org/10.18653/v1/2020.acl-main.465
%P 5210-5217

Markdown (Informal)

[How Can We Accelerate Progress Towards Human-like Linguistic Generalization?](https://aclanthology.org/2020.acl-main.465) (Linzen, ACL 2020)

• How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (Linzen, ACL 2020)

ACL

• Tal Linzen. 2020. How Can We Accelerate Progress Towards Human-like Linguistic Generalization?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5210–5217, Online. Association for Computational Linguistics.