<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="hahn-etal-2021-sensitivity">
    <titleInfo>
        <title>Sensitivity as a Complexity Measure for Sequence Classification Tasks</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Michael</namePart>
        <namePart type="family">Hahn</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Dan</namePart>
        <namePart type="family">Jurafsky</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Richard</namePart>
        <namePart type="family">Futrell</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>We introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of Boolean function sensitivity. The sensitivity of a function, given a distribution over input sequences, quantifies the number of disjoint subsets of the input sequence that can each be individually changed to change the output. We argue that standard sequence classification methods are biased towards learning low-sensitivity functions, so that tasks requiring high sensitivity are more difficult. To that end, we show analytically that simple lexical classifiers can only express functions of bounded sensitivity, and we show empirically that low-sensitivity functions are easier to learn for LSTMs. We then estimate sensitivity on 15 NLP tasks, finding that sensitivity is higher on challenging tasks collected in GLUE than on simple text classification tasks, and that sensitivity predicts the performance both of simple lexical classifiers and of vanilla BiLSTMs without pretrained contextualized embeddings. Within a task, sensitivity predicts which inputs are hard for such simple models. Our results suggest that the success of massively pretrained contextual representations stems in part because they provide representations from which information can be extracted by low-sensitivity decoders.</abstract>
    <identifier type="citekey">hahn-etal-2021-sensitivity</identifier>
    <identifier type="doi">10.1162/tacl_a_00403</identifier>
    <location>
        <url>https://aclanthology.org/2021.tacl-1.53</url>
    </location>
    <part>
        <date>2021</date>
        <detail type="volume"><number>9</number></detail>
        <extent unit="page">
            <start>891</start>
            <end>908</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Sensitivity as a Complexity Measure for Sequence Classification Tasks
%A Hahn, Michael
%A Jurafsky, Dan
%A Futrell, Richard
%J Transactions of the Association for Computational Linguistics
%D 2021
%V 9
%I MIT Press
%C Cambridge, MA
%F hahn-etal-2021-sensitivity
%X We introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of Boolean function sensitivity. The sensitivity of a function, given a distribution over input sequences, quantifies the number of disjoint subsets of the input sequence that can each be individually changed to change the output. We argue that standard sequence classification methods are biased towards learning low-sensitivity functions, so that tasks requiring high sensitivity are more difficult. To that end, we show analytically that simple lexical classifiers can only express functions of bounded sensitivity, and we show empirically that low-sensitivity functions are easier to learn for LSTMs. We then estimate sensitivity on 15 NLP tasks, finding that sensitivity is higher on challenging tasks collected in GLUE than on simple text classification tasks, and that sensitivity predicts the performance both of simple lexical classifiers and of vanilla BiLSTMs without pretrained contextualized embeddings. Within a task, sensitivity predicts which inputs are hard for such simple models. Our results suggest that the success of massively pretrained contextual representations stems in part because they provide representations from which information can be extracted by low-sensitivity decoders.
%R 10.1162/tacl_a_00403
%U https://aclanthology.org/2021.tacl-1.53
%U https://doi.org/10.1162/tacl_a_00403
%P 891-908

Markdown (Informal)

[Sensitivity as a Complexity Measure for Sequence Classification Tasks](https://aclanthology.org/2021.tacl-1.53) (Hahn et al., TACL 2021)

• Sensitivity as a Complexity Measure for Sequence Classification Tasks (Hahn et al., TACL 2021)

ACL

• Michael Hahn, Dan Jurafsky, and Richard Futrell. 2021. Sensitivity as a Complexity Measure for Sequence Classification Tasks. Transactions of the Association for Computational Linguistics, 9:891–908.