<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="li-etal-2021-differentiable">
    <titleInfo>
        <title>Differentiable Subset Pruning of Transformer Heads</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jiaoda</namePart>
        <namePart type="family">Li</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ryan</namePart>
        <namePart type="family">Cotterell</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mrinmaya</namePart>
        <namePart type="family">Sachan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Multi-head attention, a collection of several attention mechanisms that independently attend to different parts of the input, is the key ingredient in the Transformer. Recent work has shown, however, that a large proportion of the heads in a Transformer’s multi-head attention mechanism can be safely pruned away without significantly harming the performance of the model; such pruning leads to models that are noticeably smaller and faster in practice. Our work introduces a new head pruning technique that we term differentiable subset pruning. ntuitively, our method learns per- head importance variables and then enforces a user-specified hard constraint on the number of unpruned heads. he importance variables are learned via stochastic gradient descent. e conduct experiments on natural language inference and machine translation; we show that differentiable subset pruning performs comparably or better than previous works while offering precise control of the sparsity level.1</abstract>
    <identifier type="citekey">li-etal-2021-differentiable</identifier>
    <identifier type="doi">10.1162/tacl_a_00436</identifier>
    <location>
        <url>https://aclanthology.org/2021.tacl-1.86</url>
    </location>
    <part>
        <date>2021</date>
        <detail type="volume"><number>9</number></detail>
        <extent unit="page">
            <start>1442</start>
            <end>1459</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Differentiable Subset Pruning of Transformer Heads
%A Li, Jiaoda
%A Cotterell, Ryan
%A Sachan, Mrinmaya
%J Transactions of the Association for Computational Linguistics
%D 2021
%V 9
%I MIT Press
%C Cambridge, MA
%F li-etal-2021-differentiable
%X Multi-head attention, a collection of several attention mechanisms that independently attend to different parts of the input, is the key ingredient in the Transformer. Recent work has shown, however, that a large proportion of the heads in a Transformer’s multi-head attention mechanism can be safely pruned away without significantly harming the performance of the model; such pruning leads to models that are noticeably smaller and faster in practice. Our work introduces a new head pruning technique that we term differentiable subset pruning. ntuitively, our method learns per- head importance variables and then enforces a user-specified hard constraint on the number of unpruned heads. he importance variables are learned via stochastic gradient descent. e conduct experiments on natural language inference and machine translation; we show that differentiable subset pruning performs comparably or better than previous works while offering precise control of the sparsity level.1
%R 10.1162/tacl_a_00436
%U https://aclanthology.org/2021.tacl-1.86
%U https://doi.org/10.1162/tacl_a_00436
%P 1442-1459

Markdown (Informal)

[Differentiable Subset Pruning of Transformer Heads](https://aclanthology.org/2021.tacl-1.86) (Li et al., TACL 2021)

• Differentiable Subset Pruning of Transformer Heads (Li et al., TACL 2021)

ACL

• Jiaoda Li, Ryan Cotterell, and Mrinmaya Sachan. 2021. Differentiable Subset Pruning of Transformer Heads. Transactions of the Association for Computational Linguistics, 9:1442–1459.