<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ganesh-etal-2021-compressing">
    <titleInfo>
        <title>Compressing Large-Scale Transformer-Based Models: A Case Study on BERT</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Prakhar</namePart>
        <namePart type="family">Ganesh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yao</namePart>
        <namePart type="family">Chen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xin</namePart>
        <namePart type="family">Lou</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mohammad</namePart>
        <namePart type="given">Ali</namePart>
        <namePart type="family">Khan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yin</namePart>
        <namePart type="family">Yang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hassan</namePart>
        <namePart type="family">Sajjad</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Preslav</namePart>
        <namePart type="family">Nakov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Deming</namePart>
        <namePart type="family">Chen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marianne</namePart>
        <namePart type="family">Winslett</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and thus are too resource- hungry and computation-intensive to suit low- capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted considerable research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.</abstract>
    <identifier type="citekey">ganesh-etal-2021-compressing</identifier>
    <identifier type="doi">10.1162/tacl_a_00413</identifier>
    <location>
        <url>https://aclanthology.org/2021.tacl-1.63</url>
    </location>
    <part>
        <date>2021</date>
        <detail type="volume"><number>9</number></detail>
        <extent unit="page">
            <start>1061</start>
            <end>1080</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
%A Ganesh, Prakhar
%A Chen, Yao
%A Lou, Xin
%A Khan, Mohammad Ali
%A Yang, Yin
%A Sajjad, Hassan
%A Nakov, Preslav
%A Chen, Deming
%A Winslett, Marianne
%J Transactions of the Association for Computational Linguistics
%D 2021
%V 9
%I MIT Press
%C Cambridge, MA
%F ganesh-etal-2021-compressing
%X Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and thus are too resource- hungry and computation-intensive to suit low- capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted considerable research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.
%R 10.1162/tacl_a_00413
%U https://aclanthology.org/2021.tacl-1.63
%U https://doi.org/10.1162/tacl_a_00413
%P 1061-1080

Markdown (Informal)

[Compressing Large-Scale Transformer-Based Models: A Case Study on BERT](https://aclanthology.org/2021.tacl-1.63) (Ganesh et al., TACL 2021)

• Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (Ganesh et al., TACL 2021)

ACL

• Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, and Marianne Winslett. 2021. Compressing Large-Scale Transformer-Based Models: A Case Study on BERT. Transactions of the Association for Computational Linguistics, 9:1061–1080.