<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="belcak-wattenhofer-2024-ultrasparsebert">
    <titleInfo>
        <title>UltraSparseBERT: 99% Conditionally Sparse Language Modelling</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Peter</namePart>
        <namePart type="family">Belcak</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Roger</namePart>
        <namePart type="family">Wattenhofer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lun-Wei</namePart>
            <namePart type="family">Ku</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Andre</namePart>
            <namePart type="family">Martins</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivek</namePart>
            <namePart type="family">Srikumar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We present UltraSparseBERT, a BERT variant that uses 0.3% of its neurons during inference while performing on par with similar BERT models. UltraSparseBERT selectively engages just 12 out of 4095 neurons for each layer inference. This is achieved by reorganizing feedforward networks into fast feedforward networks (FFFs).To showcase but one benefit of high sparsity, we provide an Intel MKL implementation achieving 78x speedup over the optimized feedforward baseline on CPUs, and an OpenAI Triton implementation performing forward passes 4.1x faster than the corresponding native GPU implementation. The training and benchmarking code is enclosed.</abstract>
    <identifier type="citekey">belcak-wattenhofer-2024-ultrasparsebert</identifier>
    <identifier type="doi">10.18653/v1/2024.acl-short.10</identifier>
    <location>
        <url>https://aclanthology.org/2024.luhme-short.10/</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>104</start>
            <end>108</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T UltraSparseBERT: 99% Conditionally Sparse Language Modelling
%A Belcak, Peter
%A Wattenhofer, Roger
%Y Ku, Lun-Wei
%Y Martins, Andre
%Y Srikumar, Vivek
%S Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand
%F belcak-wattenhofer-2024-ultrasparsebert
%X We present UltraSparseBERT, a BERT variant that uses 0.3% of its neurons during inference while performing on par with similar BERT models. UltraSparseBERT selectively engages just 12 out of 4095 neurons for each layer inference. This is achieved by reorganizing feedforward networks into fast feedforward networks (FFFs).To showcase but one benefit of high sparsity, we provide an Intel MKL implementation achieving 78x speedup over the optimized feedforward baseline on CPUs, and an OpenAI Triton implementation performing forward passes 4.1x faster than the corresponding native GPU implementation. The training and benchmarking code is enclosed.
%R 10.18653/v1/2024.acl-short.10
%U https://aclanthology.org/2024.luhme-short.10/
%U https://doi.org/10.18653/v1/2024.acl-short.10
%P 104-108

Markdown (Informal)

[UltraSparseBERT: 99% Conditionally Sparse Language Modelling](https://aclanthology.org/2024.luhme-short.10/) (Belcak & Wattenhofer, ACL 2024)

• UltraSparseBERT: 99% Conditionally Sparse Language Modelling (Belcak & Wattenhofer, ACL 2024)

ACL

• Peter Belcak and Roger Wattenhofer. 2024. UltraSparseBERT: 99% Conditionally Sparse Language Modelling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 104–108, Bangkok, Thailand. Association for Computational Linguistics.