<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yang-etal-2023-dynamic">
    <titleInfo>
        <title>Dynamic Stashing Quantization for Efficient Transformer Training</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Guo</namePart>
        <namePart type="family">Yang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Daniel</namePart>
        <namePart type="family">Lo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Robert</namePart>
        <namePart type="family">Mullins</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yiren</namePart>
        <namePart type="family">Zhao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Large Language Models (LLMs) have demonstrated impressive performance on a range of Natural Language Processing (NLP) tasks. Unfortunately, the immense amount of computations and memory accesses required for LLM training makes them prohibitively expensive in terms of hardware cost, and thus challenging to deploy in use cases such as on-device learning. In this paper, motivated by the observation that LLM training is memory-bound, we propose a novel dynamic quantization strategy, termed Dynamic Stashing Quantization (DSQ), that puts a special focus on reducing the memory operations, but also enjoys the other benefits of low precision training, such as the reduced arithmetic cost. We conduct a thorough study on two translation tasks (trained-from-scratch) and three classification tasks (fine-tuning). DSQ reduces the amount of arithmetic operations by 20.95\times and the number of DRAM operations by 2.55\times on IWSLT17 compared to the standard 16-bit fixed-point, which is widely used in on-device learning.</abstract>
    <identifier type="citekey">yang-etal-2023-dynamic</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.489</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.489</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>7329</start>
            <end>7336</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Dynamic Stashing Quantization for Efficient Transformer Training
%A Yang, Guo
%A Lo, Daniel
%A Mullins, Robert
%A Zhao, Yiren
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F yang-etal-2023-dynamic
%X Large Language Models (LLMs) have demonstrated impressive performance on a range of Natural Language Processing (NLP) tasks. Unfortunately, the immense amount of computations and memory accesses required for LLM training makes them prohibitively expensive in terms of hardware cost, and thus challenging to deploy in use cases such as on-device learning. In this paper, motivated by the observation that LLM training is memory-bound, we propose a novel dynamic quantization strategy, termed Dynamic Stashing Quantization (DSQ), that puts a special focus on reducing the memory operations, but also enjoys the other benefits of low precision training, such as the reduced arithmetic cost. We conduct a thorough study on two translation tasks (trained-from-scratch) and three classification tasks (fine-tuning). DSQ reduces the amount of arithmetic operations by 20.95\times and the number of DRAM operations by 2.55\times on IWSLT17 compared to the standard 16-bit fixed-point, which is widely used in on-device learning.
%R 10.18653/v1/2023.findings-emnlp.489
%U https://aclanthology.org/2023.findings-emnlp.489
%U https://doi.org/10.18653/v1/2023.findings-emnlp.489
%P 7329-7336

Markdown (Informal)

[Dynamic Stashing Quantization for Efficient Transformer Training](https://aclanthology.org/2023.findings-emnlp.489) (Yang et al., Findings 2023)

• Dynamic Stashing Quantization for Efficient Transformer Training (Yang et al., Findings 2023)

ACL

• Guo Yang, Daniel Lo, Robert Mullins, and Yiren Zhao. 2023. Dynamic Stashing Quantization for Efficient Transformer Training. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7329–7336, Singapore. Association for Computational Linguistics.