<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="munoz-etal-2024-sqft">
    <titleInfo>
        <title>SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Juan</namePart>
        <namePart type="given">Pablo</namePart>
        <namePart type="family">Munoz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jinjie</namePart>
        <namePart type="family">Yuan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nilesh</namePart>
        <namePart type="family">Jain</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yaser</namePart>
            <namePart type="family">Al-Onaizan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Large pre-trained models (LPMs), such as large language models, have become ubiquitous and are employed in many applications. These models are often adapted to a desired domain or downstream task through a fine-tuning stage. This paper proposes SQFT, an end-to-end solution for low-precision sparse parameter-efficient fine-tuning of LPMs, allowing for effective model manipulation in resource-constrained environments. Additionally, an innovative strategy enables the merging of sparse weights with low-rank adapters without losing sparsity and accuracy, overcoming the limitations of previous approaches. SQFT also addresses the challenge of having quantized weights and adapters with different numerical precisions, enabling merging in the desired numerical format without sacrificing accuracy. Multiple adaptation scenarios, models, and comprehensive sparsity levels demonstrate the effectiveness of SQFT.</abstract>
    <identifier type="citekey">munoz-etal-2024-sqft</identifier>
    <identifier type="doi">10.18653/v1/2024.findings-emnlp.749</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-emnlp.749/</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>12817</start>
            <end>12832</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models
%A Munoz, Juan Pablo
%A Yuan, Jinjie
%A Jain, Nilesh
%Y Al-Onaizan, Yaser
%Y Bansal, Mohit
%Y Chen, Yun-Nung
%S Findings of the Association for Computational Linguistics: EMNLP 2024
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F munoz-etal-2024-sqft
%X Large pre-trained models (LPMs), such as large language models, have become ubiquitous and are employed in many applications. These models are often adapted to a desired domain or downstream task through a fine-tuning stage. This paper proposes SQFT, an end-to-end solution for low-precision sparse parameter-efficient fine-tuning of LPMs, allowing for effective model manipulation in resource-constrained environments. Additionally, an innovative strategy enables the merging of sparse weights with low-rank adapters without losing sparsity and accuracy, overcoming the limitations of previous approaches. SQFT also addresses the challenge of having quantized weights and adapters with different numerical precisions, enabling merging in the desired numerical format without sacrificing accuracy. Multiple adaptation scenarios, models, and comprehensive sparsity levels demonstrate the effectiveness of SQFT.
%R 10.18653/v1/2024.findings-emnlp.749
%U https://aclanthology.org/2024.findings-emnlp.749/
%U https://doi.org/10.18653/v1/2024.findings-emnlp.749
%P 12817-12832

Markdown (Informal)

[SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models](https://aclanthology.org/2024.findings-emnlp.749/) (Munoz et al., Findings 2024)

• SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models (Munoz et al., Findings 2024)

ACL

• Juan Pablo Munoz, Jinjie Yuan, and Nilesh Jain. 2024. SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12817–12832, Miami, Florida, USA. Association for Computational Linguistics.