<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lawton-etal-2023-neural">
    <titleInfo>
        <title>Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Neal</namePart>
        <namePart type="family">Lawton</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Anoop</namePart>
        <namePart type="family">Kumar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Govind</namePart>
        <namePart type="family">Thattai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Aram</namePart>
        <namePart type="family">Galstyan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Greg</namePart>
        <namePart type="family">Ver Steeg</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jordan</namePart>
            <namePart type="family">Boyd-Graber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Naoaki</namePart>
            <namePart type="family">Okazaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Parameter-efficient tuning (PET) methods fit pre-trained language models (PLMs) to downstream tasks by either computing a small compressed update for a subset of model parameters, or appending and fine-tuning a small number of new model parameters to the pre-trained network. Hand-designed PET architectures from the literature perform well in practice, but have the potential to be improved via automated neural architecture search (NAS). We propose an efficient NAS method for learning PET architectures via structured and unstructured pruning. We present experiments on GLUE demonstrating the effectiveness of our algorithm and discuss how PET architectural design choices affect performance in practice.</abstract>
    <identifier type="citekey">lawton-etal-2023-neural</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-acl.539</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-acl.539</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>8506</start>
            <end>8515</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models
%A Lawton, Neal
%A Kumar, Anoop
%A Thattai, Govind
%A Galstyan, Aram
%A Ver Steeg, Greg
%Y Rogers, Anna
%Y Boyd-Graber, Jordan
%Y Okazaki, Naoaki
%S Findings of the Association for Computational Linguistics: ACL 2023
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F lawton-etal-2023-neural
%X Parameter-efficient tuning (PET) methods fit pre-trained language models (PLMs) to downstream tasks by either computing a small compressed update for a subset of model parameters, or appending and fine-tuning a small number of new model parameters to the pre-trained network. Hand-designed PET architectures from the literature perform well in practice, but have the potential to be improved via automated neural architecture search (NAS). We propose an efficient NAS method for learning PET architectures via structured and unstructured pruning. We present experiments on GLUE demonstrating the effectiveness of our algorithm and discuss how PET architectural design choices affect performance in practice.
%R 10.18653/v1/2023.findings-acl.539
%U https://aclanthology.org/2023.findings-acl.539
%U https://doi.org/10.18653/v1/2023.findings-acl.539
%P 8506-8515

Markdown (Informal)

[Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models](https://aclanthology.org/2023.findings-acl.539) (Lawton et al., Findings 2023)

• Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models (Lawton et al., Findings 2023)

ACL

• Neal Lawton, Anoop Kumar, Govind Thattai, Aram Galstyan, and Greg Ver Steeg. 2023. Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8506–8515, Toronto, Canada. Association for Computational Linguistics.