<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="audibert-etal-2023-low">
    <titleInfo>
        <title>Low-Rank Updates of pre-trained Weights for Multi-Task Learning</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Alexandre</namePart>
        <namePart type="family">Audibert</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Massih</namePart>
        <namePart type="given">R</namePart>
        <namePart type="family">Amini</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Konstantin</namePart>
        <namePart type="family">Usevich</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marianne</namePart>
        <namePart type="family">Clausel</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jordan</namePart>
            <namePart type="family">Boyd-Graber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Naoaki</namePart>
            <namePart type="family">Okazaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Multi-Task Learning used with pre-trained models has been quite popular in the field of Natural Language Processing in recent years. This framework remains still challenging due to the complexity of the tasks and the challenges associated with fine-tuning large pre-trained models. In this paper, we propose a new approach for Multi-task learning which is based on stacking the weights of Neural Networks as a tensor. We show that low-rank updates in the canonical polyadic tensor decomposition of this tensor of weights lead to a simple, yet efficient algorithm, which without loss of performance allows to reduce considerably the model parameters. We investigate the interactions between tasks inside the model as well as the inclusion of sparsity to find the best tensor rank and to increase the compression rate. Our strategy is consistent with recent efforts that attempt to use constraints to fine-tune some model components. More precisely, we achieve equivalent performance as the state-of-the-art on the General Language Understanding Evaluation benchmark by training only 0.3 of the parameters per task while not modifying the baseline weights.</abstract>
    <identifier type="citekey">audibert-etal-2023-low</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-acl.476</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-acl.476</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>7544</start>
            <end>7554</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Low-Rank Updates of pre-trained Weights for Multi-Task Learning
%A Audibert, Alexandre
%A Amini, Massih R.
%A Usevich, Konstantin
%A Clausel, Marianne
%Y Rogers, Anna
%Y Boyd-Graber, Jordan
%Y Okazaki, Naoaki
%S Findings of the Association for Computational Linguistics: ACL 2023
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F audibert-etal-2023-low
%X Multi-Task Learning used with pre-trained models has been quite popular in the field of Natural Language Processing in recent years. This framework remains still challenging due to the complexity of the tasks and the challenges associated with fine-tuning large pre-trained models. In this paper, we propose a new approach for Multi-task learning which is based on stacking the weights of Neural Networks as a tensor. We show that low-rank updates in the canonical polyadic tensor decomposition of this tensor of weights lead to a simple, yet efficient algorithm, which without loss of performance allows to reduce considerably the model parameters. We investigate the interactions between tasks inside the model as well as the inclusion of sparsity to find the best tensor rank and to increase the compression rate. Our strategy is consistent with recent efforts that attempt to use constraints to fine-tune some model components. More precisely, we achieve equivalent performance as the state-of-the-art on the General Language Understanding Evaluation benchmark by training only 0.3 of the parameters per task while not modifying the baseline weights.
%R 10.18653/v1/2023.findings-acl.476
%U https://aclanthology.org/2023.findings-acl.476
%U https://doi.org/10.18653/v1/2023.findings-acl.476
%P 7544-7554

Markdown (Informal)

[Low-Rank Updates of pre-trained Weights for Multi-Task Learning](https://aclanthology.org/2023.findings-acl.476) (Audibert et al., Findings 2023)

• Low-Rank Updates of pre-trained Weights for Multi-Task Learning (Audibert et al., Findings 2023)

ACL

• Alexandre Audibert, Massih R Amini, Konstantin Usevich, and Marianne Clausel. 2023. Low-Rank Updates of pre-trained Weights for Multi-Task Learning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7544–7554, Toronto, Canada. Association for Computational Linguistics.