<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="o-neill-etal-2022-aligned">
    <titleInfo>
        <title>Aligned Weight Regularizers for Pruning Pretrained Neural Networks</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">James</namePart>
        <namePart type="family">O’ Neill</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sourav</namePart>
        <namePart type="family">Dutta</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Haytham</namePart>
        <namePart type="family">Assem</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2022</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Smaranda</namePart>
            <namePart type="family">Muresan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Preslav</namePart>
            <namePart type="family">Nakov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aline</namePart>
            <namePart type="family">Villavicencio</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dublin, Ireland</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel self-distillation based pruning strategy, whereby the representational similarity between the pruned and unpruned versions of the same network is maximized. Unlike previous approaches that treat distillation and pruning separately, we use distillation to inform the pruning criteria, without requiring a separate student network as in knowledge distillation. We show that the proposed cross-correlation objective for self-distilled pruning implicitly encourages sparse solutions, naturally complementing magnitude-based pruning criteria. Experiments on the GLUE and XGLUE benchmarks show that self-distilled pruning increases mono- and cross-lingual language model performance. Self-distilled pruned models also outperform smaller Transformers with an equal number of parameters and are competitive against (6 times) larger distilled networks. We also observe that self-distillation (1) maximizes class separability, (2) increases the signal-to-noise ratio, and (3) converges faster after pruning steps, providing further insights into why self-distilled pruning improves generalization.</abstract>
    <identifier type="citekey">o-neill-etal-2022-aligned</identifier>
    <identifier type="doi">10.18653/v1/2022.findings-acl.267</identifier>
    <location>
        <url>https://aclanthology.org/2022.findings-acl.267</url>
    </location>
    <part>
        <date>2022-05</date>
        <extent unit="page">
            <start>3391</start>
            <end>3401</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Aligned Weight Regularizers for Pruning Pretrained Neural Networks
%A O’ Neill, James
%A Dutta, Sourav
%A Assem, Haytham
%Y Muresan, Smaranda
%Y Nakov, Preslav
%Y Villavicencio, Aline
%S Findings of the Association for Computational Linguistics: ACL 2022
%D 2022
%8 May
%I Association for Computational Linguistics
%C Dublin, Ireland
%F o-neill-etal-2022-aligned
%X Pruning aims to reduce the number of parameters while maintaining performance close to the original network. This work proposes a novel self-distillation based pruning strategy, whereby the representational similarity between the pruned and unpruned versions of the same network is maximized. Unlike previous approaches that treat distillation and pruning separately, we use distillation to inform the pruning criteria, without requiring a separate student network as in knowledge distillation. We show that the proposed cross-correlation objective for self-distilled pruning implicitly encourages sparse solutions, naturally complementing magnitude-based pruning criteria. Experiments on the GLUE and XGLUE benchmarks show that self-distilled pruning increases mono- and cross-lingual language model performance. Self-distilled pruned models also outperform smaller Transformers with an equal number of parameters and are competitive against (6 times) larger distilled networks. We also observe that self-distillation (1) maximizes class separability, (2) increases the signal-to-noise ratio, and (3) converges faster after pruning steps, providing further insights into why self-distilled pruning improves generalization.
%R 10.18653/v1/2022.findings-acl.267
%U https://aclanthology.org/2022.findings-acl.267
%U https://doi.org/10.18653/v1/2022.findings-acl.267
%P 3391-3401

Markdown (Informal)

[Aligned Weight Regularizers for Pruning Pretrained Neural Networks](https://aclanthology.org/2022.findings-acl.267) (O’ Neill et al., Findings 2022)

• Aligned Weight Regularizers for Pruning Pretrained Neural Networks (O’ Neill et al., Findings 2022)

ACL

• James O’ Neill, Sourav Dutta, and Haytham Assem. 2022. Aligned Weight Regularizers for Pruning Pretrained Neural Networks. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3391–3401, Dublin, Ireland. Association for Computational Linguistics.