<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="bahri-etal-2022-sharpness">
    <titleInfo>
        <title>Sharpness-Aware Minimization Improves Language Model Generalization</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Dara</namePart>
        <namePart type="family">Bahri</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hossein</namePart>
        <namePart type="family">Mobahi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yi</namePart>
        <namePart type="family">Tay</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Smaranda</namePart>
            <namePart type="family">Muresan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Preslav</namePart>
            <namePart type="family">Nakov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aline</namePart>
            <namePart type="family">Villavicencio</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dublin, Ireland</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The allure of superhuman-level capabilities has led to considerable interest in language models like GPT-3 and T5, wherein the research has, by and large, revolved around new model architectures, training tasks, and loss objectives, along with substantial engineering efforts to scale up model capacity and dataset size. Comparatively little work has been done to improve the generalization of these models through better optimization. In this work, we show that Sharpness-Aware Minimization (SAM), a recently proposed optimization procedure that encourages convergence to flatter minima, can substantially improve the generalization of language models without much computational overhead. We show that SAM is able to boost performance on SuperGLUE, GLUE, Web Questions, Natural Questions, Trivia QA, and TyDiQA, with particularly large gains when training data for these tasks is limited.</abstract>
    <identifier type="citekey">bahri-etal-2022-sharpness</identifier>
    <identifier type="doi">10.18653/v1/2022.acl-long.508</identifier>
    <location>
        <url>https://aclanthology.org/2022.acl-long.508</url>
    </location>
    <part>
        <date>2022-05</date>
        <extent unit="page">
            <start>7360</start>
            <end>7371</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Sharpness-Aware Minimization Improves Language Model Generalization
%A Bahri, Dara
%A Mobahi, Hossein
%A Tay, Yi
%Y Muresan, Smaranda
%Y Nakov, Preslav
%Y Villavicencio, Aline
%S Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
%D 2022
%8 May
%I Association for Computational Linguistics
%C Dublin, Ireland
%F bahri-etal-2022-sharpness
%X The allure of superhuman-level capabilities has led to considerable interest in language models like GPT-3 and T5, wherein the research has, by and large, revolved around new model architectures, training tasks, and loss objectives, along with substantial engineering efforts to scale up model capacity and dataset size. Comparatively little work has been done to improve the generalization of these models through better optimization. In this work, we show that Sharpness-Aware Minimization (SAM), a recently proposed optimization procedure that encourages convergence to flatter minima, can substantially improve the generalization of language models without much computational overhead. We show that SAM is able to boost performance on SuperGLUE, GLUE, Web Questions, Natural Questions, Trivia QA, and TyDiQA, with particularly large gains when training data for these tasks is limited.
%R 10.18653/v1/2022.acl-long.508
%U https://aclanthology.org/2022.acl-long.508
%U https://doi.org/10.18653/v1/2022.acl-long.508
%P 7360-7371

Markdown (Informal)

[Sharpness-Aware Minimization Improves Language Model Generalization](https://aclanthology.org/2022.acl-long.508) (Bahri et al., ACL 2022)

• Sharpness-Aware Minimization Improves Language Model Generalization (Bahri et al., ACL 2022)

ACL

• Dara Bahri, Hossein Mobahi, and Yi Tay. 2022. Sharpness-Aware Minimization Improves Language Model Generalization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7360–7371, Dublin, Ireland. Association for Computational Linguistics.