<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="stacey-rei-2024-distilling">
    <titleInfo>
        <title>Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Joe</namePart>
        <namePart type="family">Stacey</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Marek</namePart>
        <namePart type="family">Rei</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics ACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lun-Wei</namePart>
            <namePart type="family">Ku</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Andre</namePart>
            <namePart type="family">Martins</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivek</namePart>
            <namePart type="family">Srikumar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand and virtual meeting</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some of the performance benefits. While this method can improve results on in-distribution examples, it does not necessarily generalise to out-of-distribution (OOD) settings. We investigate two complementary methods for improving the robustness of the resulting student models on OOD domains. The first approach augments the distillation with generated unlabeled examples that match the target distribution. The second method upsamples data points among the training set that are similar to the target distribution. When applied on the task of natural language inference (NLI), our experiments on MNLI show that distillation with these modifications outperforms previous robustness solutions. We also find that these methods improve performance on OOD domains even beyond the target domain.</abstract>
    <identifier type="citekey">stacey-rei-2024-distilling</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-acl.132</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>2239</start>
            <end>2258</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation
%A Stacey, Joe
%A Rei, Marek
%Y Ku, Lun-Wei
%Y Martins, Andre
%Y Srikumar, Vivek
%S Findings of the Association for Computational Linguistics ACL 2024
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand and virtual meeting
%F stacey-rei-2024-distilling
%X Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some of the performance benefits. While this method can improve results on in-distribution examples, it does not necessarily generalise to out-of-distribution (OOD) settings. We investigate two complementary methods for improving the robustness of the resulting student models on OOD domains. The first approach augments the distillation with generated unlabeled examples that match the target distribution. The second method upsamples data points among the training set that are similar to the target distribution. When applied on the task of natural language inference (NLI), our experiments on MNLI show that distillation with these modifications outperforms previous robustness solutions. We also find that these methods improve performance on OOD domains even beyond the target domain.
%U https://aclanthology.org/2024.findings-acl.132
%P 2239-2258

Markdown (Informal)

[Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation](https://aclanthology.org/2024.findings-acl.132) (Stacey & Rei, Findings 2024)

• Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation (Stacey & Rei, Findings 2024)

ACL

• Joe Stacey and Marek Rei. 2024. Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation. In Findings of the Association for Computational Linguistics ACL 2024, pages 2239–2258, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.