<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="weber-etal-2021-better">
    <titleInfo>
        <title>It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Verena</namePart>
        <namePart type="family">Weber</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Enrico</namePart>
        <namePart type="family">Piovano</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Melanie</namePart>
        <namePart type="family">Bradford</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Eduard</namePart>
            <namePart type="family">Dragut</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yunyao</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Lucian</namePart>
            <namePart type="family">Popa</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Slobodan</namePart>
            <namePart type="family">Vucetic</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>When a NLU model is updated, new utter- ances must be annotated to be included for training. However, manual annotation is very costly. We evaluate a semi-supervised learning workflow with a human in the loop in a produc- tion environment. The previous NLU model predicts the annotation of the new utterances, a human then reviews the predicted annotation. Only when the NLU prediction is assessed as incorrect the utterance is sent for human anno- tation. Experimental results show that the pro- posed workflow boosts the performance of the NLU model while significantly reducing the annotation volume. Specifically, in our setup, we see improvements of up to 14.16% for a recall-based metric and up to 9.57% for a F1- score based metric, while reducing the annota- tion volume by 97% and overall cost by 60% for each iteration.</abstract>
    <identifier type="citekey">weber-etal-2021-better</identifier>
    <identifier type="doi">10.18653/v1/2021.dash-1.2</identifier>
    <location>
        <url>https://aclanthology.org/2021.dash-1.2</url>
    </location>
    <part>
        <date>2021-06</date>
        <extent unit="page">
            <start>8</start>
            <end>15</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models
%A Weber, Verena
%A Piovano, Enrico
%A Bradford, Melanie
%Y Dragut, Eduard
%Y Li, Yunyao
%Y Popa, Lucian
%Y Vucetic, Slobodan
%S Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances
%D 2021
%8 June
%I Association for Computational Linguistics
%C Online
%F weber-etal-2021-better
%X When a NLU model is updated, new utter- ances must be annotated to be included for training. However, manual annotation is very costly. We evaluate a semi-supervised learning workflow with a human in the loop in a produc- tion environment. The previous NLU model predicts the annotation of the new utterances, a human then reviews the predicted annotation. Only when the NLU prediction is assessed as incorrect the utterance is sent for human anno- tation. Experimental results show that the pro- posed workflow boosts the performance of the NLU model while significantly reducing the annotation volume. Specifically, in our setup, we see improvements of up to 14.16% for a recall-based metric and up to 9.57% for a F1- score based metric, while reducing the annota- tion volume by 97% and overall cost by 60% for each iteration.
%R 10.18653/v1/2021.dash-1.2
%U https://aclanthology.org/2021.dash-1.2
%U https://doi.org/10.18653/v1/2021.dash-1.2
%P 8-15

Markdown (Informal)

[It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models](https://aclanthology.org/2021.dash-1.2) (Weber et al., DaSH 2021)

• It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models (Weber et al., DaSH 2021)

ACL

• Verena Weber, Enrico Piovano, and Melanie Bradford. 2021. It is better to Verify: Semi-Supervised Learning with a human in the loop for large-scale NLU models. In Proceedings of the Second Workshop on Data Science with Human in the Loop: Language Advances, pages 8–15, Online. Association for Computational Linguistics.