<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="griesshaber-etal-2020-fine">
    <titleInfo>
        <title>Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Daniel</namePart>
        <namePart type="family">Grießhaber</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Johannes</namePart>
        <namePart type="family">Maucher</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ngoc</namePart>
        <namePart type="given">Thang</namePart>
        <namePart type="family">Vu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 28th International Conference on Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <publisher>International Committee on Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Barcelona, Spain (Online)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recently, leveraging pre-trained Transformer based language models in down stream, task specific models has advanced state of the art results in natural language understanding tasks. However, only a little research has explored the suitability of this approach in low resource settings with less than 1,000 training data points. In this work, we explore fine-tuning methods of BERT - a pre-trained Transformer based language model - by utilizing pool-based active learning to speed up training while keeping the cost of labeling new data constant. Our experimental results on the GLUE data set show an advantage in model performance by maximizing the approximate knowledge gain of the model when querying from the pool of unlabeled data. Finally, we demonstrate and analyze the benefits of freezing layers of the language model during fine-tuning to reduce the number of trainable parameters, making it more suitable for low-resource settings.</abstract>
    <identifier type="citekey">griesshaber-etal-2020-fine</identifier>
    <identifier type="doi">10.18653/v1/2020.coling-main.100</identifier>
    <location>
        <url>https://aclanthology.org/2020.coling-main.100</url>
    </location>
    <part>
        <date>2020-12</date>
        <extent unit="page">
            <start>1158</start>
            <end>1171</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning
%A Grießhaber, Daniel
%A Maucher, Johannes
%A Vu, Ngoc Thang
%S Proceedings of the 28th International Conference on Computational Linguistics
%D 2020
%8 December
%I International Committee on Computational Linguistics
%C Barcelona, Spain (Online)
%F griesshaber-etal-2020-fine
%X Recently, leveraging pre-trained Transformer based language models in down stream, task specific models has advanced state of the art results in natural language understanding tasks. However, only a little research has explored the suitability of this approach in low resource settings with less than 1,000 training data points. In this work, we explore fine-tuning methods of BERT - a pre-trained Transformer based language model - by utilizing pool-based active learning to speed up training while keeping the cost of labeling new data constant. Our experimental results on the GLUE data set show an advantage in model performance by maximizing the approximate knowledge gain of the model when querying from the pool of unlabeled data. Finally, we demonstrate and analyze the benefits of freezing layers of the language model during fine-tuning to reduce the number of trainable parameters, making it more suitable for low-resource settings.
%R 10.18653/v1/2020.coling-main.100
%U https://aclanthology.org/2020.coling-main.100
%U https://doi.org/10.18653/v1/2020.coling-main.100
%P 1158-1171

Markdown (Informal)

[Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning](https://aclanthology.org/2020.coling-main.100) (Grießhaber et al., COLING 2020)

• Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning (Grießhaber et al., COLING 2020)

ACL

• Daniel Grießhaber, Johannes Maucher, and Ngoc Thang Vu. 2020. Fine-tuning BERT for Low-Resource Natural Language Understanding via Active Learning. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1158–1171, Barcelona, Spain (Online). International Committee on Computational Linguistics.