<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="edman-etal-2021-importance">
    <titleInfo>
        <title>The Importance of Context in Very Low Resource Language Modeling</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Lukas</namePart>
        <namePart type="family">Edman</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Antonio</namePart>
        <namePart type="family">Toral</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gertjan</namePart>
        <namePart type="family">van Noord</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 18th International Conference on Natural Language Processing (ICON)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Sivaji</namePart>
            <namePart type="family">Bandyopadhyay</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sobha</namePart>
            <namePart type="given">Lalitha</namePart>
            <namePart type="family">Devi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Pushpak</namePart>
            <namePart type="family">Bhattacharyya</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>NLP Association of India (NLPAI)</publisher>
            <place>
                <placeTerm type="text">National Institute of Technology Silchar, Silchar, India</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low-resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model’s performance in the low-resource setting, finding that limiting the model’s self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.</abstract>
    <identifier type="citekey">edman-etal-2021-importance</identifier>
    <location>
        <url>https://aclanthology.org/2021.icon-main.12</url>
    </location>
    <part>
        <date>2021-12</date>
        <extent unit="page">
            <start>86</start>
            <end>92</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T The Importance of Context in Very Low Resource Language Modeling
%A Edman, Lukas
%A Toral, Antonio
%A van Noord, Gertjan
%Y Bandyopadhyay, Sivaji
%Y Devi, Sobha Lalitha
%Y Bhattacharyya, Pushpak
%S Proceedings of the 18th International Conference on Natural Language Processing (ICON)
%D 2021
%8 December
%I NLP Association of India (NLPAI)
%C National Institute of Technology Silchar, Silchar, India
%F edman-etal-2021-importance
%X This paper investigates very low resource language model pretraining, when less than 100 thousand sentences are available. We find that, in very low-resource scenarios, statistical n-gram language models outperform state-of-the-art neural models. Our experiments show that this is mainly due to the focus of the former on a local context. As such, we introduce three methods to improve a neural model’s performance in the low-resource setting, finding that limiting the model’s self-attention is the most effective one, improving on downstream tasks such as NLI and POS tagging by up to 5% for the languages we test on: English, Hindi, and Turkish.
%U https://aclanthology.org/2021.icon-main.12
%P 86-92

Markdown (Informal)

[The Importance of Context in Very Low Resource Language Modeling](https://aclanthology.org/2021.icon-main.12) (Edman et al., ICON 2021)

• The Importance of Context in Very Low Resource Language Modeling (Edman et al., ICON 2021)

ACL

• Lukas Edman, Antonio Toral, and Gertjan van Noord. 2021. The Importance of Context in Very Low Resource Language Modeling. In Proceedings of the 18th International Conference on Natural Language Processing (ICON), pages 86–92, National Institute of Technology Silchar, Silchar, India. NLP Association of India (NLPAI).