<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kalyaeva-etal-2021-behavior">
    <titleInfo>
        <title>Behavior of Modern Pre-trained Language Models Using the Example of Probing Tasks</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ekaterina</namePart>
        <namePart type="family">Kalyaeva</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Oleg</namePart>
        <namePart type="family">Durandin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alexey</namePart>
        <namePart type="family">Malafeev</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Ruslan</namePart>
            <namePart type="family">Mitkov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Galia</namePart>
            <namePart type="family">Angelova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>INCOMA Ltd.</publisher>
            <place>
                <placeTerm type="text">Held Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Modern transformer-based language models are revolutionizing NLP. However, existing studies into language modelling with BERT have been mostly limited to English-language material and do not pay enough attention to the implicit knowledge of language, such as semantic roles, presupposition and negations, that can be acquired by the model during training. Thus, the aim of this study is to examine behavior of the model BERT in the task of masked language modelling and to provide linguistic interpretation to the unexpected effects and errors produced by the model. For this purpose, we used a new Russian-language dataset based on educational texts for learners of Russian and annotated with the help of the National Corpus of the Russian language. In terms of quality metrics (the proportion of words, semantically related to the target word), the multilingual BERT is recognized as the best model. Generally, each model has distinct strengths in relation to a certain linguistic phenomenon. These observations have meaningful implications for research into applied linguistics and pedagogy, contribute to dialogue system development, automatic exercise making, text generation and potentially could improve the quality of existing linguistic technologies</abstract>
    <identifier type="citekey">kalyaeva-etal-2021-behavior</identifier>
    <location>
        <url>https://aclanthology.org/2021.ranlp-1.75</url>
    </location>
    <part>
        <date>2021-09</date>
        <extent unit="page">
            <start>664</start>
            <end>670</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Behavior of Modern Pre-trained Language Models Using the Example of Probing Tasks
%A Kalyaeva, Ekaterina
%A Durandin, Oleg
%A Malafeev, Alexey
%Y Mitkov, Ruslan
%Y Angelova, Galia
%S Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)
%D 2021
%8 September
%I INCOMA Ltd.
%C Held Online
%F kalyaeva-etal-2021-behavior
%X Modern transformer-based language models are revolutionizing NLP. However, existing studies into language modelling with BERT have been mostly limited to English-language material and do not pay enough attention to the implicit knowledge of language, such as semantic roles, presupposition and negations, that can be acquired by the model during training. Thus, the aim of this study is to examine behavior of the model BERT in the task of masked language modelling and to provide linguistic interpretation to the unexpected effects and errors produced by the model. For this purpose, we used a new Russian-language dataset based on educational texts for learners of Russian and annotated with the help of the National Corpus of the Russian language. In terms of quality metrics (the proportion of words, semantically related to the target word), the multilingual BERT is recognized as the best model. Generally, each model has distinct strengths in relation to a certain linguistic phenomenon. These observations have meaningful implications for research into applied linguistics and pedagogy, contribute to dialogue system development, automatic exercise making, text generation and potentially could improve the quality of existing linguistic technologies
%U https://aclanthology.org/2021.ranlp-1.75
%P 664-670

Markdown (Informal)

[Behavior of Modern Pre-trained Language Models Using the Example of Probing Tasks](https://aclanthology.org/2021.ranlp-1.75) (Kalyaeva et al., RANLP 2021)

• Behavior of Modern Pre-trained Language Models Using the Example of Probing Tasks (Kalyaeva et al., RANLP 2021)

ACL

• Ekaterina Kalyaeva, Oleg Durandin, and Alexey Malafeev. 2021. Behavior of Modern Pre-trained Language Models Using the Example of Probing Tasks. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 664–670, Held Online. INCOMA Ltd..