<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="puccetti-etal-2021-bert">
    <titleInfo>
        <title>How Do BERT Embeddings Organize Linguistic Knowledge?</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Giovanni</namePart>
        <namePart type="family">Puccetti</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alessio</namePart>
        <namePart type="family">Miaschi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Felice</namePart>
        <namePart type="family">Dell’Orletta</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Eneko</namePart>
            <namePart type="family">Agirre</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Marianna</namePart>
            <namePart type="family">Apidianaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Ivan</namePart>
            <namePart type="family">Vulić</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Several studies investigated the linguistic information implicitly encoded in Neural Language Models. Most of these works focused on quantifying the amount and type of information available within their internal representations and across their layers. In line with this scenario, we proposed a different study, based on Lasso regression, aimed at understanding how the information encoded by BERT sentence-level representations is arrange within its hidden units. Using a suite of several probing tasks, we showed the existence of a relationship between the implicit knowledge learned by the model and the number of individual units involved in the encodings of this competence. Moreover, we found that it is possible to identify groups of hidden units more relevant for specific linguistic properties.</abstract>
    <identifier type="citekey">puccetti-etal-2021-bert</identifier>
    <identifier type="doi">10.18653/v1/2021.deelio-1.6</identifier>
    <location>
        <url>https://aclanthology.org/2021.deelio-1.6</url>
    </location>
    <part>
        <date>2021-06</date>
        <extent unit="page">
            <start>48</start>
            <end>57</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T How Do BERT Embeddings Organize Linguistic Knowledge?
%A Puccetti, Giovanni
%A Miaschi, Alessio
%A Dell’Orletta, Felice
%Y Agirre, Eneko
%Y Apidianaki, Marianna
%Y Vulić, Ivan
%S Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures
%D 2021
%8 June
%I Association for Computational Linguistics
%C Online
%F puccetti-etal-2021-bert
%X Several studies investigated the linguistic information implicitly encoded in Neural Language Models. Most of these works focused on quantifying the amount and type of information available within their internal representations and across their layers. In line with this scenario, we proposed a different study, based on Lasso regression, aimed at understanding how the information encoded by BERT sentence-level representations is arrange within its hidden units. Using a suite of several probing tasks, we showed the existence of a relationship between the implicit knowledge learned by the model and the number of individual units involved in the encodings of this competence. Moreover, we found that it is possible to identify groups of hidden units more relevant for specific linguistic properties.
%R 10.18653/v1/2021.deelio-1.6
%U https://aclanthology.org/2021.deelio-1.6
%U https://doi.org/10.18653/v1/2021.deelio-1.6
%P 48-57

Markdown (Informal)

[How Do BERT Embeddings Organize Linguistic Knowledge?](https://aclanthology.org/2021.deelio-1.6) (Puccetti et al., DeeLIO 2021)

• How Do BERT Embeddings Organize Linguistic Knowledge? (Puccetti et al., DeeLIO 2021)

ACL

• Giovanni Puccetti, Alessio Miaschi, and Felice Dell’Orletta. 2021. How Do BERT Embeddings Organize Linguistic Knowledge?. In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 48–57, Online. Association for Computational Linguistics.