<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="sadeq-etal-2022-informask">
    <titleInfo>
        <title>InforMask: Unsupervised Informative Masking for Language Model Pretraining</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nafis</namePart>
        <namePart type="family">Sadeq</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Canwen</namePart>
        <namePart type="family">Xu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Julian</namePart>
        <namePart type="family">McAuley</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yoav</namePart>
            <namePart type="family">Goldberg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zornitsa</namePart>
            <namePart type="family">Kozareva</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yue</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, United Arab Emirates</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Masked language modeling is widely used for pretraining large language models for natural language understanding (NLU). However, random masking is suboptimal, allocating an equal masking rate for all tokens. In this paper, we propose InforMask, a new unsupervised masking strategy for training masked language models. InforMask exploits Pointwise Mutual Information (PMI) to select the most informative tokens to mask. We further propose two optimizations for InforMask to improve its efficiency. With a one-off preprocessing step, InforMask outperforms random masking and previously proposed masking strategies on the factual recall benchmark LAMA and the question answering benchmark SQuAD v1 and v2.</abstract>
    <identifier type="citekey">sadeq-etal-2022-informask</identifier>
    <identifier type="doi">10.18653/v1/2022.emnlp-main.395</identifier>
    <location>
        <url>https://aclanthology.org/2022.emnlp-main.395</url>
    </location>
    <part>
        <date>2022-12</date>
        <extent unit="page">
            <start>5866</start>
            <end>5878</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T InforMask: Unsupervised Informative Masking for Language Model Pretraining
%A Sadeq, Nafis
%A Xu, Canwen
%A McAuley, Julian
%Y Goldberg, Yoav
%Y Kozareva, Zornitsa
%Y Zhang, Yue
%S Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
%D 2022
%8 December
%I Association for Computational Linguistics
%C Abu Dhabi, United Arab Emirates
%F sadeq-etal-2022-informask
%X Masked language modeling is widely used for pretraining large language models for natural language understanding (NLU). However, random masking is suboptimal, allocating an equal masking rate for all tokens. In this paper, we propose InforMask, a new unsupervised masking strategy for training masked language models. InforMask exploits Pointwise Mutual Information (PMI) to select the most informative tokens to mask. We further propose two optimizations for InforMask to improve its efficiency. With a one-off preprocessing step, InforMask outperforms random masking and previously proposed masking strategies on the factual recall benchmark LAMA and the question answering benchmark SQuAD v1 and v2.
%R 10.18653/v1/2022.emnlp-main.395
%U https://aclanthology.org/2022.emnlp-main.395
%U https://doi.org/10.18653/v1/2022.emnlp-main.395
%P 5866-5878

Markdown (Informal)

[InforMask: Unsupervised Informative Masking for Language Model Pretraining](https://aclanthology.org/2022.emnlp-main.395) (Sadeq et al., EMNLP 2022)

• InforMask: Unsupervised Informative Masking for Language Model Pretraining (Sadeq et al., EMNLP 2022)

ACL

• Nafis Sadeq, Canwen Xu, and Julian McAuley. 2022. InforMask: Unsupervised Informative Masking for Language Model Pretraining. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5866–5878, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.