<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="mircea-cheung-2023-balaur">
    <titleInfo>
        <title>Balaur: Language Model Pretraining with Lexical Semantic Relations</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Andrei</namePart>
        <namePart type="family">Mircea</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jackie</namePart>
        <namePart type="family">Cheung</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Lexical semantic relations (LSRs) characterize meaning relationships between words and play an important role in systematic generalization on lexical inference tasks. Notably, several tasks that require knowledge of hypernymy still pose a challenge for pretrained language models (LMs) such as BERT, underscoring the need to better align their linguistic behavior with our knowledge of LSRs. In this paper, we propose Balaur, a model that addresses this challenge by modeling LSRs directly in the LM’s hidden states throughout pretraining. Motivating our approach is the hypothesis that the internal representations of LMs can provide an interface to their observable linguistic behavior, and that by controlling one we can influence the other. We validate our hypothesis and demonstrate that Balaur generally improves the performance of large transformer-based LMs on a comprehensive set of hypernymy-informed tasks, as well as on the original LM objective. Code and data are made available at https://github.com/mirandrom/balaur</abstract>
    <identifier type="citekey">mircea-cheung-2023-balaur</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.674</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.674</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>10054</start>
            <end>10070</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Balaur: Language Model Pretraining with Lexical Semantic Relations
%A Mircea, Andrei
%A Cheung, Jackie
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F mircea-cheung-2023-balaur
%X Lexical semantic relations (LSRs) characterize meaning relationships between words and play an important role in systematic generalization on lexical inference tasks. Notably, several tasks that require knowledge of hypernymy still pose a challenge for pretrained language models (LMs) such as BERT, underscoring the need to better align their linguistic behavior with our knowledge of LSRs. In this paper, we propose Balaur, a model that addresses this challenge by modeling LSRs directly in the LM’s hidden states throughout pretraining. Motivating our approach is the hypothesis that the internal representations of LMs can provide an interface to their observable linguistic behavior, and that by controlling one we can influence the other. We validate our hypothesis and demonstrate that Balaur generally improves the performance of large transformer-based LMs on a comprehensive set of hypernymy-informed tasks, as well as on the original LM objective. Code and data are made available at https://github.com/mirandrom/balaur
%R 10.18653/v1/2023.findings-emnlp.674
%U https://aclanthology.org/2023.findings-emnlp.674
%U https://doi.org/10.18653/v1/2023.findings-emnlp.674
%P 10054-10070

Markdown (Informal)

[Balaur: Language Model Pretraining with Lexical Semantic Relations](https://aclanthology.org/2023.findings-emnlp.674) (Mircea & Cheung, Findings 2023)

• Balaur: Language Model Pretraining with Lexical Semantic Relations (Mircea & Cheung, Findings 2023)

ACL

• Andrei Mircea and Jackie Cheung. 2023. Balaur: Language Model Pretraining with Lexical Semantic Relations. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10054–10070, Singapore. Association for Computational Linguistics.