<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="urbizu-etal-2023-enough">
    <titleInfo>
        <title>Not Enough Data to Pre-train Your Language Model? MT to the Rescue!</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Gorka</namePart>
        <namePart type="family">Urbizu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Iñaki</namePart>
        <namePart type="family">San Vicente</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xabier</namePart>
        <namePart type="family">Saralegi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ander</namePart>
        <namePart type="family">Corral</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Anna</namePart>
            <namePart type="family">Rogers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jordan</namePart>
            <namePart type="family">Boyd-Graber</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Naoaki</namePart>
            <namePart type="family">Okazaki</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Toronto, Canada</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In recent years, pre-trained transformer-based language models (LM) have become a key resource for implementing most NLP tasks. However, pre-training such models demands large text collections not available in most languages. In this paper, we study the use of machine-translated corpora for pre-training LMs. We answer the following research questions: RQ1: Is MT-based data an alternative to real data for learning a LM?; RQ2: Can real data be complemented with translated data and improve the resulting LM? In order to validate these two questions, several BERT models for Basque have been trained, combining real data and synthetic data translated from Spanish.The evaluation carried out on 9 NLU tasks indicates that models trained exclusively on translated data offer competitive results. Furthermore, models trained with real data can be improved with synthetic data, although further research is needed on the matter.</abstract>
    <identifier type="citekey">urbizu-etal-2023-enough</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-acl.235</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-acl.235</url>
    </location>
    <part>
        <date>2023-07</date>
        <extent unit="page">
            <start>3826</start>
            <end>3836</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Not Enough Data to Pre-train Your Language Model? MT to the Rescue!
%A Urbizu, Gorka
%A San Vicente, Iñaki
%A Saralegi, Xabier
%A Corral, Ander
%Y Rogers, Anna
%Y Boyd-Graber, Jordan
%Y Okazaki, Naoaki
%S Findings of the Association for Computational Linguistics: ACL 2023
%D 2023
%8 July
%I Association for Computational Linguistics
%C Toronto, Canada
%F urbizu-etal-2023-enough
%X In recent years, pre-trained transformer-based language models (LM) have become a key resource for implementing most NLP tasks. However, pre-training such models demands large text collections not available in most languages. In this paper, we study the use of machine-translated corpora for pre-training LMs. We answer the following research questions: RQ1: Is MT-based data an alternative to real data for learning a LM?; RQ2: Can real data be complemented with translated data and improve the resulting LM? In order to validate these two questions, several BERT models for Basque have been trained, combining real data and synthetic data translated from Spanish.The evaluation carried out on 9 NLU tasks indicates that models trained exclusively on translated data offer competitive results. Furthermore, models trained with real data can be improved with synthetic data, although further research is needed on the matter.
%R 10.18653/v1/2023.findings-acl.235
%U https://aclanthology.org/2023.findings-acl.235
%U https://doi.org/10.18653/v1/2023.findings-acl.235
%P 3826-3836

Markdown (Informal)

[Not Enough Data to Pre-train Your Language Model? MT to the Rescue!](https://aclanthology.org/2023.findings-acl.235) (Urbizu et al., Findings 2023)

• Not Enough Data to Pre-train Your Language Model? MT to the Rescue! (Urbizu et al., Findings 2023)

ACL

• Gorka Urbizu, Iñaki San Vicente, Xabier Saralegi, and Ander Corral. 2023. Not Enough Data to Pre-train Your Language Model? MT to the Rescue!. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3826–3836, Toronto, Canada. Association for Computational Linguistics.