<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="rauscher-etal-2026-transferring">
    <titleInfo>
        <title>Transferring Scientific English Pre-Trained Language Models to Multiple Languages Using Cross-Lingual Transfer</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nikolas</namePart>
        <namePart type="given">Ching-Pu</namePart>
        <namePart type="family">Rauscher</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Fabio</namePart>
        <namePart type="family">Barth</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Georg</namePart>
        <namePart type="family">Rehm</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Georg</namePart>
            <namePart type="family">Rehm</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Stefan</namePart>
            <namePart type="family">Dietze</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Danilo</namePart>
            <namePart type="family">Dessi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Diana</namePart>
            <namePart type="family">Maynard</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sonja</namePart>
            <namePart type="family">Schimmler</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>ELRA Language Resources Association (ELRA)</publisher>
            <place>
                <placeTerm type="text">Palma, Mallorca (Spain)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In this paper, we present a pipeline for domain-adaptive pre-training and cross-lingual transfer of scientific language models from English to non-English languages. Starting from the multilingual scientific corpus SciLaD, we construct a cleaned English pre-training split and continually pre-train a T5-base encoder–decoder model, resulting in EN-T5-Sci. Our model achieves consistent zero-shot improvements on the Global-MMLU English benchmark, outperforming its base model, with particularly strong gains in STEM and Social Sciences. Despite its moderate size, it performs comparably to the much larger BLOOM model on scientific categories. Building on EN-T5-Sci, we transfer scientific knowledge to German, Japanese, Russian, Polish, Spanish, and Portuguese using the WECHSEL method. Our approach reinitializes language-specific embedding layers via aligned static embeddings while retaining the pre-trained Transformer weights, yielding six monolingual scientific T5 models. In zero-shot evaluation in each respective language, the transferred models generally outperform monolingual baselines. These results demonstrate that scientific domain knowledge acquired through English pre-training can be effectively transferred across languages, enabling competitive non-English scientific language models without training large multilingual systems from scratch.</abstract>
    <identifier type="citekey">rauscher-etal-2026-transferring</identifier>
    <identifier type="doi">10.63317/43tu6dfduwvw</identifier>
    <location>
        <url>https://aclanthology.org/2026.nslp-1.26/</url>
    </location>
    <part>
        <date>2026-05</date>
        <extent unit="page">
            <start>261</start>
            <end>269</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Transferring Scientific English Pre-Trained Language Models to Multiple Languages Using Cross-Lingual Transfer
%A Rauscher, Nikolas Ching-Pu
%A Barth, Fabio
%A Rehm, Georg
%Y Rehm, Georg
%Y Dietze, Stefan
%Y Dessi, Danilo
%Y Maynard, Diana
%Y Schimmler, Sonja
%S Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026
%D 2026
%8 May
%I ELRA Language Resources Association (ELRA)
%C Palma, Mallorca (Spain)
%F rauscher-etal-2026-transferring
%X In this paper, we present a pipeline for domain-adaptive pre-training and cross-lingual transfer of scientific language models from English to non-English languages. Starting from the multilingual scientific corpus SciLaD, we construct a cleaned English pre-training split and continually pre-train a T5-base encoder–decoder model, resulting in EN-T5-Sci. Our model achieves consistent zero-shot improvements on the Global-MMLU English benchmark, outperforming its base model, with particularly strong gains in STEM and Social Sciences. Despite its moderate size, it performs comparably to the much larger BLOOM model on scientific categories. Building on EN-T5-Sci, we transfer scientific knowledge to German, Japanese, Russian, Polish, Spanish, and Portuguese using the WECHSEL method. Our approach reinitializes language-specific embedding layers via aligned static embeddings while retaining the pre-trained Transformer weights, yielding six monolingual scientific T5 models. In zero-shot evaluation in each respective language, the transferred models generally outperform monolingual baselines. These results demonstrate that scientific domain knowledge acquired through English pre-training can be effectively transferred across languages, enabling competitive non-English scientific language models without training large multilingual systems from scratch.
%R 10.63317/43tu6dfduwvw
%U https://aclanthology.org/2026.nslp-1.26/
%U https://doi.org/10.63317/43tu6dfduwvw
%P 261-269

Markdown (Informal)

[Transferring Scientific English Pre-Trained Language Models to Multiple Languages Using Cross-Lingual Transfer](https://aclanthology.org/2026.nslp-1.26/) (Rauscher et al., NSLP 2026)

• Transferring Scientific English Pre-Trained Language Models to Multiple Languages Using Cross-Lingual Transfer (Rauscher et al., NSLP 2026)

ACL

• Nikolas Ching-Pu Rauscher, Fabio Barth, and Georg Rehm. 2026. Transferring Scientific English Pre-Trained Language Models to Multiple Languages Using Cross-Lingual Transfer. In Proceedings of Natural Scientific Language Processing (NSLP) @ LREC 2026, pages 261–269, Palma, Mallorca (Spain). ELRA Language Resources Association (ELRA).