<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kunz-2025-train">
    <titleInfo>
        <title>Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jenny</namePart>
        <namePart type="family">Kunz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Richard</namePart>
            <namePart type="family">Johansson</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sara</namePart>
            <namePart type="family">Stymne</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>University of Tartu Library</publisher>
            <place>
                <placeTerm type="text">Tallinn, Estonia</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">978-9908-53-109-0</identifier>
    </relatedItem>
    <abstract>Smaller LLMs still face significant challenges even in medium-resourced languages, particularly when it comes to language-specific knowledge – a problem not easily resolved with machine-translated data. In this case study on Icelandic, we aim to enhance the generation performance of an LLM by specialising it using unstructured text corpora. A key focus is on preventing interference with the models’ capabilities of handling longer context during this adaptation. Through ablation studies using various parameter-efficient fine-tuning (PEFT) methods and setups, we find that increasing the number of trainable parameters leads to better and more robust language adaptation. LoRAs placed in the feed-forward layers and bottleneck adapters show promising results with sufficient parameters, while prefix tuning and (IA)³ are not suitable. Although improvements are consistent in 0-shot summarisation, some adapted models struggle with longer context lengths, an issue that can be mitigated by adapting only the final layers.</abstract>
    <identifier type="citekey">kunz-2025-train</identifier>
    <location>
        <url>https://aclanthology.org/2025.nodalida-1.35/</url>
    </location>
    <part>
        <date>2025-03</date>
        <extent unit="page">
            <start>323</start>
            <end>330</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT
%A Kunz, Jenny
%Y Johansson, Richard
%Y Stymne, Sara
%S Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)
%D 2025
%8 March
%I University of Tartu Library
%C Tallinn, Estonia
%@ 978-9908-53-109-0
%F kunz-2025-train
%X Smaller LLMs still face significant challenges even in medium-resourced languages, particularly when it comes to language-specific knowledge – a problem not easily resolved with machine-translated data. In this case study on Icelandic, we aim to enhance the generation performance of an LLM by specialising it using unstructured text corpora. A key focus is on preventing interference with the models’ capabilities of handling longer context during this adaptation. Through ablation studies using various parameter-efficient fine-tuning (PEFT) methods and setups, we find that increasing the number of trainable parameters leads to better and more robust language adaptation. LoRAs placed in the feed-forward layers and bottleneck adapters show promising results with sufficient parameters, while prefix tuning and (IA)³ are not suitable. Although improvements are consistent in 0-shot summarisation, some adapted models struggle with longer context lengths, an issue that can be mitigated by adapting only the final layers.
%U https://aclanthology.org/2025.nodalida-1.35/
%P 323-330

Markdown (Informal)

[Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT](https://aclanthology.org/2025.nodalida-1.35/) (Kunz, NoDaLiDa 2025)

• Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT (Kunz, NoDaLiDa 2025)

ACL

• Jenny Kunz. 2025. Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 323–330, Tallinn, Estonia. University of Tartu Library.