<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="szep-etal-2026-fine">
    <titleInfo>
        <title>Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Marton</namePart>
        <namePart type="family">Szep</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Daniel</namePart>
        <namePart type="family">Rueckert</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rüdiger</namePart>
        <namePart type="family">von Eisenhart-Rothe</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Florian</namePart>
        <namePart type="family">Hinterwimmer</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Transactions of the Association for Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Fine-tuning large language models (LLMs) with limited data poses a practical challenge in low-resource languages, specialized domains, and constrained deployment settings. While pre-trained LLMs provide strong foundations, effective adaptation under data scarcity requires focused and efficient fine-tuning techniques. This paper presents a structured and practical survey of recent methods for fine-tuning LLMs in data-scarce scenarios. We systematically review parameter-efficient fine-tuning techniques that lower training and deployment costs, domain and cross-lingual adaptation methods for both encoder and decoder models, and model specialization strategies. We further examine preference alignment approaches that guide model behavior using limited human or synthetic feedback, emphasizing sample and compute efficiency. Throughout, we highlight empirical trade-offs, selection criteria, and best practices for choosing suitable techniques based on task constraints, including model scaling, data scaling, and the mitigation of catastrophic forgetting. The aim is to equip researchers and practitioners with actionable insights for effectively fine-tuning LLMs when data and resources are limited.</abstract>
    <identifier type="citekey">szep-etal-2026-fine</identifier>
    <identifier type="doi">10.1162/tacl.a.627</identifier>
    <location>
        <url>https://aclanthology.org/2026.tacl-1.17/</url>
    </location>
    <part>
        <date>2026</date>
        <detail type="volume"><number>14</number></detail>
        <extent unit="page">
            <start>341</start>
            <end>377</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide
%A Szep, Marton
%A Rueckert, Daniel
%A von Eisenhart-Rothe, Rüdiger
%A Hinterwimmer, Florian
%J Transactions of the Association for Computational Linguistics
%D 2026
%V 14
%I MIT Press
%C Cambridge, MA
%F szep-etal-2026-fine
%X Fine-tuning large language models (LLMs) with limited data poses a practical challenge in low-resource languages, specialized domains, and constrained deployment settings. While pre-trained LLMs provide strong foundations, effective adaptation under data scarcity requires focused and efficient fine-tuning techniques. This paper presents a structured and practical survey of recent methods for fine-tuning LLMs in data-scarce scenarios. We systematically review parameter-efficient fine-tuning techniques that lower training and deployment costs, domain and cross-lingual adaptation methods for both encoder and decoder models, and model specialization strategies. We further examine preference alignment approaches that guide model behavior using limited human or synthetic feedback, emphasizing sample and compute efficiency. Throughout, we highlight empirical trade-offs, selection criteria, and best practices for choosing suitable techniques based on task constraints, including model scaling, data scaling, and the mitigation of catastrophic forgetting. The aim is to equip researchers and practitioners with actionable insights for effectively fine-tuning LLMs when data and resources are limited.
%R 10.1162/tacl.a.627
%U https://aclanthology.org/2026.tacl-1.17/
%U https://doi.org/10.1162/tacl.a.627
%P 341-377

Markdown (Informal)

[Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide](https://aclanthology.org/2026.tacl-1.17/) (Szep et al., TACL 2026)

• Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide (Szep et al., TACL 2026)

ACL

• Marton Szep, Daniel Rueckert, Rüdiger von Eisenhart-Rothe, and Florian Hinterwimmer. 2026. Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide. Transactions of the Association for Computational Linguistics, 14:341–377.