<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="curci-etal-2026-practical">
    <titleInfo>
        <title>Practical Guidelines for Model Merging in LLMs Pre-Training</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Giuseppe</namePart>
        <namePart type="family">Curci</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Stefano</namePart>
        <namePart type="family">Simonazzi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Andrea</namePart>
        <namePart type="family">molinari</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Andrea</namePart>
        <namePart type="family">Zugarini</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yunyao</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Georg</namePart>
            <namePart type="family">Rehm</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mei</namePart>
            <namePart type="family">Tu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">San Diego, California, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-394-4</identifier>
    </relatedItem>
    <abstract>Model merging is widely used to combine fine-tuned models trained with different data distributions, tasks, or hyperparameters, yet its role during LLM pre-training remains underexplored. We systematically study checkpoint merging across training phases, focusing on the transition from stable to decaying learning rates. Across multiple scales, we find that simple averaging methods consistently improve performance during stable learning rate regimes, but gains sharply diminish during decay. We link this effect to reduced checkpoint diversity and show that merging effectiveness correlates with parameter-space variation. Strategies such as synthetic variability, task-vector merging, and cross-run merging yield only modest improvements. Our results provide practical insights on when merging is most effective in large-scale pre-training.</abstract>
    <identifier type="citekey">curci-etal-2026-practical</identifier>
    <location>
        <url>https://aclanthology.org/2026.acl-industry.105/</url>
    </location>
    <part>
        <date>2026-07</date>
        <extent unit="page">
            <start>1519</start>
            <end>1532</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Practical Guidelines for Model Merging in LLMs Pre-Training
%A Curci, Giuseppe
%A Simonazzi, Stefano
%A molinari, Andrea
%A Zugarini, Andrea
%Y Li, Yunyao
%Y Rehm, Georg
%Y Tu, Mei
%S Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026)
%D 2026
%8 July
%I Association for Computational Linguistics
%C San Diego, California, USA
%@ 979-8-89176-394-4
%F curci-etal-2026-practical
%X Model merging is widely used to combine fine-tuned models trained with different data distributions, tasks, or hyperparameters, yet its role during LLM pre-training remains underexplored. We systematically study checkpoint merging across training phases, focusing on the transition from stable to decaying learning rates. Across multiple scales, we find that simple averaging methods consistently improve performance during stable learning rate regimes, but gains sharply diminish during decay. We link this effect to reduced checkpoint diversity and show that merging effectiveness correlates with parameter-space variation. Strategies such as synthetic variability, task-vector merging, and cross-run merging yield only modest improvements. Our results provide practical insights on when merging is most effective in large-scale pre-training.
%U https://aclanthology.org/2026.acl-industry.105/
%P 1519-1532

Markdown (Informal)

[Practical Guidelines for Model Merging in LLMs Pre-Training](https://aclanthology.org/2026.acl-industry.105/) (Curci et al., ACL 2026)

• Practical Guidelines for Model Merging in LLMs Pre-Training (Curci et al., ACL 2026)

ACL

• Giuseppe Curci, Stefano Simonazzi, Andrea molinari, and Andrea Zugarini. 2026. Practical Guidelines for Model Merging in LLMs Pre-Training. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), pages 1519–1532, San Diego, California, USA. Association for Computational Linguistics.