<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ueda-etal-2026-merging">
    <titleInfo>
        <title>Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Kentaro</namePart>
        <namePart type="family">Ueda</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">François</namePart>
        <namePart type="family">Portet</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hirohiko</namePart>
        <namePart type="family">Suwa</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Keiichi</namePart>
        <namePart type="family">Yasumoto</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the Fifteenth Language Resources and Evaluation Conference</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Stelios</namePart>
            <namePart type="family">Piperidis</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Núria</namePart>
            <namePart type="family">Bel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Henk</namePart>
            <namePart type="family">van den Heuvel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nancy</namePart>
            <namePart type="family">Ide</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Simon</namePart>
            <namePart type="family">Krek</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Antonio</namePart>
            <namePart type="family">Toral</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>ELRA Language Resource Association</publisher>
            <place>
                <placeTerm type="text">Palma de Mallorca, Spain</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>While LLMs excel at general tasks, they struggle in specialized domains like finance, requiring diverse skills in domain knowledge, mathematical reasoning, and multilingual processing. Merging domain-specific Continual Pre-training (CPT) “experts” offers a practical alternative to costly and unstable multi-skill training. However, unlike established Supervised Fine-Tuning (SFT) model-based merging, CPT model merging remains largely unexplored. We address this gap by creating financial LLMs from experts in finance, math, and Japanese. We propose a three-stage evaluation focusing on knowledge recovery, complementarity, and emergence, and assess three merging methods (Task Arithmetic, TIES, and DARE-TIES) on a comprehensive financial benchmark curated from 18 tasks across 8 established datasets. Results show that merging an expert with its base model recovers general knowledge lost during CPT, while merging experts improves performance and can yield emergent cross-domain skills. Among the methods, Task Arithmetic performs strongly but is hyperparameter-sensitive, whereas TIES is more robust. Our findings also suggest that while model similarity correlates with merging success, emergent skills depend on more complex factors. This work presents the first foundational analysis of CPT model merging, establishing a principled framework and providing clear guidance for building multi-skill LLMs from existing assets.</abstract>
    <identifier type="citekey">ueda-etal-2026-merging</identifier>
    <identifier type="doi">10.63317/3cyxer74uvyp</identifier>
    <location>
        <url>https://aclanthology.org/2026.lrec-1.794/</url>
    </location>
    <part>
        <date>2026-05</date>
        <extent unit="page">
            <start>10114</start>
            <end>10129</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance
%A Ueda, Kentaro
%A Portet, François
%A Suwa, Hirohiko
%A Yasumoto, Keiichi
%Y Piperidis, Stelios
%Y Bel, Núria
%Y van den Heuvel, Henk
%Y Ide, Nancy
%Y Krek, Simon
%Y Toral, Antonio
%S Proceedings of the Fifteenth Language Resources and Evaluation Conference
%D 2026
%8 May
%I ELRA Language Resource Association
%C Palma de Mallorca, Spain
%F ueda-etal-2026-merging
%X While LLMs excel at general tasks, they struggle in specialized domains like finance, requiring diverse skills in domain knowledge, mathematical reasoning, and multilingual processing. Merging domain-specific Continual Pre-training (CPT) “experts” offers a practical alternative to costly and unstable multi-skill training. However, unlike established Supervised Fine-Tuning (SFT) model-based merging, CPT model merging remains largely unexplored. We address this gap by creating financial LLMs from experts in finance, math, and Japanese. We propose a three-stage evaluation focusing on knowledge recovery, complementarity, and emergence, and assess three merging methods (Task Arithmetic, TIES, and DARE-TIES) on a comprehensive financial benchmark curated from 18 tasks across 8 established datasets. Results show that merging an expert with its base model recovers general knowledge lost during CPT, while merging experts improves performance and can yield emergent cross-domain skills. Among the methods, Task Arithmetic performs strongly but is hyperparameter-sensitive, whereas TIES is more robust. Our findings also suggest that while model similarity correlates with merging success, emergent skills depend on more complex factors. This work presents the first foundational analysis of CPT model merging, establishing a principled framework and providing clear guidance for building multi-skill LLMs from existing assets.
%R 10.63317/3cyxer74uvyp
%U https://aclanthology.org/2026.lrec-1.794/
%U https://doi.org/10.63317/3cyxer74uvyp
%P 10114-10129

Markdown (Informal)

[Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance](https://aclanthology.org/2026.lrec-1.794/) (Ueda et al., LREC 2026)

• Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance (Ueda et al., LREC 2026)

ACL

• Kentaro Ueda, François Portet, Hirohiko Suwa, and Keiichi Yasumoto. 2026. Merging Continual Pretraining Models for Domain-Specialized LLMs: A Case Study in Finance. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 10114–10129, Palma de Mallorca, Spain. ELRA Language Resource Association.