<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="choshen-etal-2023-start">
    <titleInfo>
        <title>Where to start? Analyzing the potential value of intermediate models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Leshem</namePart>
        <namePart type="family">Choshen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Elad</namePart>
        <namePart type="family">Venezian</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Shachar</namePart>
        <namePart type="family">Don-Yehiya</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Noam</namePart>
        <namePart type="family">Slonim</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yoav</namePart>
        <namePart type="family">Katz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset. Here, we perform a systematic analysis of this intertraining scheme, over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed independently for the target dataset under consideration, and for a base model being considered as a starting point. Hence, a performant model is generally strong, even if its training data was not aligned with the target dataset. Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings. Last, we release an updating ranking of best models in the HuggingFace hub per architecture.</abstract>
    <identifier type="citekey">choshen-etal-2023-start</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.90</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.90</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>1446</start>
            <end>1470</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Where to start? Analyzing the potential value of intermediate models
%A Choshen, Leshem
%A Venezian, Elad
%A Don-Yehiya, Shachar
%A Slonim, Noam
%A Katz, Yoav
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F choshen-etal-2023-start
%X Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset. Here, we perform a systematic analysis of this intertraining scheme, over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed independently for the target dataset under consideration, and for a base model being considered as a starting point. Hence, a performant model is generally strong, even if its training data was not aligned with the target dataset. Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings. Last, we release an updating ranking of best models in the HuggingFace hub per architecture.
%R 10.18653/v1/2023.emnlp-main.90
%U https://aclanthology.org/2023.emnlp-main.90
%U https://doi.org/10.18653/v1/2023.emnlp-main.90
%P 1446-1470

Markdown (Informal)

[Where to start? Analyzing the potential value of intermediate models](https://aclanthology.org/2023.emnlp-main.90) (Choshen et al., EMNLP 2023)

• Where to start? Analyzing the potential value of intermediate models (Choshen et al., EMNLP 2023)

ACL

• Leshem Choshen, Elad Venezian, Shachar Don-Yehiya, Noam Slonim, and Yoav Katz. 2023. Where to start? Analyzing the potential value of intermediate models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1446–1470, Singapore. Association for Computational Linguistics.