<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="mehrafarin-etal-2022-importance">
    <titleInfo>
        <title>On the Importance of Data Size in Probing Fine-tuned Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Houman</namePart>
        <namePart type="family">Mehrafarin</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sara</namePart>
        <namePart type="family">Rajaee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mohammad</namePart>
        <namePart type="given">Taher</namePart>
        <namePart type="family">Pilehvar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2022</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dublin, Ireland</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. However, these studies often neglect the role of the size of the dataset on which the model is fine-tuned. In this paper, we highlight the importance of this factor and its undeniable role in probing performance. We show that the extent of encoded linguistic knowledge depends on the number of fine-tuning samples. The analysis also reveals that larger training data mainly affects higher layers, and that the extent of this change is a factor of the number of iterations updating the model during fine-tuning rather than the diversity of the training samples. Finally, we show through a set of experiments that fine-tuning data size affects the recoverability of the changes made to the model’s linguistic knowledge.</abstract>
    <identifier type="citekey">mehrafarin-etal-2022-importance</identifier>
    <identifier type="doi">10.18653/v1/2022.findings-acl.20</identifier>
    <location>
        <url>https://aclanthology.org/2022.findings-acl.20</url>
    </location>
    <part>
        <date>2022-05</date>
        <extent unit="page">
            <start>228</start>
            <end>238</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T On the Importance of Data Size in Probing Fine-tuned Models
%A Mehrafarin, Houman
%A Rajaee, Sara
%A Pilehvar, Mohammad Taher
%S Findings of the Association for Computational Linguistics: ACL 2022
%D 2022
%8 May
%I Association for Computational Linguistics
%C Dublin, Ireland
%F mehrafarin-etal-2022-importance
%X Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. However, these studies often neglect the role of the size of the dataset on which the model is fine-tuned. In this paper, we highlight the importance of this factor and its undeniable role in probing performance. We show that the extent of encoded linguistic knowledge depends on the number of fine-tuning samples. The analysis also reveals that larger training data mainly affects higher layers, and that the extent of this change is a factor of the number of iterations updating the model during fine-tuning rather than the diversity of the training samples. Finally, we show through a set of experiments that fine-tuning data size affects the recoverability of the changes made to the model’s linguistic knowledge.
%R 10.18653/v1/2022.findings-acl.20
%U https://aclanthology.org/2022.findings-acl.20
%U https://doi.org/10.18653/v1/2022.findings-acl.20
%P 228-238

Markdown (Informal)

[On the Importance of Data Size in Probing Fine-tuned Models](https://aclanthology.org/2022.findings-acl.20) (Mehrafarin et al., Findings 2022)

• On the Importance of Data Size in Probing Fine-tuned Models (Mehrafarin et al., Findings 2022)

ACL

• Houman Mehrafarin, Sara Rajaee, and Mohammad Taher Pilehvar. 2022. On the Importance of Data Size in Probing Fine-tuned Models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 228–238, Dublin, Ireland. Association for Computational Linguistics.