From Teacher to Student: Tracking Memorization Through Model Distillation

Simardeep Singh

doi:10.18653/v1/2025.l2m2-1.6

From Teacher to Student: Tracking Memorization Through Model Distillation

Correct Metadata for

Important: The Anthology treat PDFs as authoritative. Please use this form only to correct data that is out of line with the PDF. See our corrections guidelines if you need to change the PDF.

Title Adjust the title. Retain tags such as <fixed-case>.

Authors Adjust author names and order to match the PDF.

Abstract Correct abstract if needed. Retain XML formatting tags such as <tex-math>.

Verification against PDF Ensure that the new title/authors match the snapshot below. (If there is no snapshot or it is too small, consult the PDF.)

Authors concatenated from the text boxes above:

ALL author names match the snapshot above—including middle initials, hyphens, and accents.

Abstract

Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security. While previous research has focused on studying memorization in pre-trained models, much less is known about how knowledge distillation (KD) affects memorization.In this study, we explore how different KD methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.This study demonstrates that distilling a larger teacher model, fine-tuned on a dataset, into a smaller variant not only lowers computational costs and model size but also significantly reduces the memorization risks compared to standard fine-tuning approaches.

Anthology ID:: 2025.l2m2-1.6
Volume:: Proceedings of the First Workshop on Large Language Model Memorization (L2M2)
Month:: August
Year:: 2025
Address:: Vienna, Austria
Editors:: Robin Jia, Eric Wallace, Yangsibo Huang, Tiago Pimentel, Pratyush Maini, Verna Dankers, Johnny Wei, Pietro Lesci
Venues:: L2M2 | WS
SIG:
Publisher:: Association for Computational Linguistics
Note:
Pages:: 78–82
Language:
URL:: https://aclanthology.org/2025.l2m2-1.6/
DOI:: 10.18653/v1/2025.l2m2-1.6
Bibkey:
Cite (ACL):: Simardeep Singh. 2025. From Teacher to Student: Tracking Memorization Through Model Distillation. In Proceedings of the First Workshop on Large Language Model Memorization (L2M2), pages 78–82, Vienna, Austria. Association for Computational Linguistics.
Cite (Informal):: From Teacher to Student: Tracking Memorization Through Model Distillation (Singh, L2M2 2025)
Copy Citation:
PDF:: https://aclanthology.org/2025.l2m2-1.6.pdf

PDF Cite Search Fix data

Export citation

BibTeX
MODS XML
Endnote
Preformatted

@inproceedings{singh-2025-teacher,
    title = "From Teacher to Student: Tracking Memorization Through Model Distillation",
    author = "Singh, Simardeep",
    editor = "Jia, Robin  and
      Wallace, Eric  and
      Huang, Yangsibo  and
      Pimentel, Tiago  and
      Maini, Pratyush  and
      Dankers, Verna  and
      Wei, Johnny  and
      Lesci, Pietro",
    booktitle = "Proceedings of the First Workshop on Large Language Model Memorization (L2M2)",
    month = aug,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.l2m2-1.6/",
    doi = "10.18653/v1/2025.l2m2-1.6",
    pages = "78--82",
    ISBN = "979-8-89176-278-7",
    abstract = "Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security. While previous research has focused on studying memorization in pre-trained models, much less is known about how knowledge distillation (KD) affects memorization.In this study, we explore how different KD methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.This study demonstrates that distilling a larger teacher model, fine-tuned on a dataset, into a smaller variant not only lowers computational costs and model size but also significantly reduces the memorization risks compared to standard fine-tuning approaches."
}

Download as File

<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="singh-2025-teacher">
    <titleInfo>
        <title>From Teacher to Student: Tracking Memorization Through Model Distillation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Simardeep</namePart>
        <namePart type="family">Singh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the First Workshop on Large Language Model Memorization (L2M2)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Robin</namePart>
            <namePart type="family">Jia</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Eric</namePart>
            <namePart type="family">Wallace</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yangsibo</namePart>
            <namePart type="family">Huang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tiago</namePart>
            <namePart type="family">Pimentel</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Pratyush</namePart>
            <namePart type="family">Maini</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Verna</namePart>
            <namePart type="family">Dankers</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Johnny</namePart>
            <namePart type="family">Wei</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Pietro</namePart>
            <namePart type="family">Lesci</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Vienna, Austria</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-278-7</identifier>
    </relatedItem>
    <abstract>Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security. While previous research has focused on studying memorization in pre-trained models, much less is known about how knowledge distillation (KD) affects memorization.In this study, we explore how different KD methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.This study demonstrates that distilling a larger teacher model, fine-tuned on a dataset, into a smaller variant not only lowers computational costs and model size but also significantly reduces the memorization risks compared to standard fine-tuning approaches.</abstract>
    <identifier type="citekey">singh-2025-teacher</identifier>
    <identifier type="doi">10.18653/v1/2025.l2m2-1.6</identifier>
    <location>
        <url>https://aclanthology.org/2025.l2m2-1.6/</url>
    </location>
    <part>
        <date>2025-08</date>
        <extent unit="page">
            <start>78</start>
            <end>82</end>
        </extent>
    </part>
</mods>
</modsCollection>

Download as File

%0 Conference Proceedings
%T From Teacher to Student: Tracking Memorization Through Model Distillation
%A Singh, Simardeep
%Y Jia, Robin
%Y Wallace, Eric
%Y Huang, Yangsibo
%Y Pimentel, Tiago
%Y Maini, Pratyush
%Y Dankers, Verna
%Y Wei, Johnny
%Y Lesci, Pietro
%S Proceedings of the First Workshop on Large Language Model Memorization (L2M2)
%D 2025
%8 August
%I Association for Computational Linguistics
%C Vienna, Austria
%@ 979-8-89176-278-7
%F singh-2025-teacher
%X Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security. While previous research has focused on studying memorization in pre-trained models, much less is known about how knowledge distillation (KD) affects memorization.In this study, we explore how different KD methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.This study demonstrates that distilling a larger teacher model, fine-tuned on a dataset, into a smaller variant not only lowers computational costs and model size but also significantly reduces the memorization risks compared to standard fine-tuning approaches.
%R 10.18653/v1/2025.l2m2-1.6
%U https://aclanthology.org/2025.l2m2-1.6/
%U https://doi.org/10.18653/v1/2025.l2m2-1.6
%P 78-82

Download as File

Markdown (Informal)

[From Teacher to Student: Tracking Memorization Through Model Distillation](https://aclanthology.org/2025.l2m2-1.6/) (Singh, L2M2 2025)

From Teacher to Student: Tracking Memorization Through Model Distillation (Singh, L2M2 2025)

ACL

Simardeep Singh. 2025. From Teacher to Student: Tracking Memorization Through Model Distillation. In Proceedings of the First Workshop on Large Language Model Memorization (L2M2), pages 78–82, Vienna, Austria. Association for Computational Linguistics.