<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ishihara-takahashi-2024-quantifying-memorization">
    <titleInfo>
        <title>Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspaper</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Shotaro</namePart>
        <namePart type="family">Ishihara</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hiromu</namePart>
        <namePart type="family">Takahashi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 17th International Natural Language Generation Conference</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Saad</namePart>
            <namePart type="family">Mahamood</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Nguyen</namePart>
            <namePart type="given">Le</namePart>
            <namePart type="family">Minh</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Daphne</namePart>
            <namePart type="family">Ippolito</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Tokyo, Japan</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Dominant pre-trained language models (PLMs) have demonstrated the potential risk of memorizing and outputting the training data. While this concern has been discussed mainly in English, it is also practically important to focus on domain-specific PLMs. In this study, we pre-trained domain-specific GPT-2 models using a limited corpus of Japanese newspaper articles and evaluated their behavior. Experiments replicated the empirical finding that memorization of PLMs is related to the duplication in the training data, model size, and prompt length, in Japanese the same as in previous English studies. Furthermore, we attempted membership inference attacks, demonstrating that the training data can be detected even in Japanese, which is the same trend as in English. The study warns that domain-specific PLMs, sometimes trained with valuable private data, can ”copy and paste” on a large scale.</abstract>
    <identifier type="citekey">ishihara-takahashi-2024-quantifying-memorization</identifier>
    <location>
        <url>https://aclanthology.org/2024.inlg-main.14</url>
    </location>
    <part>
        <date>2024-09</date>
        <extent unit="page">
            <start>165</start>
            <end>179</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspaper
%A Ishihara, Shotaro
%A Takahashi, Hiromu
%Y Mahamood, Saad
%Y Minh, Nguyen Le
%Y Ippolito, Daphne
%S Proceedings of the 17th International Natural Language Generation Conference
%D 2024
%8 September
%I Association for Computational Linguistics
%C Tokyo, Japan
%F ishihara-takahashi-2024-quantifying-memorization
%X Dominant pre-trained language models (PLMs) have demonstrated the potential risk of memorizing and outputting the training data. While this concern has been discussed mainly in English, it is also practically important to focus on domain-specific PLMs. In this study, we pre-trained domain-specific GPT-2 models using a limited corpus of Japanese newspaper articles and evaluated their behavior. Experiments replicated the empirical finding that memorization of PLMs is related to the duplication in the training data, model size, and prompt length, in Japanese the same as in previous English studies. Furthermore, we attempted membership inference attacks, demonstrating that the training data can be detected even in Japanese, which is the same trend as in English. The study warns that domain-specific PLMs, sometimes trained with valuable private data, can ”copy and paste” on a large scale.
%U https://aclanthology.org/2024.inlg-main.14
%P 165-179

Markdown (Informal)

[Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspaper](https://aclanthology.org/2024.inlg-main.14) (Ishihara & Takahashi, INLG 2024)

• Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspaper (Ishihara & Takahashi, INLG 2024)

ACL

• Shotaro Ishihara and Hiromu Takahashi. 2024. Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspaper. In Proceedings of the 17th International Natural Language Generation Conference, pages 165–179, Tokyo, Japan. Association for Computational Linguistics.