<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="hidayat-etal-2025-simulating">
    <titleInfo>
        <title>Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Naila</namePart>
        <namePart type="given">Shafirni</namePart>
        <namePart type="family">Hidayat</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Muhammad</namePart>
        <namePart type="given">Dehan</namePart>
        <namePart type="family">Al Kautsar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alfan</namePart>
        <namePart type="given">Farizki</namePart>
        <namePart type="family">Wicaksono</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Fajri</namePart>
        <namePart type="family">Koto</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Mousumi</namePart>
            <namePart type="family">Akter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Tahiya</namePart>
            <namePart type="family">Chowdhury</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Steffen</namePart>
            <namePart type="family">Eger</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Christoph</namePart>
            <namePart type="family">Leiter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juri</namePart>
            <namePart type="family">Opitz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Erion</namePart>
            <namePart type="family">Çano</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Mumbai, India</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-305-0</identifier>
    </relatedItem>
    <abstract>The performance of large language models (LLMs) continues to improve, as reflected in rising scores on standard benchmarks. However, the lack of transparency around training data raises concerns about potential overlap with evaluation sets and the fairness of reported results. Although prior work has proposed methods for detecting data leakage, these approaches primarily focus on identifying outliers and have not been evaluated under controlled simulated leakage conditions. In this work, we compare existing leakage detection techniques, namely permutation and n-gram-based methods, under a continual pretraining setup that simulates real-world leakage scenarios, and additionally explore a lightweight method we call semi-half question. We further introduce two efficient extensions, permutation-R and permutation-Q. While semi-half offers a low-cost alternative, our analysis shows that the n-gram method consistently achieves the highest F1-Score, performing competitively with permutation-Q. We also refine these techniques to support instance-level detection and reduce computational overhead. Leveraging the best-performing method, we create cleaned versions of MMLU and HellaSwag, and re-evaluate several LLMs. Our findings present a practical path toward more reliable and transparent evaluations, and we recommend contamination checks as a standard practice before releasing benchmark results.</abstract>
    <identifier type="citekey">hidayat-etal-2025-simulating</identifier>
    <location>
        <url>https://aclanthology.org/2025.eval4nlp-1.3/</url>
    </location>
    <part>
        <date>2025-12</date>
        <extent unit="page">
            <start>21</start>
            <end>39</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation
%A Hidayat, Naila Shafirni
%A Al Kautsar, Muhammad Dehan
%A Wicaksono, Alfan Farizki
%A Koto, Fajri
%Y Akter, Mousumi
%Y Chowdhury, Tahiya
%Y Eger, Steffen
%Y Leiter, Christoph
%Y Opitz, Juri
%Y Çano, Erion
%S Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems
%D 2025
%8 December
%I Association for Computational Linguistics
%C Mumbai, India
%@ 979-8-89176-305-0
%F hidayat-etal-2025-simulating
%X The performance of large language models (LLMs) continues to improve, as reflected in rising scores on standard benchmarks. However, the lack of transparency around training data raises concerns about potential overlap with evaluation sets and the fairness of reported results. Although prior work has proposed methods for detecting data leakage, these approaches primarily focus on identifying outliers and have not been evaluated under controlled simulated leakage conditions. In this work, we compare existing leakage detection techniques, namely permutation and n-gram-based methods, under a continual pretraining setup that simulates real-world leakage scenarios, and additionally explore a lightweight method we call semi-half question. We further introduce two efficient extensions, permutation-R and permutation-Q. While semi-half offers a low-cost alternative, our analysis shows that the n-gram method consistently achieves the highest F1-Score, performing competitively with permutation-Q. We also refine these techniques to support instance-level detection and reduce computational overhead. Leveraging the best-performing method, we create cleaned versions of MMLU and HellaSwag, and re-evaluate several LLMs. Our findings present a practical path toward more reliable and transparent evaluations, and we recommend contamination checks as a standard practice before releasing benchmark results.
%U https://aclanthology.org/2025.eval4nlp-1.3/
%P 21-39

Markdown (Informal)

[Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation](https://aclanthology.org/2025.eval4nlp-1.3/) (Hidayat et al., Eval4NLP 2025)

• Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation (Hidayat et al., Eval4NLP 2025)

ACL

• Naila Shafirni Hidayat, Muhammad Dehan Al Kautsar, Alfan Farizki Wicaksono, and Fajri Koto. 2025. Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation. In Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems, pages 21–39, Mumbai, India. Association for Computational Linguistics.