<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="vu-etal-2023-koala">
    <titleInfo>
        <title>Koala: An Index for Quantifying Overlaps with Pre-training Corpora</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Thuy-Trang</namePart>
        <namePart type="family">Vu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xuanli</namePart>
        <namePart type="family">He</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gholamreza</namePart>
        <namePart type="family">Haffari</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ehsan</namePart>
        <namePart type="family">Shareghi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yansong</namePart>
            <namePart type="family">Feng</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Els</namePart>
            <namePart type="family">Lefever</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In very recent years more attention has been placed on probing the role of pre-training data in Large Language Models (LLMs) downstream behaviour. Despite the importance, there is no public tool that supports such analysis of pre-training corpora at large scale. To help research in this space, we launch Koala, a searchable index over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. In its first release we index the public proportion of OPT 175B, GPT-3, GPT-Neo, GPT-Neo, LLaMA, BERT, ELECTRA, RoBERTA, XLNet pre-training corpora. Koala provides a framework to do forensic analysis on the current and future benchmarks as well as to assess the degree of memorization in the output from the LLMs. Koala is available for public use at https://koala-index.erc.monash.edu/.</abstract>
    <identifier type="citekey">vu-etal-2023-koala</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-demo.7</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-demo.7</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>90</start>
            <end>98</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Koala: An Index for Quantifying Overlaps with Pre-training Corpora
%A Vu, Thuy-Trang
%A He, Xuanli
%A Haffari, Gholamreza
%A Shareghi, Ehsan
%Y Feng, Yansong
%Y Lefever, Els
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F vu-etal-2023-koala
%X In very recent years more attention has been placed on probing the role of pre-training data in Large Language Models (LLMs) downstream behaviour. Despite the importance, there is no public tool that supports such analysis of pre-training corpora at large scale. To help research in this space, we launch Koala, a searchable index over large pre-training corpora using lossless compressed suffix arrays with highly efficient compression rate and search support. In its first release we index the public proportion of OPT 175B, GPT-3, GPT-Neo, GPT-Neo, LLaMA, BERT, ELECTRA, RoBERTA, XLNet pre-training corpora. Koala provides a framework to do forensic analysis on the current and future benchmarks as well as to assess the degree of memorization in the output from the LLMs. Koala is available for public use at https://koala-index.erc.monash.edu/.
%R 10.18653/v1/2023.emnlp-demo.7
%U https://aclanthology.org/2023.emnlp-demo.7
%U https://doi.org/10.18653/v1/2023.emnlp-demo.7
%P 90-98

Markdown (Informal)

[Koala: An Index for Quantifying Overlaps with Pre-training Corpora](https://aclanthology.org/2023.emnlp-demo.7) (Vu et al., EMNLP 2023)

• Koala: An Index for Quantifying Overlaps with Pre-training Corpora (Vu et al., EMNLP 2023)

ACL

• Thuy-Trang Vu, Xuanli He, Gholamreza Haffari, and Ehsan Shareghi. 2023. Koala: An Index for Quantifying Overlaps with Pre-training Corpora. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 90–98, Singapore. Association for Computational Linguistics.