<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yauney-etal-2023-data">
    <titleInfo>
        <title>Data Similarity is Not Enough to Explain Language Model Performance</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Gregory</namePart>
        <namePart type="family">Yauney</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Emily</namePart>
        <namePart type="family">Reif</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">David</namePart>
        <namePart type="family">Mimno</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Large language models achieve high performance on many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this variance: a task with data that is more similar to a model’s pretraining data is assumed to be easier for that model. We test whether distributional and example-specific similarity measures (embedding-, token- and model-based) correlate with language model performance through a large-scale comparison of the Pile and C4 pretraining datasets with downstream benchmarks. Similarity correlates with performance for multilingual datasets, but in other benchmarks, we surprisingly find that similarity metrics are not correlated with accuracy or even each other. This suggests that the relationship between pretraining data and downstream tasks is more complex than often assumed.</abstract>
    <identifier type="citekey">yauney-etal-2023-data</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.695</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.695</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>11295</start>
            <end>11304</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Data Similarity is Not Enough to Explain Language Model Performance
%A Yauney, Gregory
%A Reif, Emily
%A Mimno, David
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F yauney-etal-2023-data
%X Large language models achieve high performance on many but not all downstream tasks. The interaction between pretraining data and task data is commonly assumed to determine this variance: a task with data that is more similar to a model’s pretraining data is assumed to be easier for that model. We test whether distributional and example-specific similarity measures (embedding-, token- and model-based) correlate with language model performance through a large-scale comparison of the Pile and C4 pretraining datasets with downstream benchmarks. Similarity correlates with performance for multilingual datasets, but in other benchmarks, we surprisingly find that similarity metrics are not correlated with accuracy or even each other. This suggests that the relationship between pretraining data and downstream tasks is more complex than often assumed.
%R 10.18653/v1/2023.emnlp-main.695
%U https://aclanthology.org/2023.emnlp-main.695
%U https://doi.org/10.18653/v1/2023.emnlp-main.695
%P 11295-11304

Markdown (Informal)

[Data Similarity is Not Enough to Explain Language Model Performance](https://aclanthology.org/2023.emnlp-main.695) (Yauney et al., EMNLP 2023)

• Data Similarity is Not Enough to Explain Language Model Performance (Yauney et al., EMNLP 2023)

ACL

• Gregory Yauney, Emily Reif, and David Mimno. 2023. Data Similarity is Not Enough to Explain Language Model Performance. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11295–11304, Singapore. Association for Computational Linguistics.