<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kirstain-etal-2022-examples">
    <titleInfo>
        <title>A Few More Examples May Be Worth Billions of Parameters</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yuval</namePart>
        <namePart type="family">Kirstain</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Patrick</namePart>
        <namePart type="family">Lewis</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sebastian</namePart>
        <namePart type="family">Riedel</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Omer</namePart>
        <namePart type="family">Levy</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2022</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yoav</namePart>
            <namePart type="family">Goldberg</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Zornitsa</namePart>
            <namePart type="family">Kozareva</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yue</namePart>
            <namePart type="family">Zhang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, United Arab Emirates</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>We investigate the dynamics of increasing the number of model parameters versus the number of labeled examples across a wide variety of tasks. Our exploration reveals that while scaling parameters consistently yields performance improvements, the contribution of additional examples highly depends on the task’s format. Specifically, in open question answering tasks, enlarging the training set does not improve performance. In contrast, classification, extractive question answering, and multiple choice tasks benefit so much from additional examples that collecting a few hundred examples is often “worth” billions of parameters. We hypothesize that unlike open question answering, which involves recalling specific information, solving strategies for tasks with a more restricted output space transfer across examples, and can therefore be learned with small amounts of labeled data.</abstract>
    <identifier type="citekey">kirstain-etal-2022-examples</identifier>
    <identifier type="doi">10.18653/v1/2022.findings-emnlp.72</identifier>
    <location>
        <url>https://aclanthology.org/2022.findings-emnlp.72</url>
    </location>
    <part>
        <date>2022-12</date>
        <extent unit="page">
            <start>1017</start>
            <end>1029</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T A Few More Examples May Be Worth Billions of Parameters
%A Kirstain, Yuval
%A Lewis, Patrick
%A Riedel, Sebastian
%A Levy, Omer
%Y Goldberg, Yoav
%Y Kozareva, Zornitsa
%Y Zhang, Yue
%S Findings of the Association for Computational Linguistics: EMNLP 2022
%D 2022
%8 December
%I Association for Computational Linguistics
%C Abu Dhabi, United Arab Emirates
%F kirstain-etal-2022-examples
%X We investigate the dynamics of increasing the number of model parameters versus the number of labeled examples across a wide variety of tasks. Our exploration reveals that while scaling parameters consistently yields performance improvements, the contribution of additional examples highly depends on the task’s format. Specifically, in open question answering tasks, enlarging the training set does not improve performance. In contrast, classification, extractive question answering, and multiple choice tasks benefit so much from additional examples that collecting a few hundred examples is often “worth” billions of parameters. We hypothesize that unlike open question answering, which involves recalling specific information, solving strategies for tasks with a more restricted output space transfer across examples, and can therefore be learned with small amounts of labeled data.
%R 10.18653/v1/2022.findings-emnlp.72
%U https://aclanthology.org/2022.findings-emnlp.72
%U https://doi.org/10.18653/v1/2022.findings-emnlp.72
%P 1017-1029

Markdown (Informal)

[A Few More Examples May Be Worth Billions of Parameters](https://aclanthology.org/2022.findings-emnlp.72) (Kirstain et al., Findings 2022)

• A Few More Examples May Be Worth Billions of Parameters (Kirstain et al., Findings 2022)

ACL

• Yuval Kirstain, Patrick Lewis, Sebastian Riedel, and Omer Levy. 2022. A Few More Examples May Be Worth Billions of Parameters. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1017–1029, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.