<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kamoda-etal-2023-test">
    <titleInfo>
        <title>Test-time Augmentation for Factual Probing</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Go</namePart>
        <namePart type="family">Kamoda</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Benjamin</namePart>
        <namePart type="family">Heinzerling</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Keisuke</namePart>
        <namePart type="family">Sakaguchi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kentaro</namePart>
        <namePart type="family">Inui</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Factual probing is a method that uses prompts to test if a language model “knows” certain world knowledge facts. A problem in factual probing is that small changes to the prompt can lead to large changes in model output. Previous work aimed to alleviate this problem by optimizing prompts via text mining or fine-tuning. However, such approaches are relation-specific and do not generalize to unseen relation types. Here, we propose to use test-time augmentation (TTA) as a relation-agnostic method for reducing sensitivity to prompt variations by automatically augmenting and ensembling prompts at test time. Experiments show improved model calibration, i.e., with TTA, model confidence better reflects prediction accuracy. Improvements in prediction accuracy are observed for some models, but for other models, TTA leads to degradation. Error analysis identifies the difficulty of producing high-quality prompt variations as the main challenge for TTA.</abstract>
    <identifier type="citekey">kamoda-etal-2023-test</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.236</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.236</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>3650</start>
            <end>3661</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Test-time Augmentation for Factual Probing
%A Kamoda, Go
%A Heinzerling, Benjamin
%A Sakaguchi, Keisuke
%A Inui, Kentaro
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F kamoda-etal-2023-test
%X Factual probing is a method that uses prompts to test if a language model “knows” certain world knowledge facts. A problem in factual probing is that small changes to the prompt can lead to large changes in model output. Previous work aimed to alleviate this problem by optimizing prompts via text mining or fine-tuning. However, such approaches are relation-specific and do not generalize to unseen relation types. Here, we propose to use test-time augmentation (TTA) as a relation-agnostic method for reducing sensitivity to prompt variations by automatically augmenting and ensembling prompts at test time. Experiments show improved model calibration, i.e., with TTA, model confidence better reflects prediction accuracy. Improvements in prediction accuracy are observed for some models, but for other models, TTA leads to degradation. Error analysis identifies the difficulty of producing high-quality prompt variations as the main challenge for TTA.
%R 10.18653/v1/2023.findings-emnlp.236
%U https://aclanthology.org/2023.findings-emnlp.236
%U https://doi.org/10.18653/v1/2023.findings-emnlp.236
%P 3650-3661

Markdown (Informal)

[Test-time Augmentation for Factual Probing](https://aclanthology.org/2023.findings-emnlp.236) (Kamoda et al., Findings 2023)

• Test-time Augmentation for Factual Probing (Kamoda et al., Findings 2023)

ACL

• Go Kamoda, Benjamin Heinzerling, Keisuke Sakaguchi, and Kentaro Inui. 2023. Test-time Augmentation for Factual Probing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3650–3661, Singapore. Association for Computational Linguistics.