<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="tao-etal-2024-inference">
    <titleInfo>
        <title>Inference and Verbalization Functions During In-Context Learning</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Junyi</namePart>
        <namePart type="family">Tao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xiaoyin</namePart>
        <namePart type="family">Chen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nelson</namePart>
        <namePart type="given">F</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yaser</namePart>
            <namePart type="family">Al-Onaizan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Large language models (LMs) are capable of in-context learning from a few demonstrations (example-label pairs) to solve new tasks during inference. Despite the intuitive importance of high-quality demonstrations, previous work has observed that, in some settings, ICL performance is minimally affected by irrelevant labels (Min et al., 2022). We hypothesize that LMs perform ICL with irrelevant labels via two sequential processes: an inference function that solves the task, followed by a verbalization function that maps the inferred answer to the label space. Importantly, we hypothesize that the inference function is invariant to remappings of the label space (e.g., “true”/“false” to “cat”/“dog”), enabling LMs to share the same inference function across settings with different label words. We empirically validate this hypothesis with controlled layer-wise interchange intervention experiments. Our findings confirm the hypotheses on multiple datasets and tasks (natural language inference, sentiment analysis, and topic classification) and further suggest that the two functions can be localized in specific layers across various open-sourced models, including GEMMA-7B, MISTRAL-7B-V0.3, GEMMA-2-27B, and LLAMA-3.1-70B.</abstract>
    <identifier type="citekey">tao-etal-2024-inference</identifier>
    <identifier type="doi">10.18653/v1/2024.findings-emnlp.957</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-emnlp.957/</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>16394</start>
            <end>16421</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Inference and Verbalization Functions During In-Context Learning
%A Tao, Junyi
%A Chen, Xiaoyin
%A Liu, Nelson F.
%Y Al-Onaizan, Yaser
%Y Bansal, Mohit
%Y Chen, Yun-Nung
%S Findings of the Association for Computational Linguistics: EMNLP 2024
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F tao-etal-2024-inference
%X Large language models (LMs) are capable of in-context learning from a few demonstrations (example-label pairs) to solve new tasks during inference. Despite the intuitive importance of high-quality demonstrations, previous work has observed that, in some settings, ICL performance is minimally affected by irrelevant labels (Min et al., 2022). We hypothesize that LMs perform ICL with irrelevant labels via two sequential processes: an inference function that solves the task, followed by a verbalization function that maps the inferred answer to the label space. Importantly, we hypothesize that the inference function is invariant to remappings of the label space (e.g., “true”/“false” to “cat”/“dog”), enabling LMs to share the same inference function across settings with different label words. We empirically validate this hypothesis with controlled layer-wise interchange intervention experiments. Our findings confirm the hypotheses on multiple datasets and tasks (natural language inference, sentiment analysis, and topic classification) and further suggest that the two functions can be localized in specific layers across various open-sourced models, including GEMMA-7B, MISTRAL-7B-V0.3, GEMMA-2-27B, and LLAMA-3.1-70B.
%R 10.18653/v1/2024.findings-emnlp.957
%U https://aclanthology.org/2024.findings-emnlp.957/
%U https://doi.org/10.18653/v1/2024.findings-emnlp.957
%P 16394-16421

Markdown (Informal)

[Inference and Verbalization Functions During In-Context Learning](https://aclanthology.org/2024.findings-emnlp.957/) (Tao et al., Findings 2024)

• Inference and Verbalization Functions During In-Context Learning (Tao et al., Findings 2024)

ACL

• Junyi Tao, Xiaoyin Chen, and Nelson F. Liu. 2024. Inference and Verbalization Functions During In-Context Learning. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16394–16421, Miami, Florida, USA. Association for Computational Linguistics.