<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="schoch-ji-2025-monte">
    <titleInfo>
        <title>Monte Carlo Sampling for Analyzing In-Context Examples</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Stephanie</namePart>
        <namePart type="family">Schoch</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yangfeng</namePart>
        <namePart type="family">Ji</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2025-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>The Sixth Workshop on Insights from Negative Results in NLP</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Aleksandr</namePart>
            <namePart type="family">Drozd</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">João</namePart>
            <namePart type="family">Sedoc</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Shabnam</namePart>
            <namePart type="family">Tafreshi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Arjun</namePart>
            <namePart type="family">Akula</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Raphael</namePart>
            <namePart type="family">Shu</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Albuquerque, New Mexico</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-240-4</identifier>
    </relatedItem>
    <abstract>Prior works have shown that in-context learning is brittle to presentation factors such as the order, number, and choice of selected examples. However, ablation-based guidance on selecting the number of examples may ignore the interplay between different presentation factors. In this work we develop a Monte Carlo sampling-based method to study the impact of number of examples while explicitly accounting for effects from order and selected examples. We find that previous guidance on how many in-context examples to select does not always generalize across different sets of selected examples and orderings, and whether one-shot settings outperform zero-shot settings is highly dependent on the selected example. Additionally, inspired by data valuation, we apply our sampling method to in-context example selection to select examples that perform well across different orderings. We find a negative result, that while performance is robust to ordering and number of examples, there is an unexpected performance degradation compared to random sampling.</abstract>
    <identifier type="citekey">schoch-ji-2025-monte</identifier>
    <identifier type="doi">10.18653/v1/2025.insights-1.7</identifier>
    <location>
        <url>https://aclanthology.org/2025.insights-1.7/</url>
    </location>
    <part>
        <date>2025-05</date>
        <extent unit="page">
            <start>63</start>
            <end>78</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Monte Carlo Sampling for Analyzing In-Context Examples
%A Schoch, Stephanie
%A Ji, Yangfeng
%Y Drozd, Aleksandr
%Y Sedoc, João
%Y Tafreshi, Shabnam
%Y Akula, Arjun
%Y Shu, Raphael
%S The Sixth Workshop on Insights from Negative Results in NLP
%D 2025
%8 May
%I Association for Computational Linguistics
%C Albuquerque, New Mexico
%@ 979-8-89176-240-4
%F schoch-ji-2025-monte
%X Prior works have shown that in-context learning is brittle to presentation factors such as the order, number, and choice of selected examples. However, ablation-based guidance on selecting the number of examples may ignore the interplay between different presentation factors. In this work we develop a Monte Carlo sampling-based method to study the impact of number of examples while explicitly accounting for effects from order and selected examples. We find that previous guidance on how many in-context examples to select does not always generalize across different sets of selected examples and orderings, and whether one-shot settings outperform zero-shot settings is highly dependent on the selected example. Additionally, inspired by data valuation, we apply our sampling method to in-context example selection to select examples that perform well across different orderings. We find a negative result, that while performance is robust to ordering and number of examples, there is an unexpected performance degradation compared to random sampling.
%R 10.18653/v1/2025.insights-1.7
%U https://aclanthology.org/2025.insights-1.7/
%U https://doi.org/10.18653/v1/2025.insights-1.7
%P 63-78

Markdown (Informal)

[Monte Carlo Sampling for Analyzing In-Context Examples](https://aclanthology.org/2025.insights-1.7/) (Schoch & Ji, insights 2025)

• Monte Carlo Sampling for Analyzing In-Context Examples (Schoch & Ji, insights 2025)

ACL

• Stephanie Schoch and Yangfeng Ji. 2025. Monte Carlo Sampling for Analyzing In-Context Examples. In The Sixth Workshop on Insights from Negative Results in NLP, pages 63–78, Albuquerque, New Mexico. Association for Computational Linguistics.