<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zheng-saparov-2023-noisy">
    <titleInfo>
        <title>Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Hongyi</namePart>
        <namePart type="family">Zheng</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Abulhair</namePart>
        <namePart type="family">Saparov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent advances in prompt engineering enable large language models (LLMs) to solve multi-hop logical reasoning problems with impressive accuracy. However, there is little existing work investigating the robustness of LLMs with few-shot prompting techniques. Therefore, we introduce a systematic approach to test the robustness of LLMs in multi-hop reasoning tasks via domain-agnostic perturbations. We include perturbations at multiple levels of abstractions (e.g. lexical perturbations such as typos, and semantic perturbations such as the inclusion of intermediate reasoning steps in the questions) to conduct behavioral analysis on the LLMs. Throughout our experiments, we find that models are more sensitive to certain perturbations such as replacing words with their synonyms. We also demonstrate that increasing the proportion of perturbed exemplars in the prompts improves the robustness of few-shot prompting methods.</abstract>
    <identifier type="citekey">zheng-saparov-2023-noisy</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.277</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.277</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>4560</start>
            <end>4568</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis
%A Zheng, Hongyi
%A Saparov, Abulhair
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F zheng-saparov-2023-noisy
%X Recent advances in prompt engineering enable large language models (LLMs) to solve multi-hop logical reasoning problems with impressive accuracy. However, there is little existing work investigating the robustness of LLMs with few-shot prompting techniques. Therefore, we introduce a systematic approach to test the robustness of LLMs in multi-hop reasoning tasks via domain-agnostic perturbations. We include perturbations at multiple levels of abstractions (e.g. lexical perturbations such as typos, and semantic perturbations such as the inclusion of intermediate reasoning steps in the questions) to conduct behavioral analysis on the LLMs. Throughout our experiments, we find that models are more sensitive to certain perturbations such as replacing words with their synonyms. We also demonstrate that increasing the proportion of perturbed exemplars in the prompts improves the robustness of few-shot prompting methods.
%R 10.18653/v1/2023.emnlp-main.277
%U https://aclanthology.org/2023.emnlp-main.277
%U https://doi.org/10.18653/v1/2023.emnlp-main.277
%P 4560-4568

Markdown (Informal)

[Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis](https://aclanthology.org/2023.emnlp-main.277) (Zheng & Saparov, EMNLP 2023)

• Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis (Zheng & Saparov, EMNLP 2023)

ACL

• Hongyi Zheng and Abulhair Saparov. 2023. Noisy Exemplars Make Large Language Models More Robust: A Domain-Agnostic Behavioral Analysis. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4560–4568, Singapore. Association for Computational Linguistics.