<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhang-etal-2024-revisiting-opro">
    <titleInfo>
        <title>Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tuo</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jinyue</namePart>
        <namePart type="family">Yuan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Salman</namePart>
        <namePart type="family">Avestimehr</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Lun-Wei</namePart>
            <namePart type="family">Ku</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Andre</namePart>
            <namePart type="family">Martins</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vivek</namePart>
            <namePart type="family">Srikumar</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Bangkok, Thailand</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Numerous recent works aim to enhance the efficacy of Large Language Models (LLMs) through strategic prompting. In particular, the Optimization by PROmpting (OPRO) approach provides state-of-the-art performance by leveraging LLMs as optimizers where the optimization task is to find instructions that maximize the task accuracy. In this paper, we revisit OPRO for automated prompting with relatively small-scale LLMs, such as LLaMa-2 family and Mistral 7B. Our investigation reveals that OPRO shows limited effectiveness in small-scale LLMs, with limited inference capabilities constraining optimization ability. We suggest future automatic prompting engineering to consider both model capabilities and computational costs. Additionally, for small-scale LLMs, we recommend direct instructions that clearly outline objectives and methodologies as robust prompt baselines, ensuring efficient and effective prompt engineering in ongoing research.</abstract>
    <identifier type="citekey">zhang-etal-2024-revisiting-opro</identifier>
    <identifier type="doi">10.18653/v1/2024.findings-acl.100</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-acl.100</url>
    </location>
    <part>
        <date>2024-08</date>
        <extent unit="page">
            <start>1727</start>
            <end>1735</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers
%A Zhang, Tuo
%A Yuan, Jinyue
%A Avestimehr, Salman
%Y Ku, Lun-Wei
%Y Martins, Andre
%Y Srikumar, Vivek
%S Findings of the Association for Computational Linguistics: ACL 2024
%D 2024
%8 August
%I Association for Computational Linguistics
%C Bangkok, Thailand
%F zhang-etal-2024-revisiting-opro
%X Numerous recent works aim to enhance the efficacy of Large Language Models (LLMs) through strategic prompting. In particular, the Optimization by PROmpting (OPRO) approach provides state-of-the-art performance by leveraging LLMs as optimizers where the optimization task is to find instructions that maximize the task accuracy. In this paper, we revisit OPRO for automated prompting with relatively small-scale LLMs, such as LLaMa-2 family and Mistral 7B. Our investigation reveals that OPRO shows limited effectiveness in small-scale LLMs, with limited inference capabilities constraining optimization ability. We suggest future automatic prompting engineering to consider both model capabilities and computational costs. Additionally, for small-scale LLMs, we recommend direct instructions that clearly outline objectives and methodologies as robust prompt baselines, ensuring efficient and effective prompt engineering in ongoing research.
%R 10.18653/v1/2024.findings-acl.100
%U https://aclanthology.org/2024.findings-acl.100
%U https://doi.org/10.18653/v1/2024.findings-acl.100
%P 1727-1735

Markdown (Informal)

[Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers](https://aclanthology.org/2024.findings-acl.100) (Zhang et al., Findings 2024)

• Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers (Zhang et al., Findings 2024)

ACL

• Tuo Zhang, Jinyue Yuan, and Salman Avestimehr. 2024. Revisiting OPRO: The Limitations of Small-Scale LLMs as Optimizers. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1727–1735, Bangkok, Thailand. Association for Computational Linguistics.