<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="luz-de-araujo-roth-2024-functionality">
    <titleInfo>
        <title>Functionality learning through specification instructions</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Pedro</namePart>
        <namePart type="given">Henrique</namePart>
        <namePart type="family">Luz De Araujo</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Benjamin</namePart>
        <namePart type="family">Roth</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2024</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yaser</namePart>
            <namePart type="family">Al-Onaizan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Mohit</namePart>
            <namePart type="family">Bansal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Yun-Nung</namePart>
            <namePart type="family">Chen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Miami, Florida, USA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Test suites assess natural language processing models’ performance on specific functionalities: cases of interest involving model robustness, fairness, or particular linguistic capabilities. This paper introduces specification instructions: text descriptions specifying fine-grained task-specific behaviors. For each functionality in a suite, we generate an instruction that describes it. We combine the specification instructions to create specification-augmented prompts, which we feed to language models pre-trained on natural instruction data.We conduct experiments to measure how optimizing for some functionalities may negatively impact functionalities that are not covered by the specification set. Our analyses across four tasks and models of diverse sizes and families show that smaller models struggle to follow specification instructions. However, larger models (\textgreater 3B params.) can benefit from specifications and—surprisingly—even generalize certain desirable behaviors across functionalities.</abstract>
    <identifier type="citekey">luz-de-araujo-roth-2024-functionality</identifier>
    <location>
        <url>https://aclanthology.org/2024.findings-emnlp.642</url>
    </location>
    <part>
        <date>2024-11</date>
        <extent unit="page">
            <start>10955</start>
            <end>10990</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Functionality learning through specification instructions
%A Luz De Araujo, Pedro Henrique
%A Roth, Benjamin
%Y Al-Onaizan, Yaser
%Y Bansal, Mohit
%Y Chen, Yun-Nung
%S Findings of the Association for Computational Linguistics: EMNLP 2024
%D 2024
%8 November
%I Association for Computational Linguistics
%C Miami, Florida, USA
%F luz-de-araujo-roth-2024-functionality
%X Test suites assess natural language processing models’ performance on specific functionalities: cases of interest involving model robustness, fairness, or particular linguistic capabilities. This paper introduces specification instructions: text descriptions specifying fine-grained task-specific behaviors. For each functionality in a suite, we generate an instruction that describes it. We combine the specification instructions to create specification-augmented prompts, which we feed to language models pre-trained on natural instruction data.We conduct experiments to measure how optimizing for some functionalities may negatively impact functionalities that are not covered by the specification set. Our analyses across four tasks and models of diverse sizes and families show that smaller models struggle to follow specification instructions. However, larger models (\textgreater 3B params.) can benefit from specifications and—surprisingly—even generalize certain desirable behaviors across functionalities.
%U https://aclanthology.org/2024.findings-emnlp.642
%P 10955-10990

Markdown (Informal)

[Functionality learning through specification instructions](https://aclanthology.org/2024.findings-emnlp.642) (Luz De Araujo & Roth, Findings 2024)

• Functionality learning through specification instructions (Luz De Araujo & Roth, Findings 2024)

ACL

• Pedro Henrique Luz De Araujo and Benjamin Roth. 2024. Functionality learning through specification instructions. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10955–10990, Miami, Florida, USA. Association for Computational Linguistics.