<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="farajidizaji-raina-2026-task">
    <titleInfo>
        <title>Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Mahdiyeh</namePart>
        <namePart type="family">Farajidizaji</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Vatsal</namePart>
        <namePart type="family">Raina</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 3rd Workshop on Language Understanding in the Human-Machine Era (LUHME 2026)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Rui</namePart>
            <namePart type="family">Sousa-Silva</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Henrique</namePart>
            <namePart type="family">Lopes Cardoso</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Maarit</namePart>
            <namePart type="family">Koponen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Antonio</namePart>
            <namePart type="family">Pareja-Lora</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-471-2</identifier>
    </relatedItem>
    <abstract>Instruction tuning is meant to make language models follow user requests, yet it is unclear whether small models comply when an instruction conflicts with their usual task behavior. We study this across three tasks, multiple-choice question answering (MCQA), sentiment classification, and mathematical question answering, by pairing a standard instruction with a conflicting non-standard one (select an incorrect option, output the opposite sentiment, or return twice the answer). This cross-task design allows us to test whether resistance to conflicting instructions is tied to specific task characteristics or reflects a broader behavioral tendency. As all predictions are scored against the original ground truth, a model that ignores the non-standard instruction still appears “accurate”. Using standard accuracy, non-standard accuracy, and an Instruction-Following Failure Rate (IFFR), we evaluate instruction-tuned Qwen models across sizes. Both standard accuracy and instruction following generally improve with scale, although the pattern is not consistent across all tasks and datasets. Small models stay competent yet routinely ignore the non-standard instruction, while larger models show a clear gap between the two settings. These findings suggest that gains in task capability do not automatically provide reliable control over model behavior. Task competence and instruction following are therefore distinct abilities, and reporting only standard accuracy hides instruction-following failures.</abstract>
    <identifier type="citekey">farajidizaji-raina-2026-task</identifier>
    <location>
        <url>https://aclanthology.org/2026.luhme-1.9/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>109</start>
            <end>120</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models
%A Farajidizaji, Mahdiyeh
%A Raina, Vatsal
%Y Sousa-Silva, Rui
%Y Lopes Cardoso, Henrique
%Y Koponen, Maarit
%Y Pareja-Lora, Antonio
%S Proceedings of the 3rd Workshop on Language Understanding in the Human-Machine Era (LUHME 2026)
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-471-2
%F farajidizaji-raina-2026-task
%X Instruction tuning is meant to make language models follow user requests, yet it is unclear whether small models comply when an instruction conflicts with their usual task behavior. We study this across three tasks, multiple-choice question answering (MCQA), sentiment classification, and mathematical question answering, by pairing a standard instruction with a conflicting non-standard one (select an incorrect option, output the opposite sentiment, or return twice the answer). This cross-task design allows us to test whether resistance to conflicting instructions is tied to specific task characteristics or reflects a broader behavioral tendency. As all predictions are scored against the original ground truth, a model that ignores the non-standard instruction still appears “accurate”. Using standard accuracy, non-standard accuracy, and an Instruction-Following Failure Rate (IFFR), we evaluate instruction-tuned Qwen models across sizes. Both standard accuracy and instruction following generally improve with scale, although the pattern is not consistent across all tasks and datasets. Small models stay competent yet routinely ignore the non-standard instruction, while larger models show a clear gap between the two settings. These findings suggest that gains in task capability do not automatically provide reliable control over model behavior. Task competence and instruction following are therefore distinct abilities, and reporting only standard accuracy hides instruction-following failures.
%U https://aclanthology.org/2026.luhme-1.9/
%P 109-120

Markdown (Informal)

[Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models](https://aclanthology.org/2026.luhme-1.9/) (Farajidizaji & Raina, LUHME 2026)

• Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models (Farajidizaji & Raina, LUHME 2026)

ACL

• Mahdiyeh Farajidizaji and Vatsal Raina. 2026. Task Competence Is Not Instruction Following: Evaluating Instruction-Conflicting Behavior in Small Language Models. In Proceedings of the 3rd Workshop on Language Understanding in the Human-Machine Era (LUHME 2026), pages 109–120, Budapest, Hungary. Association for Computational Linguistics.