<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="akli-etal-2026-natural">
    <titleInfo>
        <title>Natural Language Defects in Code Generation Prompts: Empirical Impact and Lightweight Detection</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Amal</namePart>
        <namePart type="family">Akli</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mike</namePart>
        <namePart type="family">Papadakis</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Maxime</namePart>
        <namePart type="family">Cordy</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Yves</namePart>
        <namePart type="family">Le Traon</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-462-0</identifier>
    </relatedItem>
    <abstract>Large language models are widely used for code generation, yet they rely on an implicit assumption that the task descriptions are sufficiently detailed and well-formed. However, in practice, users may provide defective descriptions, which can have a strong effect on code correctness. To address this issue, we conduct a large-scale empirical analysis evaluating the robustness of 10 code LLMs across 3 benchmarks under three natural language defect types: Lexical Vagueness (LV), Under-Specification (US), and Syntax/Formatting (SF). Our results show that defect impact is not uniform: Under-Specification causes Pass@1 drops of up to 15.3% with no model immune, including state-of-the-art reasoning models; Lexical Vagueness produces moderate, benchmark-dependent degradation; and surface formatting noise has a negligible effect. Crucially, robustness is shaped more by the properties of the task description than by the size or architecture of the model, in contrast to prior perturbation studies reporting uniform degradation across models. Second, to address these defects before they reach the model, we develop SpecValidator, a lightweight LoRA-fine-tuned classifier based on a 1.5B code model, achieving F1 = 0.84 and MCC = 0.79, significantly outperforming few-shot GPT-5-mini (F1 = 0.43) and Claude-Sonnet-4 (F1 = 0.56), and demonstrating that strong code generation ability does not transfer to prompt meta-reasoning. Perhaps more importantly, our analysis indicates that SpecValidator can generalize to unseen issues and detect unknown Under-Specification defects in the original (real) descriptions of the benchmarks used.</abstract>
    <identifier type="citekey">akli-etal-2026-natural</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-emnlp.1908/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>40649</start>
            <end>40663</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Natural Language Defects in Code Generation Prompts: Empirical Impact and Lightweight Detection
%A Akli, Amal
%A Papadakis, Mike
%A Cordy, Maxime
%A Le Traon, Yves
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F akli-etal-2026-natural
%X Large language models are widely used for code generation, yet they rely on an implicit assumption that the task descriptions are sufficiently detailed and well-formed. However, in practice, users may provide defective descriptions, which can have a strong effect on code correctness. To address this issue, we conduct a large-scale empirical analysis evaluating the robustness of 10 code LLMs across 3 benchmarks under three natural language defect types: Lexical Vagueness (LV), Under-Specification (US), and Syntax/Formatting (SF). Our results show that defect impact is not uniform: Under-Specification causes Pass@1 drops of up to 15.3% with no model immune, including state-of-the-art reasoning models; Lexical Vagueness produces moderate, benchmark-dependent degradation; and surface formatting noise has a negligible effect. Crucially, robustness is shaped more by the properties of the task description than by the size or architecture of the model, in contrast to prior perturbation studies reporting uniform degradation across models. Second, to address these defects before they reach the model, we develop SpecValidator, a lightweight LoRA-fine-tuned classifier based on a 1.5B code model, achieving F1 = 0.84 and MCC = 0.79, significantly outperforming few-shot GPT-5-mini (F1 = 0.43) and Claude-Sonnet-4 (F1 = 0.56), and demonstrating that strong code generation ability does not transfer to prompt meta-reasoning. Perhaps more importantly, our analysis indicates that SpecValidator can generalize to unseen issues and detect unknown Under-Specification defects in the original (real) descriptions of the benchmarks used.
%U https://aclanthology.org/2026.findings-emnlp.1908/
%P 40649-40663

Markdown (Informal)

[Natural Language Defects in Code Generation Prompts: Empirical Impact and Lightweight Detection](https://aclanthology.org/2026.findings-emnlp.1908/) (Akli et al., Findings 2026)

• Natural Language Defects in Code Generation Prompts: Empirical Impact and Lightweight Detection (Akli et al., Findings 2026)

ACL

• Amal Akli, Mike Papadakis, Maxime Cordy, and Yves Le Traon. 2026. Natural Language Defects in Code Generation Prompts: Empirical Impact and Lightweight Detection. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 40649–40663, Budapest, Hungary. Association for Computational Linguistics.