<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kanda-etal-2026-workflowperturb">
    <titleInfo>
        <title>WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Madhav</namePart>
        <namePart type="family">Kanda</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sharad</namePart>
        <namePart type="family">Agarwal</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rodrigo</namePart>
        <namePart type="family">Fonseca</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alok</namePart>
        <namePart type="given">Gautam</namePart>
        <namePart type="family">Kumbhare</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Pedro</namePart>
        <namePart type="family">Las-Casas</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Pepa</namePart>
            <namePart type="family">Atanasova</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gholamreza</namePart>
            <namePart type="family">Haffari</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Minghan</namePart>
            <namePart type="family">Wang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-460-6</identifier>
    </relatedItem>
    <abstract>Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, DevOps, and enterprise orchestration. Operating them exposes a recurring change-management problem. Routine updates, such as re-running an input, swapping the LLM, or refactoring an agent’s prompt or orchestration code, often produce workflows that differ substantially from validated references. Engineers then lack a principled way to decide whether a change is safe to ship. Automatic workflow evaluation is the natural tool, but in practice metric scores are poorly calibrated, and a numeric change rarely communicates the severity of the underlying degradation. We introduce WorkflowPerturb, a controlled benchmark that applies realistic, graded perturbations to golden workflows. It contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, Description Changes) at severities of 10%, 30%, and 50%. We benchmark multiple metric families, analyzing their sensitivity and calibration using expected score trajectories and instance-level alert rates. Our results characterize systematic differences across families and support severity-aware interpretation of workflow evaluation scores in change-management settings. WorkflowPerturb is publicly available at https://huggingface.co/datasets/microsoft/WorkflowPerturb.</abstract>
    <identifier type="citekey">kanda-etal-2026-workflowperturb</identifier>
    <location>
        <url>https://aclanthology.org/2026.emnlp-industry.25/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>394</start>
            <end>411</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics
%A Kanda, Madhav
%A Agarwal, Sharad
%A Fonseca, Rodrigo
%A Kumbhare, Alok Gautam
%A Las-Casas, Pedro
%Y Atanasova, Pepa
%Y Haffari, Gholamreza
%Y Wang, Minghan
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-460-6
%F kanda-etal-2026-workflowperturb
%X Multi-agent LLM systems that generate structured workflows from natural-language requests are now deployed in production across cloud automation, DevOps, and enterprise orchestration. Operating them exposes a recurring change-management problem. Routine updates, such as re-running an input, swapping the LLM, or refactoring an agent’s prompt or orchestration code, often produce workflows that differ substantially from validated references. Engineers then lack a principled way to decide whether a change is safe to ship. Automatic workflow evaluation is the natural tool, but in practice metric scores are poorly calibrated, and a numeric change rarely communicates the severity of the underlying degradation. We introduce WorkflowPerturb, a controlled benchmark that applies realistic, graded perturbations to golden workflows. It contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, Description Changes) at severities of 10%, 30%, and 50%. We benchmark multiple metric families, analyzing their sensitivity and calibration using expected score trajectories and instance-level alert rates. Our results characterize systematic differences across families and support severity-aware interpretation of workflow evaluation scores in change-management settings. WorkflowPerturb is publicly available at https://huggingface.co/datasets/microsoft/WorkflowPerturb.
%U https://aclanthology.org/2026.emnlp-industry.25/
%P 394-411

Markdown (Informal)

[WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics](https://aclanthology.org/2026.emnlp-industry.25/) (Kanda et al., EMNLP 2026)

• WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics (Kanda et al., EMNLP 2026)

ACL

• Madhav Kanda, Sharad Agarwal, Rodrigo Fonseca, Alok Gautam Kumbhare, and Pedro Las-Casas. 2026. WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 394–411, Budapest, Hungary. Association for Computational Linguistics.