<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="yang-etal-2026-mcpgen">
    <titleInfo>
        <title>MCPGen: Benchmarking LLMs on Executable MCP Workflow Development</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Yingxuan</namePart>
        <namePart type="family">Yang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jiaqi</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Lirui</namePart>
        <namePart type="family">Guan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Jiaye</namePart>
        <namePart type="family">Gao</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Weiwen</namePart>
        <namePart type="family">Liu</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Weinan</namePart>
        <namePart type="family">Zhang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ying</namePart>
        <namePart type="family">Wen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2026-10</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2026</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Hung-Yi</namePart>
            <namePart type="family">Lee</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sunipa</namePart>
            <namePart type="family">Dev</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jessy</namePart>
            <namePart type="family">Li</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Desmond</namePart>
            <namePart type="family">Elliott</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Budapest, Hungary</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
        <identifier type="isbn">979-8-89176-462-0</identifier>
    </relatedItem>
    <abstract>We study whether LLMs can produce executable workflow artifacts that remain consistent across graph structure, tool implementation, schema bindings, and runtime wiring. In this setting, correctness depends on cross-layer consistency: a workflow may be structurally plausible, yet still fail because tool implementations, schema bindings, or runtime execution do not align. Existing benchmarks largely evaluate these capabilities in isolation or rely on trajectory-level proxies, leaving open whether generated workflow artifacts execute end-to-end. We introduce MCPGen, an executable benchmark for Model Context Protocol (MCP) workflow development. MCPGen contains 100 self-contained MCP projects across 16 application domains and evaluates three diagnostic tasks: workflow reconstruction, tool creation, and backward-compatible workflow extension. We evaluate 11 representative LLMs in a single-turn foundation-model setting, assessing generated artifacts through static analysis, unit and integration tests, and process-isolated end-to-end execution. Models reach 88.5% on workflow reconstruction, but no model exceeds 57% end-to-end execution success. Per-tool unit-test pass rates reach 63.8%, while project-level integration success does not exceed 45%, suggesting that integration remains a major bottleneck even when isolated tool tests pass. Code and data are available at https://github.com/zoe-yyx/MCPGen_Official.</abstract>
    <identifier type="citekey">yang-etal-2026-mcpgen</identifier>
    <location>
        <url>https://aclanthology.org/2026.findings-emnlp.1358/</url>
    </location>
    <part>
        <date>2026-10</date>
        <extent unit="page">
            <start>28846</start>
            <end>28859</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T MCPGen: Benchmarking LLMs on Executable MCP Workflow Development
%A Yang, Yingxuan
%A Liu, Jiaqi
%A Guan, Lirui
%A Gao, Jiaye
%A Liu, Weiwen
%A Zhang, Weinan
%A Wen, Ying
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Findings of the Association for Computational Linguistics: EMNLP 2026
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-462-0
%F yang-etal-2026-mcpgen
%X We study whether LLMs can produce executable workflow artifacts that remain consistent across graph structure, tool implementation, schema bindings, and runtime wiring. In this setting, correctness depends on cross-layer consistency: a workflow may be structurally plausible, yet still fail because tool implementations, schema bindings, or runtime execution do not align. Existing benchmarks largely evaluate these capabilities in isolation or rely on trajectory-level proxies, leaving open whether generated workflow artifacts execute end-to-end. We introduce MCPGen, an executable benchmark for Model Context Protocol (MCP) workflow development. MCPGen contains 100 self-contained MCP projects across 16 application domains and evaluates three diagnostic tasks: workflow reconstruction, tool creation, and backward-compatible workflow extension. We evaluate 11 representative LLMs in a single-turn foundation-model setting, assessing generated artifacts through static analysis, unit and integration tests, and process-isolated end-to-end execution. Models reach 88.5% on workflow reconstruction, but no model exceeds 57% end-to-end execution success. Per-tool unit-test pass rates reach 63.8%, while project-level integration success does not exceed 45%, suggesting that integration remains a major bottleneck even when isolated tool tests pass. Code and data are available at https://github.com/zoe-yyx/MCPGen_Official.
%U https://aclanthology.org/2026.findings-emnlp.1358/
%P 28846-28859

Markdown (Informal)

[MCPGen: Benchmarking LLMs on Executable MCP Workflow Development](https://aclanthology.org/2026.findings-emnlp.1358/) (Yang et al., Findings 2026)

• MCPGen: Benchmarking LLMs on Executable MCP Workflow Development (Yang et al., Findings 2026)

ACL

• Yingxuan Yang, Jiaqi Liu, Lirui Guan, Jiaye Gao, Weiwen Liu, Weinan Zhang, and Ying Wen. 2026. MCPGen: Benchmarking LLMs on Executable MCP Workflow Development. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 28846–28859, Budapest, Hungary. Association for Computational Linguistics.