Agent-Readiness of Synthesized API Tools: Parametric Gravity and the Limits of Documentation Grounding

Teodora Stereciu, Baohao Liao, Christian Herold, Shahram Khadivi, Christof Monz


Abstract
We study whether LLM agents can autonomously synthesize deployment-ready MCP (Model Context Protocol) servers from raw REST API documentation, and whether the resulting servers are reliable enough for downstream agentic task completion. Evaluating 8 synthesis models across 14 APIs under three documentation conditions — full docs, no docs, and counterfactually mutated docs — we find that synthesis is broadly feasible: 11 of 14 APIs achieve sufficient tool coverage, and synthesized servers match or exceed human-authored references. The dominant mediating factor is not model capability but parametric gravity: for well-known APIs (GitHub, Stripe, Shopify), training-time knowledge overrides in-context documentation, rendering documentation provision neutral or harmful. A counterfactual mutation study confirms this: high-familiarity APIs show near-zero adherence to renamed parameters, while low-familiarity APIs show near-complete adherence. Logit-lens probing shows the representations of memory and context become increasingly separable in the final transformer layers, consistent with deep feed-forward MLP modules as the site of parametric association storage. Synthesis coverage (SYNTH) correlates strongly with agent task completion (r = 0.937), providing a reliable pre-flight quality signal without requiring agent evaluation.
Anthology ID:
2026.realm-1.22
Volume:
Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026)
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Ehsan Kamalloo, Nicolas Gontier, Nikolai Rozanov, Maryam Hashemzadeh, Akhil Arora
Venues:
REALM | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
323–340
Language:
URL:
https://aclanthology.org/2026.realm-1.22/
DOI:
Bibkey:
Cite (ACL):
Teodora Stereciu, Baohao Liao, Christian Herold, Shahram Khadivi, and Christof Monz. 2026. Agent-Readiness of Synthesized API Tools: Parametric Gravity and the Limits of Documentation Grounding. In Proceedings of the Second Workshop for REsearch on Agent Language Models (REALM 2026), pages 323–340, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Agent-Readiness of Synthesized API Tools: Parametric Gravity and the Limits of Documentation Grounding (Stereciu et al., REALM 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.realm-1.22.pdf