When Do Prompt-Side Agent Playbooks Transfer?Accuracy, Cost, and Runtime Shift in Agent Deployment

Weihong Lin, Xiangzheng Zhang, Lin Sun


Abstract
Prompt-side playbooks can improve tool-using language agents without retraining, but their portability beyond the source setting is unclear. We study frozen playbook transfer under a shared distill–validate–transfer protocol. On ALFWorld, transfer is beneficial under controlled greedy decoding and, in one near-budget-matched comparison, distilled guidance outperforms five fixed demonstrations. On TAU2-Bench, a prespecified aggregate contrast supports a modest average matched-domain advantage, but global Holm correction retains only one of 135 route-level effects; the remaining grid provides descriptive evidence of compatibility-sensitive heterogeneity. On XBench-DeepSearch, one artifact–runtime pairing preserves useful first-try heuristics while producing repeated queries, delayed stopping, and substantial cost inflation after a context-runtime shift. Across benchmarks, transferred and target-derived playbooks both require target-side validation of success, termination, protocol compatibility, and cost. Frozen transfer is therefore a conditional cold-start option, not a reuse-by-default strategy or a universally preferable alternative to target-side redistillation.
Anthology ID:
2026.emnlp-industry.251
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Pepa Atanasova, Gholamreza Haffari, Minghan Wang
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
4243–4263
Language:
URL:
https://aclanthology.org/2026.emnlp-industry.251/
DOI:
Bibkey:
Cite (ACL):
Weihong Lin, Xiangzheng Zhang, and Lin Sun. 2026. When Do Prompt-Side Agent Playbooks Transfer?Accuracy, Cost, and Runtime Shift in Agent Deployment. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 4243–4263, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
When Do Prompt-Side Agent Playbooks Transfer?Accuracy, Cost, and Runtime Shift in Agent Deployment (Lin et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-industry.251.pdf