Frustratingly Easy Agentic RAG via Portable Skills: A Case Study in Customer Support

Istok Bako, Bonan Min, Qihong Shao, Claire Cheng, Yashar Mehdad


Abstract
Enterprise workflow automation typically requires retrieving a complete and accurate set of relevant information from large operational document corpora (SOPs, policy documents) given a specific scenario or request. In customer support (CS), for example, human specialists must determine the correct sequence of resolution steps for each case, a task that demands precise grounding in policy documentation. Meeting this bar in production typically requires complex document chunking, indexing, and retrieval pipelines, often layered with learned re-rankers, query rewriters, and bespoke agentic RAG infrastructure.

We present a simpler alternative: a portable skill consisting of a natural-language instruction, lightweight tools (a retrieval API, format-checking scripts), and a corpus enriched with an LLM-generated navigation layer, which is a structured "Wikipedia-style" index over the underlying documents. This agentic RAG approach is straightforward to implement (1-2 engineer days), portable across capable agentic harnesses (e.g., Claude Code), and performs strongly on the challenging task of CS resolution workflow generation, achieving 99.2% procedural equivalence to human-approved ground truth and 83.3% evidence sufficiency. Our analysis yields a concrete deployment insight: a skill-based agentic RAG implementation over a corpus augmented with a lightweight navigation layer can substitute for complex RAG infrastructure, shifting the primary maintenance burden from model artifacts to text and data curation.
Anthology ID:
2026.emnlp-industry.156
Volume:
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Pepa Atanasova, Gholamreza Haffari, Minghan Wang
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2647–2661
Language:
URL:
https://aclanthology.org/2026.emnlp-industry.156/
DOI:
Bibkey:
Cite (ACL):
Istok Bako, Bonan Min, Qihong Shao, Claire Cheng, and Yashar Mehdad. 2026. Frustratingly Easy Agentic RAG via Portable Skills: A Case Study in Customer Support. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 2647–2661, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
Frustratingly Easy Agentic RAG via Portable Skills: A Case Study in Customer Support (Bako et al., EMNLP 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.emnlp-industry.156.pdf