When Tool-Backed Skill Retrieval Fails: Source-Style Collapse in Executable Capability Retrieval

Yiqi Liu, Joseph James, Yang Wang, Chenghao Xiao, Chenghua Lin


Abstract
Large-scale agents increasingly rely on retrieval to access external capabilities. We study this retrieval gate in structured tools and APIs, a measurable class of tool-backed executable skills that must be surfaced before an agent can plan, incorporate, or act. In this setting the retrieval layer can silently fail even when the capability corpus is fixed: on ToolRet, a retriever fine-tuned on one source-specific slice collapses on another source-specific slice of the same benchmark, with FT-1100 falling to 0.7% coverage on APIGen despite its higher lexical overlap with the gold tools. We call this failure mode source-style collapse. Query-side TF-IDF centroid distance identifies low-coverage transfer sources more reliably than semantic- or length-based proxies, providing a lightweight compatibility signal over a fixed tool corpus. We propose ToolScout, a source-aware routing method that uses this signal as a routing criterion: on the mixed 4,996-query stream, TF-IDF-based routing raises coverage from 22.3% to 86.1%, and across five collapsed sources, 20-shot source-specific adaptation, using 20 training queries per source, raises the coverage-weighted global top-1 proxy from 1.3% to 53.9%. The same failure and routing behaviors persist under executable skill-card rendering, showing that source-style collapse also occurs with this capability representation.
Anthology ID:
2026.findings-aacl.60
Volume:
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Month:
November
Year:
2026
Address:
Hengqin, China
Editors:
Yue Zhang, Ruifeng Xu, Ayu Purwarianti, Tong Rong, Jun Suzuki
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1181–1203
Language:
URL:
https://aclanthology.org/2026.findings-aacl.60/
DOI:
Bibkey:
Cite (ACL):
Yiqi Liu, Joseph James, Yang Wang, Chenghao Xiao, and Chenghua Lin. 2026. When Tool-Backed Skill Retrieval Fails: Source-Style Collapse in Executable Capability Retrieval. In Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026, pages 1181–1203, Hengqin, China. Association for Computational Linguistics.
Cite (Informal):
When Tool-Backed Skill Retrieval Fails: Source-Style Collapse in Executable Capability Retrieval (Liu et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-aacl.60.pdf