STARS: Skill Triage via Activation Risk Scoring for LLM Agents

Guijia Zhang, Shu Yang, Xilin Gong, Di Wang


Abstract
The risk of an agent skill call depends not on the skill alone but on the invocation context: the same file-reading skill is benign when listing logs, yet dangerous when tainted content steers the agent toward secret exfiltration. We formalize invocation-time auditing: intercept a proposed skill call before execution and score its risk from the user request, the candidate skill, and the runtime context. STARS implements this as a three-stage pipeline that fuses a static capability prior with a request-conditioned invocation scorer. The primary configuration, STARS-LLM, uses an LLM judge; STARS-Rules is a deterministic, API-free alternative. On SIA-Bench, a 3,000-record synthetic benchmark with a 180-record human check at Fleiss κ = 0.709, request-conditioned scoring improves high-risk retrieval on held-out indirect prompt injection, where static metadata cannot see the attack. On a frozen 500/100 scoring subset, STARS-LLM raises rules-only HR-AUPRC from 0.297 to 0.728 in-distribution and from 0.504 to 0.757 on held-out IPI. That gain is in ranking: under frozen thresholds, held-out classification accuracy does not improve. SIA-Bench is a ranking stress test, not an estimate of real-world prevalence.
Anthology ID:
2026.findings-emnlp.891
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
18874–18890
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.891/
DOI:
10.18653/2026.findings-emnlp.891
Bibkey:
Cite (ACL):
Guijia Zhang, Shu Yang, Xilin Gong, and Di Wang. 2026. STARS: Skill Triage via Activation Risk Scoring for LLM Agents. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 18874–18890, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
STARS: Skill Triage via Activation Risk Scoring for LLM Agents (Zhang et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.891.pdf