Agentic AI systems powered by large language models (LLMs) are rapidly evolving into autonomous decision-making systems, exposing attack vectors beyond those of traditional LLM vulnerabilities. Existing security evaluations are often tied to specific implementations or domains, limiting unified comparison across heterogeneous systems. To address this gap, we introduce RIFT-Bench, a representation-driven methodology for dynamic red-teaming that enables unified evaluations across diverse agentic architectures. Building on a novel hierarchical representation, RIFT-Bench operates in two automated phases: Discovery, which extracts system structure, and Scanning, which executes adaptive adversarial attacks. It directly evaluates the examined system using 105 adaptive adversarial probes spanning diverse attack vectors and objectives. We demonstrate the effectiveness of the proposed evaluation pipeline across 45 agentic systems spanning a diverse range of implementations, showing that the approach generalizes effectively to heterogeneous agentic architectures. Beyond systems and attacks, RIFT-Bench also supports direct evaluation of mitigation strategies. These key capabilities make RIFT-Bench a scalable foundation for security evaluation of agentic AI systems in practice. Infrastructure code and benchmark artifacts are available at https://github.com/FujitsuResearch/RIFT-Bench.

AI agents combine LLMs with external tools to solve complex tasks. While tools extend agents’ capabilities, improper permissions introduce risks such as indirect prompt injection and tool misuse. We characterize these failures as unbalanced tool-driven agency, a unified view in which agents either retain unnecessary permissions (excessive agency) or fail to invoke required tools (insufficient agency), increasing risk and reducing performance. We introduce AgenTRIM, a framework that addresses this imbalance without altering agent reasoning, through complementary offline and online stages. Offline, it reconstructs the agent’s tool interface from code and validates tool behavior via controlled executions and trace analysis. At runtime, it enforces per-step least-privilege access through adaptive capability filtering and isolated validation of tool calls, implicitly decomposing tasks without explicit planning. On the AgentDojo benchmark, AgenTRIM substantially reduces attack success while improving task performance over the baseline, unlike other leading defenses. To assess robustness beyond prompt injection, we conduct additional experiments covering description-based attacks and policy enforcement. Together, these results establish AgenTRIM as a general, capability-preserving framework for mitigating tool-driven risks in LLM-based agents.

Automated LLM vulnerability scanners are increasingly used to assess security risks by measuring different attack type success rates (ASR). Yet the validity of these measurements hinges on an often-overlooked component: the evaluator who determines whether an attack has succeeded. In this study, we demonstrate that commonly used open-source scanners exhibit measurement instability that depends on the evaluator component. Consequently, changing the evaluator while keeping the attacks and model outputs constant can significantly alter the reported ASR. To tackle this problem, we present a two-phase, reliability-aware evaluation framework. In the first phase, we quantify evaluator disagreement to identify attack categories where ASR reliability cannot be assumed. In the second phase, we propose a verification-based evaluation method where evaluators are validated by an independent verifier, enabling reliability assessment without relying on extensive human annotation. Applied to the widely used Garak scanner, we observe that 22 of 25 attack categories exhibit evaluator instability, reflected in high disagreement among evaluators. Our approach raises evaluator accuracy from 72% to 89% while enabling selective deployment to control cost and computational overhead. We further quantify evaluator uncertainty in ASR estimates, showing that reported vulnerability scores can vary by up to ±33% depending on the evaluator. Our results indicate that the outputs of vulnerability scanners are highly sensitive to the choice of evaluators. Our framework offers a practical approach to quantify unreliable evaluations and enhance the reliability of measurements in automated LLM security assessments.