Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assumption through silence, ambiguity, or contradiction, admitting multiple defensible readings that a single gold trajectory cannot capture. Auditing two 𝜏²-bench domains, we develop a taxonomy of such policy loopholes and show that affected tasks produce unreliable scores: they lower scores across different models in different ways and make every model less consistent across repeated trials. A cross-domain comparison reveals that exploitability requires both policy ambiguity and tool permissiveness: when policy complexity exceeds what tools can enforce, agents resolve gaps inconsistently and scores become unreliable. Policy specification quality sets the ceiling on evaluation quality. Benchmark developers should audit policies before collecting gold annotations.

Large Language Model (LLM) agents increasingly operate in multi-step, multi-turn settings that combine user interaction, tool use, state changes, and communication. Most agent evaluation still emphasizes task completion, while procedure-aware checks remain fragmented. We conduct a systematic survey and introduce ℱ^*, a systematic organizing map that maps scattered procedural checks into a shared audit space spanning integrity relations and quality dimensions. Applying this framework to recent agent-evaluation work reveals three procedural blind spots: most work remains outcome-focused or procedurally under-specified, procedure-aware checks mostly remain local spot-checks, and communication grounding lags action compliance. To close these gaps, we propose four new Natural Language Processing (NLP) missions around communication grounding, constraint semantics, reasoning faithfulness, and procedural equity. This survey provides a comprehensive overview of procedure-aware agent evaluation, offering a common vocabulary and coverage map that clarify what current benchmarks can certify and provide clear guidelines for future research.

Imagine two users contacting the same AI travel agent on the same day, making the same request under identical policy conditions, differing only in the demographic information they provide. The governing policy is explicit and contains no demographic distinction. Should the agent respond identically? We present an empirical study of how demographic context influences the behaviour of policy-governed LLM agents. Building on an existing agentic evaluation framework, we augment policy-governed tasks with controlled demographic cues while keeping policies and task environments fixed, enabling behavioural differences to be attributed solely to demographic signals. To evaluate procedural fairness, we introduce three procedural dimensions: Policy Application Fairness (PAF), Procedural Effort Equity (PEE), and User Fulfillment Parity (UFP), unified into the Demographic Policy Fairness Index (DPFI). Across more than twenty thousand simulated interactions spanning multiple service domains and frontier models, all evaluated systems exhibit systematic demographic disparities that conventional task-success metrics fail to capture. Our analysis identifies three distinct failure modes with different procedural dynamics, demonstrating that aggregate success metrics can obscure how disparities emerge. Overall, the findings suggest that demographic bias in policy-governed LLM agents manifests through procedural behaviour rather than a single uniform failure pattern. The proposed framework provides a foundation for systematically auditing procedural fairness in policy-driven LLM agents.