Eoin Thomas

Author directory

2026

Assessing regulatory compliance for high-risk AI systems requires systematic identification of technical gaps such as missing controls, inadequate processes, or shallow evaluation of practices. Compliance officers typically administer structured questionnaires collecting evidence of current practices, where organizations respond with descriptions of their existing processes, technical capabilities, and architectural choices. However, translating these compliance questionnaire responses into typed, actionable gap diagnoses remains unsystematic. In this paper, we present a literature-grounded taxonomy of technical compliance gaps covering areas such as accuracy, robustness, logging, and human oversight, and embed it in GapToCompliance, a two-stage agentic framework coordinating two specialized agents: a Gap Extraction Agent that performs hierarchical reasoning from compliance areas to specific gap types, and a Remediation Planning Agent that ranks remediation actions from a remediation catalog via multi-signal scoring combining semantic similarity, regulatory alignment, and effort estimation. We evaluate this framework against four automatic compliance gap detection baselines across 9 AI systems and 6 AI families. Our approach achieves a gap-type F1 of 0.831 (vs. 0.533 for the strongest baseline) and remediation P@5 of 0.872 (vs. 0.635 for retrieval methods), demonstrating promising potential for automated compliance workflows. The structured agent coordination reveals a path toward human-in-the-loop compliance systems where automated gap extraction reduces expert burden while maintaining auditability, and the taxonomy-driven design enables extension to emerging regulatory frameworks beyond the EU AI Act.
Large Language Model (LLM) agents increasingly operate in multi-step, multi-turn settings that combine user interaction, tool use, state changes, and communication. Most agent evaluation still emphasizes task completion, while procedure-aware checks remain fragmented. We conduct a systematic survey and introduce ℱ*, a systematic organizing map that maps scattered procedural checks into a shared audit space spanning integrity relations and quality dimensions. Applying this framework to recent agent-evaluation work reveals three procedural blind spots: most work remains outcome-focused or procedurally under-specified, procedure-aware checks mostly remain local spot-checks, and communication grounding lags action compliance. To close these gaps, we propose four new Natural Language Processing (NLP) missions around communication grounding, constraint semantics, reasoning faithfulness, and procedural equity. This survey provides a comprehensive overview of procedure-aware agent evaluation, offering a common vocabulary and coverage map that clarify what current benchmarks can certify and provide clear guidelines for future research.
Imagine two users contacting the same AI travel agent on the same day, making the same request under identical policy conditions, differing only in the demographic information they provide. The governing policy is explicit and contains no demographic distinction. Should the agent respond identically? We present an empirical study of how demographic context influences the behaviour of policy-governed LLM agents. Building on an existing agentic evaluation framework, we augment policy-governed tasks with controlled demographic cues while keeping policies and task environments fixed, enabling behavioural differences to be attributed solely to demographic signals. To evaluate procedural fairness, we introduce three procedural dimensions: Policy Application Fairness (PAF), Procedural Effort Equity (PEE), and User Fulfillment Parity (UFP), unified into the Demographic Policy Fairness Index (DPFI). Across more than twenty thousand simulated interactions spanning multiple service domains and frontier models, all evaluated systems exhibit systematic demographic disparities that conventional task-success metrics fail to capture. Our analysis identifies three distinct failure modes with different procedural dynamics, demonstrating that aggregate success metrics can obscure how disparities emerge. Overall, the findings suggest that demographic bias in policy-governed LLM agents manifests through procedural behaviour rather than a single uniform failure pattern. The proposed framework provides a foundation for systematically auditing procedural fairness in policy-driven LLM agents.