Umakanta Maharana
Author directory2026
OrgAccess: A Benchmark for Role-Based Access Control in Organization Scale LLMs
Debdeep Sanyal | Umakanta Maharana | Hong Ming Tan | Yash Sinha | Shirish Karande | Mohan Kankanhalli | Dhruv Kumar | Murari Mandal
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Debdeep Sanyal | Umakanta Maharana | Hong Ming Tan | Yash Sinha | Shirish Karande | Mohan Kankanhalli | Dhruv Kumar | Murari Mandal
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Conversational language models are increasingly deployed as central knowledge repositories and "conversational operating systems" in enterprise environments, where they must comply with role-based access control (RBAC) and organizational hierarchies. However, evaluating models on natural language permission compliance is difficult due to the highly proprietary nature of real-world corporate policies, and existing benchmarks do not capture the semantic nuance of overlapping or conflicting rules within free-text queries. To address this gap, we introduce **OrgAccess**, a benchmark designed to evaluate LLMs on their capacity to adhere to complex organizational hierarchies under natural language constraints. Grounded in NIST cybersecurity frameworks and validated via a multi-round expert Delphi process, OrgAccess defines 40 standard permission types across three difficulty tiers (70,000 total instances), simulating scenarios with up to five concurrent, potentially conflicting permissions. Through extensive evaluations of 16 frontier and open-source models, we systematically assess their classification accuracy and diagnostic failure modes on permission-aware query routing. Our findings reveal a pronounced performance decline: as rule complexity increases, even flagship models fail to resolve combinatorial constraint interactions, frequently yielding over-conservative "False Partial" predictions due to safety-tuning hedging. Specifically, the strongest evaluated model achieves an F1-score of only 0.27 on our hardest split. This demonstrates that current instruction-following and safety alignment paradigms do not inherently translate to robust, multi-constraint rule compliance. More broadly, OrgAccess establishes a formal, reproducible testing framework to diagnose these constraint-adherence deficits and support the systematic integration of conversational agents into highly regulated, multi-user collaborative environments. The dataset is available on the anonymous project page.
2025
Investigating Pedagogical Teacher and Student LLM Agents: Genetic Adaptation Meets Retrieval-Augmented Generation Across Learning Styles
Debdeep Sanyal | Agniva Maiti | Umakanta Maharana | Dhruv Kumar | Ankur Mali | C. Lee Giles | Murari Mandal
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Debdeep Sanyal | Agniva Maiti | Umakanta Maharana | Dhruv Kumar | Ankur Mali | C. Lee Giles | Murari Mandal
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Effective teaching necessitates adapting pedagogical strategies to the inherent diversity of students, encompassing variations in aptitude, learning styles, and personality, a critical challenge in education and teacher training. Large Language Models (LLMs) offer a powerful tool to simulate complex classroom dynamics, providing a controlled environment for exploring optimal teaching patterns. However, existing simulation frameworks often fall short by neglecting comprehensive student modeling beyond basic knowledge states and, more importantly, by lacking mechanisms for teachers to dynamically adapt their approach based on student feedback and collective performance. Addressing these limitations, we propose a simulation framework that integrates LLM-based diverse student agents with a self-evolving teacher agent. We use genetic algorithms to automatically tune and optimize the teacher’s pedagogical parameters based on simulated student performance, enabling the teacher agent to discover and refine teaching patterns tailored to specific class characteristics. Complementing this, we introduce Persona-RAG, a novel Retrieval-Augmented Generation method specifically designed for personalized knowledge retrieval in pedagogical contexts, allowing students to retrieve information as per their learning styles. We show how Persona-RAG remains competitive with standard RAG baselines in accurately retrieving relevant information while adding a touch of personalization for students. Crucially, we perform extensive experiments and highlight the different patterns learnt by the teacher agent while optimizing over classes with students of various learning styles. Our work presents a significant step towards creating adaptive educational technologies and improving teacher training through realistic, data-driven simulation.