Murari Mandal
Author directory2026
Beyond Accuracy: Diagnosing Algebraic Reasoning Failures in LLMs Across Nine Complexity Dimensions
Parth Shrikant Patil | Yash Sinha | Murari Mandal | Dhruv Kumar
The Proceedings of the 4th Workshop on Mathematical Natural Language Processing (MathNLP 2026)
Parth Shrikant Patil | Yash Sinha | Murari Mandal | Dhruv Kumar
The Proceedings of the 4th Workshop on Mathematical Natural Language Processing (MathNLP 2026)
Aggregate mathematical accuracy does not usually identify the source of a model failure. In algebraic reasoning, errors may arise from nesting depth, operator identity, simultaneous intermediate values, branch-local composition, repeated-token counting, sequential dependence, or numeric magnitude. We introduce a nine-dimension algebraic complexity framework that converts these factors into controlled diagnostic variables. For each dimension, a parametric generator increases one target variable while holding the remaining variables at minimum values. Every retained problem is verified by a computer algebra system. We evaluate eight models, including six standard instruction- tuned models and two reasoning-specialised models. The results show that standard models exhibit a scale-limited D4 working-memory collapse in the 20–40 parallel-branch region. Notably, Qwen3 235B does not escape the transition band observed for Llama 3 8B, despite its much larger parameter scale. Reasoning- specialised models shift this limit to higher levels, but they do not remove it. D5 branch-local composition and D7 repeated-token counting produce distinct failures that are not reducible to aggregate length or depth.
FinBalance: A Multi-Document Accounting Reconciliation Benchmark
Sasank Tumpati | Devansh Agarwal | Ayush Kedia | Arjun Neekhra | Murari Mandal | Krishna Garg | Yash Sinha | Suman Gupta | Dhruv Kumar
Findings of the Association for Computational Linguistics: EMNLP 2026
Sasank Tumpati | Devansh Agarwal | Ayush Kedia | Arjun Neekhra | Murari Mandal | Krishna Garg | Yash Sinha | Suman Gupta | Dhruv Kumar
Findings of the Association for Computational Linguistics: EMNLP 2026
Existing financial-NLP benchmarks mostly evaluate prepared artifacts such as filings, tables, or extracted values. Real accounting begins earlier: source documents must be reconciled into cited journal entries, aggregated into a balance sheet, and checked for contradictions. We introduce FINBALANCE, a multi-document accounting reconciliation benchmark built from source-document bundles across eight industries, three period types, and five difficulty levels. Human-authored business scenarios, accounting policies, tax/FX treatments, document schemas, distractors, and inconsistency templates are composed by a deterministic generator whose ledger produces journal entries, balance sheets, and 23 inconsistency-code labels. On a 710-record evaluation split, six contemporary large language models (LLMs) reach at most 46% exact final-balance-sheet accuracy. For four models, balance sheets reconstructed from their journal entries score 26–41 percentage points higher than their own reported balance sheets. Models often recover numerically plausible entries but fail to bind them to supporting documents and aggregate them consistently. Citation-pressure prompting barely changes document-linking errors, while ledger-feedback ablations substantially improve reported balance sheets and expose inconsistency-detection trade-offs. Expert finance reviewers validate the benchmark design and labels.
When Reject Turns into Accept: Quantifying the Vulnerability of LLM-Based Scientific Reviewers to Indirect Prompt Injection
Devanshu Sahoo | Manish Prasad | Vasudev Majhi | Jahnvi Singh | Vinay Chamola | Yash Sinha | Murari Mandal | Dhruv Kumar
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Devanshu Sahoo | Manish Prasad | Vasudev Majhi | Jahnvi Singh | Vinay Chamola | Yash Sinha | Murari Mandal | Dhruv Kumar
Findings of the Association for Computational Linguistics: IJCNLP-AACL 2026
Driven by surging submission volumes, scientific peer review has catalyzed two parallel trends: individual over-reliance on LLMs and institutional AI-powered assessment systems. This study investigates the robustness of "LLM-as-a-Judge" systems to adversarial PDF manipulation via invisible text injections and layout-aware encoding attacks. We specifically target the distinct incentive of flipping "Reject” decisions to "Accept,” a vulnerability that fundamentally compromises scientific integrity. To measure this, we introduce the Weighted Adversarial Vulnerability Score (WAVS), a novel metric that quantifies susceptibility by weighting score inflation against the severity of decision shifts relative to ground truth. We adapt 15 domain-specific attack strategies, ranging from semantic persuasion to cognitive obfuscation, and evaluate them across 13 diverse language models (including GPT-5 and DeepSeek) using a curated dataset of 200 official and real-world accepted, rejected, and template papers (e.g., ICLR OpenReview). Our results demonstrate that obfuscation techniques like "Maximum Mark Magyk" successfully manipulate scores, achieving an increase in acceptance rates of up to 86.26% in open-source models, while exposing distinct "reasoning traps" in proprietary systems. We release our complete dataset and injection framework to facilitate further research on the topic (https://github.com/Manishprsd563/scientific-reviewer-jailbreak).
"I Strongly Suspect This Website Is a Scam": Benchmarking PII Leakage and Detection without Defense in Autonomous Web Agents
Soham Roy | Sarthakbrata Halder | Arya Bharaty | Vaibhav Bhaskar | Yash Sinha | Dhruv Kumar | Srikant Panda | Murari Mandal
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Soham Roy | Sarthakbrata Halder | Arya Bharaty | Vaibhav Bhaskar | Yash Sinha | Dhruv Kumar | Srikant Panda | Murari Mandal
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Deceptive web content, pervasive across the internet and commonly known as social-engineering attacks, manipulates autonomous web agents into submitting users’ personally identifiable information (PII) to attacker-controlled endpoints. In this paper, we show that these attacks are highly effective at extracting the most sensitive data an agent holds for its user (SSN, card number and CVV, bank account, passwords, API keys; the critical tier), posing a severe risk to deployed agentic systems. To quantify this risk, we introduce Scammer4U, a pre-registered benchmark of 91 attacker-controlled websites spanning 8 attack vectors and 16 site categories on an eight-axis taxonomy, together with 10 benign twins of those sites with the attack removed and the legitimate task intact. 60 of the 91 differ from a sibling site on exactly one axis, so any leakage difference within a pair is attributable to that single design choice. Under no privacy guidance, critical-tier leakage reaches 54–93% across four frontier agents, against 0% on the benign twins, so the leakage is caused by the attack and not by ordinary form-filling. Escalating prompt-level mitigation yields sharply model-dependent reductions across the four families and, pooled across them, remains insufficient to reliably prevent critical PII submission. Most critically, we identify a detection–action gap: agents whose reasoning an independent LLM judge confirms has flagged the site as suspicious still submit critical PII in 35.9% of sessions, versus 66.1% when no suspicion is verbalized, a gap of 30.2 percentage points that holds across all four model families. Defenses that gate on the agent’s own recognition of an attack are therefore gating on the wrong signal. This motivates intercepting outbound submissions at the point they leave the browser, independently of the agent’s reasoning loop.
Format Collapse in Policy Optimization for LLM Reasoning
Debdeep Sanyal | Aakash Sen Sharma | Saurabh Deshpande | Dhruv Kumar | Murari Mandal
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Debdeep Sanyal | Aakash Sen Sharma | Saurabh Deshpande | Dhruv Kumar | Murari Mandal
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Policy optimization (PO) is widely used to post-train language models for reasoning, often with a rigid <think><answer> format. We study what PO learns when this constraint is relaxed. We evaluate a relaxed reward across six models and four benchmarks, then audit learned formats on GSM8K and controlled variants across three on-policy algorithms. In these audits, accepting either explicit reasoning or a direct answer pushes the policy toward the shortest valid format: an <answer>-only response. On GSM8K, the collapsed Qwen-2.5 7B policy produces answer-only outputs in 98.6% of generations and loses 7.1 pass@5 points relative to a strict-format baseline. Controlled rewards show this is not a regex artifact: with nested formats, PO learns easy requirements before harder ones despite a 4× hard-format reward; with mutually exclusive equal-reward formats, the audited Qwen-2.5 7B final policy assigns more than 99% of generations to the easiest format. A sample-gated gradient account fits the pattern: formats receive positive updates only when sampled, so early support can outweigh reward scale. KL-controlled runs show collapse appears only when the policy can move far enough from initialization. The result identifies a practical failure mode for RLVR reward design.
OrgAccess: A Benchmark for Role-Based Access Control in Organization Scale LLMs
Debdeep Sanyal | Umakanta Maharana | Hong Ming Tan | Yash Sinha | Shirish Karande | Mohan Kankanhalli | Dhruv Kumar | Murari Mandal
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Debdeep Sanyal | Umakanta Maharana | Hong Ming Tan | Yash Sinha | Shirish Karande | Mohan Kankanhalli | Dhruv Kumar | Murari Mandal
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Conversational language models are increasingly deployed as central knowledge repositories and "conversational operating systems" in enterprise environments, where they must comply with role-based access control (RBAC) and organizational hierarchies. However, evaluating models on natural language permission compliance is difficult due to the highly proprietary nature of real-world corporate policies, and existing benchmarks do not capture the semantic nuance of overlapping or conflicting rules within free-text queries. To address this gap, we introduce **OrgAccess**, a benchmark designed to evaluate LLMs on their capacity to adhere to complex organizational hierarchies under natural language constraints. Grounded in NIST cybersecurity frameworks and validated via a multi-round expert Delphi process, OrgAccess defines 40 standard permission types across three difficulty tiers (70,000 total instances), simulating scenarios with up to five concurrent, potentially conflicting permissions. Through extensive evaluations of 16 frontier and open-source models, we systematically assess their classification accuracy and diagnostic failure modes on permission-aware query routing. Our findings reveal a pronounced performance decline: as rule complexity increases, even flagship models fail to resolve combinatorial constraint interactions, frequently yielding over-conservative "False Partial" predictions due to safety-tuning hedging. Specifically, the strongest evaluated model achieves an F1-score of only 0.27 on our hardest split. This demonstrates that current instruction-following and safety alignment paradigms do not inherently translate to robust, multi-constraint rule compliance. More broadly, OrgAccess establishes a formal, reproducible testing framework to diagnose these constraint-adherence deficits and support the systematic integration of conversational agents into highly regulated, multi-user collaborative environments. The dataset is available on the anonymous project page.
Method Ranking Inverts Across Targets: A Matched-Grid Evaluation of Jailbreak Attacks on Open-Weight Vision-Language Models
Venkat Aniruddh Thamatam | Vasudev Majhi | Aaryan Nagda | Jagat Sesh Challa | Murari Mandal | Yash Sinha | Dhruv Kumar
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Venkat Aniruddh Thamatam | Vasudev Majhi | Aaryan Nagda | Jagat Sesh Challa | Murari Mandal | Yash Sinha | Dhruv Kumar
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Jailbreak attacks are usually compared by their attack success rate (ASR), with a higher ASR read as a stronger attack. But an ASR mixes an attack’s general strength with its fit to the particular model it was measured on, and it also depends on the judge that scored it. We separate these factors with a matched evaluation grid: five attacks on eight open-weight vision–language models, 200 HarmBench behaviours per cell, every turn scored by three judges (Qwen2.5-7B, Qwen2.5-72B and Llama-Guard-3), and each ASR decomposed into general strength, target susceptibility and target-specific fit. Attack rankings invert across targets: FigStep leads PAIR on InternVL2.5-8B (13.0% vs. 3.5%, paired p=4×10-10) and trails it on Qwen2.5-VL-7B, in the same direction under all three judges. Across the grid, the target-specific component is statistically significant under every judge (p<10-12). The judge matters as well: agreement between judges is low (Cohen’s 𝜅 between 0.03 and 0.29), and a larger judge with the same rubric changes which attack ranks first on three of eight targets. A higher ASR therefore shows that an attack succeeded more often on a given model under a given judge, but not whether its advantage is general or specific to that model. We release the protocol and ∼20,400 trajectories with per-turn scores under all three judges
Anchored Turns: A Length-Driven Gradient in Chain-of-Thought Causal Necessity in Sharded Arithmetic Conversations
Dhruv Gupta | Vasudev Majhi | Jagat Sesh Challa | Murari Mandal | Dhruv Kumar
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Dhruv Gupta | Vasudev Majhi | Jagat Sesh Challa | Murari Mandal | Dhruv Kumar
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Chain-of-thought (CoT) faithfulness is measured almost exclusively in single-turn settings, leaving open how the reasoning-trace-to-answer relationship evolves as conversations lengthen. We introduce turn-level CoT counterfactual deletion: at every assistant turn we regenerate the answer at five truncation levels of the current-turn thinking block under greedy decoding, and label a turn anchored—a descriptive label for causal redundancy of the trace, not a mechanistic claim—when all five regenerations agree on the same answer. Across 67 multi-turn sharded-GSM8K conversations from DeepSeek-R1-Distill-Qwen-7B, the anchored fraction grows 3.9× from short (< 10 turns, 7.2%) to long (> 20 turns, 28.4%). Strict early < mid < late monotone ordering holds in five of seven reasoning models from three base families and four post-training pipelines (binomial sign test, p=0.002). A supplementary pooled GEE across all seven models converges on this direction (β=1.12, p=5.1×10⁻⁸, n=2,838 turn-level rows). A surface-feature predictor flags anchored turns at AUC= 0.71 (+0.07 over a length-only baseline) without additional generation, providing a cheap online signal for CoT monitors calibrated on short conversations. Single-turn CoT benchmarks, by construction, cannot detect this gradient.
2025
Nine Ways to Break Copyright Law and Why Our LLM Won’t: A Fair Use Aligned Generation Framework
Aakash Sen Sharma | Debdeep Sanyal | Priyansh Srivastava | Sundar Athreya H | Shirish Karande | Mohan Kankanhalli | Murari Mandal
Findings of the Association for Computational Linguistics: EMNLP 2025
Aakash Sen Sharma | Debdeep Sanyal | Priyansh Srivastava | Sundar Athreya H | Shirish Karande | Mohan Kankanhalli | Murari Mandal
Findings of the Association for Computational Linguistics: EMNLP 2025
Large language models (LLMs) commonly risk copyright infringement by reproducing protected content verbatim or with insufficient transformative modifications, posing significant ethical, legal, and practical concerns. Current inference-time safeguards predominantly rely on restrictive refusal-based filters, often compromising the practical utility of these models. To address this, we collaborated closely with intellectual property experts to develop LAW-LM (Legally Aware Language Model), a legally-grounded framework explicitly designed to align LLM outputs with fair-use doctrine. Central to our method is FairUseDB, a carefully constructed dataset containing 18,000 expert-validated examples covering nine realistic infringement scenarios. Leveraging this dataset, we apply Direct Preference Optimization (DPO) to fine-tune open-source LLMs, encouraging them to produce legally compliant and practically useful alternatives rather than resorting to blunt refusal. Recognizing the shortcomings of traditional evaluation metrics, we propose new measures: Weighted Penalty Utility and Compliance Aware Harmonic Mean (CAH) to balance infringement risk against response utility. Extensive quantitative experiments coupled with expert evaluations confirm that LAW-LM substantially reduces problematic outputs compared to state-of-the-art approaches, while preserving real-world usability.
ReviewEval: An Evaluation Framework for AI-Generated Reviews
Madhav Krishan Garg | Tejash Prasad | Tanmay Singhal | Chhavi Kirtani | Murari Mandal | Dhruv Kumar
Findings of the Association for Computational Linguistics: EMNLP 2025
Madhav Krishan Garg | Tejash Prasad | Tanmay Singhal | Chhavi Kirtani | Murari Mandal | Dhruv Kumar
Findings of the Association for Computational Linguistics: EMNLP 2025
The escalating volume of academic research, coupled with a shortage of qualified reviewers, necessitates innovative approaches to peer review. In this work, we propose: (1) ReviewEval, a comprehensive evaluation framework for AI-generated reviews that measures alignment with human assessments, verifies factual accuracy, assesses analytical depth, identifies degree of constructiveness and adherence to reviewer guidelines; and (2) ReviewAgent, an LLM-based review generation agent featuring a novel alignment mechanism to tailor feedback to target conferences and journals, along with a self-refinement loop that iteratively optimizes its intermediate outputs and an external improvement loop using ReviewEval to improve upon the final reviews. ReviewAgent improves actionable insights by 6.78% and 47.62% over existing AI baselines and expert reviews respectively. Further, it boosts analytical depth by 3.97% and 12.73%, enhances adherence to guidelines by 10.11% and 47.26% respectively. This paper establishes essential metrics for AI-based peer review and substantially enhances the reliability and impact of AI-generated reviews in academic research.
Investigating Pedagogical Teacher and Student LLM Agents: Genetic Adaptation Meets Retrieval-Augmented Generation Across Learning Styles
Debdeep Sanyal | Agniva Maiti | Umakanta Maharana | Dhruv Kumar | Ankur Mali | C. Lee Giles | Murari Mandal
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Debdeep Sanyal | Agniva Maiti | Umakanta Maharana | Dhruv Kumar | Ankur Mali | C. Lee Giles | Murari Mandal
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Effective teaching necessitates adapting pedagogical strategies to the inherent diversity of students, encompassing variations in aptitude, learning styles, and personality, a critical challenge in education and teacher training. Large Language Models (LLMs) offer a powerful tool to simulate complex classroom dynamics, providing a controlled environment for exploring optimal teaching patterns. However, existing simulation frameworks often fall short by neglecting comprehensive student modeling beyond basic knowledge states and, more importantly, by lacking mechanisms for teachers to dynamically adapt their approach based on student feedback and collective performance. Addressing these limitations, we propose a simulation framework that integrates LLM-based diverse student agents with a self-evolving teacher agent. We use genetic algorithms to automatically tune and optimize the teacher’s pedagogical parameters based on simulated student performance, enabling the teacher agent to discover and refine teaching patterns tailored to specific class characteristics. Complementing this, we introduce Persona-RAG, a novel Retrieval-Augmented Generation method specifically designed for personalized knowledge retrieval in pedagogical contexts, allowing students to retrieve information as per their learning styles. We show how Persona-RAG remains competitive with standard RAG baselines in accurately retrieving relevant information while adding a touch of personalization for students. Crucially, we perform extensive experiments and highlight the different patterns learnt by the teacher agent while optimizing over classes with students of various learning styles. Our work presents a significant step towards creating adaptive educational technologies and improving teacher training through realistic, data-driven simulation.
Search
Co-authors
- Dhruv Kumar 10
- Yash Sinha 6
- Debdeep Sanyal 4
- Vasudev Majhi 3
- Jagat Sesh Challa 2
- Mohan Kankanhalli 2
- Shirish Karande 2
- Umakanta Maharana 2
- Aakash Sen Sharma 2
- Devansh Agarwal 1
- Arya Bharaty 1
- Vaibhav Bhaskar 1
- Vinay Chamola 1
- Saurabh Deshpande 1
- Krishna Garg 1
- Madhav Krishan Garg 1
- C. Lee Giles 1
- Dhruv Gupta 1
- Suman Gupta 1
- Sundar Athreya H 1
- Sarthakbrata Halder 1
- Ayush Kedia 1
- Chhavi Kirtani 1
- Agniva Maiti 1
- Ankur Mali 1
- Aaryan Nagda 1
- Arjun Neekhra 1
- Srikant Panda 1
- Parth Shrikant Patil 1
- Manish Prasad 1
- Tejash Prasad 1
- Soham Roy 1
- Devanshu Sahoo 1
- Jahnvi Singh 1
- Tanmay Singhal 1
- Priyansh Srivastava 1
- Hong Ming Tan 1
- Venkat Aniruddh Thamatam 1
- Sasank Tumpati 1