Amit Roy-Chowdhury
Author directory2026
GRAIL: Grounded Visual Alignment of Medical Vision-Language Models for Radiology Report Generation
Sarosij Bose | Ravi K. Rajendran | Biplob Debnath | Konstantinos Karydis | Amit Roy-Chowdhury | Srimat Chakradhar
Findings of the Association for Computational Linguistics: EMNLP 2026
Sarosij Bose | Ravi K. Rajendran | Biplob Debnath | Konstantinos Karydis | Amit Roy-Chowdhury | Srimat Chakradhar
Findings of the Association for Computational Linguistics: EMNLP 2026
Radiology Report Generation (RRG) aims to reduce the burden of clinical documentation, yet generating reports that remain visually grounded in the input image remains challenging. Current medical vision-language models (Med-VLMs) often rely on curated preference data, retrieval augmentation, or large-scale labeled corpora, but remain prone to hallucinated findings due to weak cross-modal alignment. To address this challenge, we propose GRAIL: Grounded Radiology Alignment of Medical Vision-Language Models for Radiology Report Generation, a two-stage multimodal alignment framework for grounded RRG. First, Clinically Informed Textual Reasoning improves report structure and medical terminology using verifiable language rewards with periodic clinical guidance. Second, Expert-guided Visual Reasoning leverages a domain expert to rank generated reports using image-report alignment scores, encouraging visually grounded report generation without curated preference pairs or retrieval data. Experiments on multiple benchmarks show that GRAIL consistently improves both report generation quality and clinical accuracy, achieving relative gains over strong post-training baselines by up to 15% in clinical accuracy and 13% in generation quality. GRAIL also outperforms proprietary general-purpose and medical VLMs, highlighting the effectiveness of self-supervised multimodal post-alignment for reducing hallucinations in visually grounded radiology report generation.
FORGE: Forensic Reasoning with Grounded Evidence
Rohit Kundu | Shan Jia | Vishal Mohanty | Athula Balachandran | Amit Roy-Chowdhury
Findings of the Association for Computational Linguistics: EMNLP 2026
Rohit Kundu | Shan Jia | Vishal Mohanty | Athula Balachandran | Amit Roy-Chowdhury
Findings of the Association for Computational Linguistics: EMNLP 2026
Forensic deepfake analysis demands more than binary classification: investigators need region-grounded natural language explanations they can verify against the image. Multimodal large language models (MLLMs) are a natural fit, but pretrained MLLMs fail systematically, producing globally coherent text that misses the small localized cues defining manipulations. We argue this is an inductive bias problem rather than a capacity issue: the image-text contrastive objective training MLLM visual encoders optimizes for whole-image semantic summaries, not patch-level forensic detail. The same mismatch explains why prior deepfake reasoning methods target either face manipulation or fully AI-generated content, never both.
We propose \textcolor{blue}{\texttt{FORGE}}, which addresses the mismatch by routing a second visual stream into the language model from a Vision-Only Model (VOM) trained on dense patch prediction rather than image-text alignment. The MLLM’s native encoder and the VOM operate on a shared patch grid, which lets us interleave their tokens with preserved spatial correspondence; we show this beats naive concatenation. A two-stage adapter training protocol (generic image-caption alignment, then joint task-specific optimization) prevents the localized stream from overfitting to training-domain manipulations. Across face-manipulated and fully synthetic content, \textcolor{blue}{\texttt{FORGE}} produces region-referential explanations answering fine-grained attribute queries (“Does the eyes/nose/mouth look real or fake?”) and substantially outperforms in-domain baselines on cross-domain evaluations; region-specific evaluation and human studies confirm explanation faithfulness.
We propose \textcolor{blue}{\texttt{FORGE}}, which addresses the mismatch by routing a second visual stream into the language model from a Vision-Only Model (VOM) trained on dense patch prediction rather than image-text alignment. The MLLM’s native encoder and the VOM operate on a shared patch grid, which lets us interleave their tokens with preserved spatial correspondence; we show this beats naive concatenation. A two-stage adapter training protocol (generic image-caption alignment, then joint task-specific optimization) prevents the localized stream from overfitting to training-domain manipulations. Across face-manipulated and fully synthetic content, \textcolor{blue}{\texttt{FORGE}} produces region-referential explanations answering fine-grained attribute queries (“Does the eyes/nose/mouth look real or fake?”) and substantially outperforms in-domain baselines on cross-domain evaluations; region-specific evaluation and human studies confirm explanation faithfulness.
2025
HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models
Trishna Chakraborty | Udita Ghosh | Xiaopan Zhang | Fahim Faisal Niloy | Yue Dong | Jiachen Li | Amit Roy-Chowdhury | Chengyu Song
Findings of the Association for Computational Linguistics: EMNLP 2025
Trishna Chakraborty | Udita Ghosh | Xiaopan Zhang | Fahim Faisal Niloy | Yue Dong | Jiachen Li | Amit Roy-Chowdhury | Chengyu Song
Findings of the Association for Computational Linguistics: EMNLP 2025
Large language models (LLMs) are increasingly being adopted as the cognitive core of embodied agents. However, inherited hallucinations, which stem from failures to ground user instructions in the observed physical environment, can lead to navigation errors, such as searching for a refrigerator that does not exist. In this paper, we present the first systematic study of hallucinations in LLM-based embodied agents performing long-horizon tasks under scene–task inconsistencies. Our goal is to understand to what extent hallucinations occur, what types of inconsistencies trigger them, and how current models respond. To achieve these goals, we construct a hallucination probing set by building on an existing benchmark, capable of inducing hallucination rates up to 40× higher than base prompts. Evaluating 12 models across two simulation environments, we find that while models exhibit reasoning, they fail to resolve scene-task inconsistencies — highlighting fundamental limitations in handling infeasible tasks. We also provide actionable insights on ideal model behavior for each scenario, offering guidance for developing more robust and reliable planning strategies.
2024
Can Textual Unlearning Solve Cross-Modality Safety Alignment?
Trishna Chakraborty | Erfan Shayegani | Zikui Cai | Nael B. Abu-Ghazaleh | M. Salman Asif | Yue Dong | Amit Roy-Chowdhury | Chengyu Song
Findings of the Association for Computational Linguistics: EMNLP 2024
Trishna Chakraborty | Erfan Shayegani | Zikui Cai | Nael B. Abu-Ghazaleh | M. Salman Asif | Yue Dong | Amit Roy-Chowdhury | Chengyu Song
Findings of the Association for Computational Linguistics: EMNLP 2024
Recent studies reveal that integrating new modalities into large language models (LLMs), such as vision-language models (VLMs), creates a new attack surface that bypasses existing safety training techniques like supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF). While further SFT and RLHF-based safety training can be conducted in multi-modal settings, collecting multi-modal training datasets poses a significant challenge. Inspired by the structural design of recent multi-modal models, where all input modalities are ultimately fused into the language space, we explore whether unlearning solely in the textual domain can be effective for cross-modality safety alignment. Our empirical evaluation across seven datasets demonstrates promising transferability — textual unlearning in VLMs significantly reduces the Attack Success Rate (ASR) to less than 8% and in some cases, even as low as nearly 2% for both text-based and vision-text-based attacks, alongside preserving the utility. Moreover, our experiments show that unlearning with a multi-modal dataset offers no potential benefits but incurs significantly increased computational demands.