Xianya Fang
Author directory2026
Beyond Superficial Unlearning: Sharpness-Aware Robust Erasure of Hallucinations in Multimodal LLMs
Xianya Fang | Feiyang Ren | Xiang Chen | Yu Tian | Zhen Bi | Haiyang Yu | Sheng-Jun Huang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Xianya Fang | Feiyang Ren | Xiang Chen | Yu Tian | Zhen Bi | Haiyang Yu | Sheng-Jun Huang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Multimodal LLMs are powerful but prone to object hallucinations, which describe non-existent entities and harm reliability. While recent unlearning methods attempt to mitigate this, we identify a critical flaw: structural fragility. We empirically demonstrate that standard erasure achieves only superficial suppression, trapping the model in sharp minima where hallucinations catastrophically resurge after lightweight relearning. To ensure geometric stability, we propose Sharpness-Aware Robust Erasure (SARE), which casts unlearning as a targeted min-max optimization problem and leverages a targeted Sharpness-Aware Minimization (Targeted-SAM) mechanism to explicitly flatten the loss landscape around hallucinated concepts. By suppressing hallucinations under simulated worst-case parameter perturbations, our framework ensures robust removal stable against weight shifts. Extensive experiments demonstrate that SARE outperforms baselines in erasure efficacy while preserving general generation quality. Crucially, it maintains persistent hallucination suppression against relearning and parameter updates, validating the effectiveness of geometric stabilization.