Anmeng Zhang

Author directory

2026

Reinforcement learning is increasingly used to align multimodal large language models (MLLMs). However, higher proxy rewards do not necessarily lead to better task performance, especially when visual evidence is difficult to verify with text-only or weakly grounded reward signals. We systematically study reward hacking in MLLM reinforcement learning by varying reward design, data ambiguity, model scale, and RL algorithm. To separate RL-induced failures from pre-existing weaknesses, we introduce the Newly Rewarded Failure Rate (NRFR), which measures the hacking rate among samples where RL achieves higher proxy reward than the SFT baseline. Our analysis yields three main findings. First, RL can actively create new failures rather than merely amplify existing ones, and the effect is especially pronounced near ambiguous decision boundaries. Second, robustness depends jointly on model scale, reward design, and RL algorithm, and scaling mitigates hacking but cannot replace reliable rewards. Third, incorporating visual evidence helps only when verification is reliable, since keyword-based verification introduces additional exploitation opportunities whereas VLM-based semantic verification can reduce reward hacking. Overall, multimodal reward hacking is a systematic consequence of optimizing imperfect reward signals, highlighting the need for rewards and verifiers that remain reliable under optimization pressure.