SubmissionNumber#=%=#21 FinalPaperTitle#=%=#Risk-Averse Failure Modes of Jensen-Shannon Distance in RLVR Distributional Alignment ShortPaperTitle#=%=# NumberOfPages#=%=#10 CopyrightSigned#=%=#Sebastian Loftus JobTitle#==# Organization#==#Université catholique de Louvain Pl. de l'Université 1, 1348 Ottignies-Louvain-la-Neuve, Belgium Abstract#==#Jensen-Shannon Distance (JSD) appears well-suited as a reward signal for aligning model predictions with human annotations, particularly for subjective tasks where label variation reflects legitimate disagreement. We demonstrate that JSD's mathematical properties create exploitable reward landscapes in reinforcement learning. Via a study on preference distribution prediction, we show that two JSD-based reward formulations both fail to achieve alignment and instead lead to models converging at conservative, near-uniform predictions that maximize expected reward without matching human behavior. We trace this failure to JSD's non-uniform sensitivity: it penalizes errors near balanced distributions less than errors near extreme distributions, creating risk asymmetry that incentivizes safe predictions over accurate ones. Our findings reveal fundamental challenges in using bounded symmetric divergence metrics as scalar rewards for distributional alignment. Author{1}{Firstname}#=%=#Sebastian Author{1}{Lastname}#=%=#Loftus Author{1}{Username}#=%=#s.loftus Author{1}{Orcid}#=%=# Author{1}{Email}#=%=#sebastian.loftus@uclouvain.be Author{1}{Affiliation}#=%=#CENTAL - UCLouvain Author{2}{Firstname}#=%=#Xinpeng Author{2}{Lastname}#=%=#Wang Author{2}{Username}#=%=#xinpeng.wang Author{2}{Orcid}#=%=# Author{2}{Email}#=%=#xinpeng@cis.lmu.de Author{2}{Affiliation}#=%=#Ludwig-Maximilians-Universitaet Muenchen Author{3}{Firstname}#=%=#Marie-Catherine Author{3}{Lastname}#=%=#de Marneffe Author{3}{Username}#=%=#mcdm Author{3}{Orcid}#=%=# Author{3}{Email}#=%=#marie-catherine.demarneffe@uclouvain.be Author{3}{Affiliation}#=%=#FNRS - UCLouvain Author{4}{Firstname}#=%=#Barbara Author{4}{Lastname}#=%=#Plank Author{4}{Username}#=%=#bplank Author{4}{Orcid}#=%=#0000-0002-4394-1965 Author{4}{Email}#=%=#bplank@gmail.com Author{4}{Affiliation}#=%=#LMU Munich ========== èéáğö