Reward-aware variants of Direct Preference Optimization (DPO), such as ODPO and MODPO, implicitly assume linear marginal utility over the preference signal: identical signal gaps induce identical gradient updates regardless of whether the pair lies in a low- or high-quality region. We propose FreeMU, a lightweight plug-and-play training module that introduces non-linear marginal utility through a pair-aware divided-difference weight applied directly to the preference margin. The same weight formula integrates with three representative DPO variants—ODPO, MODPO, and β-DPO—with each instantiation differing only in the reward source and its normalisation. We first instantiate FreeMU on the two explicit-score variants (Likert labels for ODPO and reward-model outputs for MODPO) and then, for variants without externally annotated scores, on β-DPO using the implicit Bradley–Terry reward induced by the policy and reference models. Across summarization, multi-objective safety alignment, and dialogue, FreeMU consistently improves over its host in LLM-based win rates and task-specific metrics with near-zero training overhead; a controlled ablation confirms that the convex curvature direction is the operative ingredient—reversing the sign to a concave utility removes the gain.