Bohao Lv
Author directory2026
MoralSteer: Safety-Preserving Fidelity Gains in Morally Complex Role-Playing via Theory-Guided On-Policy Self-Distillation
Shaojie Shi | Teqi Hao | Bohao Lv | Rui Xu | Zhengyu Shi | Xu Yinghui
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Shaojie Shi | Teqi Hao | Bohao Lv | Rui Xu | Zhengyu Shi | Xu Yinghui
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Aligned LLMs systematically fail at morally complex role-playing: villains collapse into prosocial defaults due to a structural safety and fidelity tension. We present MoralSteer, a theory-guided on-policy self-distillation framework that improves morally complex role-play fidelity while preserving the safety behavior of aligned base models. MoralSteer treats Moral Foundations Theory (MFT) reasoning chains as privileged information for a frozen teacher, introduces a moral-complexity-adaptive KL interpolation 𝛼(c)=𝜎(M(c)-𝜏) to prevent prosocial collapse, and couples this with M(c)-adaptive trajectory mixing to mitigate rollout degeneration. Across four base models, MoralSteer improves Moral RolePlay L3+L4 scores by +5.8∼+9.9%; across five safety benchmarks, the observed rate differences remain within ±1pp of the aligned bases. We find that villain failure concentrates in Care/Fairness foundations (67% of the L4 gap) and that aligned models perform trait substitution rather than deletion. Our code is available at https://github.com/Sii-yuning/MoralSteer.