From Evasive Servants to Dignified Peers via Dual Preference Optimization

Xiangqi Wang, Yue Huang, Haomin Zhuang, Kehan Guo, Xiangliang Zhang


Abstract
Current aligned language models exhibit a dual failure mode we term the Evasive Servant: they sycophantically validate flawed user beliefs while deflecting responsibility with boilerplate disclaimers. We propose the Dignified Peer framework, which targets servility through anti-sycophancy and trustworthiness, and evasiveness through empathy and creativity. Operationalizing this framework requires addressing three concrete gaps: lack of fine-grained per-dimension supervision, objective imbalance during multi-objective optimization, and confound-prone evaluation. We introduce the PersonaKnob dataset, which encodes a compositional partial-order structure of multi-persona preferences, and pair it with Lag-DPO, a tolerance-constrained Lagrangian DPO algorithm that dynamically rebalances dimensions to mitigate collapse. For evaluation, we adapt the Many-Facet Rasch Model (MFRM) to disentangle model capability from judge severity, rubric difficulty, and question complexity, and we validate MFRM scores against human pairwise preferences (Spearman 𝜌 ∈ [0.67, 0.85] across dimensions). Across two open-source backbones, Lag-DPO improves all four dimensions in distribution and reduces sycophancy while preserving safe-refusal balance on out-of-distribution benchmarks; we report these results with explicit weight-sensitivity and uncertainty analyses rather than claim a fully resolved Dignified Peer.
Anthology ID:
2026.findings-emnlp.197
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2026
Month:
October
Year:
2026
Address:
Budapest, Hungary
Editors:
Hung-Yi Lee, Sunipa Dev, Jessy Li, Desmond Elliott
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
4164–4186
Language:
URL:
https://aclanthology.org/2026.findings-emnlp.197/
DOI:
Bibkey:
Cite (ACL):
Xiangqi Wang, Yue Huang, Haomin Zhuang, Kehan Guo, and Xiangliang Zhang. 2026. From Evasive Servants to Dignified Peers via Dual Preference Optimization. In Findings of the Association for Computational Linguistics: EMNLP 2026, pages 4164–4186, Budapest, Hungary. Association for Computational Linguistics.
Cite (Informal):
From Evasive Servants to Dignified Peers via Dual Preference Optimization (Wang et al., Findings 2026)
Copy Citation:
PDF:
https://aclanthology.org/2026.findings-emnlp.197.pdf