Nguyen Viet Ha
Author directory2026
Not All Verifiable Preference Pairs Are Equal: Support-Aware Filtering for Stable Iterative DPO
Viet Anh Ngo | Long Tri Thai Son | Nguyen Viet Duc | Dong Minh Duc | Quang Minh Tran | Nguyen Thi Thuy Linh | Nguyen Viet Ha
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
Viet Anh Ngo | Long Tri Thai Son | Nguyen Viet Duc | Dong Minh Duc | Quang Minh Tran | Nguyen Thi Thuy Linh | Nguyen Viet Ha
Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
Direct Preference Optimization (DPO) is well suited to verifiable reasoning tasks because preference pairs can be constructed automatically from answer correctness. However, in iterative DPO, correctness alone does not determine whether a pair provides a useful training signal. We distinguish two regimes according to the origin of the chosen response: On-policy pairs, where both the chosen and rejected responses are sampled from the current policy, and Fallback pairs, where the policy fails to generate a correct response and the chosen completion is supplied by a dataset-provided reference solution. Based on this distinction, we propose support-aware filtering, which applies margin-based filtering to On-policy pairs and a likelihood-based support gate to Fallback pairs. Experiments on GSM8K and MATH-500 with Qwen2.5-style 0.5B, 1.5B, and 3B models show that support-aware filtering yields its clearest gains on MATH-500 at 1.5B and 3B, while GSM8K and the 0.5B MATH-500 setting are more mixed. Overall, our findings suggest that preference margins in iterative verifiable DPO are pair-origin dependent, and that chosen-response support is a useful signal for stabilizing fallback supervision.