Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generation

Yikun Wang, Rui Zheng, Haoming Li, Qi Zhang, Tao Gui, Fei Liu


Abstract
Customizing LLMs for a specific task involves separating high-quality responses from lower-quality ones. This skill can be developed using supervised fine-tuning with extensive human preference data. However, obtaining a large volume of expert-annotated data is costly for most tasks. In this paper, we explore a novel method to optimize LLMs using ranking metrics. This method trains the model to prioritize the best responses from a pool of candidates created for a particular task. Rather than a traditional full ordering, we advocate for a partial ordering, as achieving consensus on the perfect order of candidate responses can be challenging. Our partial ordering is more robust, less sensitive to noise, and can be achieved with limited human annotations or through heuristic methods. We test our system’s improved response generation ability using benchmark datasets, including textual entailment and multi-document question answering. We conduct ablation studies to understand crucial factors, such as how to gather candidate responses for a specific task, determine their most suitable order, and balance supervised fine-tuning with ranking metrics. Our approach, named RESCUE, offers a promising avenue for enhancing the response generation and task accuracy of LLMs.
Anthology ID:
2024.acl-srw.32
Volume:
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Xiyan Fu, Eve Fleisig
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
261–272
Language:
URL:
https://aclanthology.org/2024.acl-srw.32
DOI:
10.18653/v1/2024.acl-srw.32
Bibkey:
Cite (ACL):
Yikun Wang, Rui Zheng, Haoming Li, Qi Zhang, Tao Gui, and Fei Liu. 2024. Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 261–272, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
Rescue: Ranking LLM Responses with Partial Ordering to Improve Response Generation (Wang et al., ACL 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.acl-srw.32.pdf