2024
pdf
bib
abs
Conditional Language Policy: A General Framework For Steerable Multi-Objective Finetuning
Kaiwen Wang
|
Rahul Kidambi
|
Ryan Sullivan
|
Alekh Agarwal
|
Christoph Dann
|
Andrea Michi
|
Marco Gelmi
|
Yunxuan Li
|
Raghav Gupta
|
Kumar Dubey
|
Alexandre Rame
|
Johan Ferret
|
Geoffrey Cideron
|
Le Hou
|
Hongkun Yu
|
Amr Ahmed
|
Aranyak Mehta
|
Leonard Hussenot
|
Olivier Bachem
|
Edouard Leurent
Findings of the Association for Computational Linguistics: EMNLP 2024
Reward-based finetuning is crucial for aligning language policies with intended behaviors (*e.g.*, creativity and safety). A key challenge is to develop steerable language models that trade-off multiple (conflicting) objectives in a flexible and efficient manner. This paper presents Conditional Language Policy (CLP), a general framework for finetuning language models on multiple objectives. Building on techniques from multi-task training and parameter-efficient finetuning, CLP learn steerable models that effectively trade-off conflicting objectives at *inference time*. Notably, this does not require training or maintaining multiple models to achieve different trade-offs between the objectives. Through extensive experiments and ablations on two summarization datasets, we show that CLP learns steerable language models that outperform and Pareto-dominate the existing approaches for multi-objective
pdf
bib
abs
Improving Multi-Agent Debate with Sparse Communication Topology
Yunxuan Li
|
Yibing Du
|
Jiageng Zhang
|
Le Hou
|
Peter Grabowski
|
Yeqing Li
|
Eugene Ie
Findings of the Association for Computational Linguistics: EMNLP 2024
Multi-agent debate has proven effective in improving large language models quality for reasoning and factuality tasks. While various role-playing strategies in multi-agent debates have been explored, in terms of the communication among agents, existing approaches adopt a brute force algorithm – each agent can communicate with all other agents. In this paper, we systematically investigate the effect of communication connectivity in multi-agent systems. Our experiments on GPT and Mistral models reveal that multi-agent debates leveraging sparse communication topology can achieve comparable or superior performance while significantly reducing computational costs. Furthermore, we extend the multi-agent debate framework to multi-modal reasoning and alignment labeling tasks, showcasing its broad applicability and effectiveness. Our findings underscore the importance of communication connectivity on enhancing the efficiency and effectiveness of the “society of minds” approach.
pdf
bib
abs
Multi-step Problem Solving Through a Verifier: An Empirical Analysis on Model-induced Process Supervision
Zihan Wang
|
Yunxuan Li
|
Yuexin Wu
|
Liangchen Luo
|
Le Hou
|
Hongkun Yu
|
Jingbo Shang
Findings of the Association for Computational Linguistics: EMNLP 2024
Process supervision, using a trained verifier to evaluate the intermediate steps generated by a reasoner, has demonstrated significant improvements in multi-step problem solving. In this paper, to avoid the expensive effort of human annotation on the verifier training data, we introduce Model-induced Process Supervision (MiPS), a novel method for automating data curation. MiPS annotates an intermediate step by sampling completions of this solution through the reasoning model, and obtaining an accuracy defined as the proportion of correct completions. Inaccuracies of the reasoner would cause MiPS underestimating the accuracy of intermediate steps, therefore, we suggest and empirically show that verification focusing on high predicted scores of the verifier shall be preferred over that of low predicted scores, contrary to prior observations on human curated data. Our approach significantly improves the performance of PaLM 2 on math and coding tasks (accuracy +0.67% on GSM8K, +4.16% on MATH, +0.92% on MBPP compared with an output supervision trained verifier). Additionally, our study demonstrates that the verifier exhibits strong generalization ability across different reasoning models.
pdf
bib
abs
Learning to Ask Denotative and Connotative Questions for Knowledge-based VQA
Xiaoying Xing
|
Peixi Xiong
|
Lei Fan
|
Yunxuan Li
|
Ying Wu
Findings of the Association for Computational Linguistics: EMNLP 2024
Large language models (LLMs) have attracted increasing attention due to its prominent performance on various tasks. Recent works seek to leverage LLMs on knowledge-based visual question answering (VQA) tasks which require common sense knowledge to answer the question about an image, since LLMs have obtained rich knowledge from large-scale training. Several methods have proposed to leverage frozen LLMs by converting visual information to textual prompts. However, how to efficiently exploit the knowledge of LLMs and bridge the disconnects between visual information and language models remain open problems. In this paper, we propose to let LLMs learn to ask (L2A) informative questions to collect essential visual information. We introduce the concepts of denotation and connotation to promote image and question understanding and provide a clear guidance with respect to the objective of question generation. In this way, the model can better capture the associations between different concepts, as well as efficiently collect both explicit information and implicit relevant information that contribute to the final answer. The experiments demonstrate that our proposed method achieves consistent performance on various knowledge-based VQA datasets.