“大语言模型在过去两年受到了极大的关注,并引起了对通用人工智能的广泛讨论。为了实现通用人工智能,合成数据被认为是其中非常关键的一环。本文将当前常见的数据合成方法归为三类,基于蒸馏的合成数据、基于模型自我进化、基于工具的合成数据。针对每一类合成数据方法,我们简要介绍了几种主流的做法,以期概览各类方法的基本思路以及异同。当前大部分合成数据方法都基于蒸馏,尽管这些方法取得了良好的效果,但其实质是将更强的大模型蒸馏到更小的大模型。这样的方法从降低大模型推理成本的角度具有实际意义,但对于进一步提升大模型能力上限作用有限。基于模型自我进化和基于工具的合成数据研究相对偏少,对于持续提升模型能力,这两个方向需要有更多探索。”

“Multi-Hop Question Answering (MHQA) is a significant area in question answering, requiringmultiple reasoning components, including document retrieval, supporting sentence prediction,and answer span extraction. In this work, we present the first application of label smoothing tothe MHQA task, aiming to enhance generalization capabilities in MHQA systems while miti-gating overfitting of answer spans and reasoning paths in the training set. We introduce a novellabel smoothing technique, F1 Smoothing, which incorporates uncertainty into the learning pro-cess and is specifically tailored for Machine Reading Comprehension (MRC) tasks. Moreover,we employ a Linear Decay Label Smoothing Algorithm (LDLA) in conjunction with curricu-lum learning to progressively reduce uncertainty throughout the training process. Experimenton the HotpotQA dataset confirms the effectiveness of our approach in improving generaliza-tion and achieving significant improvements, leading to new state-of-the-art performance on theHotpotQA leaderboard.”