Large Language Models (LLMs) have displayed remarkable capabilities across diverse domains, but their training remains resource – and time-intensive, requiring massive computational resources and careful orchestration of training procedures. Model souping – the practice of averaging weights from multiple models of the same architecture – has emerged as a promising pre- and post-training technique that can enhance performance without expensive retraining. We observe that previous souping approaches can lead to collapse in precision-sensitive LLM capabilities. In this paper, we introduce Soup of Category Experts (SoCE), a principled approach for model souping to overcome this shortcoming. The proposed method utilizes benchmark composition to identify optimal model candidates and applies non-uniform weighted averaging to maximize performance. Contrary to previous approaches, our method leverages the observation that different clusters (or categories) of points within a benchmark often exhibit low inter-correlations in model performance. SoCE identifies "expert" models for each weakly-correlated category cluster and combines them using optimized weighted averaging rather than uniform weights. We demonstrate that SoCE improves performance and robustness across multiple domains achieves state-of-the-art results on the Berkeley Function Calling Leaderboard.

Language technologies contribute to promoting multilingualism and linguistic diversity around the world. However, only a very small number of the over 7000 languages of the world are represented in the rapidly evolving language technologies and applications. In this paper we look at the relation between the types of languages, resources, and their representation in NLP conferences to understand the trajectory that different languages have followed over time. Our quantitative investigation underlines the disparity between languages, especially in terms of their resources, and calls into question the “language agnostic” status of current models and systems. Through this paper, we attempt to convince the ACL community to prioritise the resolution of the predicaments highlighted here, so that no language is left behind.