Large Language Models (LLMs) have displayed remarkable capabilities across diverse domains, but their training remains resource – and time-intensive, requiring massive computational resources and careful orchestration of training procedures. Model souping – the practice of averaging weights from multiple models of the same architecture – has emerged as a promising pre- and post-training technique that can enhance performance without expensive retraining. We observe that previous souping approaches can lead to collapse in precision-sensitive LLM capabilities. In this paper, we introduce Soup of Category Experts (SoCE), a principled approach for model souping to overcome this shortcoming. The proposed method utilizes benchmark composition to identify optimal model candidates and applies non-uniform weighted averaging to maximize performance. Contrary to previous approaches, our method leverages the observation that different clusters (or categories) of points within a benchmark often exhibit low inter-correlations in model performance. SoCE identifies "expert" models for each weakly-correlated category cluster and combines them using optimized weighted averaging rather than uniform weights. We demonstrate that SoCE improves performance and robustness across multiple domains achieves state-of-the-art results on the Berkeley Function Calling Leaderboard.

Large language models are increasingly used through multi-turn refinement loops, but most evidence for iterative improvement comes from settings with informative feedback, external tools, full conversation history, or structured self-critique. Less is known about a configuration, uninformative memoryless refinement, where a model repeatedly revises only its previous output under vague instructions such as “improve it”. In this paper, we establish a framework to analyze when this uninformative memoryless refinement configuration improves LLM responses across ideation, coding, and mathematical reasoning. We evaluate seven LLMs on a 450-task pool under vague and domain-specific feedback. We score every turn and measure both task outcomes and trajectory dynamics, including semantic drift, volatility, lexical novelty, and output growth. This trajectory view reveals a consistent discovery–retention gap: models often produce better intermediate outputs before overwriting them. The effect is most severe in coding, where correctness falls from 40.7% at T1 to 2.5% at T12, while best@12 exceeds final@12 by 42.5 percentage points. Math shows model- and prompt-dependent behavior, with GPT-OSS-20B improving under elaboration but several models drifting downward. Ideation benefits from explicit steering, but vague feedback produces rhetorical inflation and feasibility loss.Our analysis stresses the importance of turn-wise, trajectory-aware evaluation of uninformative memoryless refinement and shows why verified intermediate outputs should be retained rather than automatically replaced by later revisions.