Ervin Dervishaj

Author directory

2026

Model merging allows extending an expert model’s capabilities by fusing its weights with another expert. However, task interference has been shown to negatively affect mergeability. To address this, prior work has mainly focused on introducing new merging methods. Despite reducing task interference, the effectiveness of these merging methods is only evident post-merging, requiring a full evaluation of the merged model, which for contemporary LLMs can be expensive. In this work, we study the overlooked predictive power of structural and functional properties of merged experts for mergeability. Our experiments with 8 expert LLMs spanning several programming languages, math, and knowledge show that activation L2 distance and activation cosine similarity emerge as the most important properties for predicting effective merging, contrasting with prior findings on vision models where gradient-based metrics dominated. We further show that task vector L2 distance, a cheaply computed structural metric, provides a mechanistic link between mergeability prediction and training dynamics: it explains why learning rate effects on mergeability are domain-dependent in LLMs, contradicting the universal “higher noise improves merging” principle from vision. This enables a practical intervention where mergeability metrics guide learning rate selection during expert training.