Model diffing studies the internal changes of a model from training to reveal learning mechanisms. To enhance interpretability, existing studies often strive for structural sparsity by removing redundant model components. However, we emphasize that for most fine-tuning methods, the fundamental source of internal change, i.e., the parameter updates, is non-sparse. The dense update mechanism contradicts the structural sparsity sought by researchers. Thus, we seek to bring sparsity to fine-tuning via an interpretability-driven method called circuit-tuning. It employs a heuristic prune-then-tune paradigm, forcing the model to optimize within a restricted subset of parameters. To test our method, we designed a novel task called Subject-Verb Disagreement (SVD) and conducted comprehensive explorations on it. Experimental results demonstrate that, compared to other fine-tuning methods, circuit-tuning achieves better sparsity in parameter updates, thus making it possible for faithful model diffing. Based on this, we performed stage-wise model diffing, revealing the evolution of model capabilities during fine-tuning from a new view. Furthermore, experiments on gender debiasing and larger models verify the practical utility of our approach. Our work provides a new methodology for model diffing-based interpretability and its applications.

Large language models (LLMs) have shown excellent mastering of human language but still struggle in real-world applications that require mathematical problem-solving. While many strategies and datasets to enhance LLMs’ mathematics are developed, it remains a challenge to simultaneously maintain and improve both language and mathematical capabilities in deployed LLM systems. In this work, we tailor the Self-Critique pipeline, which addresses the challenge in the feedback learning stage of LLM alignment. We first train a general Math-Critique model from the LLM itself to provide feedback signals. Then, we sequentially employ rejective fine-tuning and direct preference optimization over the LLM’s own generations for data collection. Based on ChatGLM3-32B, we conduct experiments on both academic and our newly created challenging dataset, MathUserEval. Results show that our pipeline significantly enhances the LLM’s mathematical problem-solving while still improving its language ability, outperforming LLMs that could be two times larger. Related techniques have been deployed to ChatGLM, an online serving LLM. Related evaluation datasets and scripts are released at https://github.com/THUDM/ChatGLM-Math.