Zhuqiang Lu
Author directory2026
Charting Empirical Patterns for LLM Fine-Tuning in Scientific Multi-Discipline Learning
Lintao Wang | Zhuqiang Lu | Yilin Zhu | Kun Hu | Zhenfei Yin | Shixiang Tang | Zhiyong Wang | Wanli Ouyang | Xinzhu Ma
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Lintao Wang | Zhuqiang Lu | Yilin Zhu | Kun Hu | Zhenfei Yin | Shixiang Tang | Zhiyong Wang | Wanli Ouyang | Xinzhu Ma
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
While large language models (LLMs) perform strongly after adaptation within individual scientific domains, their learning dynamics in multi-disciplinary contexts remain poorly understood, despite the promise of improved generalization and broader applicability through cross-domain knowledge synergy. We present a systematic study of multi-discipline LLM fine-tuning across five scientific disciplines and compare the learning dynamics of full fine-tuning, LoRA, LoRA-MoE, and LoRA composition. In our naturally imbalanced setting, we show that multi-disciplinary learning is substantially more variable than single-discipline training and distill four empirical patterns: (1) Balance-then-Diversity: increasing scarce-domain exposure helps, but broad source-pool diversity must be preserved; (2) Merge-then-Align: general instruction data restores instruction-following behavior needed for effective joint adaptation; (3) Optimize-then-Scale: increasing trainable capacity offers limited gains without prior design optimization; and (4) Share-then-Specialize: asymmetric LoRA-MoE improves multi-discipline performance through a shared low-rank projection. Together, these patterns form a practical recipe for principled multi-discipline fine-tuning and provide actionable guidance for developing generalizable scientific LLMs.