Zhuqiang Lu

Author directory

2026

While large language models (LLMs) perform strongly after adaptation within individual scientific domains, their learning dynamics in multi-disciplinary contexts remain poorly understood, despite the promise of improved generalization and broader applicability through cross-domain knowledge synergy. We present a systematic study of multi-discipline LLM fine-tuning across five scientific disciplines and compare the learning dynamics of full fine-tuning, LoRA, LoRA-MoE, and LoRA composition. In our naturally imbalanced setting, we show that multi-disciplinary learning is substantially more variable than single-discipline training and distill four empirical patterns: (1) Balance-then-Diversity: increasing scarce-domain exposure helps, but broad source-pool diversity must be preserved; (2) Merge-then-Align: general instruction data restores instruction-following behavior needed for effective joint adaptation; (3) Optimize-then-Scale: increasing trainable capacity offers limited gains without prior design optimization; and (4) Share-then-Specialize: asymmetric LoRA-MoE improves multi-discipline performance through a shared low-rank projection. Together, these patterns form a practical recipe for principled multi-discipline fine-tuning and provide actionable guidance for developing generalizable scientific LLMs.