MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation

Yusheng Liao, Shuyang Jiang, Zhe Chen, Yu Wang, Yanfeng Wang


Abstract
Large language models (LLMs) have shown substantial progress in natural language understanding and generation, proving valuable especially in the medical field. Despite advancements, challenges persist due to the complexity and diversity inherent in medical tasks, which can be categorized as knowledge-intensive tasks and alignment-required tasks. Previous approaches either ignore the latter task or focus on a minority of tasks and hence lose generalization. To address these drawbacks, we propose a progressive fine-tuning pipeline. This pipeline employs a and a to encode diverse knowledge in the first stage and filter out detrimental information. In the second stage, we drop the to avoid the interference of suboptimal representation and leverage an additional alignment module optimized towards an orthogonal direction to the knowledge space to mitigate knowledge forgetting. Based on this two-stage paradigm, we proposed a Medical LLM through decoupling Clinical Alignment and Knowledge Aggregation (), which is designed to achieve promising performance on over 20 medical tasks, as well as results on specific medical alignment tasks. Various model sizes of (1.8B, 7B, 14B) all demonstrate significant improvements over existing models with similar model sizes. Our code and datasets are available at https://github.com/BlueZeros/MedCare.
Anthology ID:
2024.findings-emnlp.619
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2024
Month:
November
Year:
2024
Address:
Miami, Florida, USA
Editors:
Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
10562–10581
Language:
URL:
https://aclanthology.org/2024.findings-emnlp.619
DOI:
10.18653/v1/2024.findings-emnlp.619
Bibkey:
Cite (ACL):
Yusheng Liao, Shuyang Jiang, Zhe Chen, Yu Wang, and Yanfeng Wang. 2024. MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10562–10581, Miami, Florida, USA. Association for Computational Linguistics.
Cite (Informal):
MedCare: Advancing Medical LLMs through Decoupling Clinical Alignment and Knowledge Aggregation (Liao et al., Findings 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.findings-emnlp.619.pdf