Hyock Ju Kwon
2024
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning
Hossein Rajabzadeh
|
Mojtaba Valipour
|
Tianshu Zhu
|
Marzieh S. Tahaei
|
Hyock Ju Kwon
|
Ali Ghodsi
|
Boxing Chen
|
Mehdi Rezagholizadeh
Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track
Finetuning large language models requires huge GPU memory, restricting the choice to acquire Larger models. While the quantized version of the Low-Rank Adaptation technique, named QLoRA, significantly alleviates this issue, finding the efficient LoRA rank is still challenging. Moreover, QLoRA is trained on a pre-defined rank and, therefore, cannot be reconfigured for its lower ranks without requiring further fine-tuning steps. This paper proposes QDyLoRA -Quantized Dynamic Low-Rank Adaptation-, as an efficient quantization approach for dynamic low-rank adaptation. Motivated by Dynamic LoRA, QDyLoRA is able to efficiently finetune LLMs on a set of pre-defined LoRA ranks. QDyLoRA enables fine-tuning Falcon-40b for ranks 1 to 64 on a single 32 GB V100-GPU through one round of fine-tuning. Experimental results show that QDyLoRA is competitive to QLoRA and outperforms when employing its optimal rank.
Search
Co-authors
- Hossein Rajabzadeh 1
- Mojtaba Valipour 1
- Tianshu Zhu 1
- Marzieh S. Tahaei 1
- Ali Ghodsi 1
- show all...