QDLoRA: Enhanced LoRA Fine-Tuning on Quantized LLMs via Integrated Low-Rank Decomposition

We propose QDLoRA, a parameter-efficient fine-tuning (PEFT) framework that integrates low-rank decomposition and quantization into the LoRA fine-tuning process for pretrained large language models (LLMs). Unlike prior methods such as LoftQ and ApiQ that rely solely on quantization and suffer performance degradation under extreme compression, QDLoRA preserves more informative structure at the same compression ratio, thereby improving fine-tuning results. To further enhance robustness, QDLoRA introduces a similarity-aware rank selection strategy and a quantization-aware initialization scheme. Experimental results on various model architectures across diverse NLP benchmarks demonstrate that QDLoRA achieves superior accuracy and efficiency compared to existing methods, particularly under limited resource budgets. The proposed method offers a practical and scalable solution for efficient fine-tuning of large language models.