LasQ: Largest Singular Components Fine-Tuning for LLMs with Quantization

Large language models (LLMs) have demonstrated strong capabilities in various industries, but as the model parameters increase, the computational cost of fine-tuning the entire model becomes extremely high. To address this challenge, we focus on applying quantization and LoRA fine-tuning together in pre-training scenarios and propose an efficient parameter fine-tuning (PEFT) method, the LasQ (Largest Singular Components Fine-tuning for LLMs with Quantization) framework. Performs singular value decomposition on the pre-trained weights after quantization, using high-order singular value components to initialize the low rank adapter. We evaluate our method in natural language understanding, question answering, summarization, and natural language generation tasks. The experiments show that our method can significantly outperform existing methods with fewer training parameters. Compared with LoftQ and QLoRA methods, it has a 2%–15% improvement, and it can even achieve equivalent LoRA fine-tuning effects and full parameter fine-tuning effects.