Knowledge Distillation-Based Federated Fine-Tuning under Resource Constraints

Model fine-tuning is an effective machine learning method that allows pre-trained models to adapt to different downstream tasks. However, in the context of federated learning, it is not realistic to directly fine-tune the pre-trained model due to the storage, computation, and communication resources constraints of user devices. The communication overhead of traditional federated learning increases with the size of the model. In addition, sharing locally trained models may pose a risk of privacy leakage. To address the aforementioned issues, we propose knowledge distillation-based federated fine-tuning. We use LoRA, a parameter efficient fine-tuning method, to reduce user storage overhead during local training epoch. We implement model-free federated learning using knowledge distillation, eliminating the impact of LoRA model dimensions on communication volume. Our algorithm effectively avoids the high communication overhead caused by multiple exchanges of pre-trained model parameters, while protecting user privacy. The experimental results on the image classification dataset SVHN and CIFAR10 demonstrate the effectiveness of our algorithm.