AdaDQ-KD: An Adaptive Dithering Quantization with Knowledge Distillation in Privacy-Preserving Federated Learning

Dithering quantization (DQ) is a promising Differential Privacy (DP) approach designed for Federated Learning (FL) to prevent privacy leakage of clients. However, DQ-FL inevitably exacerbates the decline in model accuracy in FL, particularly under conditions characterized by straggler effects and statistical heterogeneity across clients. In this paper, we propose a novel Adaptive Dithering Quantization with Knowledge Distillation (AdaDQ-KD) algorithm to improve the training efficiency within a privacy-preserving FL framework. Specifically, an adaptive dithering quantization scheme is developed, which dynamically tunes the dithering quantization intervals to decrease communication overhead based on the clients' local delays. Moreover, we introduce a novel training strategy through a knowledge distillation module, which integrates a feature distillation loss into the training loss function to mitigate the accuracy loss induced by noise injection and gradient quantization. Theoretical analysis is conducted to evaluate the privacy guarantees, efficiency and convergence properties of the proposed AdaDQ-KD algorithm. Experimental results demonstrate that the AdaDQ-KD algorithm outperforms state-of-the-art (SOTA) methods across various settings, achieving a better trade-off among privacy, training efficiency, and model accuracy.