Deep Learning Algorithms Design and Implementation Based on Differential Privacy

Deep learning models bear the risks of privacy leakage. Attackers can obtain sensitive information contained in training data with some techniques. However, existing differentially private methods such as Differential Privacy-Stochastic Gradient Descent (DP-SGD) and Differential Privacy-Generative Adversarial Network (DP-GAN) are not very efficient as they require to perform sampling multiple times. More importantly, DP-GAN algorithm need public data to set gradient clipping threshold. In this paper, we introduce our refined algorithms to tackle these problems. First, we employ random shuffling instead of random sampling to improve training efficiency. We also test Gaussian and Laplace Mechanisms for clipping gradients and injecting noise. Second, we employ zero Concentrated Differential Privacy (zCDP) to compute overall privacy budget. Finally, we adopt dynamical gradient clipping in DP-GAN algorithm. During each iteration, we random sample training examples and set the average gradients norm as the new threshold. This not only makes the algorithm more robust but also doesn’t increase the overall privacy budget. We experiment with our algorithms on MNIST data sets and demonstrate the accuracies. In our refined DP-SGD algorithm, we achieve test accuracy of 96.58%. In our refined DP-GAN algorithm, we adopt the synthetic data to train models and reach test accuracy of 91.64%. The results show that our approach ensures model usability and provides the capability of privacy protection.