FedTC: Enabling Communication-Efficient Federated Learning via Transform Coding

Federated learning (FL) enables distributed training via periodically synchronizing model updates among participants. Communication overhead becomes a dominant constraint of FL since participating clients usually suffer from limited bandwidth. To tackle this issue, top-k based gradient compression techniques are broadly explored in FL context, manifesting powerful capabilities in reducing gradient volumes via picking significant entries. However, previous studies are primarily conducted on the raw gradients where massive spatial redundancies exist and positions of non-zero (top-k) entries vary greatly between gradients, which both impede the achievement of deeper compressions. Top-k may also degrade the performance of trained models due to biased gradient estimations. Targeting the above issues, we propose FedTC, a novel transform coding based compression framework. FedTC transforms gradients into a new domain with more compact energy distributions, which facilitates reducing spatial redundancies and biases in subsequent sparsification. Furthermore, non-zero entries across clients from different rounds become highly aligned in the transform domain, motivating us to partition the gradients into smaller entry blocks with various alignment levels to better exploit these alignments. Lastly, positions and values of non-zero entries are independently compressed in a block-wise manner with our customized designs, through which a higher compression ratio is achieved. Theoretical analysis and extensive experiments consistently demonstrate the effectiveness of our approach.