Fed-e³: A tri-level federated fairness evaluation framework with statistical guarantees and privacy preservation

Federated Learning has rapidly gained industrial attention due to its privacy protection advantages. However, accurately evaluating whether federated models exhibit bias in real deployment environments faces challenges, as each client typically possesses only limited private data. To address this, we propose Fed- e ³ , a tri-level federated fairness evaluation framework that integrates private data from multiple clients while preserving privacy, enabling comprehensive assessment of model fairness with statistical guarantees. Specifically, Fed- e ³ ensures privacy by exchanging only intermediate statistical parameters, which are protected by homomorphic encryption. The statistical guarantees are realized through a two-stage hypothesis testing strategy with a controllable Type I error rate and the construction of high-confidence statistical intervals. Based on the principle of ”estimating population from samples,” it infers model fairness performance in real environments from limited data. The framework’s two-stage evaluation strategy not only detects fairness issues but also classifies model bias into heterogeneous bias (inconsistent across clients) and systematic bias (consistent across all clients). Furthermore, we quantify the degree of model bias by constructing said statistical confidence intervals, thereby establishing the tri-level evaluation mechanism of “detection-classification-quantification”. Through Monte Carlo simulations, we validate the theoretical performance and sensitivity of Fed- e ³ ’s detection strategy. Extensive experiments on real-world datasets from five critical decision-making domains further confirm its effectiveness. Compared to existing methods, Fed- e ³ achieves 1.8 × higher stability of evaluation under data distribution shifts, while reducing the evaluation bias by 64 %.