Federated Semantic Synergistic Alignment for Food Image Classification under Class Imbalance

Federated Learning (FL) enables distributed model training while preserving data privacy, making it highly applicable to privacy-sensitive tasks such as food image classification. However, challenges such as class imbalance, vacant classes, and high computational and communication costs hinder its performance. This study proposes a Federated Semantic Synergistic Alignment model (FedSSA), which leverages CLIP's multimodal semantic knowledge to enhance food image classification under label-skewed scenarios. FedSSA comprises three core stages: Multimodal Semantic Knowledge Construction (MSKC), which utilizes CLIP to generate multidimensional semantic features encompassing appearance, texture, context, and other attributes, providing prior knowledge for vacant classes; Semantic-Guided Knowledge Transfer (SGKT), which aligns local model outputs with global semantic features via KL divergence to improve classification of vacant classes; and Adaptive Feature Projection Optimization (AFPO), which dynamically adjusts the feature space and reduces client-side computational and communication costs through server-side feature dissemination. Experiments on the Food101, ISIAFood200, and VireoFood172 datasets demonstrate that FedSSA significantly outperforms state-of-the-art methods, including FedAvg, MOON, FedMR, and FedVLS, in severe class imbalance scenarios. Moreover, FedSSA achieves a computational complexity of only 1/16th that of CLIP, reduces inference time to 0.15 seconds, and lowers storage requirements to approximately 200 MB, highlighting its efficiency and robustness.