Harnessing Asynchrony to Balance Modalities in Multi-modal Federated Learning

Multi-Modal Federated Learning enables clients to collaboratively train multi-modal models without sharing raw data. In practice, it suffers from modality laziness, where dominant modalities overshadow weaker ones, and asynchronous modality availability, where modalities arrive at clients at different times. Existing modality balancing methods assume synchronous access to all modalities in each round, making them unfit for asynchronous arrivals. We present MBA (Modality Balancing via Asynchrony), a lightweight framework that exploits asynchrony to combat modality laziness under feature-level fusion. First, clients perform opportunistic local balancing, where early-arriving modalities create uni-modal feature anchors to regularize multi-modal local updates without idle waiting. Then the server adopts balance-aware asynchronous aggregation, which estimates and corrects global modality imbalance via staleness-weighted updates. Experiments show that MBA improves both accuracy and efficiency, demonstrating that asynchrony can be harnessed to achieve balanced multi-modal federated learning.