LMC: Lightweight Message Collection for Distributed Training of Deep Learning Models

As the scale of deep learning models continues to expand, distributed training has become an inevitable trend, which is not only time-consuming but also expensive. Therefore, effective debugging and tuning are crucial for the distributed training, where the messages among training processes can be a useful information. However, there remains a lack of generalpurpose communication logging systems that can efficiently capture message contents while maintaining compatibility with mainstream distributed deep learning frameworks. This paper proposes LMC, a lightweight communication collection system designed for distributed deep learning workloads. LMC employs non-intrusive function interception to transparently capture collective communication calls and associates each operation with runtime semantic context through dynamic context injection. To mitigate the impact of massive messages of frequent collective communications during the training, LMC incorporates several message-reduction approaches. It also supports configurable logging granularity and generates structured, analyzable logs. Experiments are conducted on a CPU-GPU cluster, and results demonstrate effectiveness of LMC.