Training Deep Neural Networks with Large-scale Datasets on Sunway High Performance Computer

It's important to obtain an accurate large-scale dataset for training deep neural networks (DNNs). But manually labeling is a time-consuming process with high labor cost. In this scenario, researchers are concerned about replacing precise data with collecting images from the internet, especially for images recognition tasks. This brings two problems: the labels of images from the web are often imprecise, and the large number of images involves large amount of computation on training. In this paper, we designed a large-scale noisy image training system based on Sunway TaihuLight supercomputer and implemented it using the Caffe framework. The system utilizes parallel processes as well as data prefetching to exploit computing power of the Sunway supercomputer. In addition, the system employs the mutual calibration training method to reduce the impact of noisy labels. Experimental results show that the system can greatly reduce the training time and has good scalability.