Optimal FPGA-oriented lightweight network architecture search under multi-objective constraints

The neural architecture search (NAS) technology has developed rapidly, but most algorithms mainly target the GPUs, causing the searched models to lack consideration of the target hardware platform. This paper proposes a novel NAS algorithm specific for the unique calculation and storage character of FPGA to accelerate the deployment efficiency of neural networks and automated build the FPGA-oriented neural structure. Firstly, we design the search space based on the lightweight candidate blocks. Then, we measure the real simulation latency of each candidate block on the FPGA platform and for the first time using it to direct the NAS process. Eventually, this paper proposed a differentiable representation function of hardware latency and built a multi-objective NAS framework based on real simulation latency and FLOPS. In addition, by analyzing the relationship between the network convergence process and the model size, we achieve an optimal trade-off balance between precision and latency and build a more suitable network structure for deployment on FPGA. Compared with state-of-the-art manual and automated lightweight models, our method has relatively better accuracy and latency performance. Extensive experiments on the ImageNet dataset show that the proposed method can achieve 2x speedup compared with the similar accuracy manual lightweight network (MobileNet V2) and even get 3x speedup compared with the equivalent auto lightweight network (DARTS).