HHA: Hybrid Control-Data Flow Heterogeneous Accelerator for Conv-transformer Neural Networks

The conv-transformer neural network (CTNN) has demonstrated exceptional performance in computer vision. When implemented on resource-constrained edge devices, CTNNs ex-hibit two critical operational challenges. The first challenge is the limited acceleration of nonlinear operators, while the second challenge is the complexity of hardware scheduling due to the variety of CTNN operators. These challenges significantly reduce the inference performance of CTNN on edge devices. We propose a hybrid control-data flow heterogeneous CTNN accelerator, HHA. To address the computational inefficiency of nonlinear operators, we propose a novel nonlinear acceleration engine(NAE) based on lookup tables(LUTs). Building on this foundation, we propose an accelerator architecture that integrates control flow and data flow in a hybrid manner, alongside a batch-by-batch heterogeneous cores scheduling algorithm(BH) to enhance the parallel computing capability of heterogeneous cores. Experimental results indicate that the NAE significantly improves the efficiency of nonlinear operations. Compared with the state-of-the-art acceleration architectures for non-linear operators, NAE achieves 1.44x, 12.52x, and 18.56x speedup in computing the Softmax, Layer Normalization(LN), and GELU operators, respectively. The BH method significantly enhances the CTNN inference speed by a factor of 3.944 through optimization of the computing and storage structure of the operators. We implemented HHA on the Xilinx UltraScale+ MPSoC ZCU102. Compared to the state-of-the-art FPGA accelerator and NVIDIA V100, HHA achieves a 1.78x and 3.74x throughput improvement and a 1.11x and 47.76x energy efficiency improvement.