A 3D Reconstruction Network Based on Multi-sensor

To reconstruct the 3D model of specific targets in real time, a multi-sensor data fusion-based 3D reconstruction algorithm is proposed in this paper. This network-based algorithm takes the camera image and lidar point-cloud data as inputs, employing RGB channel and lidar channel to process each type of data separately, and finally obtains the targets’ dense depth map by fusion. In RGB channel, the transformer network rather than CNN (convolutional neural network) is used to obtain multi-scale image features with global receptive field and high resolution, and generate monocular depth, guidance map and semantic segmentation. In the lidar channel, the sparse lidar data is fused with the guidance map to generate the final prediction of dense depth. In the test, our algorithm achieved a high ranking on the leaderboard. In application, under the condition of equal reconstruction quality, a five times faster speed is obtained in 3D reconstruction comparing to the traditional image-based method.