Fusion Strategy of Multi-sensor Based Object Detection for Self-driving Vehicles

Lidar and optical camera are common sensors in the sensor layer of autopilot system. Lidar can use depth data to obtain accurate relative distance and contour information of obstacles, which is not easily affected by external light conditions. Optical camera can obtain rich object/environment semantic information through high-resolution image, which is relatively mature in technology. The two different sensors are highly complementary, and previous studies show that the fusion of laser point cloud and image data can greatly improve the efficiency of object detection in out door environment. In this paper, a deep convolutional neural network detection model based on Lidar and image information features layered fusion is studied. We try different fusion depth at the CNN model to seek the best solution according to the detection performance. The experimental results on the KITTI dataset show that the detection accuracy of the fusion based on YOLOv3 is 1.08% higher than original model. Another small scale experiment with our own self-driving platform on local area also show the final fusion model can achieve better detection accuracy in real road condition.