A Camera and LiDAR Data Fusion Method for Railway Object Detection

Object detection on railway tracks, which is crucial for train operational safety, face numerous challenges such as multiple types of objects and the complexity of train running environment. In this study, a multi-sensor framework is proposed to fuse camera and LiDAR data for the detection of objects on railway track including small obstacles and forward trains. The framework involves a two-stage process: region of interest extraction and object detection. In the first stage, a multi-scale prediction network is designed to achieve pixel level segmentation of the railway track and forward train via the image. In the second stage, LiDAR data is used to estimate the distance to the train and detect small obstacles in the railway track area which is extracted from the first stage. Experimental results show that the region of interest extraction method achieves desirable accuracy for railway track and train segmentation; and the proposed fusion method outperforms the one based on camera or LiDAR alone for small obstacles and forward train detection. Moreover, in practice the proposed framework has been successfully applied on the Hong Kong Metro TSUEN WAN line and the Beijing Metro YANFANG line.