Visual relocalization using long-short term memory fully convolutional network

This paper tackles the problem of camera relocalization using a single image. We formulate this problem as a regression problem and directly learn the mapping between am image and its pose by a new Deep Neural Network (DNN) architecture in an end-To-end manner. The main contribution of this work is the proposed network, called Long-Short Term Memory Fully Convolutional Network (LSTMFCN), which consists of a Fully Convolutional Network (FCN) as the feature extractor and a Long-Short Term Memory (LSTM) as the pooling layer to aggregate information across the image. In contrast to the previous DNN-based relocalization algorithms that only consider a small patch of the image, the new network has a much larger receptive field. This can avoid the aperture problem and can make it more robust to partial occlusion and moving objects. Besides, we adopt the shortcut connection to fuse features from different layers, and introduce the Error of Average Pose (EAP) into the cost function. Moreover, we show that our algorithm can be viewed as a keyframe-based relocalization algorithm, if we treat the training samples as keyframes. But unlike the traditional keyframe-based algorithms whose computational time and storage will increase as the size of the scene enlarges, our algorithm has constant computational time and storage. We investigate different network structures and parameter settings, and compare our algorithm with the previous algorithms by experiments. The experimental results show that our algorithm significantly outperforms the state-of-The-Art DNN-based algorithm and achieves real time.