LGFCTR: Local and Global Feature Convolutional Transformer for Image Matching

Image matching that establishing correspondences across images is a challenging task under extreme conditions. Capturing local and global features simultaneously is an important way to mitigate such an issue according to the human-intuitive insight that human vision relies on both local appearances and global structures for matching. However, recent methods were still stuck in two issues: (1) CNN-based encoders only extract local features; (2) transformers lack locality and rely on explicit positional encoding. Inspired by the locality and implicit positional encoding of convolutions, a novel convolutional transformer LGFCTR is proposed to capture both local contexts and global structures more sufficiently for detector-free matching. Firstly, a universal FPN-like framework captures global structures in self-encoder as well as cross-decoder by transformers and compensates local contexts as well as implicit positional encoding by convolutions. Secondly, a novel Convolutional Transformer Module explores multi-scale long-range dependencies in a grouped manner and further aggregates local information as well as imposing neighborhood constraints for locality enhancement. Finally, a novel regression-based Sub-pixel Refinement Module exploits the whole fine-grained window features for fine-level positional deviation regression. LGFCTR achieves establishing robust and accurate correspondences across illumination variations, viewpoint changes, and scale differences. LGFCTR outperforms state-of-the-art methods by 1.14% and 0.7% in AUC@10°respectively on two general benchmarks of relative pose estimation. LGFCTR also surpasses the second place by 0.0223 on a benchmark of feature matching. LGFCTR consistently demonstrates superior performances on a wide range of benchmarks, including feature matching (HPatches), homography estimation (HPatches), relative pose estimation (MegaDepth and YFCC100M), and visual localization (Aachen Day-Night v1.1).