Weakly supervised object-aware convolutional neural networks for semantic feature matching

We address the task of establishing visual correspondences between two images depicting main objects of the same semantic category. This task encounters various challenges such as background clutter, intra-class variation, and viewpoint variations. Existing works are dominated by end-to-end training methods that rely on redundant calculation or large amounts of manual annotations, and cannot generalize to unseen object categories. In this paper, we propose to construct a weakly supervised object-aware convolutional neural network architecture for semantic feature matching, while being trainable end-to-end without the requirement for manual annotations. The main component of this architecture is a similarity filter module containing a trainable neural nearest neighbors network. Since training data for semantic feature matching is rather limited, we introduce a simple and effective foreground selection strategy to produce the foreground masks. Using these masks as a form of weak supervision signal for correspondence task and tackle the background clutter. Extensive experiments illustrate that the proposed approach outperforms the state-of-the-art methods for semantic feature matching on multiple public standard benchmark datasets.