Global Context Enhanced Multi-modal Fusion for Referring Image Segmentation

The referring image segmentation is a challenging task which aims to segment the object of interest in an image according to a natural language expression. Most existing works directly concatenate the global language representation with local visual features, and follow by a convolutional operation to fuse two modalities. These works ignore that the global contextual information from vision is essential for vision-language fusing and inferring the referred objects. The global context can establish a perception of the full image, thus it’s fusion with global language representation is beneficial to reduce mislabeled pixels of similar objects in an image. To address aforementioned issue, we propose a global fusion network (GFNet), which is composed of visual guided global fusion module and language guided global fusion module. By modeling the expression-region interactions, two modules can aggregate the expression-related visual contextual information and fuse it with global representation of language expression. Moreover, to alleviate the distribution differences between two modalities, we introduce a channel-wise self-gate on visual-language concatenated features. We validate the proposed network on four standard datasets, the experimental results show that our approach outperforms state-of-the-art methods.