Unsupervised Cycle-Consistent Generative Adversarial Networks for Pan Sharpening

Deep learning-based pan sharpening has received significant research interest in recent years. Most of the existing methods fall into the supervised learning framework in which they downsample the multispectral (MS) and panchromatic (PAN) images and regard the original MS images as ground truths to form training samples based on Wald's protocol. Although impressive performance could be achieved, they have difficulties when generalizing to the original full-scale images due to the scale gap, which makes them lack of practicability. In this article, we propose an unsupervised generative adversarial framework that learns from the full-scale images without the ground truths to alleviate this problem. We first extract the modality-specific features from the PAN and MS images with a two-stream generator, perform fusion in the feature domain, and then reconstruct the pan-sharpened images. Furthermore, we introduce a novel hybrid loss based on the cycle-consistency and adversarial scheme to improve the performance. Comparison experiments with the state-of-the-art methods are conducted on GaoFen-2 (GF-2) and WorldView-3 satellites. Results demonstrate that the proposed method can greatly improve the pan-sharpening performance on the full-scale images, which clearly shows its practical value. Codes are available at http://github.com/zhysora/UCGAN.