CU-NET+: DEEP FULLY INTERPRETABLE NETWORK FOR MULTI-MODAL IMAGE RESTORATION

The network interpretability is critical in computer vision related tasks, especially for tasks involving multiple modalities. For multi-modal image restoration, one recent method, CU-Net, introduces an interpretable network based on a multi-modal convolutional sparse coding model. However, its network architecture does not mimic in full the proposed sparse model. In this paper, we overcome the limitation of CU-Net by using recurrent scheme, and this leads to a fully interpretable network which we call CU-Net+. In addition, we relax the constraint on the number of common and unique features in CU-Net, for making it more consistent with real condition. The effectiveness of the proposed CU-Net+ is evaluated on RGB guided depth image super-resolution and flash guided non-flash image denoising tasks. The numerical results show that CU-Net+ outperforms other interpretable or non-interpretable methods, with 0.16 RMSE and 0.66 dB PSNR improvement over CU-Net for the two mentioned tasks, respectively. Code is available at https://github.com/JingyiXu404/CU-Net-plus.