Cycle-CNN for colorization towards real monochrome-color camera systems

Colorization in monochrome-color camera systems aims to colorize the gray image I_Gfrom the monochrome camera using the color image R_Cfrom the color camera as reference. Since monochrome cameras have better imaging quality than color cameras, the colorization can help obtain higher quality color images. Related learning based methods usually simulate the monochrome-color camera systems to generate the synthesized data for training, due to the lack of ground-truth color information of the gray image in the real data. However, the methods that are trained relying on the synthesized data may get poor results when colorizing real data, because the synthesized data may deviate from the real data. We present a new CNN model, named cycle CNN, which can directly use the real data from monochrome-color camera systems for training. In detail, we use the colorization CNN model to do the colorization twice. First, we colorize I_Gusing RC as reference to obtain the first-time colorization result I_C. Second, we colorize the de-colored map of R_C, i.e. RG, using the first-time colorization result I_Cas reference to obtain the second-time colorization result R_C. In this way, for the second-time colorization result R_C, we use the original color map R_Cas ground-truth and introduce the cycle consistency loss to push R_C≈ R_C. Also, for the first-time colorization result I_C, we propose a structure similarity loss to encourage the luminance maps between I_Gand I_Cto have similar structures. In addition, we introduce a spatial smoothness loss within the colorization CNN model to encourage spatial smoothness of the colorization result. Combining all these losses, we could train the colorization CNN model using the real data in the absence of the ground-truth color information of I_G. Experimental results show that we can outperform related methods largely for colorizing real data.