Pyramid convolutional network for colorization in monochrome-color multi-lens camera system

We study the problem of colorization in the monochrome-color multi-lens camera system. The recent convolutional network (CNN) based method learns a 3-D weight volume to solve this problem and gets very high accuracy. But the model size is very big due to the large-displacement problem, i.e. there are large displacements between some pixels in the input gray image and the pixels in the reference image that could provide correct colors. To overcome the limitations, we improve the recent CNN based method and propose to combine pyramid processing with CNNs for colorization. At each level of the pyramid, our method warps the reference image using the estimated warping information map from the previous level so that we can learn a much more compact 3-D weight volume for colorization. We also compute an update to the warping information map by a Markov Random Field method at each level. With the pyramid CNN structure, our model has much smaller model size, and experimental results show that our method outperforms all of the state-of-the-art methods in accuracy as well.