Improving Multi-view Stereo with Contextual 2D-3D Skip Connection

Learning-based methods have shown their strong competitiveness in estimating voxel for multi-view stereo. However, due to the modality gap between 2D and 3D space, the quality of the estimated 3D object is limited by the reconstruction of some detailed structures. To tackle this problem, we regard the 3D voxel reconstruction as a semantic segmentation task where skip connections between the 2D encoder and 2D decoder are usually utilized to incorporate significant contextual, aiming to segment more details. Thus, we propose an approach to improve the multi-view 3D voxel reconstruction via contextual 2D-3D skip connection. In our method, a 2D-3D skip connection branch embedded with feature visual hull is designed and plugged into the standard 2D encoder-3D decoder reconstruction architecture, which enables 2D contextual information to be effectively transmitted into the 3D domain. Then, an attention-guided module is designed to adaptively combine the transmitted features with the original 3D decoded features. Finally, a 3D RNN layer is built at the end of network to aggregate individual 3D features from different views. Extensive results have shown that the contextual information from our 2D-3D skip connections can significantly improve the reconstruction performance, especially for the detailed structures recovering.