Adaptive Cost Aggregation in Iterative Depth Estimation for Efficient Multi-view Stereo

The deep multi-view stereo (MVS) approaches generally construct 3D cost volumes to regularize and regress the depth map. These methods are limited with high-resolution outputs since the memory and time costs grow cubically as the volume resolution increases. In this paper, we presented an multi-stage iterative depth map estimation method for MVS. In our network, the cost volume is iteratively processed by lightweight 2D convolution based GRU modules, and the multi-stage coarse-to-fine structure is adopted to speed up the depth estimation process. To further improve the 3D reconstruction quality, we make improvements from two different perspectives of adaptive cost aggregation: a view-adaptive weighting module is proposed to account for the occlusion problem in cost volume fusion, and a spatial-adaptive deformable geometric feature encoding module is introduced to the cost volume feature encoding before feeding into GRUs for stronger modeling capability. Experiments on the DTU dataset demonstrated the effectiveness of the proposed network in accuracy with remarkable efficiency performance.