HFHuman: High-Fidelity Human Reconstruction From Single Image With Multi-Modality Fusion

Accurately reconstructing high-fidelity human models from single images is critical for virtual reality applications. Existing methods often rely on 3D features from the estimated parametric human model to provide geometric priors. This approach addresses challenges such as missing limbs or deformations, which often arise due to viewpoint limitations and self-occlusion. However, accurately predicting 3D features from monocular images remains a significant challenge. This limitation poses difficulties for the fusion of 2D and 3D information. In this paper, we introduce HFHuman, a novel approach for high-fidelity human reconstruction from a single image using multi-modality fusion. HFHuman effectively fuses multiple modalities, including geometric and depth, directly from images. Our method introduces three key innovations: (1) a depth and geometric parallel reconstruction framework that simultaneously handles whole-body geometry and detailed depth reconstruction, refining a parameterized 3D human model under progressive depth guidance; (2) a pixel-voxel feature fusion strategy that combines pixel-aligned features with voxel-aligned features using a multi-modality adaptor; and (3) a depth-refined technique that integrates RGB imagery with surface normals and depth mapping. By addressing the challenge of blending 2D and 3D modalities, HFHuman results in more accurate and realistic human reconstructions. Experimental results demonstrate that HFHuman outperforms state-of-the-art methods, setting a new standard for realistic 3D human body reconstruction.