Hand dynamic 3D reconstruction using multiple keypoint-to-keypoint correspondences

We propose a new approach to robustly reconstruct non-rigid hand geometries and motions from single-view depth sequences. On the basis of the assumption that the correspondence between any pair of key points is consistent across all hand poses, a smooth hand mesh template is adopted to provide the shape and topology prior, and several energy functions are introduced to formulate these 3D keypoint-to-keypoint correspondences, which are then incorporated into the available non-rigid registration pipeline to achieve an accurate surface fit. The non-rigid hand motion is captured by minimizing the distortion error between a hand template and the partial input. An iterative solver is employed to gradually refine deformations around joint regions with explicit correspondences, leading to fast convergence and plausible deformations. Extensive experiments on hand depth sequences with noise demonstrate that our approach is capable of reconstructing accurate hand motions, and is robust for large deformations and occlusions.