A Virtual Hand Based Manipulation Learning Framework for Dexterous Hand

Dexterous hand programming is considered to be generally difficult suffering from high degree of freedom. Vision-based manipulation learning provides an efficient way to support automatic programming by human demonstration. However, complex visual perception is tightly coupled with the kinematics structure of dexterous hand in existing solutions, which makes it hard to migrate the developed algorithm from one kind of dexterous hand to another due to their kinematic difference. To address the issue, we introduce an intermediary called virtual hand and a new paradigm of motion mapping, serving as the link between human hand and dexterous hand. The virtual hand is generated in real time by hand pose estimation to track the motion of human hand. Then the joint configurations of dexterous hand and robotic arm can be calculated using geometry and optimization method based on the motion mapping paradigm we proposed. The method we introduced decouples the visual perception and kinematics calculation, which brings flexibility and low cost. The results of our preliminary experiments show the effectiveness of our method.