Fingertip detection and tracking based on Siamese network

Fingertip detection and tracking is an important task in human-computer interaction which has high requirements for the speed and accuracy. Due to the fingertip is small and easy to be confused with other joints of the hand, it brings great challenges to the detection and tracking. The common object detection and tracking algorithms are difficult to meet the requirements. Our work can achieve the purpose of accurately detecting and tracking the fingertip of the index finger in the video, and has good real-time performance. In our work, we first detect and track the hand, then we add a segmentation module to the work. The fingertip can be extracted from the mask of hand. We use SSD to detect the hand in the first frame of the video, as the initialization of the tracking. Then we use SiamMask to track and get the mask of the hand. Finally, we detect and track the fingertip of the index finger in the segmentation. Our method has been tested on our own dataset and has achieved good accuracy and real-time performance. The AP has reached 98% at OKS(0.5) and the speed has reached 30fps.