Scene-graph-driven semantic feature matching for monocular digestive endoscopy

Background and objective: Registration of the preoperative 3D model with the video of the digestive tract is the key task in endoscopy surgical navigation. Accurate 3D reconstruction of soft tissue surfaces is essential to complete registration. However, existing feature matching methods still fall short of desirable performance, due to the soft tissue deformation and smooth but less-textured surface. Methods: In this paper, we present a new semantic description based on the scene graph to integrate contour features and SIFT features. Firstly, we construct the semantic feature descriptor using the SIFT features and dense points in the contour regions to obtain more dense point feature matching. Secondly, we design a clustering algorithm based on the proposed semantic feature descriptor. Finally, we apply the semantic description to the structure from motion (SfM) reconstruction framework. Results: Our techniques are validated by the phantom tests and real surgery videos. We compare our approaches with other typical methods in contour extraction, feature matching, and SfM reconstruction. On average, the feature matching accuracy reaches 75.6% and improves 16.6% in pose estimation. In addition, 39.8% of sparse points are increased in SfM results, and 35.31% more valid points are obtained for the DenseDescriptorNet training in 3D reconstruction. Conclusions: The new semantic feature description has the potential to reveal more accurate and dense feature correspondence and provides local semantic information in feature matching. Our experiments on the clinical dataset demonstrate the effectiveness and robustness of the novel approach.