Development and validation of a deep learning system for ascites cytopathology interpretation

Background: Early diagnosis of Peritoneal metastasis (PM) is clinically significant regarding optimal treatment selection and avoidance of unnecessary surgical procedures. Cytopathology plays an important role in early screening of PM. We aimed to develop a deep learning (DL) system to achieve intelligent cytopathology interpretation, especially in ascites cytopathology. Methods: The original ascites cytopathology image dataset consists of 139 patients’ original hematoxylin–eosin (HE) and Papanicolaou (PAP) Staining images. DL system was developed using transfer learning (TL) to achieve cell detection and classification. Pre-trained alexnet, vgg16, goolenet, resnet18 and resnet50 models were studied. Cell detection dataset consists of 176 cropped images with 6573 annotated cell bounding boxes. Cell classification data set consists of 487 cropped images with 18,558 and 6089 annotated malignant and benign cells in total, respectively. Results: We established a novel ascites cytopathology image dataset and achieved automatically cell detection and classification. DetectionNet based on Faster R-CNN using pre-trained resnet18 achieved cell detection with 87.22% of cells’ Intersection of Union (IoU) bigger than the threshold of 0.5. The mean average precision (mAP) was 0.8316. The ClassificationNet based on resnet50 achieved the greatest performance in cell classification with AUC = 0.8851, Precision = 96.80%, FNR = 4.73%. The DL system integrating the separately trained DetectionNet and Classificationnet showed great performance in the cytopathology image interpretation. Conclusions: We demonstrate that the integration of DL can improve the efficiency of healthcare. The DL system we developed using TL techniques achieved accurate cytopathology interpretation, and had great potential to be integrated into clinician workflow.