Zero-shot assembly anomaly detection based on virtual-real difference comparison

In the assembly process of complex products, detecting the assembly states is crucial, as it affects the final product quality. Currently, deep learning-based methods have been widely employed for assembly anomaly detection. However, most learning-based methods use supervised learning and rely on large amounts of training data, making them difficult to adapt to the diversity of complex assembly states. This is because they require collecting data from multiple perspectives and states, followed by labeling such data, which is a costly task. To address these issues, we propose a zero-shot assembly anomaly detection method based on virtual-real difference comparison. It utilizes the CAD model rendered image as the reference and detects anomaly by segmenting the differences between the real image and virtual rendered image. Firstly, we adopt a keypoint-based coarse-to-fine alignment method for virtual and real images, which estimates the rough pose of prominent objects and performs fine alignment via feature point matching. Then, visual foundation models are used to extract common features from virtual and real images. The similarity of virtual and real features is calculated to obtain pixel-level pseudo-anomaly regions. Furthermore, we utilize SAM (Segment Anything) to segment virtual and real images, and perform geometric intersection matching between the SAM-based masks and the pseudo-anomaly mask, as well as semantic matching of virtual and real regions, which can obtain object-level anomaly masks. Finally, experimental verification is conducted on a complex product assembly dataset, and the results demonstrate that the proposed method has competitive detection accuracy, which proves the effectiveness and feasibility of the proposed method.