Fusing human action recognition and object detection for human-robot collaborative assembly

Human-Robot Collaboration Assembly (HRCA) provides a solution to complex product assembly tasks in modern industrial manufacturing. A key point is how to enable robots to recognize assembly tasks being executed by assemblers through autonomous visual perception technology, and plan collaborative assembly tasks according to perception results in conjunction with assembly task procedures. First, to improve the cognitive ability of a robot in HRCA, an efficient method of assembly task recognition is proposed, which decouples assembly task recognition into two independent tasks: assembly action recognition and assembly object detection. Meanwhile, an integrated neural network model that is able to recognize assembly action while detecting assembly objects is proposed. Assembly task recognition is a prerequisite for a robot to plan and perform assisting tasks. Second, to improve response speeds and grasp success rates of a robot in HRCA, a two-stage object 6-DOF grasp pose detection algorithm is proposed. In the first stage, an object detection algorithm based on RGB images is used to recognize and locate target objects rapidly. In the second stage, the target object region is mapped into 3D point cloud data, which is used to predict stable grasp poses, so as to achieve orderly grasping of target objects. Finally, the effectiveness of the HRCA framework proposed is verified using a planetary gear reducer assembly scenario.