Effective Crash Recovery of Robot Software Programs in ROS

Modern robot systems use various software programs to autonomously perform different kinds of tasks. However, due to the risks of possible faults and errors, a robotic software program can inevitably crash in some cases, causing that the robot system fails to perform the current task. Thus, for robustness, the crashed program should be correctly recovered to continue the failed task. For this purpose, ROS provides a default restart method to automatically restart crashed programs. However, our case studies of typical ROS programs show that the restart method can perform incorrect crash recovery, and it can even cause the robot to perform dangerous behaviors, because this method loses the program's important data that was stored before the crash and is used after recovery. To solve this problem, we develop a practical approach named RORY, to perform effective crash recovery of robot software programs in ROS. RORY uses a hybrid checkpoint-replay method, and it is generic to different ROS programs by considering ROS properties. We evaluate RORY on 6 common ROS programs, and show that RORY performs correct crash recovery in both virtual and realistic environments with modest overhead. The comparison experiments indicate that RORY outperforms the restart, checkpoint-alone and replay-alone methods.