Scene graph enhanced retrieval-augmented generation for unsafe behavior detection in industry

Unsafe Behavior Detection is a process of detecting behaviors that can make human injury or equipment damage according to typical Occupational Safety and Health Act specification. It is a crucial step to ensure the protection of personnel and resources during various industrial activities. Existing methods basically rely on computer vision models or multi-modal models to detect entities and their relations from a surveillance image. However, the images for different kinds of unsafe behaviors usually involve complex interconnection of human, machine, and materials, and have the characteristic of long-tail distribution. Hence, this paper proposed a scene graph enhanced retrieval-augmented generation method for unsafe behavior detection in industrial field. First, a self-annotation pipeline is presented to establish an unsafe behavior detection dataset especially in industrial workspace. Second, an improved scene graph generation model is proposed to extract relations between human, machines and tools as a sub-graph. Combined with knowledge graph based on the Occupational Safety and Health Act specification, unsafe behavior can be located. Third, a multi-agent voting strategy is designed to guide LLM to provide explanations on the unsafe behavior according to the sub-graph obtained by the improved scene graph generation model and the knowledge graph with the Occupational Safety and Health Act specification. Experimental results in typical industrial manufacturing scenarios demonstrate that the proposed method with less than half parameters can improve the precision of unsafe behavior detection by 16.6% compare to 11 state-of-the-art vision-based methods and 9 large language model-based methods.