QANPRODrive: A VLM-based clustering approach for safety-critical driving scenarios

In recent years, significant progress has been made in End-to-End autonomous driving technology. However, the majority of widely used autonomous driving datasets focus on conventional driving scenarios. While autonomous driving models perform well when trained on such common road conditions, they struggle to adequately reflect their capability in safety-critical scenarios, thereby limiting model performance optimization. To address this issue, this paper proposes a model for the automatic mining of safety-critical scenarios based on a visual language model (VLM). This approach leverages the VLM's robust scene comprehension and reasoning capabilities, employing predefined prompts to guide the model in generating detailed descriptions of driving scenarios. Combined with a rule-based classifier, it automatically filters scene descriptions to select key scenario subsets such as intersections, turns, construction zones, and overtaking maneuvers. Experiments demonstrate that the proposed framework effectively exposes performance deficiencies in advanced End-to-End driving models. The most pronounced degradation occurs in overtaking scenarios, with collision rates increasing by 130% and L2 error rising by 15%. These metrics reveal distinct vulnerabilities across different types of scenarios, indicating significant shortcomings in the models' generalization and robustness.