Towards Secure and Robust Vision-Language Models in Autonomous Driving: A Survey for Perception-Oriented and Decision-Oriented Attacks

Vision-language models (VLMs) have been widely adopted in autonomous driving (AD) for their strong generalization capabilities and interpretability. However, they remain inherently vulnerable to robustness challenges, particularly adversarial examples, which pose significant safety risks. This paper presents a comprehensive survey of attacks on VLMs in AD, introducing a novel taxonomy that categorizes attacks into two types: perception-oriented attacks, which manipulate input data to mislead the perception process, and decision-oriented attacks, which exploit vulnerabilities in decision-making and interaction processes. Following this classification, the paper conducts a systematic review of existing attack methods, their limitations, and the key challenges they present. It also explores future directions and potential applications to enhance the robustness and performance of VLMs in AD. This work aims to lay a foundation for developing more secure and reliable VLMs for AD systems.