LDP: Latent Diffusion-based Adversarial Purification towards Transformer-based Visual Encoders

Adversarial purification has emerged as a critical defense mechanism against various adversarial attacks on deep neural networks, however, such category of diffusion-based purification in the pixel space confronts two challenges while applying into today's large vision transformers: i) the distributional divergence between adversarial and clean examples in high-dimensional image manifold; and ii) the prohibitive computational cost while processing high-resolution images. To address the predicament, we in this paper focus on the transformer-based visual encoders commonly employed in large Vision-Language Models (VLMs), and propose a novel Latent Diffusion-based Purification (LDP) mechanism through leveraging the latent space to pave the gap between adversarial distribution and clean distribution. Resorting to projecting adversarial input into a low-dimensional latent representation, our proposed LDP not only suppresses the off-manifold perturbations to achieve accelerated denoising, but also preserves critical semantic features by aligning input with the priori high-quality visual representations. Multi-facet experiments over both proactive robustness enhancement and post-attack purification demonstrate that our LDP has a superior performance in terms of effectiveness and efficiency.