SifterNet: Model-Agnostic Defense against Backdoor Attack in Vision Large Model

Vision models have been applied into urban computing, vision-language navigation, intelligent transportation, etc. Nevertheless, various convolution neural networks (CNN)-based vision models, or even the recently-developed vision Transformer-based large models all encounter security and privacy issues. In this paper, aiming at resisting backdoor attacks in these vision models, we proposes a generalized and model-agnostic trigger-purification approach resorting to the classic Ising model in physics. To date, existing trigger detection/removal studies usually require to know the detailed knowledge of target model in advance, access to a large number of clean samples or even model-retraining authorization, which brings the huge inconvenience for practical applications, especially in case of inaccessibility to the target model. Thereby, an ideal countermeasure ought to eliminate the implanted trigger without regarding whatever the target models are. To this end, a lightweight and black-box defense approach SifterNet is proposed through leveraging the memorization-association functionality of Hopfield network, by which the triggers of input samples can be effectively purified in a proper manner. The main novelty of our proposed approach lies in the introduction of ideology of Ising model. A set of experiments also validate the effectiveness of our approach in terms of proper trigger purification and high accuracy achievement, and compared to the state-of-the-art baselines, our proposed SiferNet has a significant superior performance under five popular backdoor attacks.