Secure Inference Method for Multimodal Large Models Based on Trusted Execution Environments

Multimodal large models have demonstrated significant advantages in edge intelligence fields such as security inspection, particularly due to their open-vocabulary object detection (OVD) capabilities. However, deploying these valuable models on edge devices introduces the risk of model theft. Existing protection schemes, unfortunately, are not compatible with multimodal models. In this paper, we present the first secure inference framework for multimodal large OVD models based on Trusted Execution Environments (TEEs). By employing orthogonal multi-adapter head combinations, the framework reduces the protection cost of the OVD backbone large model components to fit within the secure memory constraints of the TEE. Additionally, a distribution-preserving obfuscation scheme is introduced to balance both security and real-time performance. Experimental results indicate that our proposed framework reduces secure memory requirements by 95% on edge devices, while ensuring reliable confidentiality, integrity, and maintaining real-time inference performance.