Evolving malware detection through instant dynamic graph inverse reinforcement learning

The rapid development and increasing evolution of malware necessitate novel defensive techniques with high accuracy and minimal false positives to safeguard information systems against potential threats. Unfortunately, current malware detection methods, primarily relying on deep learning to identify malicious fingerprints from extensive training sets, prove ineffective against few-shot and evolving malware variants. To address these challenges, this paper introduces MalIRL, which designs a model-free inverse reinforcement learning (IRL) mechanism to automatically capture the evolving attack intent of malware. Specifically, MalIRL explores six representative categories of malware actions and employs sliding windows to organically divide the massive malware execution event stream into multiple attack stages, achieving a reduced action space and state space. To model dynamic malicious environments, MalIRL proposes an instant dynamic heterogeneous graph representation learning technique. This technique learns state representations of different malware attack stages, enhancing detection accuracy and efficiency by incrementally capturing the newly added contextual semantics of diverse malware entities and relations. In experiments with three real-world malware datasets, MalIRL surpasses existing state-of-the-art methods, particularly in few-shot malware detection scenarios, MalIRL exhibits performance benefits of up to 18.9%.