VuLASTE: Long Sequence Model with Abstract Syntax Tree Embedding for Vulnerability Detection

In this paper, we present a model named VuLASTE, treating vulnerability detection as a specialized text classification task. To address the vocabulary explosion problem, VuLASTE utilizes a byte-level BPE algorithm from natural language processing. Within VuLASTE, we introduce a novel AST path embedding to represent source code nesting information. Additionally, we employ a combination of global and dilated window attention from Longformer to extract long sequence semantics from source code. To tackle the issue of data imbalance, a common challenge in vulnerability detection datasets, we employ focal loss as a loss function. This ensures that the model prioritizes poorly classified cases during training. To evaluate our model’s performance on real-world source code, we construct a cross-language and multi-repository vulnerability dataset from the Github Security Advisory Database. VuLASTE achieves top 50, top 100, top 200, and top 500 hits of 29, 51, 86, and 228, respectively, surpassing state-of-the-art researches.