Light-weight Unsupervised Anomaly Detection for Encrypted Malware Traffic

Users and businesses in the network frequently suffer from attacks by malware like privacy breach. While encrypted traffic protects users and businesses, it also provides convenience for attackers to avoid detection. Existing anomaly detection systems use supervised learning with high-dimension features and employ experts for labeling. However, our exploration reveals that high-dimension features will reduce the efficiency of the classification model. Besides, their training needs abundant high-quality labels, which is difficult to obtain in practice. Facing these challenges, in this paper, we propose an unsupervised anomaly detection method, which adopts the three-layer Autoencoder for feature compression to improve model running efficiency and employs the classical Kmeans algorithm to achieve unsupervised classification. When training the Autoencoder, we only use the normal encrypted traffic. We compare the performance of our method against the state-of-the-art anomaly detection algorithms using open encrypted malware traffic data set. The results demonstrate that our method can achieve the Fl-measure of 0.95, which is competitive with supervised learning algorithms.