Deep learning models for malware traffic detection and classification: a focus on pruning and hyperband for resource optimization

Sheetekela, Sanette Penelao

Deep learning models for malware traffic detection and classification: a focus on pruning and hyperband for resource optimization

Thesis / Dissertation

2026

Publisher

University of Cape Town

Department

Department of Computer Science

Faculty

Faculty of Science

Abstract

Network traffic encryption is increasingly being adopted as the norm for communication technologies because of its capacity to protect privacy. Conversely, concealing data from unintended recipients presents an additional issue in the communication system, as encrypted data makes it challenging to detect malware using traditional methods such as deep packet inspection. Deep learning (DL) has emerged as a powerful technique for detecting malware in encrypted traffic flows. However, their high processing needs make it hard to use them in places with limited resources, such as Internet of Things (IoT) and edge devices. This study investigated the effectiveness of post-training weight and neuron pruning on three DL models—Multilayer Perceptron (MLP), One-dimensional convolutional Neural Networks (1D-CNN), and Long Short-term Memory Networks (LSTM)—to optimize resource usage while maintaining detection accuracy. In addition, the study integrated Hyperband, a resource-efficient hyperparameter tuning method that dynamically allocates computational resources based on intermediate performance evaluations to enhance pruned models. The CTU-13 dataset was used to assess how pruning rates influence detection performance and resource efficiency, with an emphasis on the trade-offs introduced by weight pruning (WP) and neuron pruning (NP). Our findings revealed that moderate pruning of 20% to 40% maintained or improved model accuracy while decreasing the model size by up to 40%, inference time by 30%, and memory usage by 10%. Among the models, 1D-CNN consistently provided the optimal balance between performance and computational efficiency. Regarding pruning techniques, structured neuron pruning (NP) has proven to be the most effective method, achieving significant reductions in model size and memory usage without compromising on detection performance. In contrast, unstructured weight pruning (WP) primarily reduced the number of active weights in the model by setting them to zero, which helped lower memory usage during processing; however, no compression was applied to the actual file size. Overall, the findings showed that adaptive tuning, when combined with pruning, offers an effective approach for building high-performing lightweight models that can be implemented in resource-constrained environments for encrypted malware traffic flow classification.