Deep learning models for malware traffic detection and classification: a focus on pruning and hyperband for resource optimization
| dc.contributor.advisor | Chavula, Josiah | |
| dc.contributor.author | Sheetekela, Sanette Penelao | |
| dc.date.accessioned | 2026-07-23T09:08:16Z | |
| dc.date.available | 2026-07-23T09:08:16Z | |
| dc.date.issued | 2026 | |
| dc.date.updated | 2026-07-23T09:06:36Z | |
| dc.description.abstract | Network traffic encryption is increasingly being adopted as the norm for communication technologies because of its capacity to protect privacy. Conversely, concealing data from unintended recipients presents an additional issue in the communication system, as encrypted data makes it challenging to detect malware using traditional methods such as deep packet inspection. Deep learning (DL) has emerged as a powerful technique for detecting malware in encrypted traffic flows. However, their high processing needs make it hard to use them in places with limited resources, such as Internet of Things (IoT) and edge devices. This study investigated the effectiveness of post-training weight and neuron pruning on three DL models—Multilayer Perceptron (MLP), One-dimensional convolutional Neural Networks (1D-CNN), and Long Short-term Memory Networks (LSTM)—to optimize resource usage while maintaining detection accuracy. In addition, the study integrated Hyperband, a resource-efficient hyperparameter tuning method that dynamically allocates computational resources based on intermediate performance evaluations to enhance pruned models. The CTU-13 dataset was used to assess how pruning rates influence detection performance and resource efficiency, with an emphasis on the trade-offs introduced by weight pruning (WP) and neuron pruning (NP). Our findings revealed that moderate pruning of 20% to 40% maintained or improved model accuracy while decreasing the model size by up to 40%, inference time by 30%, and memory usage by 10%. Among the models, 1D-CNN consistently provided the optimal balance between performance and computational efficiency. Regarding pruning techniques, structured neuron pruning (NP) has proven to be the most effective method, achieving significant reductions in model size and memory usage without compromising on detection performance. In contrast, unstructured weight pruning (WP) primarily reduced the number of active weights in the model by setting them to zero, which helped lower memory usage during processing; however, no compression was applied to the actual file size. Overall, the findings showed that adaptive tuning, when combined with pruning, offers an effective approach for building high-performing lightweight models that can be implemented in resource-constrained environments for encrypted malware traffic flow classification. | |
| dc.identifier.apacitation | Sheetekela, S. P. (2026). <i>Deep learning models for malware traffic detection and classification: a focus on pruning and hyperband for resource optimization</i>. (). University of Cape Town ,Faculty of Science ,Department of Computer Science. Retrieved from http://hdl.handle.net/11427/43661 | en_ZA |
| dc.identifier.chicagocitation | Sheetekela, Sanette Penelao. <i>"Deep learning models for malware traffic detection and classification: a focus on pruning and hyperband for resource optimization."</i> ., University of Cape Town ,Faculty of Science ,Department of Computer Science, 2026. http://hdl.handle.net/11427/43661 | en_ZA |
| dc.identifier.citation | Sheetekela, S.P. 2026. Deep learning models for malware traffic detection and classification: a focus on pruning and hyperband for resource optimization. . University of Cape Town ,Faculty of Science ,Department of Computer Science. http://hdl.handle.net/11427/43661 | en_ZA |
| dc.identifier.ris | TY - Thesis / Dissertation AU - Sheetekela, Sanette Penelao AB - Network traffic encryption is increasingly being adopted as the norm for communication technologies because of its capacity to protect privacy. Conversely, concealing data from unintended recipients presents an additional issue in the communication system, as encrypted data makes it challenging to detect malware using traditional methods such as deep packet inspection. Deep learning (DL) has emerged as a powerful technique for detecting malware in encrypted traffic flows. However, their high processing needs make it hard to use them in places with limited resources, such as Internet of Things (IoT) and edge devices. This study investigated the effectiveness of post-training weight and neuron pruning on three DL models—Multilayer Perceptron (MLP), One-dimensional convolutional Neural Networks (1D-CNN), and Long Short-term Memory Networks (LSTM)—to optimize resource usage while maintaining detection accuracy. In addition, the study integrated Hyperband, a resource-efficient hyperparameter tuning method that dynamically allocates computational resources based on intermediate performance evaluations to enhance pruned models. The CTU-13 dataset was used to assess how pruning rates influence detection performance and resource efficiency, with an emphasis on the trade-offs introduced by weight pruning (WP) and neuron pruning (NP). Our findings revealed that moderate pruning of 20% to 40% maintained or improved model accuracy while decreasing the model size by up to 40%, inference time by 30%, and memory usage by 10%. Among the models, 1D-CNN consistently provided the optimal balance between performance and computational efficiency. Regarding pruning techniques, structured neuron pruning (NP) has proven to be the most effective method, achieving significant reductions in model size and memory usage without compromising on detection performance. In contrast, unstructured weight pruning (WP) primarily reduced the number of active weights in the model by setting them to zero, which helped lower memory usage during processing; however, no compression was applied to the actual file size. Overall, the findings showed that adaptive tuning, when combined with pruning, offers an effective approach for building high-performing lightweight models that can be implemented in resource-constrained environments for encrypted malware traffic flow classification. DA - 2026 DB - OpenUCT DP - University of Cape Town KW - Encrypted Traffic Flows KW - Deep Learning KW - Malware Detection KW - Post-training Pruning KW - Weight Pruning KW - Neuron Pruning KW - Hyperband Optimization KW - Binary Classification KW - Multiclass Classification KW - Resource-constrained Environments LK - https://open.uct.ac.za PB - University of Cape Town PY - 2026 T1 - Deep learning models for malware traffic detection and classification: a focus on pruning and hyperband for resource optimization TI - Deep learning models for malware traffic detection and classification: a focus on pruning and hyperband for resource optimization UR - http://hdl.handle.net/11427/43661 ER - | en_ZA |
| dc.identifier.uri | http://hdl.handle.net/11427/43661 | |
| dc.identifier.vancouvercitation | Sheetekela SP. Deep learning models for malware traffic detection and classification: a focus on pruning and hyperband for resource optimization. []. University of Cape Town ,Faculty of Science ,Department of Computer Science, 2026 [cited yyyy month dd]. Available from: http://hdl.handle.net/11427/43661 | en_ZA |
| dc.language.iso | en | |
| dc.language.rfc3066 | eng | |
| dc.publisher.department | Department of Computer Science | |
| dc.publisher.faculty | Faculty of Science | |
| dc.publisher.institution | University of Cape Town | |
| dc.subject | Encrypted Traffic Flows | |
| dc.subject | Deep Learning | |
| dc.subject | Malware Detection | |
| dc.subject | Post-training Pruning | |
| dc.subject | Weight Pruning | |
| dc.subject | Neuron Pruning | |
| dc.subject | Hyperband Optimization | |
| dc.subject | Binary Classification | |
| dc.subject | Multiclass Classification | |
| dc.subject | Resource-constrained Environments | |
| dc.title | Deep learning models for malware traffic detection and classification: a focus on pruning and hyperband for resource optimization | |
| dc.type | Thesis / Dissertation | |
| dc.type.qualificationlevel | Masters | |
| dc.type.qualificationlevel | MSc |