Machine Learning-Based Model for Identification of Malicious Encrypted Files

Abstract

The widespread use of encryption to protect digital data and communication has posed new challenges for cybersecurity, as attackers increasingly exploit encrypted files to hide malicious content that bypasses traditional signature-based detection systems. This research develops a machine learning model designed to detect harmful encrypted files, overcoming the weaknesses of conventional methods that fail to analyze encrypted data effectively. Since encryption conceals the payloads of malicious files, advanced classification models are essential for accurate identification. The study applies three supervised learning algorithms Convolutional Neural Network (CNN), Support Vector Machine (SVM), and Recurrent Neural Network (RNN) to train and evaluate models capable of differentiating between legitimate and malicious encrypted files. The dataset was obtained from the Canadian Institute for Cybersecurity (CIC), and model performance was assessed using confusion matrix–based evaluation metrics implemented in Python. Among the models tested, SVM achieved the highest detection accuracy of 98.7%, outperforming RNN (91%) and CNN (74.3%). These results indicate that SVM delivers superior classification efficiency, particularly in handling complex and high-dimensional data features. The study concludes that integrating such models into enterprise cybersecurity systems can significantly enhance threat detection and protection mechanisms. It further recommends exploring hybrid deep learning strategies to improve adaptability, reduce false positives, and enable real-time detection of malicious encrypted files in rapidly changing digital environments.