A Review of Outlier Detection Techniques in Cybersecurity: A Machine Learning Perspective

Abstract

Outlier detection has emerged as a critical component of modern cybersecurity systems, enabling the timely identification of anomalous behavior to prevent breaches, detect insider threats, and mitigate zero-day attacks. Traditional rule-based systems are proving inadequate for the increasing scale and complexity of cyber threats, prompting the integration of machine learning (ML) techniques to enhance detection accuracy and adaptability. This review paper synthesizes existing studies from 2000 to 2025, identified using keyword-based searches in Scopus, IEEE Xplore, and Google Scholar. The selection criteria focused on relevance, recency, and applications of ML-based outlier detection in cybersecurity. We categorize outlier detection methods into statistical, distance-based, density-based, clustering-based, and ML-driven approaches, and discuss their applications in intrusion detection, malware analysis, phishing detection, and Internet of Things (IoT) security. Additionally, the paper addresses commonly used datasets and evaluation metrics, challenges such as class imbalance and concept drift, and future research directions, including explainable AI and adversarial robustness. By synthesizing the current landscape and identifying research gaps, this review aims to guide the development of intelligent, scalable, and interpretable outlier detection systems for cybersecurity.