A Machine Learning-Based Approach with the Cyber Security Big Data (CSBD) System

Abstract

This paper presents a comprehensive analysis of various advanced intrusion detection systems (IDS) and their methodologies, emphasizing the integration of machine learning algorithms and feature selection techniques to enhance network security. It reviews several innovative approaches proposed in recent studies, each offering unique strategies to improve the accuracy and efficiency of intrusion detection in information technology networks. This paper introduces the Cyber Security Big Data (CSBD) system, a novel approach aimed at enhancing security within big data environments. The CSBD system serves as a platform for selecting appropriate security services, with a key focus on establishing and verifying secure communication. The methodology involves network-level security, where user requests are authenticated using a machine learning algorithm at the Defense Policy Unit (DPU). The approach includes data collection from the CSE-CIC-IDS2018 dataset, pre-processing steps such as dropping unnecessary columns and data standardization, and a combined weighted feature extraction and classification method. The system's effectiveness is evaluated using various performance parameters including accuracy, precision, recall, and F1-score, with results demonstrating the system's high effectiveness in detecting various types of network activities and attacks.