Feature Engineering in Machine Learning for Advanced Threat Detection

Abstract

This study investigates the role of advanced feature engineering techniques in enhancing the accuracy, robustness, and interpretability of machine learning-based cyber threat detection systems. Building on a foundational framework that emphasizes the extraction of behavioral features for anomaly classification, this research proposes an enhanced approach. The proposed framework integrates domain-specific heuristics, protocol-aware attributes, and explainability techniques such as SHapley Additive exPlanations (SHAP) to strengthen intelligent threat detection capabilities. By refining traditional feature extraction pipelines and incorporating SHAP values, the framework offers human-understandable insights into model predictions, fostering trust among security analysts and supporting real-time decision-making in complex cyber environments. Rigorous experimental evaluations using diverse, real-world cybersecurity datasets demonstrate the framework's effectiveness, particularly in detecting stealthy, low-frequency, and novel threats that often evade conventional systems. The interpretable feature attributions further enhance forensic analysis, enabling security teams to trace, validate, and respond to threats with precision and contextual understanding. This work extends prior foundational contributions, presenting a scalable and interpretable framework that advances the field of cyber threat detection. The findings underscore the importance of merging domain expertise with explainable artificial intelligence to address the challenges posed by increasingly sophisticated cyber threats.