Program Characterization for Software Exploitation Detection

Software exploitation is an ever-growing problem. Signature-based exploitation detection techniques have not been effective as malicious actors continuously develop circumvention techniques. Current ML-based (signature-less) exploitation detection research is limited in quantity and use cases. Key to the success of any ML model is the characteristics used to depict program behaviour (i.e., features). Current work on using ML for software exploitation is focused on novelty ML algorithms while neglecting program characterization and under-reporting the approach for data preparation. There are two main competing program characterization techniques, micro-architecture independent (MAI) and micro-architecture dependent (MAD) techniques. This study evaluates MAI program characterization techniques for use with ML-based exploitation detection. A publicly available runtime-based traces of 11 Windows applications under buffer-overflow exploitation is used to replicate the feature engineering work found in research that uses MAI for ML-based exploitation detection. The performance and feature importance are evaluated with two different ensemble ML models (Random Forests and XGBoost). The results demonstrate that, although 0% FPR has been achieved in all datasets, MAI features that are purely fine-grained in nature can achieve a maximum recall value of 100% and an average recall of 40%, respectively. While features that contain a higher coarse-grained to fine-grained features ratio can achieve a maximum recall of 100% with an average value of 62%. The study provides a detailed discussion of the feature importance and reveals that the most important features relate to memory traffic characteristics.