Abstract
Nowadays, executable packing remains an open issue in its detection especially when it comes to static analysis. Packing is significantly used in malware to hide malicious code from detection systems. These last years, many studies about static packing detection addressed this problem with heuristics and machine learning, considering different ad hoc techniques, algorithms and feature sets but very few addressed it from the adversarial point of view, that is, how to fool heuristics by altering samples with targeted modifications. The objective of this work is to study to what extent it is easy to evade detection by open source static detectors that are commonly used by the community by applying alterations on packed samples, which require only slight adaptations of the related packers, resulting in evasion. An adversarial setting from the problem-space perspective is addressed by using realistic modifications of binary samples that target common significant features. For this purpose, alterations and datasets are composed and static detection is applied using the experimental toolkit Packing Box. Results of alterations are shown, in terms of information gain of features and accuracy of detection, on open source static packing detectors. Finally, their significant effects are highlighted and their effectiveness is evaluated.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Yet Another Markup Language.
References
Anderson, H.S., Kharkar, A., Filar, B., Evans, D., Roth, P.: Learning to evade static PE machine learning malware models via reinforcement learning (2018)
Bat-Erdene, M., Park, H., Li, H., Lee, H., Choi, M.S.: Entropy analysis to classify unknown packing algorithms for malware detection. Int. J. Inf. Secur. 16, 227–248 (2016)
Bertrand Van Ouytsel, C.H., Dam, K.H.T., Legay, A.: Analysis of machine learning approaches to packing detection. Comput. Secur. 136, 103536 (2023)
Biondi, F., Enescu, M.A., Given-Wilson, T., Legay, A., Noureddine, L., Verma, V.: Effective, efficient, and robust packing detection and classification. Comput. Secur. 85, 436–451 (2019)
Carlini, N., et al.: On evaluating adversarial robustness (2019)
Choi, M.J., Bang, J., Kim, J., Kim, H., Moon, Y.S., Díaz-Verdejo, J.: All-in-one framework for detection, unpacking, and verification for malware analysis. Secur. Commun. Netw. (2019)
Choi, Y.S., Kim, I.K., Oh, J.T., Ryou, J.C.: PE file header analysis-based packed PE file detection technique (PHAD). In: International Symposium on Computer Science and its Applications (2008)
Demetrio, L., Biggio, B., Roli, F.: Practical attacks on machine learning: a case study on adversarial windows malware. IEEE Secur. Priv. 20, 77–85 (2022)
D’Hondt, A., Bertrand Van Ouytsel, C.H., Legay, A.: Experimental toolkit for manipulating executable packing. In: International Conference on Risks and Security of Internet and Systems (2023)
Fang, Z., Wang, J., Li, B., Wu, S., Zhou, Y., Huang, H.: Evading anti-malware engines with deep reinforcement learning. IEEE (2019)
Han, S., Lee, K., Lee, S.: Packed PE file detection for malware forensics. In: International Conference on Computational Science and Its Applications (2009)
Kancherla, K., Donahue, J., Mukkamala, S.: Packer identification using Byte plot and Markov plot. J. Comput. Virol. Hacking Tech. 12, 101–111 (2015)
Khan, M., Akram, M., Riaz, N.: A comparative analysis of software protection schemes. Int. Arab J. Inf. Technol. (2014)
Ling, X., et al.: Adversarial attacks against windows PE malware detection: a survey of the state-of-the-art. Comput. Secur. 128, 103134 (2021)
Lyda, R., Hamrock, J.: Using entropy analysis to find encrypted and packed malware. IEEE Secur. Priv. 5, 40–45 (2007)
Pareek, H., Arora, R., Singh, A.: A heuristics-based static analysis approach for detecting packed PE binaries. Int. J. Secur. Appl. 7, 257–268 (2013)
Shin, D., Im, C., Jeong, H., Kim, j., Won, D.: The new signature generation method based on an unpacking algorithm and procedure for a packer detection. In: International Journal of Advanced Science and Technology (2011)
Ugarte-Pedrero, X., Balzarotti, D., Santos, I., Bringas, P.G.: SoK: deep packer inspection: a longitudinal study of the complexity of run-time packers. In: IEEE Symposium on Security and Privacy (2015)
Wu, C., Shi, J., Yang, Y., Li, W.: Enhancing machine learning based malware detection model by reinforcement learning. In: International Conference on Communication and Network Security (2018)
Acknowledgements
The authors are funded by the CyberExcellence project (RW, Convention 2110186). They want to express their gratitude to Romain Jennes who made a significant contribution through his master’s thesis.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
D’Hondt, A., Bertrand Van Ouytsel, C.H., Legay, A. (2024). Extended Abstract: Evading Packing Detection: Breaking Heuristic-Based Static Detectors. In: Maggi, F., Egele, M., Payer, M., Carminati, M. (eds) Detection of Intrusions and Malware, and Vulnerability Assessment. DIMVA 2024. Lecture Notes in Computer Science, vol 14828. Springer, Cham. https://doi.org/10.1007/978-3-031-64171-8_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-64171-8_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-64170-1
Online ISBN: 978-3-031-64171-8
eBook Packages: Computer ScienceComputer Science (R0)