Abstract
Machine Learning (ML) models are susceptible to evasion attacks. Evasion accuracy is typically assessed using aggregate evasion rate, and it is an open question whether aggregate evasion rate enables feature-level diagnosis on the effect of adversarial perturbations on evasive predictions. In this paper, we introduce a novel framework that harnesses explainable ML methods to guide high-fidelity assessment of ML evasion attacks. Our framework enables explanation-guided correlation analysis between pre-evasion perturbations and post-evasion explanations. Towards systematic assessment of ML evasion attacks, we propose and evaluate a novel suite of model-agnostic metrics for sample-level and dataset-level correlation analysis. Using malware and image classifiers, we conduct comprehensive evaluations across diverse model architectures and complementary feature representations. Our explanation-guided correlation analysis reveals correlation gaps between adversarial samples and the corresponding perturbations performed on them. Using a case study on explanation-guided evasion, we show the broader usage of our methodology for assessing robustness of ML models .
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
CNET freeware site (2020). https://download.cnet.com/s/software/windows/?licenseType=Free
Cuckoo sandbox (2020). https://cuckoosandbox.org
LIEF project (2020). https://github.com/lief-project/LIEF
Virus share (2020). https://virusshare.com
Virus total (2020). https://www.virustotal.com/gui/home/upload
Ali, A., Eshete, B.: Best-effort adversarial approximation of black-box malware classifiers. In: Park, N., Sun, K., Foresti, S., Butler, K., Saxena, N. (eds.) SecureComm 2020. LNICST, vol. 335, pp. 318–338. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-63086-7_18
Anderson, H.S., Roth, P.: EMBER: an open dataset for training static PE malware machine learning models. ArXiv e-prints (2018)
Anderson, H.S., Kharkar, A., Filar, B., Evans, D., Roth, P.: Learning to evade static PE machine learning malware models via reinforcement learning. CoRR arXiv:1801.08917 (2018)
Apruzzese, G., Andreolini, M., Marchetti, M., Venturi, A., Colajanni, M.: Deep reinforcement adversarial learning against botnet evasion attacks. IEEE Trans. Netw. Serv. Manage. 17, 1975–1987 (2020)
Biggio, B., Roli, F.: Wild patterns: ten years after the rise of adversarial machine learning. Pattern Recogn. 84, 317–331 (2018)
Carlini, N., Wagner, D.A.: Towards evaluating the robustness of neural networks. In: IEEE SP, pp. 39–57 (2017)
Dahl, G.E., Yu, D., Deng, L., Acero, A.: Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Trans. Audio Speech Lang. Process. 20(1), 30–42 (2012)
Demetrio, L., Biggio, B., Lagorio, G., Roli, F., Armando, A.: Explaining vulnerabilities of deep learning to adversarial malware binaries. In: Proceedings of the Third Italian Conference on Cyber Security (2019)
Demontis, A., et al.: Yes, machine learning can be more secure! A case study on android malware detection. IEEE TDSC 16(4), 711–724 (2019)
Fan, M., Wei, W., Xie, X., Liu, Y., Guan, X., Liu, T.: Can we trust your explanations? Sanity checks for interpreters in android malware analysis. IEEE Trans. Inf. Forensics Secur. 16, 838–853 (2021)
Fidel, G., Bitton, R., Shabtai, A.: When explainability meets adversarial learning: Detecting adversarial examples using SHAP signatures. In: IEEE IJCNN, pp. 1–8 (2020)
Fong, R.C., Vedaldi, A.: Interpretable explanations of black boxes by meaningful perturbation. In: IEEE ICCV, pp. 3449–3457 (2017)
Gao, F., et al.: DeepCC: a novel deep learning-based framework for cancer molecular subtype classification. Oncogenesis 8(9), 1–12 (2019)
Ghorbani, A., Abid, A., Zou, J.: Interpretation of neural networks is fragile. In: AAAI, vol. 33 (2017)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: ICLR (2015)
Grosse, K., Papernot, N., Manoharan, P., Backes, M., McDaniel, P.: Adversarial examples for malware detection. In: Foley, S.N., Gollmann, D., Snekkenes, E. (eds.) ESORICS 2017. LNCS, vol. 10493, pp. 62–79. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66399-9_4
Guidotti, R., Monreale, A., Ruggieri, S., Pedreschi, D., Turini, F., Giannotti, F.: Local rule-based explanations of black box decision systems. CoRR arXiv:1805.10820 (2018)
Guo, W., Mu, D., Xu, J., Su, P., Wang, G., Xing, X.: LEMNA: explaining deep learning based security applications. In: ACM SIGSAC CCS, pp. 364–379 (2018)
Han, D., et al.: Practical traffic-space adversarial attacks on learning-based nidss. CoRR arXiv:2005.07519 (2020)
Heo, J., Joo, S., Moon, T.: Fooling neural network interpretations via adversarial model manipulation (2019)
Hu, W., Tan, Y.: Generating adversarial malware examples for black-box attacks based on GAN. CoRR arXiv:1702.05983 (2017)
Kolosnjaji, B., et al.: Adversarial malware binaries: evading deep learning for malware detection in executables. In: EUSIPCO, pp. 533–537 (2018)
Kreuk, F., Barak, A., Aviv-Reuven, S., Baruch, M., Pinkas, B., Keshet, J.: Deceiving end-to-end deep learning malware detectors using adversarial examples (2018)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)
Kurakin, A., Goodfellow, I.J., Bengio, S.: Adversarial machine learning at scale. CoRR arXiv:1611.01236 (2016)
LeCun, Y., Cortes, C., Burges, C.J.: The MNIST database of handwritten digits (2020). http://yann.lecun.com/exdb/mnist/
Li, J., Monroe, W., Jurafsky, D.: Understanding neural networks through representation erasure. CoRR arXiv:1612.08220 (2016)
Lundberg, S.M., Lee, S.: A unified approach to interpreting model predictions. In: NeurIPS, pp. 4765–4774 (2017)
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. CoRR arXiv:1706.06083 (2017)
Papernot, N., McDaniel, P.D., Goodfellow, I.J.: Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. CoRR arXiv:1605.07277 (2016)
Papernot, N., McDaniel, P.D., Goodfellow, I.J., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against deep learning systems using adversarial examples. CoRR arXiv:1602.02697 (2016)
Pierazzi, F., Pendlebury, F., Cortellazzi, J., Cavallaro, L.: Intriguing properties of adversarial ML attacks in the problem space. In: IEEE SP (2020)
Raff, E., Barker, J., Sylvester, J., Brandon, R., Catanzaro, B., Nicholas, C.K.: Malware detection by eating a whole EXE. In: AAAI Workshops, pp. 268–276 (2018)
Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: explaining the predictions of any classifier. In: ACM SIGKDD, pp. 1135–1144 (2016)
Ribeiro, M.T., Singh, S., Guestrin, C.: Anchors: high-precision model-agnostic explanations. In: AAAI, pp. 1527–1535 (2018)
Rosenberg, I., Shabtai, A., Rokach, L., Elovici, Y.: Generic black-box end-to-end attack against state of the art API call based malware classifiers. In: Bailey, M., Holz, T., Stamatogiannakis, M., Ioannidis, S. (eds.) RAID 2018. LNCS, vol. 11050, pp. 490–510. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00470-5_23
Sallab, A.E., Abdou, M., Perot, E., Yogamani, S.K.: Deep reinforcement learning framework for autonomous driving. CoRR arXiv:1704.02532 (2017)
Shapley, L.: A value for n-person games (1953)
Shrikumar, A., Greenside, P., Kundaje, A.: Learning important features through propagating activation differences. In: ICML, pp. 3145–3153 (2017)
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps. In: ICLR Workshop Track Proceedings (2014)
Smilkov, D., Thorat, N., Kim, B., Viégas, F.B., Wattenberg, M.: SmoothGrad: removing noise by adding noise. CoRR arXiv:1706.03825 (2017)
Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.A.: Striving for simplicity: the all convolutional net. In: ICLR Workshop Track Proceedings (2015)
Srndic, N., Laskov, P.: Practical evasion of a learning-based classifier: a case study. In: IEEE SP, pp. 197–211 (2014)
Suciu, O., Coull, S.E., Johns, J.: Exploring adversarial examples in malware detection. In: IEEE SP Workshops, pp. 8–14 (2019)
Warnecke, A., Arp, D., Wressnegger, C., Rieck, K.: Evaluating explanation methods for deep learning in security. In: IEEE EuroSP, pp. 158–174 (2020)
Xu, W., Qi, Y., Evans, D.: Automatically evading classifiers: a case study on PDF malware classifiers. In: NDSS (2016)
Yang, W., Kong, D., Xie, T., Gunter, C.A.: Malware detection in adversarial settings: exploiting feature evolutions and confusions in android apps. In: ACSAC, pp. 288–302 (2017)
Zhang, X., Wang, N., Shen, H., Ji, S., Luo, X., Wang, T.: Interpretable deep learning under fire. In: USENIX Security, pp. 1659–1676 (2020)
Acknowledgements
We thank our shepherd Giovanni Apruzzese and the anonymous reviewers for their insightful feedback that immensely improved this paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Amich, A., Eshete, B. (2021). Explanation-Guided Diagnosis of Machine Learning Evasion Attacks. In: Garcia-Alfaro, J., Li, S., Poovendran, R., Debar, H., Yung, M. (eds) Security and Privacy in Communication Networks. SecureComm 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 398. Springer, Cham. https://doi.org/10.1007/978-3-030-90019-9_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-90019-9_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-90018-2
Online ISBN: 978-3-030-90019-9
eBook Packages: Computer ScienceComputer Science (R0)