More Web Proxy on the site http://driver.im/

research-article

Public Access

LEMNA: Explaining Deep Learning based Security Applications

Authors:

Xinyu XingAuthors Info & Claims

CCS '18: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security

Pages 364 - 379

https://doi.org/10.1145/3243734.3243792

Published: 15 October 2018 Publication History

Abstract

While deep learning has shown a great potential in various domains, the lack of transparency has limited its application in security or safety-critical areas. Existing research has attempted to develop explanation techniques to provide interpretable explanations for each classification decision. Unfortunately, current methods are optimized for non-security tasks ( e.g., image analysis). Their key assumptions are often violated in security applications, leading to a poor explanation fidelity. In this paper, we propose LEMNA, a high-fidelity explanation method dedicated for security applications. Given an input data sample, LEMNA generates a small set of interpretable features to explain how the input sample is classified. The core idea is to approximate a local area of the complex deep learning decision boundary using a simple interpretable model. The local interpretable model is specially designed to (1) handle feature dependency to better work with security applications ( e.g., binary code analysis); and (2) handle nonlinear local boundaries to boost explanation fidelity. We evaluate our system using two popular deep learning applications in security (a malware classifier, and a function start detector for binary reverse-engineering). Extensive evaluations show that LEMNA's explanation has a much higher fidelity level compared to existing methods. In addition, we demonstrate practical use cases of LEMNA to help machine learning developers to validate model behavior, troubleshoot classification errors, and automatically patch the errors of the target models.

Supplementary Material

MP4 File (p364-guo.mp4)

Download
351.96 MB

References

[1]

2014. Mimcus. https://github.com/srndic/mimicus. (2014).

[2]

Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad Rieck, and CERT Siemens. 2014. DREBIN: Effective and Explainable Detection of Android Malware in Your Pocket. In Proceedings of the 20th Network and Distributed System Security Symposium (NDSS).

[3]

Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. 2015. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one (2015).

[4]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).

[5]

Tiffany Bao, Johnathon Burket, Maverick Woo, Rafael Turner, and David Brumley. 2014. Byteweight: Learning to recognize functions in binary code. In Proceedings of the 23rd USENIX Security Symposium (USENIX Security).

Digital Library

[6]

Osbert Bastani, Carolyn Kim, and Hamsa Bastani. 2017. Interpreting blackbox models via model extraction. arXiv preprint arXiv:1705.08504 (2017).

[7]

Konstantin Berlin, David Slater, and Joshua Saxe. 2015. Malicious behavior detection using windows audit logs. In Proceedings of the 8th Workshop on Artificial Intelligence and Security (AISec).

Digital Library

[8]

Arjun Nitin Bhagoji, Daniel Cullina, and Prateek Mittal. 2017. Dimensionality reduction as a defense against evasion attacks on machine learning classifiers. arXiv preprint arXiv:1704.02654 (2017).

[9]

Xiaoyu Cao and Neil Zhenqiang Gong. 2017. Mitigating evasion attacks to deep neural networks via region-based classification. In Proceedings of the 33rd Annual Computer Security Applications Conference (ACSAC).

Digital Library

[10]

Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In Proceedings of the 36th IEEE Symposium on Security and Privacy (S&P).

Digital Library

[11]

Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In Proceedings of the 38th IEEE Symposium on Security and Privacy (S&P).

[12]

Gert Cauwenberghs and Tomaso Poggio. 2000. Incremental and decremental support vector machine learning. In Proceedings of the 13th Conference on Neural Information Processing Systems (NIPS).

Digital Library

[13]

Peng Chen and Hao Chen. 2018. Angora: Efficient Fuzzing by Principled Search. In Proceedings of the 39th IEEE Symposium on Security and Privacy (S&P).

[14]

François Chollet et al. 2017. Keras. (2017).

[15]

Zheng Leong Chua, Shiqi Shen, Prateek Saxena, and Zhenkai Liang. 2017. Neural Nets Can Learn Function Type Signatures From Binaries. In Proceedings of the 26th USENIX Security Symposium (USENIX Security).

Digital Library

[16]

George E Dahl, Jack W Stokes, Li Deng, and Dong Yu. 2013. Large-scale malware classification using random projections and neural networks. In Proceedings of the 38th International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[17]

R.C. Fong and A. Vedaldi. 2017. Interpretable Explanations of Black Boxes by Meaningful Perturbation. In Proceedings of the 16th International Conference on Computer Vision (ICCV).

[18]

Chuang Gan, Naiyan Wang, Yi Yang, Dit-Yan Yeung, and Alex G Hauptmann. 2015. Devnet: A deep event network for multimedia event detection and evidence recounting. In Proceedings of the 28th Conference on Computer Vision and Pattern Recognition. (CVPR).

[19]

Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. 2018. AI 2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation. In Proceedings of the 39th IEEE Symposium on Security and Privacy (S&P).

[20]

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the 3rd International Conference on Learning Representations (ICLR).

[21]

Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick McDaniel. 2016. Adversarial perturbations against deep neural networks for malware classification. arXiv preprint arXiv:1606.04435 (2016).

[22]

The Santa Cruz Operation Inc. 1997. System V application binary interface. (1997).

[23]

Anil K. Jain and B. Chandrasekaran. 1982. Dimensionality and Sample Size Considerations in Pattern Recognition Practice. Handbook of Statistics (1982).

[24]

Ahmad Javaid, Quamar Niyaz, Weiqing Sun, and Mansoor Alam. 2016. A deep learning approach for network intrusion detection system. In Proceedings of the 9th International Conference on Bio-inspired Information and Communications Technologies (BIONETICS).

Digital Library

[25]

Justin Johnson, Andrej Karpathy, and Li Fei-Fei. 2016. Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the 29th Conference on Computer Vision and Pattern Recognition (CVPR).

[26]

Ian T Jolliffe. 1986. Principal component analysis and factor analysis. In Principal component analysis.

[27]

Michael I Jordan and Robert A Jacobs. 1994. Hierarchical mixtures of experts and the EM algorithm. Neural computation (1994).

Digital Library

[28]

Abbas Khalili and Jiahua Chen. 2007. Variable selection in finite mixture of regression models. Journal of the american Statistical association (2007).

[29]

Pang Wei Koh and Percy Liang. 2017. Understanding Black-box Predictions via Influence Functions. In Proceedings of the 34th International Conference on Machine Learning (ICML).

Digital Library

[30]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th Conference on Neural Information Processing Systems (NIPS).

Digital Library

[31]

Himabindu Lakkaraju, Stephen H Bach, and Jure Leskovec. 2016. Interpretable decision sets: A joint framework for description and prediction. In Proceedings of the 22nd International Conference on Knowledge Discovery and Data Mining (KDD).

Digital Library

[32]

Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. Understanding Neural Networks through Representation Erasure. arXiv preprint arXiv:1612.08220 (2016).

[33]

Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. 2017. Delving into transferable adversarial examples and black-box attacks. In Proceedings of the 5th International Conference on Learning Representations (ICLR).

[34]

Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 30th Conference on Neural Information Processing Systems (NIPS).

[35]

Mengersen K. Marin, J.M. and C.P. Robert. 2005. Bayesian modelling and inference on mixtures of distributions. Handbook of statistics (2005).

[36]

Dongyu Meng and Hao Chen. 2017. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 24th ACM Conference on Computer and Communications Security (CCS).

Digital Library

[37]

Bengt Muthén and Kerby Shedden. 1999. Finite mixture modeling with mixture outcomes using the EM algorithm. Biometrics (1999).

[38]

In Jae Myung. 2003. Tutorial on maximum likelihood estimation. Journal of mathematical Psychology (2003).

Digital Library

[39]

Bruno A Olshausen and David J Field. 1996. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature (1996).

[40]

Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. 2016. Distillation as a defense to adversarial perturbations against deep neural networks. In Proceedings of the 37th IEEE Symposium on Security and Privacy (S&P).

[41]

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. Journal of machine learning research (2011).

Digital Library

Cited By

Xu LWang BZhao DWu X(2025)DAN: Neural network based on dual attention for anomaly detection in ICSExpert Systems with Applications10.1016/j.eswa.2024.125766263(125766)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125766
Di Bonito LCampanile LDi Natale FMastroianni MIacono M(2024)eXplainable Artificial Intelligence in Process Engineering: Promises, Facts, and Current LimitationsApplied System Innovation10.3390/asi70601217:6(121)Online publication date: 30-Nov-2024
https://doi.org/10.3390/asi7060121
Arreche OGuntur TAbdallah M(2024)XAI-IDS: Toward Proposing an Explainable Artificial Intelligence Framework for Enhancing Network Intrusion Detection SystemsApplied Sciences10.3390/app1410417014:10(4170)Online publication date: 14-May-2024
https://doi.org/10.3390/app14104170
Show More Cited By

Index Terms

LEMNA: Explaining Deep Learning based Security Applications
1. Security and privacy
  1. Software and application security
    1. Software reverse engineering

Recommendations

Causal generative explainers using counterfactual inference: a case study on the Morpho-MNIST dataset
Abstract
In this paper, we propose leveraging causal generative learning as an interpretable tool for explaining image classifiers. Specifically, we present a generative counterfactual inference approach to study the influence of visual features (pixels) ...
RiftNeXt™: Explainable Deep Neural RF Scene Classification
WiseML '21: Proceedings of the 3rd ACM Workshop on Wireless Security and Machine Learning

We propose a framework, RiftNeXtTM, to perform radio frequency (RF) scene context change detection and classification with Expert driven neural explainability. Our approach uses a deep learning based classifier to perform spectrum monitoring of Wi-Fi ...
Counterfactual explanations and how to find them: literature review and benchmarking
Abstract
Interpretable machine learning aims at unveiling the reasons behind predictions returned by uninterpretable classifiers. One of the most valuable types of explanation consists of counterfactuals. A counterfactual explanation reveals what should ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

CCS '18: Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security

October 2018

2359 pages

ISBN:9781450356930

DOI:10.1145/3243734

General Chairs:
David Lie
University of Toronto
,
Mohammad Mannan
Concordia University
,
Program Chairs:
Michael Backes
CISPA Helmholtz Center i.G.
,
XiaoFeng Wang
Indiana University

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSAC: ACM Special Interest Group on Security, Audit, and Control

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Paper

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

CCS '18

Sponsor:

SIGSAC

CCS '18: 2018 ACM SIGSAC Conference on Computer and Communications Security

October 15 - 19, 2018

Toronto, Canada

Acceptance Rates

CCS '18 Paper Acceptance Rate 134 of 809 submissions, 17%;

Overall Acceptance Rate 1,261 of 6,999 submissions, 18%

Upcoming Conference

CCS '25

Sponsor:
sigsac

ACM SIGSAC Conference on Computer and Communications Security

October 13 - 17, 2025

Taipei , Taiwan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

188
Total Citations
View Citations
5,774
Total Downloads

Downloads (Last 12 months)1,028
Downloads (Last 6 weeks)123

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xu LWang BZhao DWu X(2025)DAN: Neural network based on dual attention for anomaly detection in ICSExpert Systems with Applications10.1016/j.eswa.2024.125766263(125766)Online publication date: Mar-2025
https://doi.org/10.1016/j.eswa.2024.125766
Di Bonito LCampanile LDi Natale FMastroianni MIacono M(2024)eXplainable Artificial Intelligence in Process Engineering: Promises, Facts, and Current LimitationsApplied System Innovation10.3390/asi70601217:6(121)Online publication date: 30-Nov-2024
https://doi.org/10.3390/asi7060121
Arreche OGuntur TAbdallah M(2024)XAI-IDS: Toward Proposing an Explainable Artificial Intelligence Framework for Enhancing Network Intrusion Detection SystemsApplied Sciences10.3390/app1410417014:10(4170)Online publication date: 14-May-2024
https://doi.org/10.3390/app14104170
Quan XXie XLiu Y(2024)Sentiment interpretability analysis on Chinese texts employing multi-task and knowledge baseFrontiers in Artificial Intelligence10.3389/frai.2023.11040646Online publication date: 5-Jan-2024
https://doi.org/10.3389/frai.2023.1104064
Tong LLv ZGuo J(2024)Application of Online Automated Segmentation and Evaluation Method in Anomaly Detection at Rail Profile Based on Pattern Matching and Complex NetworksISIJ International10.2355/isijinternational.ISIJINT-2024-00364:10(1528-1537)Online publication date: 15-Aug-2024
https://doi.org/10.2355/isijinternational.ISIJINT-2024-003
Zhang BGao YKuang BYu CFu ASusilo W(2024)A Survey on Advanced Persistent Threat Detection: A Unified Framework, Challenges, and CountermeasuresACM Computing Surveys10.1145/370074957:3(1-36)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3700749
Lee JChen SMordahl ALiu CYang WWei S(2024)Automated Testing Linguistic Capabilities of NLP ModelsACM Transactions on Software Engineering and Methodology10.1145/367245533:7(1-33)Online publication date: 14-Jun-2024
https://dl.acm.org/doi/10.1145/3672455
Chen SLi ZYang WLiu C(2024)DeciX: Explain Deep Learning Based Code Generation ApplicationsProceedings of the ACM on Software Engineering10.1145/36608141:FSE(2424-2446)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660814
Han DWang ZFeng RJin MChen WWang KWang SYang JShi XYin XLiu YLuo BLiao XXu JKirda ELie D(2024)Rules Refine the Riddle: Global Explanation for Deep Learning-Based Anomaly Detection in Security ApplicationsProceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security10.1145/3658644.3670375(4509-4523)Online publication date: 2-Dec-2024
https://dl.acm.org/doi/10.1145/3658644.3670375
Kasarapu SBhusal DRastogi NPudukotai Dinakarrao S(2024)Comprehensive Analysis of Consistency and Robustness of Machine Learning Models in Malware DetectionProceedings of the Great Lakes Symposium on VLSI 202410.1145/3649476.3658725(477-482)Online publication date: 12-Jun-2024
https://dl.acm.org/doi/10.1145/3649476.3658725
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents