More Web Proxy on the site http://driver.im/

research-article

Open access

Multiple Level Action Embedding for Penetration Testing

Authors:

Hoang Viet Nguyen,

Hai Ngoc Nguyen,

Tetsutaro UeharaAuthors Info & Claims

ICFNDS '20: Proceedings of the 4th International Conference on Future Networks and Distributed Systems

Article No.: 53, Pages 1 - 9

https://doi.org/10.1145/3440749.3442660

Published: 13 May 2021 Publication History

All formats PDF

Abstract

Penetration Testing (PT) is one of the most effective and widely used methods to increase the defence of a system by looking for potential vulnerabilities. Reinforcement learning (RL), a powerful type of machine learning in self-decision making, is demonstrated to be applicable in PT to increase automation as well as reduce implementation costs. However, RL algorithms are still having difficulty on PT problems which have large network size and high complexity. This paper proposes a multiple level action embedding applied with Wolpertinger architect (WA) to enhance the accuracy and performance of the RL, especially in large and complicated environments. The main purpose of the action embedding is to be able to represent the elements in the RL action space as an n-dimensional vector while preserving their properties and accurately representing the relationship between them. Experiments are conducted to evaluate the logical accuracy of the action embedding. The deep Q-network algorithm is also used as a baseline for comparing with WA using the multiple level action embedding.

References

[1]

Sahibsingh A Dudani. 1976. The distance-weighted k-nearest-neighbor rule. IEEE Transactions on Systems, Man, and Cybernetics4 (1976), 325–327.

[2]

Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. 2015. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679(2015).

[3]

Mohamed C Ghanem and Thomas M Chen. 2020. Reinforcement learning for efficient network penetration testing. Information 11, 1 (2020), 6.

[4]

Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining. 855–864.

Digital Library

[5]

Hado V Hasselt. 2010. Double Q-learning. In Advances in neural information processing systems. 2613–2621.

[6]

Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. 1996. Reinforcement learning: A survey. Journal of artificial intelligence research 4 (1996), 237–285.

Digital Library

[7]

Xiaoyuan Liang, Xunsheng Du, Guiling Wang, and Zhu Han. 2018. Deep reinforcement learning for traffic light control in vehicular networks. arXiv preprint arXiv:1803.11115(2018).

[8]

Long-Ji Lin. 1992. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning 8, 3-4 (1992), 293–321.

[9]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. 3111–3119.

[10]

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In International conference on machine learning. 1928–1937.

Digital Library

[11]

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602(2013).

[12]

S Ozkan. 2011. CVE Details: The ultimate security vulnerability datasource. Retrieved March 20, 2020 from http://www.cvedetails.com

[13]

Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2015. Prioritized experience replay. arXiv preprint arXiv:1511.05952(2015).

[14]

Jonathon Schwartz and Hanna Kurniawati. 2019. Autonomous penetration testing using reinforcement learning. arXiv preprint arXiv:1905.05965(2019).

[15]

Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. 2016. Dueling network architectures for deep reinforcement learning. In International conference on machine learning. 1995–2003.

[16]

Zhiheng Zhao, Yi Liang, and Xiaoming Jin. 2018. Handling large-scale action space in deep Q network. In 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD). IEEE, 93–96.

Cited By

Liu HLiu CWu XQu YLiu H(2024)An Automated Penetration Testing Framework Based on Hierarchical Reinforcement LearningElectronics10.3390/electronics1321431113:21(4311)Online publication date: 2-Nov-2024
https://doi.org/10.3390/electronics13214311
Nayab SWotawa F(2024)Testing and Reinforcement Learning - A Structured Literature Review2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C63300.2024.00049(326-335)Online publication date: 1-Jul-2024
https://doi.org/10.1109/QRS-C63300.2024.00049
Venturi AAndreolini MMarchetti MColajanni M(2024)Assessing generalizability of Deep Reinforcement Learning algorithms for Automated Vulnerability Assessment and Penetration TestingArray10.1016/j.array.2024.10036524(100365)Online publication date: Dec-2024
https://doi.org/10.1016/j.array.2024.100365
Show More Cited By

Recommendations

Proposal and evaluation of deep exploitation-oriented learning under multiple reward environment
Abstract
Recently, deep reinforcement learning (DRL) has attracted considerable attention. The well-known deep Q-network (DQN) architecture successfully combines deep learning and Q-learning which is a representative reinforcement learning (RL) ...
Path planning in an unknown environment based on deep reinforcement learning with prior knowledge

Path planning in an unknown environment is a basic task for mobile robots to complete tasks. As a typical deep reinforcement learning, deep Q-network (DQN) algorithm has gained wide popularity in path planning tasks due to its self-learning and ...
Deep deformable Q-Network: an extension of deep Q-Network
WI '17: Proceedings of the International Conference on Web Intelligence

The performance of Deep Reinforcement Learning (DRL) algorithms is usually constrained by instability and variability. In this work, we present an extension of Deep Q-Network (DQN) called Deep Deformable Q-Network which is based on deformable ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICFNDS '20: Proceedings of the 4th International Conference on Future Networks and Distributed Systems

November 2020

313 pages

ISBN:9781450388863

DOI:10.1145/3440749

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICFNDS '20

ICFNDS '20: The 4th International Conference on Future Networks and Distributed Systems

November 26 - 27, 2020

St.Petersburg, Russian Federation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
906
Total Downloads

Downloads (Last 12 months)245
Downloads (Last 6 weeks)27

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu HLiu CWu XQu YLiu H(2024)An Automated Penetration Testing Framework Based on Hierarchical Reinforcement LearningElectronics10.3390/electronics1321431113:21(4311)Online publication date: 2-Nov-2024
https://doi.org/10.3390/electronics13214311
Nayab SWotawa F(2024)Testing and Reinforcement Learning - A Structured Literature Review2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C63300.2024.00049(326-335)Online publication date: 1-Jul-2024
https://doi.org/10.1109/QRS-C63300.2024.00049
Venturi AAndreolini MMarchetti MColajanni M(2024)Assessing generalizability of Deep Reinforcement Learning algorithms for Automated Vulnerability Assessment and Penetration TestingArray10.1016/j.array.2024.10036524(100365)Online publication date: Dec-2024
https://doi.org/10.1016/j.array.2024.100365
Nguyen HUehara T(2023)Multilayer Action Representation based on MITRE ATT&CK for Automated Penetration TestingJournal of Information Processing10.2197/ipsjjip.31.56231(562-577)Online publication date: 2023
https://doi.org/10.2197/ipsjjip.31.562
Li LEl Rami JTaylor ARao JKunz T(2023)Unified Emulation-Simulation Training Environment for Autonomous Cyber AgentsMachine Learning for Networking10.1007/978-3-031-36183-8_9(130-144)Online publication date: 7-Jul-2023
https://doi.org/10.1007/978-3-031-36183-8_9
Wang WSun DJiang FChen XZhu C(2022)Research and Challenges of Reinforcement Learning in Cyber Defense Decision-Making for Intranet SecurityAlgorithms10.3390/a1504013415:4(134)Online publication date: 18-Apr-2022
https://doi.org/10.3390/a15040134
Husen AChaudary MAhmad F(2022)A Survey on Requirements of Future Intelligent Networks: Solutions and Future Research DirectionsACM Computing Surveys10.1145/352410655:4(1-61)Online publication date: 21-Nov-2022
https://dl.acm.org/doi/10.1145/3524106
Nguyen HUehara T(2022)Hierarchical Action Embedding for Effective Autonomous Penetration Testing2022 IEEE 22nd International Conference on Software Quality, Reliability, and Security Companion (QRS-C)10.1109/QRS-C57518.2022.00030(152-157)Online publication date: Dec-2022
https://doi.org/10.1109/QRS-C57518.2022.00030
Almazrouei OMagalingam P(2022)The Internet of Things Network Penetration Testing Model Using Attack Graph Analysis2022 International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT)10.1109/ISMSIT56059.2022.9932758(360-368)Online publication date: 20-Oct-2022
https://doi.org/10.1109/ISMSIT56059.2022.9932758
Greco CFortino GCrispo BChoo K(2022)AI-enabled IoT penetration testing: state-of-the-art and research challengesEnterprise Information Systems10.1080/17517575.2022.213001417:9Online publication date: 10-Oct-2022
https://doi.org/10.1080/17517575.2022.2130014

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents