Cited By
View all- Saravanan SPrakash G(2021)A Comprehensive Survey on Big Data Technology Based Cybersecurity Analytics SystemsApplied Soft Computing and Communication Networks10.1007/978-981-33-6173-7_9(123-143)Online publication date: 2-Jul-2021
Survey of data intensive computing technologies application to to security log data management
Authors: 
Anne Tall, 
Jun Wang, Dezhi HanAuthors Info & Claims
Pages 268 - 273
Abstract
Data intensive computing research and technology developments offer the potential of providing significant improvements in several security log management challenges. Approaches to address the complexity, timeliness, expense, diversity, and noise issues have been identified. These improvements are motivated by the increasingly important role of analytics. Machine learning and expert systems that incorporate attack patterns are providing greater detection insights. Finding actionable indicators requires the analysis to combine security event log data with other network data such and access control lists, making the big-data problem even bigger. Automation of threat intelligence is recognized as not complete with limited adoption of standards. With limited progress in anomaly signature detection, movement towards using expert systems has been identified as the path forward. Techniques focus on matching behaviors of attackers to patterns of abnormal activity in the network. The need to stream, parse, and analyze large volumes of small, semi-structured data files can be feasibly addressed through a variety of techniques identified by researchers. This report highlights research in key areas, including protection of the data, performance of the systems and network bandwidth utilization.
References
[1]
Apache Hadoop, https://hadoop.apache.org/
[2]
Spark, https://spark.apache.org
[3]
Chuvakin, A.; Schmidt, K.; Phillips, C.; Logging and Log Management: The Authoritative Guide to Understanding the Concepts Surrounding Logging and Log Management, Elsevier, 2013.
[4]
Apache Flume, https://flume.apache.org/
[5]
Beaver, D.; Hutchinson, S.; Elasticsearch, Logstash, and Kibana (ELK), Carnegie Mellon University Software Engineering Institute FloCon 2015, January 2015. http://www.cert.org/flocon/past-conferences.cfm
[6]
Logstash, Elastic, https://www.elastic.co/products/logstash
[7]
Yen, T.; Oprea, A.; Onarlioglu, K.; Leetham, T.; Robertson, W.; Juels, A.; Kirda, E.; "Beehive: Large-Scale Log Analysis for Detecting Suspicious Activity in Enterprise Networks," ACM, ACSAC, December 2013.
[8]
Goldberg, J., SPLUNK, "Big Data for Security: How Can I Put Big Data To Work For Me?" and "Good Guys Vs Bad Guys: Using Big Data to Counteract Advanced Threats," RSA Conference Europe 2013, October 2013.
[9]
Fu, Q.; Lou, J.; Wang, Y.; Li, J.; "Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis," 2009 IEEE International Conference on Data Mining, 2009.
[10]
Myers, J.; Grimaila, M.; Mills, R.; "Log-Based Distributed Security Event Detection Using Simple Event Correlator," Proceedings of the 44th Hawaii International Conference on System Science, 2011.
[11]
Poletto, M., "Data mining for security at Google," Google Security Team - Stanford CS259D, 28 Oct 2014.
[12]
Bhatt, P.; Yano Toshiro E.; Gustavsson, P.; "Towards a Framework to Detect Multi-Stage Advanced Persistent Threats Attacks," 2014 IEEE 8th International Symposium on Service Oriented Systems Engineering, 2014.
[13]
Leskovec, J.; Rajaraman, A.; Ullman, J.; Mining of Massive Datasets, Second Edition, Cambridge University Press, 2014.
[14]
Vaarandi, R.; Pihelgas, M.; "Using Security Logs for Collecting and Reporting Technical Security Metrics," 2014 IEEE Military Communications Conferences, 2014.
[15]
Marty, R., PixlCloud - Data Visualization, https://about.me/raffy
[16]
Mackey, G.; Sehrish, S.; Wang, J, "Improving Metadata Management for Small Files in HDFS," Proceedings of IEEE International Conference on Cluster Computing and Workshops, August 2009.
[17]
Dong, B.; Qiu, J.; Zheng, Q.; Zhong, X.; Li, J.; Li, Y.; "A Novel Approach to Improving the Efficiency of Storing and Accessing Small Files on Hadoop: a Case Study by PowerPoint Files," 2010 IEEE International Conference on Services Computing (SCC), 2010.
[18]
Yang, F.; Liu, H.; Zhao, Z.; "Research on Cloud-Based Mass Log Data Management Mechanism," Journal of Computers, Vol 9, No 6, June 2014.
[19]
Zhou, W.; Zhan, J.; Meng, D.; Xu, D.; Zhang, Z., "LogMaster: Mining Event Correlations in Logs of Large scale Cluster Systems," Cornell University Library, January 2013.
[20]
Ferrera, P.; dePrado, I.; Palacios, E.; Fernandez-Marquez, J.; Seruguendo, G.; "Tuple MapReduce: Beyond classic MapReduce," 2012 IEEE 12 International Conference on Data Mining, 2012.
[21]
Wang, R; Enck, W.; Reeves, D.; Zhang, X.; Ning, P.; Xu, D.; Zhou, W.; Azab, A.; "EASEAndroid: Automatic Policy Analysis and Refinement for Security Enhanced Android via Larg-Scale Semi Supervised Learning," Samsung Research America and North Carolina state University, 24th UNENIX Security Symposium, 12 August 2015.
[22]
Adversarial Tactic, Techniques and Common Knowledge (ATT&CK), https://attack.mitre.org/
[23]
Common Attack Pattern Enumeration and Classification (CAPEC), https://capec.mitre.org/
[24]
Structured Threat Information eXpression (STIX), http://stixproject.github.io/
[25]
Ray, I.; Belyaev, K.; Strizhov, M.; Mulamba, D.; Rajaram, M.; "Secure Logging as a Service - Delegating Log Management to the Cloud," IEEE Systems Journal, Vol. 7, No. 2, June 2013.
[26]
Chen, G.; Wu, S.; Wang, Y.; "The Evolvement of Big Data Systems: From the Perspective of an Information Security Application," Elsevier - Big Data Research, 23 March 2015.
[27]
Marchal, S.; Jiang, X.; State, R.; Engel, T.; "A Big Data Architecture for Large Scale Security Monitoring," 2014 IEEE International Congress on Big Data, 2014.
[28]
Bandre, S.; Nandimath, J.; "Design Considerations of Network Intrusion Detection System using Hadoop and GPGPU," IEEE 2015 International Conference on Pervasive Computing (ICPC), 2015.
- Survey of data intensive computing technologies application to to security log data management
Recommendations
A Spark-Based Big Data Platform for Massive Remote Sensing Data Processing
ICDS 2015: Proceedings of the Second International Conference on Data Science - Volume 9208With the fast development of remote sensing techniques, the volume of acquired data grows exponentially. This brings a big challenge to process massive remote sensing data. In the paper, an in-memory computing framework is proposed to address this ...
An experimental survey on big data frameworks
AbstractRecently, increasingly large amounts of data are generated from a variety of sources.Existing data processing technologies are not suitable to cope with the huge amounts of generated data. Yet, many research works focus on Big Data, a buzzword ...
Highlights- An overview of most popular Big Data frameworks.
- A categorization of the presented frameworks and techniques.
- An extensive set of experiments to evaluate the studied Big Data frameworks.
- A description of best practices related ...
A Performance Study of Big Spatial Data Systems
BigSpatial '18: Proceedings of the 7th ACM SIGSPATIAL International Workshop on Analytics for Big Geospatial DataWith the accelerated growth in spatial data volume, being generated from a wide variety of sources, the need for efficient storage, retrieval, processing and analyzing of spatial data is ever more important. Hence, spatial data processing system has ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2016 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 06 December 2016
Check for updates
Author Tags
Qualifiers
- Short-paper
Funding Sources
Conference
UCC '16
Sponsor:
UCC '16: 9th International Conference on Utility and Cloud Computing
December 6 - 9, 2016
Shanghai, China
Acceptance Rates
Overall Acceptance Rate 27 of 93 submissions, 29%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 501Total Downloads
- Downloads (Last 12 months)57
- Downloads (Last 6 weeks)9
Reflects downloads up to 14 Dec 2024
Other Metrics
Citations
Cited By
View all- Saravanan SPrakash G(2021)A Comprehensive Survey on Big Data Technology Based Cybersecurity Analytics SystemsApplied Soft Computing and Communication Networks10.1007/978-981-33-6173-7_9(123-143)Online publication date: 2-Jul-2021
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in