More Web Proxy on the site http://driver.im/

article

Online anomaly detection for multi-source VMware using a distributed streaming framework

Authors:

Mohiuddin Solaimani,

Mohammed Iftekhar,

Bhavani Thuraisingham,

Sadi Evren SekerAuthors Info & Claims

Software—Practice & Experience, Volume 46, Issue 11

Pages 1479 - 1497

https://doi.org/10.1002/spe.2390

Published: 01 November 2016 Publication History

Abstract

Anomaly detection refers to the identification of patterns in a dataset that do not conform to expected patterns. Such non-conformant patterns typically correspond to samples of interest and are assigned to different labels in different domains, such as outliers, anomalies, exceptions, and malware. A daunting challenge is to detect anomalies in rapid voluminous streams of data.

References

[1]

Solaimani M, Khan L, Thuraisingham B. Real-time anomaly detection over VMware performance data using storm. The 15th IEEE International Conference on Information Reuse and Integration, San Francisco, USA, 2014; pp.458-465.

[2]

Solaimani M, Iftekhar M, Khan L, Thuraisingham B, Ingram JB. Spark-based anomaly detection over multi-source VMware performance data in real-time. In the Proceeding of 2014 IEEE Symposium Series on Computational Intelligence: Orlando, Florida, USA, 2014; pp.66-73.

[3]

Mustafa A, Solaimani M, Khan L, Chiang K, Ingram J. Host-based anomalous behavior detection using cluster-level markov networks. Tenth Annual IFIP WG 11.9 International Conference on Digital Forensics, Vienna University of Technology: Vienna, Austria, 2014.

[4]

Mustafa A, Haque A, Khan L, Baron M, Thuraisingham B. Evolving stream classification using change detection. 10th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing, October 22-25, 2014, Miami, Florida, USA, 2014; pp.154-162.

[5]

Yao Y, Sharma A, Golubchik L, Govindan R. Online anomaly detection for sensor systems: a simple and efficient approach. Performance Evaluation. 2010; Volume 67 Issue 11: pp.1059-1075.

Digital Library

[6]

Lee W, Stolfo SJ, Chan PK, Eskin E, Fan W, Miller M, Hershkop S, Zhang J. Real time data mining-based intrusion detection. In Darpa Information Survivability Conference & Exposition II, 2001. Discex'01. Proceedings, Vol.¿1, IEEE, Anaheim, CA, USA, 2001; pp.89-100.

[7]

Abuaitah GR. Anomalies in sensor network deployments: analysis, modeling, and detection. Ph.D. dissertation, Wright State University, OH, USA, 2013.

[8]

Hayes MA, Capretz MAM. Contextual anomaly detection in big sensor data. In Big Data BigData Congress, 2014 IEEE International Congress on: IEEE, 2014; pp.64-71.

Digital Library

[9]

Janeja VP, Azari A, Namayanja JM, Heilig B. B-dids: mining anomalies in a big-distributed intrusion detection system. In Big Data Big Data, 2014 IEEE International Conference on, 2014; pp.32-34.

[10]

Savage D, Zhang X, Yu X, Chou P, Wang Q. Anomaly detection in online social networks. Social Networks. 2014; Volume 39 Issue 0: pp.62-70.

[11]

Apache hadoop. Available at: "http://hadoop.apache.org/" {Accessed 15 October 2013}.

[12]

Dean J, Ghemawat S. Mapreduce: simplified data processing on large clusters. Communications of the ACM. 2008; Volume 51 Issue 1: pp.107-113.

Digital Library

[13]

Apache hbase. Available at: "https://hbase.apache.org/" {Accessed 12 October 2015}.

[14]

Apache mahout. Available at: "https://mahout.apache.org/" {Accessed 1 February 2014}.

[15]

Chang F, Dean J, Ghemawat S, Hsieh W. C, Wallach D. A, Burrows M, Chandra T, Fikes A, Gruber R. E. Bigtable: a distributed storage system for structured data. ACM Transactions on Computer Systems TOCS. 2008; Volume 26 Issue 2: pp.4-4.

Digital Library

[16]

Storm - distributed and fault-tolerant realtime computation. Available at: "http://storm.incubator.apache.org/" {Accessed 8 December 2013}.

[17]

S4. Available at: "http://incubator.apache.org/s4" {Accessed June 2013}.

[18]

Apache Spark. Available at: "http://spark.apache.org/" {Accessed 30 May 2014}.

[19]

Apache Spark. Available at: "http://spark.apache.org/streaming/" {Accessed 30 May 2014}.

[20]

Balasingam B, Sankavaram MS, Choi K, Ayala DFM, Sidoti D, Pattipati K, Willett P, Lintz C, Commeau G, Dorigo F, Fahrny J. Online anomaly detection in big data. Information Fusion Fusion, 2014 17th International Conference, Salamanca, Spain, 2014; pp.1-8.

[21]

Camacho J, Macia-Fernandez G, Diaz-Verdejo J, Garcia-Teodoro P. Tackling the big data 4 VS for anomaly detection. Computer Communications Workshops INFOCOM WKSHPS, 2014 IEEE Conference on, Toronto, Canada, 2014; pp.500-505.

[22]

Apache Kafka. Available at: "http://kafka.apache.org/".

[23]

Message broker. Available at: "http://en.wikipedia.org/wiki/Message_{b}roker" {Accessed 7 December 2015}.

[24]

Publish subscribe pattern. Available at: "http://en.wikipedia.org/wiki/Publish-subscribe_{p}attern" {Accessed 18 December 2015}.

[25]

Charikar M, Chekuri C, Feder T, Motwani R. Incremental clustering and dynamic information retrieval. SIAM Journal on Computing. 2004; Volume 33 Issue 6: pp.1417-1440.

Digital Library

[26]

VMware. Automating the virtual datacenter. Available at: "https://www.vmware.com/files/pdf/avd_{w}p.pdf".

[27]

Apache zookeeper TM. Available at: "http://zookeeper.apache.org/doc/r3.3.4/" {Accessed 28 November 2011}.

[28]

Apache hadoop nextgen mapreduce YARN. Available at: "http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html" {Accessed 29 June 2015}.

[29]

Zaharia M, Chowdhury M, Das T, Dave A, Ma J, McCauley M, Franklin MJ, Shenker S, Stoica I. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. 9th USENIX Conference on Networked Systems Design and Implementation: San Jose, CA, USA, 2012; pp.2-2.

Digital Library

[30]

Zaharia M, Chowdhury M, Das T, Dave A, Ma J, Mccauley M, Franklin M, Shenker S, Stoica I. Fast and interactive analytics over hadoop data with spark. USENIX; ligin 37.4 2012; pp.45-51.

[31]

Zaharia M, Das T, Li H, Shenker S, Stoica I. Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters. 4th USENIX Conference on Hot Topics in Cloud Ccomputing, USENIX Association, Boston, AA, USA, 2012; pp.10-10.

Digital Library

[32]

Why use resource pools ? Available at: "http://pubs.vmware.com/vsphere-4-esx-vcenter/index.jsp?topic=/com.vmware.vsphere.resourcemanagement.doc_{4}0/managing_{r}esource_{p}ools/c_{w}hy_{u}se_{r}esource_{p}ools.html".

[33]

VMware. Using DRS affinity rules. Available at: "http://pubs.vmware.com/vsphere-51/index.jsp".

[34]

Tan Kumar S. Introduction to Data Mining. no. 3, Pearson Education, Limited: NY, USA, 2005.

[35]

VMware. Vsphere ESX and ESXI info center. Available at: "http://www.vmware.com/products/esxi-and-esx/overview".

[36]

Cooper BF, Silberstein A, Tam E, Ramakrishnan R, Sears R. Benchmarking cloud serving systems with YCSB. In Proceedings of the 1st ACM Symposium on Cloud Computing: ACM, Indianapolis, IN, USA, 2010; pp.143-154.

Digital Library

[37]

Kaufman L, Rousseeuw PJ. Finding Groups in Data: An Introduction to Cluster Analysis, <bookSeriesTitle>Wiley series in probability and mathematical statistics. Applied probability and statistics</bookSeriesTitle>. Wiley, 2005.

[38]

Assent I, Kranen P, Baldauf C, Seidl T. Anyout: anytime outlier detection on streaming data. In Database Systems for Advanced Applications Springer, 2012; pp.228-242.

Digital Library

[39]

Frey E. Bashreduce, 2009. Available at: "http://rcrowley.org/2009/06/27/bashreduce" {Accessed 27 June 2009}.

[40]

Yu L, Lan Z. A scalable, non-parametric anomaly detection framework for hadoop. In Proceedings of the 2013 ACM Cloud and Autonomic Computing Conference, ACM: Miamy, FL, USA, 2013; pp.22-22.

Digital Library

[41]

Gupta M, Sharma AB, Chen H, Jiang G. Context-aware time series anomaly detection for complex systems. In WORKSHOP NOTES, Austin, TX, USA, 2013; pp.14-14.

Cited By

Shih WYang CJiang CKristiani E(2023)Implementation and visualization of a netflow log data lake system for cyberattack detection using distributed deep learningThe Journal of Supercomputing10.1007/s11227-022-04802-y79:5(4983-5012)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1007/s11227-022-04802-y
Liu MXue ZXu XZhong CChen J(2018)Host-Based Intrusion Detection System with System CallsACM Computing Surveys10.1145/321430451:5(1-36)Online publication date: 19-Nov-2018
https://dl.acm.org/doi/10.1145/3214304

Online anomaly detection for multi-source VMware using a distributed streaming framework
1. Computing methodologies

Recommendations

Scotty: General and Efficient Open-source Window Aggregation for Stream Processing Systems

Window aggregation is a core operation in data stream processing. Existing aggregation techniques focus on reducing latency, eliminating redundant computations, or minimizing memory usage. However, each technique operates under different assumptions ...
Speeding up the multimedia feature extraction: a comparative study on the big data approach

The current explosion of multimedia data is significantly increasing the amount of potential knowledge. However, to get to the actual information requires to apply novel content-based techniques which in turn require time consuming extraction of ...
Human-machine interactive streaming anomaly detection by online self-adaptive forest
Abstract
Anomaly detectors are used to distinguish differences between normal and abnormal data, which are usually implemented by evaluating and ranking the anomaly scores of each instance. A static unsupervised streaming anomaly detector is difficult to ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Software

Software Volume 46, Issue 11

November 2016

143 pages

ISSN:0038-0644

EISSN:1097-024X

Issue’s Table of Contents

Publisher

John Wiley & Sons, Inc.

United States

Publication History

Published: 01 November 2016

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Shih WYang CJiang CKristiani E(2023)Implementation and visualization of a netflow log data lake system for cyberattack detection using distributed deep learningThe Journal of Supercomputing10.1007/s11227-022-04802-y79:5(4983-5012)Online publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1007/s11227-022-04802-y
Liu MXue ZXu XZhong CChen J(2018)Host-Based Intrusion Detection System with System CallsACM Computing Surveys10.1145/321430451:5(1-36)Online publication date: 19-Nov-2018
https://dl.acm.org/doi/10.1145/3214304

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents