More Web Proxy on the site http://driver.im/

article

Hunting attacks in the dark: clustering and correlation analysis for unsupervised anomaly detection

Authors:

Romain Fontugne,

Kensuke Fukuda,

Philippe OwezarskiAuthors Info & Claims

Networks, Volume 25, Issue 5

Pages 283 - 305

https://doi.org/10.1002/nem.1903

Published: 01 September 2015 Publication History

Abstract

Network anomalies and attacks represent a serious challenge to ISPs, who need to cope with an increasing number of unknown events that put their networks' integrity at risk. Most of the network anomaly detection systems proposed so far employ a supervised strategy to accomplish their task, using either signature-based detection methods or supervised-learning techniques. The former fails to detect unknown anomalies, exposing the network to severe consequences; the latter requires labeled traffic, which is difficult and expensive to produce. In this paper, we introduce a powerful unsupervised approach to detect and characterize network anomalies in the dark, that is, without relying on signatures or labeled traffic. Unsupervised detection is accomplished by means of robust clustering techniques, combining subspace clustering with correlation analysis to blindly identify anomalies. To alleviate network operator's post-processing tasks and to speed up the deployment of effective countermeasures, anomaly ranking and characterization are automatically performed on the detected events. The system is extensively tested with real traffic from the Widely Integrated Distributed Environment backbone network, spanning 6years of flows captured from a trans-Pacific link between Japan and the USA, using the MAWILab framework for ground-truth generation. We additionally evaluate the proposed approach with synthetic data, consisting of traffic from an operational network with synthetic attacks. Finally, we compare the performance of the unsupervised detection against different previously used unsupervised detection techniques, as well as against multiple anomaly detectors used in MAWILab. Copyright © 2015 John Wiley & Sons, Ltd.

References

[1]

Casas P, Mazel J, Owezarski P. Steps towards autonomous network security: unsupervised detection of network attacks. In Proceedings of the 4th IFIP International Conference on New Technologies, Mobility and Security,Paris, France, 2011; pp.1-5.

[2]

Mazel J, Casas P, Labit Y, Owezarski P. Sub-space clustering, inter-clustering results association & anomaly correlation for unsupervised network anomaly detection. In Proceedings of the 7th International Conference on Network and Service Management,Paris, France, 2011; pp.1-8.

[3]

Cho K, Mitsuya K, Kato A. Traffic data repository at the wide project. In Proceedings of the Annual Conference on USENIX Annual Technical Conference,San Diego, CA, USA, 2000; pp.51-56.

Digital Library

[4]

Fontugne R, Borgnat P, Abry P, Fukuda K. MAWILab: combining diverse anomaly detectors for automated anomaly labeling and performance benchmarking. In Proceedings of the 10th International Conference on Emerging Networking Experiments and Technologies CoNEXT,Philadelphia, PA, USA, 2010; pp.8:1-8:12.

[5]

Fontugne R, Fukuda K. A Hough-transform-based anomaly detector with an adaptive time interval. ACM SIGAPP Applied Computing Review. 2011; Volume 11 Issue 3: pp.41-51.

[6]

Dewaele G, Fukuda K, Borgnat P, Abry P, Cho K. Extracting hidden anomalies using sketch and non Gaussian multiresolution statistical detection procedures. In Proceedings of the 2007 Workshop on Large Scale Attack Defense LSAD,Kyoto, Japan, 2007; pp.145-152.

Digital Library

[7]

Kanda Y, Fontugne R, Fukuda K, Sugawara T. ADMIRE: Anomaly detection method using entropy-based PCA with three-step sketches. Computer Communications. 2013; Volume 36 Issue 5: pp.575-588.

[8]

Brauckhoff D, Dimitropoulos X, Wagner A, Salamatian K. Anomaly extraction in backbone networks using association rules. IEEE/ACM Transactions on Networking. 2012; Volume 20 Issue 6: pp.1788-1799.

[9]

Portnoy L, Eskin E, Stolfo S. Intrusion detection with unlabeled data using clustering. In Proceedings of ACM CSS Workshop on Data Mining Applied to Security,Philadelphia, PA, USA, 2001.

[10]

Eskin E, Arnold A, Prerau M, Portnoy L, Stolfo S. A geometric framework for unsupervised anomaly detection: detecting intrusions in unlabeled data. Applications of Data Mining in Computer Security. 2002; Volume 6: pp.77-101.

[11]

Leung K, Leckie C. Unsupervised anomaly detection in network intrusion detection using clusters. In Proceedings of the Twenty-Eighth Australasian Conference on Computer Science,Darlinghurst, Australia, 2005; pp.333-342.

Digital Library

[12]

Lakhina A, Crovella M, Diot C. Diagnosing network-wide traffic anomalies. In Proceedings of the 4th Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications SIGCOMM,Portland, Oregon, USA, 2004; pp.219-230.

Digital Library

[13]

Lakhina A, Crovella M, Diot C. Mining anomalies using traffic feature distributions. In Proceedings of the 2005 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications SIGCOMM,Philadelphia, PA, USA, 2005; pp.217-228.

[14]

METROlogy for SECurity and QOS, 2007. Available from: "http://projects.laas.fr/METROSEC/" {26 August 2014}.

[15]

Barford P, Kline J, Plonka D, Ron A. A signal analysis of network traffic anomalies. In Proceedings of the 2nd ACM SIGCOMM Workshop on Internet Measurement IMC,Marseille, France, 2002; pp.71-82.

Digital Library

[16]

Brutlag JD. Aberrant behavior detection in time series for network monitoring. In Proceedings of the 14th USENIX Conference on System Administration,New Orleans, Louisiana, USA, 2000; pp.139-146.

Digital Library

[17]

Fontugne R, Abry P, Fukuda K, Borgnat P, Mazel J, Wendt H, Veitch D. Random projection and multiscale wavelet leader based anomaly detection and address identification in Internet traffic. In Proceedings of the 40th IEEE International Conference on Acoustics, Speech, and Signal Processing ICASSP,Brisbane, Australia, 2015.

[18]

Soule A, Salamatian K, Taft N. Combining filtering and statistical methods for anomaly detection. In Proceedings of the 5th ACM SIGCOMM Conference on Internet Measurement IMC,Berkeley, CA, USA, 2005; pp.331-344.

[19]

Krishnamurthy B, Sen S, Zhang Y, Chen Y. Sketch-based change detection: methods, evaluation, and applications. In Proceedings of the 3rd ACM SIGCOMM Conference on Internet Measurement IMC,Miami Beach, FL, USA, 2003; pp.234-247.

Digital Library

[20]

Silveira F, Diot C, Taft N, Govindan R. ASTUTE: detecting a different class of traffic anomalies. In Proceedings of the ACM SIGCOMM 2010 Conference,New Delhi, India, 2010; pp.267-278.

[21]

Chandola V, Banerjee A, Kumar V. Anomaly detection: a survey. ACM Computing Surveys. 2009; Volume 41 Issue 3: pp.15:1-15:58.

[22]

Marnerides A, Schaeffer-Filho A, Mauthe A. Traffic anomaly diagnosis in internet backbone networks: a survey. Computer Networks. 2014; Volume 73 Issue 0: pp.224-243.

[23]

Xu K, Zhang ZL, Bhattacharyya S. Internet traffic behavior profiling for network security monitoring. IEEE/ACM Transactions on Networking. 2008; Volume 16 Issue 6: pp.1241-1252.

[24]

Fernandes G, Owezarski P. Automated classification of network traffic anomalies. In Proceedings of 5th Conference on Security and Privacy in Communication Networks SecurComm,Athens, Greece, 2009; pp.91-100.

[25]

Silveira F, Diot C. URCA: Pulling out anomalies by their root causes. In Proceedings of the 29th Conference on Information Communications INFOCOM,San Diego, CA, USA, 2010; pp.1-9.

[26]

Tellenbach B, Burkhart M, Schatzmann D, Gugelmann D, Sornette D. Accurate network anomaly classification with generalized entropy metrics. Computer Networks. 2011; Volume 55 Issue 15: pp.3485-3502.

[27]

Brownlee N. One-way traffic monitoring with iatmon. In Proceedings of the 13th International Conference on Passive and Active Measurement,Vienna, Austria, 2012; pp.179-188.

Digital Library

[28]

Glatz E, Dimitropoulos X. Classifying Internet one-way traffic. In Proceedings of the 12th ACM Conference on Internet Measurement Conference,Boston, Massachusetts, USA, 2012; pp.37-50.

[29]

Fiadino P, D'Alconzo A, Bar A, Finamore A, Casas P. On the detection of network traffic anomalies in content delivery network services. In 2014 26th International Teletraffic Congress ITC,Karlskrona, Sweden, 2014; pp.1-9.

[30]

Coluccia A, D'alconzo A, Ricciato F. Distribution-based anomaly detection via generalized likelihood ratio test: A general maximum entropy approach. Computer Networks. 2013; Volume 57 Issue 17: pp.3446-3462.

[31]

Gamer T. Collaborative anomaly-based detection of large-scale internet attacks. Computer Networks. 2012; Volume 56 Issue 1: pp.169-185.

[32]

Zhang Y. 2013. An adaptive flow counting method for anomaly detection in SDN. In Proceedings of the Ninth ACM Conference on Emerging Networking Experiments and Technologies, <bookSeriesTitle>CoNEXT '13</bookSeriesTitle>ACM: New York, NY, USA; pp.25-30.

[33]

Mirkovic J, Reiher P. A taxonomy of DDoS attack and DDoS defense mechanisms. ACM SIGCOMM Computer Communication Review. 2004; Volume 34 Issue 2: pp.39-53.

[34]

Barnett RJ, Irwin B. Towards a taxonomy of network scanning techniques. In Proceedings of the 2008 South African Institute of Computer Scientists and Information Technologists on IT SAICSIT,Wilderness, South Africa, 2008; pp.1-7.

[35]

Plonka D, Barford P. Network anomaly confirmation, diagnosis and remediation. In Proceedings of the 47th Annual Allerton Conference on Communication, Control, and Computing CCC,Monticello, Illinois, USA, 2009; pp.128-135.

Digital Library

[36]

Borgnat P, Dewaele G, Fukuda K, Abry P, Cho K. Seven years and one day: sketching the evolution of Internet traffic. In Proceedings of the 28th Conference on Information Communications INFOCOM,Rio de Janeiro, Brazil, 2009; pp.711-719.

[37]

Allman M, Paxson V, Terrell J. A brief history of scanning. In Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement IMC,San Diego, CA, USA, 2007; pp.77-82.

Digital Library

[38]

Shon T, Moon J. A hybrid machine learning approach to network anomaly detection. Information Sciences. 2007; Volume 177: pp.3799-3821.

[39]

Duffield N, Haffner P, Krishnamurthy B, Ringberg H. Rule-based anomaly detection on IP flows. In Proceedings of the 28th Conference on Information Communications INFOCOM,Rio de Janeiro, Brazil, 2009; pp.424-432.

[40]

Bhuyan MH, Bhattacharyya DK, Kalita JK. Towards an unsupervised method for network anomaly detection in large datasets. Computing and Informatics. 2014; Volume 33 Issue 1: pp.1-34.

[41]

Novakov S, Lung CH, Lambadaris I, Seddigh N. A hybrid technique using PCA and wavelets in network traffic anomaly detection. International Journal of Mobile Computing and Multimedia Communications. 2014; Volume 6 Issue 1: pp.17-53.

[42]

Parsons L, Haque E, Liu H. Subspace clustering for high dimensional data: a review. SIGKDD Exploration Newsletter. 2004; Volume 6: pp.90-105.

[43]

Fred ALN, Jain AK. Combining multiple clusterings using evidence accumulation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2005; Volume 27: pp.835-850.

[44]

Cormode G, Muthukrishnan S. What's new: finding significant differences in network data streams. IEEE/ACM Transactions on Networking. 2005; Volume 13: pp.1219-1232.

[45]

Jain AK. Data clustering: 50 years beyond k-means. Pattern Recognition Letters. 2010; Volume 31: pp.651-666.

[46]

Strehl A, Ghosh J. Cluster ensembles-a knowledge reuse framework for combining multiple partitions. Journal of Machine Learning Research. 2003; Volume 3: pp.583-617.

[47]

Ester M, Kriegel HP, S J, Xu X. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining KDD,Portland, OR, USA, 1996; pp.226-231.

[48]

Agrawal R, Gehrke J, Gunopulos D, Raghavan P. Automatic subspace clustering of high dimensional data for data mining applications. SIGMOD Record. 1998; Volume 27 Issue 2: pp.94-105.

[49]

Dean J, Ghemawat S. MapReduce: simplified data processing on large clusters. Communications of the ACM. 2008; Volume 51 Issue 1: pp.107-113.

[50]

Müller E, Günnemann S, Assent I, Seidl T. Evaluating clustering in subspace projections of high dimensional data. Proceedings of VLDB Endowment. 2009; Volume 2: pp.1270-1281.

[51]

Moise G, Zimek A, Kröger P, Kriegel HP, Sander J. Subspace and projected clustering: experimental evaluation and analysis. Knowledge and Information Systems. 2009; Volume 21: pp.299-326.

Cited By

Qureshi MYan JCheng YYeganeh SSeung YCardwell NDe Bruijn WJacobson VKaur JWetherall DVahdat ASchulzrinne HKohler EMaltz DMisra V(2023)Fathom: Understanding Datacenter Application Network PerformanceProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604815(394-405)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604815
Shi YShen H(2022)Unsupervised anomaly detection for network traffic using artificial immune networkNeural Computing and Applications10.1007/s00521-022-07156-x34:15(13007-13027)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.1007/s00521-022-07156-x
Putina ARossi D(2021)Online Anomaly Detection Leveraging Stream-Based Clustering and Real-Time TelemetryIEEE Transactions on Network and Service Management10.1109/TNSM.2020.303701918:1(839-854)Online publication date: 1-Mar-2021
https://dl.acm.org/doi/10.1109/TNSM.2020.3037019
Show More Cited By

Hunting attacks in the dark: clustering and correlation analysis for unsupervised anomaly detection
1. Networks
  1. Network services

Recommendations

Hunting attacks in the dark: clustering and correlation analysis for unsupervised anomaly detection

Network anomalies and attacks represent a serious challenge to ISPs, who need to cope with an increasing number of unknown events that put their networks' integrity at risk. Most of the network anomaly detection systems proposed so far employ a ...
Hunting attacks in the dark: clustering and correlation analysis for unsupervised anomaly detection

Network anomalies and attacks represent a serious challenge to ISPs, who need to cope with an increasing number of unknown events that put their networks' integrity at risk. Most of the network anomaly detection systems proposed so far employ a ...
Hunting attacks in the dark: clustering and correlation analysis for unsupervised anomaly detection

Network anomalies and attacks represent a serious challenge to ISPs, who need to cope with an increasing number of unknown events that put their networks' integrity at risk. Most of the network anomaly detection systems proposed so far employ a ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Networks

Networks Volume 25, Issue 5

September 2015

100 pages

ISSN:0028-3045

EISSN:1097-0037

Issue’s Table of Contents

Publisher

Wiley-Interscience

United States

Publication History

Published: 01 September 2015

Author Tag

unsupervised anomaly detection & characterization, clustering, outliers detection, anomaly correlation, filtering rules, MAWILab

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Qureshi MYan JCheng YYeganeh SSeung YCardwell NDe Bruijn WJacobson VKaur JWetherall DVahdat ASchulzrinne HKohler EMaltz DMisra V(2023)Fathom: Understanding Datacenter Application Network PerformanceProceedings of the ACM SIGCOMM 2023 Conference10.1145/3603269.3604815(394-405)Online publication date: 10-Sep-2023
https://dl.acm.org/doi/10.1145/3603269.3604815
Shi YShen H(2022)Unsupervised anomaly detection for network traffic using artificial immune networkNeural Computing and Applications10.1007/s00521-022-07156-x34:15(13007-13027)Online publication date: 1-Aug-2022
https://dl.acm.org/doi/10.1007/s00521-022-07156-x
Putina ARossi D(2021)Online Anomaly Detection Leveraging Stream-Based Clustering and Real-Time TelemetryIEEE Transactions on Network and Service Management10.1109/TNSM.2020.303701918:1(839-854)Online publication date: 1-Mar-2021
https://dl.acm.org/doi/10.1109/TNSM.2020.3037019
Diab DAsSadhan BBinsalleeh HLambotharan SKyriakopoulos KGhafir I(2021)Denial of service detection using dynamic time warpingInternational Journal of Network Management10.1002/nem.215931:6Online publication date: 2-Nov-2021
https://dl.acm.org/doi/10.1002/nem.2159
AsSadhan BAlShaalan RDiab DAlzoghaiby AAlshebeili SAl-Muhtadi JBin-Abbas HEl-Samie F(2020)A robust anomaly detection method using a constant false alarm rate approachMultimedia Tools and Applications10.1007/s11042-020-08653-879:17-18(12727-12750)Online publication date: 1-May-2020
https://dl.acm.org/doi/10.1007/s11042-020-08653-8
AsSadhan BAlzoghaiby ABinsalleeh HKyriakopoulos KLambotharan S(2020)Network anomaly detection using a cross‐correlation‐based long‐range dependence analysisInternational Journal of Network Management10.1002/nem.212930:6Online publication date: 4-Nov-2020
https://dl.acm.org/doi/10.1002/nem.2129

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents