Abstract
Unsupervised anomaly detection can detect attacks without the need for clean or labeled training data. This paper studies the application of clustering to unsupervised anomaly detection (ACUAD). Data records are mapped to a feature space. Anomalies are detected by determining which points lie in the sparse regions of the feature space. A critical element for this method to be effective is the definition of the distance function between data records. We propose a unified normalization distance framework for records with numeric and nominal features mixed data. A heuristic method that computes the distance for nominal features is proposed, taking advantage of an important characteristic of nominal features—their probability distribution. Then, robust methods are proposed for mapping numeric features and computing their distance, these being able to tolerate the impact of the value difference in scale and diversification among features, and outliers introduced by intrusions. Empirical experiments with the KDD 1999 dataset showed that ACUAD can detect intrusions with relatively low false alarm rates compared with other approaches.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Cansado, A., Soto, A., 2008. Unsupervised anomaly detection in large databases using Bayesian networks. Appl. Artif. Intell., 22(4):309–330. [doi:10.1080/08839510801972801]
Eskin, E., 2000. Anomaly Detection over Noisy Data Using Learned Probability Distributions. Proc. Int. Conf. on Machine Learning, p.255–262. [doi:10.1109/ICCSA.2008.70]
Eskin, E., Arnold, A., Prerau, M., Portony, L., Stolfo, S., 2002. A Geometric Framework for Unsupervised Anomaly Detection: Detecting Intrusions in Unlabeled Data. In: Barbara, E., Jajodia, S. (Eds.), Applications of Data Mining in Computer Security. Kluwer Academic Publishers, Norwell, MA, USA, p.272.
Ismail, A.S.H., Abdullah, A.H., Bak, K.B.A., Nqudi, M.A., Dahlan, D., Chimphlee, W., 2008. A Novel Method for Unsupervised Anomaly Detection Using Unlabelled Data. Proc. Int. Conf. on Computational Sciences and Its Applications., p.252–260. [doi:10.1109/ICCSA.2008.70]
Knorr, E.M., 2002. Outliers and Data Mining: Finding Exceptions in Data. PhD Thesis, University of British Columbia, Canada, p.74.
Kwitt, R., Hofmann, U., 2007. Unsupervised Anomaly Detection in Network Traffic by Means of Robust PCA. Proc. Int. Multi-Conf. on Computing in the Global Information Technology, p.37–41. [doi:10.1109/ICCGI.2007.62]
Leung, K., Leckie, C., 2005. Unsupervised Anomaly Detection in Network Intrusion Detection Using Clusters. Proc. 28th Australasian Conf. on Computer Science, 102:333–342.
Author information
Authors and Affiliations
Corresponding author
Additional information
Project supported by the PhD Foundation of Engineering and Commerce College, South-Central University for Nationalities, China
Rights and permissions
About this article
Cite this article
Cai, Lz., Chen, J., Ke, Y. et al. A new data normalization method for unsupervised anomaly intrusion detection. J. Zhejiang Univ. - Sci. C 11, 778–784 (2010). https://doi.org/10.1631/jzus.C0910625
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/jzus.C0910625