More Web Proxy on the site http://driver.im/

research-article

Anomaly Detection in Internet of Things Based on Logs Using Machine Learning and Deep Learning Techniques

Authors:

Nguyen Huy-Trung,

Nguyen Viet QuocAuthors Info & Claims

CNIOT '23: Proceedings of the 2023 4th International Conference on Computing, Networks and Internet of Things

Pages 969 - 974

https://doi.org/10.1145/3603781.3604223

Published: 27 July 2023 Publication History

Abstract

Engineers (developers or operators) can comprehend the condition of the system and spot odd behaviors like malware attacks and system failures by using log data that records critical events and system status. However, in the 4.0 era, IoT devices are expected to explode in number, and a large amount of data is generated from IoT devices. If something goes wrong, engineers will spend a lot of time manually processing expansive sums of log information. Therefore, it is vital to develop automated methods for for log-based anomaly detection, machine learning, and deep learning applications. However, with IoT device log data, how compelling are straightforward deep learning and machine learning models, and which approach will be more reasonable? This work is for research and evaluation of machine learning and deep learning models with two actual log datasets. The machine learning algorithms like RF, kNN, XGBoost models are trained on two actual log datasets based on the log parsers. Ensemble classifier, XGBoost got the best results with Accuracy, precision, and F1-score best at 99.9%, 99.8%, and 99.9%, respectively. We expect that the discoveries of our think about will be very beneficial for both professionals and analysts seeking after this interesting field.

References

[1]

“41.6 billion IoT devices will be generating 79.4 zettabytes of data in 2025, Truy cp: https://www.helpnetsecurity.com/2019/06/21/connected-iot-devices-forecast/. Ngày truy cp: 10/3/2022.”

[2]

“Singh, S., Sheng, Q. Z., Benkhelifa, E., & Lloret, J. (2020). Guest Editorial: Energy Management, Protocols, and Security for the Next-Generation Networks and Internet of Things. IEEE Trans. Ind. Informatics, 16(5), 3515-3520.”

[3]

“Mi, H., Wang, H., Zhou, Y., Lyu, M. R. T., & Cai, H. (2013). Toward fine-grained, unsupervised, scalable performance diagnosis for production cloud computing systems. IEEE Transactions on Parallel and Distributed Systems, 24(6), 1245-1255.”

[4]

“Zhu, J., He, S., Liu, J., He, P., Xie, Q., Zheng, Z., & Lyu, M. R. (2019, May). Tools and benchmarks for automated log parsing. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (pp. 121-130). IEEE.”

[5]

“Shu, X., Smiy, J., Yao, D. D., & Lin, H. (2013, December). Massive distributed and parallel log analysis for organizational security. In 2013 IEEE Globecom Workshops (GC Wkshps) (pp. 194-199). IEEE.”

[6]

“W. Meng, Y. Liu, Y. Zhu, S. Zhang, D. Pei, Y. Liu, Y. Chen, R. Zhang, S. Tao, P. Sun, and R. Zhou, ‘Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs,’ in IJCAI. ijcai.org, 2019, pp. 4739–4745.”

[7]

“Zhang, X., Xu, Y., Lin, Q., Qiao, B., Zhang, H., Dang, Y., ... & Zhang, D. (2019, August). Robust log-based anomaly detection on unstable log data. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (pp. 807-817).”

[8]

“Q. Lin, H. Zhang, J. Lou, Y. Zhang, and X. Chen, ‘Log clustering based problem identification for online service systems,’ in ICSE (Companion Volume). ACM, 2016, pp. 102–111.”

[9]

“W. Meng, Y. Liu, Y. Zhu, S. Zhang, D. Pei, Y. Liu, Y. Chen, R. Zhang, S. Tao, P. Sun, and R. Zhou, ‘Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs,’ in IJCAI. ijcai.org, 2019, pp. 4739–4745.”

[10]

“Han, S., Wu, Q., & Yang, Y. (2022). Machine learning for Internet of things anomaly detection under low-quality data. International Journal of Distributed Sensor Networks, 18(10), 15501329221133765.”

[11]

“Granlund, O. (2019). Unsupervised anomaly detection on log-based time series data (Dissertation). Retrieved from http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-265534.”

[12]

“Yadav, R. B., Kumar, P. S., & Dhavale, S. V. (2020, June). A survey on log anomaly detection using deep learning. In 2020 8th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions)(ICRITO) (pp. 1215-1220). IEEE.”

[13]

“Chalapathy, R., & Chawla, S. (2019). Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407.”

[14]

“Du, M., Li, F., Zheng, G., & Srikumar, V. (2017, October). Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security (pp. 1285-1298).”

[15]

“Meng, W., Liu, Y., Zhu, Y., Zhang, S., Pei, D., Liu, Y., ... & Zhou, R. (2019, August). LogAnomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs. In IJCAI (Vol. 19, No. 7, pp. 4739-4745).”

[16]

“Le, V. H., & Zhang, H. (2021, November). Log-based anomaly detection without log parsing. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) (pp. 492-504). IEEE.”

[17]

Protogerou, A., Papadopoulos, S., Drosou, A., Tzovaras, D., & Refanidis, I. (2021). A graph neural network method for distributed anomaly detection in IoT. Evolving Systems, 12, 19-36.”

[18]

“Wang, J., Tang, Y., He, S., Zhao, C., Sharma, P. K., Alfarraj, O., & Tolba, A. (2020). LogEvent2vec: LogEvent-to-vector based anomaly detection for large-scale logs in internet of things. Sensors, 20(9), 2451.”

[19]

“Xin, Y., Kong, L., Liu, Z., Chen, Y., Li, Y., Zhu, H., ... & Wang, C. (2018). Machine learning and deep learning methods for cybersecurity. Ieee access, 6, 35365-35381.”

[20]

“A. Oliner and J. Stearley, ‘What supercomputers say: A study of five system logs,’ in 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN’07). IEEE, 2007, pp. 575– 584.”

[21]

“He, P., Zhu, J., Zheng, Z., & Lyu, M. R. (2017, June). Drain: An online log parsing approach with fixed depth tree. In 2017 IEEE international conference on web services (ICWS) (pp. 33-40). IEEE.”

Cited By

Nguyen HLe HMinh BVan Huan PAnh Tan N(2024)Towards an Approach for Representing Log Event in Anomaly Detection2024 International Conference on Expert Clouds and Applications (ICOECA)10.1109/ICOECA62351.2024.00030(96-101)Online publication date: 18-Apr-2024
https://doi.org/10.1109/ICOECA62351.2024.00030
Lupton SWashizaki HYoshioka NFukazawa Y(2024)Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection MethodsIEEE Access10.1109/ACCESS.2024.338728712(78193-78218)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3387287

Index Terms

Anomaly Detection in Internet of Things Based on Logs Using Machine Learning and Deep Learning Techniques

Index terms have been assigned to the content through auto-classification.

Recommendations

Cloud-based multiclass anomaly detection and categorization using ensemble learning
Abstract
The world of the Internet and networking is exposed to many cyber-attacks and threats. Over the years, machine learning models have progressed to be integrated into many scenarios to detect anomalies accurately. This paper proposes a novel ...
Beginning Anomaly Detection Using Python-Based Deep Learning: With Keras and PyTorch
Federated deep learning for anomaly detection in the internet of things
Abstract
Privacy has emerged as a top worry as a result of the development of zero-day hacks because IoT devices produce and transmit sensitive information through the regular internet. This study suggests a deep neural network (DNN) and federated ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

CNIOT '23: Proceedings of the 2023 4th International Conference on Computing, Networks and Internet of Things

May 2023

1025 pages

ISBN:9798400700705

DOI:10.1145/3603781

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CNIOT'23

CNIOT'23: 2023 4th International Conference on Computing, Networks and Internet of Things

May 26 - 28, 2023

Xiamen, China

Acceptance Rates

Overall Acceptance Rate 39 of 82 submissions, 48%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
96
Total Downloads

Downloads (Last 12 months)75
Downloads (Last 6 weeks)10

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Nguyen HLe HMinh BVan Huan PAnh Tan N(2024)Towards an Approach for Representing Log Event in Anomaly Detection2024 International Conference on Expert Clouds and Applications (ICOECA)10.1109/ICOECA62351.2024.00030(96-101)Online publication date: 18-Apr-2024
https://doi.org/10.1109/ICOECA62351.2024.00030
Lupton SWashizaki HYoshioka NFukazawa Y(2024)Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection MethodsIEEE Access10.1109/ACCESS.2024.338728712(78193-78218)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3387287

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents