LogExpert: Log-based Recommended Resolutions Generation using Large Language Model
Authors: 
Jiabo Wang, Guojun Chu, Jingyu Wang, Haifeng Sun, Qi Qi, Yuanyi Wang, Ji Qi, Jianxin LiaoAuthors Info & Claims
Pages 42 - 46
Abstract
Software logs play a vital role in ensuring the reliability and availability of large-scale software systems. In recent years, researchers have made significant efforts to build log analysis approaches to manage software systems. However, these approaches focus on log compression, log parsing and log anomaly detection. In the current context, engineers continue to spend substantial time and effort on resolving errors once anomalous logs have been detected. To achieve truly automated software system management and high-level Artificial Intelligence for IT Operations (AIOps), it's necessary to bridge the gap between anomalous logs and their resolutions.
In this paper, we propose a novel framework LogExpert to automatically generate recommended resolutions for anomalous logs. Specifically, we build a log recognizer to utilize the wealth of software knowledge in technical forums such as Stack Overflow (SO). In addition, LogExpert combines the great power of a Large Language Model (LLM) with domain-specific knowledge to generate the resolution. We conducted a preliminary evaluation of our framework on datasets from SO. Our log recognizer achieves the F1 score of 0.936. Our lexical metrics and human evaluation show the overall LogExpert framework achieves excellent performance in log-based resolution generation.
References
[1]
Toufique Ahmed, Supriyo Ghosh, Chetan Bansal, Thomas Zimmermann, Xuchao Zhang, and Saravan Rajmohan. 2023. Recommending Root-Cause and Mitigation Steps for Cloud Incidents Using Large Language Models. In Proceedings of the 45th International Conference on Software Engineering (Melbourne, Victoria, Australia) (ICSE '23). IEEE Press, 1737--1749.
[2]
Yinfang Chen, Huaibing Xie, Minghua Ma, Yu Kang, Xin Gao, Liu Shi, Yunjie Cao, Xuedong Gao, Hao Fan, Ming Wen, et al. 2023. Empowering Practical Root Cause Analysis by Large Language Models for Cloud Incidents. arXiv preprint arXiv:2305.15778 (2023).
[3]
Guojun Chu, Jingyu Wang, Qi Qi, Haifeng Sun, Shimin Tao, and Jianxin Liao. 2021. Prefix-Graph: A Versatile Log Parsing Approach Merging Prefix Tree with Probabilistic Graph. In 2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE, 2411--2422.
[4]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171--4186.
[5]
Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. 2024. Large Language Models are Few-Shot Summarizers: Multi-Intent Comment Generation via In-Context Learning. (2024).
[6]
Yintong Huo, Yuxin Su, Cheryl Lee, and Michael R Lyu. 2023. SemParser: A Semantic Parser for Log Analytics. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 881--893.
[7]
Jiajun Jiang, Weihai Lu, Junjie Chen, Qingwei Lin, Pu Zhao, Yu Kang, Hongyu Zhang, Yingfei Xiong, Feng Gao, Zhangwei Xu, et al. 2020. How to mitigate the incident? an effective troubleshooting guide recommendation technique for online service systems. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1410--1420.
[8]
Bonan Kou, Muhao Chen, and Tianyi Zhang. 2023. Automated Summarization of Stack Overflow Posts. arXiv preprint arXiv:2305.16680 (2023).
[9]
Yukyung Lee, Jina Kim, and Pilsung Kang. 2023. Lanobert: System log anomaly detection based on bert masked language model. Applied Soft Computing 146 (2023), 110689.
[10]
Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74--81.
[11]
Jinyang Liu, Jieming Zhu, Shilin He, Pinjia He, Zibin Zheng, and Michael R Lyu. 2019. Logzip: Extracting hidden structures via iterative clustering for log compression. In 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 863--873.
[12]
Yudong Liu, Xu Zhang, Shilin He, Hongyu Zhang, Liqun Li, Yu Kang, Yong Xu, Minghua Ma, Qingwei Lin, Yingnong Dang, et al. 2022. Uniparser: A unified log parser for heterogeneous log data. In Proceedings of the ACM Web Conference 2022. 1893--1901.
[13]
Ruchi Mahindru, Harshit Kumar, and Sahil Bansal. 2021. Log anomaly to resolution: AI based proactive incident remediation. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 1353--1357.
[14]
Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, et al. 2019. Loganomaly: Unsupervised detection of sequential and quantitative anomalies in unstructured logs. In IJCAI, Vol. 19. 4739--4745.
[15]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311--318.
[16]
Antonio Pecchia, Marcello Cinque, Gabriella Carrozza, and Domenico Cotroneo. 2015. Industry practices and event logging: Assessment of a critical software development process. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. IEEE, 169--178.
[17]
Aidi Pi, Wei Chen, Shaoqi Wang, and Xiaobo Zhou. 2019. Semantic-aware workflow construction and analysis for distributed data analytics systems. In Proceedings of the 28th International Symposium on High-Performance Parallel and Distributed Computing. 255--266.
[18]
Jeniya Tabassum, Mounica Maddela, Wei Xu, and Alan Ritter. 2020. Code and Named Entity Recognition in StackOverflow. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 4913--4926.
[19]
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023).
Index Terms
- LogExpert: Log-based Recommended Resolutions Generation using Large Language Model
Recommendations
GenGLAD: A Generated Graph Based Log Anomaly Detection Framework
Smart Computing and CommunicationAbstractInformation systems record the current states and the access records in logs, so logs become the data basis for detecting anomalies of system security. To realize log anomaly detection, frameworks based on text, sequence, and graph are applied. ...
HEART: Heterogeneous Log Anomaly Detection Using Robust Transformers
Discovery ScienceAbstractLog sequences generated by heterogeneous systems are critical for understanding computer system behaviour and ensuring operational and security integrity. However, the diverse formats, structures, and content of logs pose challenges for ...
Log Attention – Assessing Software Releases with Attention-Based Log Anomaly Detection
Service-Oriented Computing – ICSOC 2021 WorkshopsAbstractA Software Engineering Manager (EM) has to cater to the demand for higher reliability and resilience in Production while simultaneously addressing the evolution of software architecture from monolithic applications to multi-cloud distributed ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
April 2024
127 pages
ISBN:9798400705007
DOI:10.1145/3639476
- Co-chairs:
- Ana Paiva,
- Rui Abreu,
- Robert Hierons,
- Henrique Madeira,
- Program Co-chairs:
- Abhik Roychoudhury,
- Margaret Storey
Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
In-Cooperation
- Faculty of Engineering of University of Porto
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
ICSE-NIER'24
Sponsor:
ICSE-NIER'24: 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results
April 14 - 20, 2024
Lisbon, Portugal
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 293Total Downloads
- Downloads (Last 12 months)293
- Downloads (Last 6 weeks)44
Reflects downloads up to 09 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in