Abstract
Detecting icons in Graphical User Interfaces (GUIs) is essential for effective application automation. This study examines the impact of different annotation methods on the performance of object detection models for icon detection in GUIs. We compared manual, automated, and hybrid annotations using three models: Faster R-CNN, YOLOv8, and YOLOv9. The results show that manual annotations achieve the highest accuracy, with YOLOv9 reaching an Average Precision (AP) of 68.23% and Faster R-CNN achieving 61.82%. Hybrid methods that combine automated annotations with manual corrections also show significant improvements, though they do not perform as well as manual annotations alone. These findings underscore the importance of high-quality, consistent annotations for training effective detection models. While we used HTML code for automated annotations to simplify the process, we encountered inconsistencies that affected model performance. This highlights the need to develop better hybrid methods tailored to specific tasks, ensuring efficiency and accuracy in data annotation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Altinbas, M.D., Serif, T.: GUI element detection from mobile UI images using YOLov5. In: Awan, I., Younas, M., Poniszewska-Marańda, A. (eds.) MobiWIS 2022. LNCS, vol. 13475, pp. 32–45. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-14391-5_3
Bisong, E., Bisong, E.: Google colaboratory. Building machine learning and deep learning models on google cloud platform: a comprehensive guide for beginners, pp. 59–64 (2019)
Chen, J., et al.: Object detection for graphical user interface: old fashioned or deep learning or a combination? In: Proceedings of the 28th ACM joint meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1202–1214 (2020)
Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vision 88, 303–338 (2010)
Fröhlich, P., Baldauf, M., Meneweger, T., Tscheligi, M., de Ruyter, B., Paternó, F.: Everyday automation experience: a research agenda. Pers. Ubiquit. Comput. 24, 725–734 (2020)
Gu, Z., Xu, Z., Chen, H., Lan, J., Meng, C., Wang, W.: Mobile user interface element detection via adaptively prompt tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11155–11164 (2023)
Jocher, G., Chaurasia, A., Qiu, J.: YOLO by Ultralytics (2023). https://github.com/ultralytics/ultralytics. Accessed 10 June 2024
Padilla, R., Netto, S.L., Da Silva, E.A.: A survey on performance metrics for object-detection algorithms. In: 2020 International Conference on Systems, Signals and Image Processing (IWSSIP), pp. 237–242. IEEE (2020)
Passini, S., Strazzari, F., Borghi, A.: Icon-function relationship in toolbar icons. Displays 29(5), 521–525 (2008)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 28 (2015)
Selcuk, B., Serif, T.: A comparison of YOLOv5 and YOLOv8 in the context of mobile UI detection. In: Younas, M., Awan, I., Grønli, T.M. (eds.) MobiWIS 2023. LNCS, vol. 13977, pp. 161–174. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-39764-6_11
Tkachenko, M., Malyuk, M., Holmanyuk, A., Liubimov, N.: Label Studio: Data labeling software (2020-2022). https://github.com/heartexlabs/label-studio. Open source software https://github.com/heartexlabs/label-studio
Wang, C.Y., Yeh, I.H., Liao, H.Y.M.: YOLOv9: learning what you want to learn using programmable gradient information. arXiv preprint arXiv:2402.13616 (2024)
Xiao, S., et al.: UI semantic component group detection: grouping UI elements with similar semantics in mobile graphical user interface. Displays 83, 102679 (2024)
Xie, M., Feng, S., Xing, Z., Chen, J., Chen, C.: UIED: a hybrid tool for GUI element detection. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1655–1659 (2020)
Acknowledgements
This research has been funded by the Spanish Ministry of Economics and Industry –grant PID2020-112726RB-I00–, the Spanish Research Agency –grant PID2023-146257OB-I00–, and Missions Science and Innovation project MIG-20211008 (INMERBOT). Also, by Principado de Asturias, grant SV-PA-21-AYUD/2021/50994, and by the Council of Gijón through the University Institute of Industrial Technology of Asturias grants SV-21-GIJON-1-19, SV-22-GIJON-1-19, SV-22-GIJON-1-22, SV-23-GIJON-1-09, and SV-23-GIJON-1-17. Finally, this research has also been funded by Fundación Universidad de Oviedo grants FUO-23-008 and FUO-22-450.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Dicu, M., González, E.G., Chira, C., Villar, J.R. (2025). The Impact of Data Annotations on the Performance of Object Detection Models in Icon Detection for GUI Images. In: Quintián, H., et al. Hybrid Artificial Intelligent Systems. HAIS 2024. Lecture Notes in Computer Science(), vol 14857. Springer, Cham. https://doi.org/10.1007/978-3-031-74183-8_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-74183-8_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-74182-1
Online ISBN: 978-3-031-74183-8
eBook Packages: Computer ScienceComputer Science (R0)