Abstract
Deep Learning techniques have become the mainstream and unquestioned standard in many fields, e.g. convolutional neural networks (CNN) for image analysis and recognition tasks. As testing and validation of graphical user interfaces (GUIs) is increasingly relying on computer vision, CNN models that predict such subjective and informal dimensions of user experience as aesthetic or complexity perception start to achieve decent accuracy. They however require huge amounts of human-labeled training data, which are costly or unavailable in the field of Human-Computer Interaction (HCI). More traditional approaches rely on manually engineered features that are extracted from UI images with domain-specific algorithms and are used in “traditional” Machine Learning models, such as feedforward artificial neural networks (ANN) that generally need fewer data. In our paper, we compare the prediction quality of CNN (a modified GoogLeNet architecture) and ANN models to predict visual perception per Aesthetics, Complexity, and Orderliness scales for about 2700 web UIs assessed by 137 users. Our results suggest that the ANN architecture produces smaller Mean Squared Error (MSE) for the training dataset size (N) available in our study, but that CNN should become superior with N > 2912. We also propose the regression model that can help HCI researchers to foretell MSE in their ML experiments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The web browser version is available at http://va.wuikb.info.
References
Oulasvirta, A., et al.: Aalto interface metrics (AIM): a service and codebase for computational GUI evaluation. In: The 31st Annual ACM Symposium on User Interface Software and Technology Adjunct Proceedings, pp. 16–19. ACM (2018)
Bakaev, M., Heil, S., Khvorostov, V., Gaedke, M.: Auto-extraction and integration of metrics for web user interfaces. J. Web Eng. 17(6), 561–590 (2018)
Bakaev, M., Speicher, M., Heil, S., Gaedke, M.: I Don’t Have That Much Data! Reusing User Behavior Models for Websites from Different Domains. In: Bielikova, M., Mikkonen, T., Pautasso, C. (eds.) ICWE 2020. LNCS, vol. 12128, pp. 146–162. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50578-3_11
Lima, A.L.D.S., Gresse von Wangenheim, C.: Assessing the visual esthetics of user interfaces: a ten-year systematic mapping. Int. J. Hum. Comput. Interact. 38(2), 144–164 (2022)
Dou, Q., Zheng, X.S., Sun, T., Heng, P.A.: Webthetics: quantifying webpage aesthetics with deep learning. Int. J. Hum Comput Stud. 124, 56–66 (2019)
Deka, B., et al.: Rico: A mobile app dataset for building data-driven design applications. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 845–854 (2017)
Bakaev M., Heil, S., Hamgushkeeva, G., Gaedke, M.: The effect of input data quality in feature-based modeling of user behavior. In: ESK International Symposium (2021) (In Print)
Ciołkosz-Styk, A., Styk, A.: Advanced image processing for maps graphical complexity estimation. In: Proceedings of the 26th International Cartographic Conference, Dresden, Germany, pp. 25–30 (2013)
Carballal, A., Santos, A., Romero, J., Machado, P., Correia, J., Castro, L.: Distinguishing paintings from photographs by complexity estimates. Neural Comput. Appl. 30(6), 1957–1969 (2016). https://doi.org/10.1007/s00521-016-2787-5
López-Rubio, J.M., Molina-Cabello, M.A., Ramos-Jiménez, G., López-Rubio, E.: Classification of Images as Photographs or Paintings by Using Convolutional Neural Networks. In: Rojas, I., Joya, G., Català, A. (eds.) IWANN 2021. LNCS, vol. 12861, pp. 432–442. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85030-2_36
Kreinovich, V.: From traditional neural networks to deep learning: towards mathematical foundations of empirical successes. In: Shahbazova, S.N., Kacprzyk, J., Balas, V.E., Kreinovich, V. (eds.) Recent Developments and the New Direction in Soft-Computing Foundations and Applications. SFSC, vol. 393, pp. 387–397. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-47124-8_31
Khan, A., Sohail, A., Zahoora, U., Qureshi, A.S.: A survey of the recent architectures of deep convolutional neural networks. Artif. Intell. Rev. 53(8), 5455–5516 (2020). https://doi.org/10.1007/s10462-020-09825-6
Talebi, H., Milanfar, P.: NIMA: Neural image assessment. IEEE Trans. Image Process. 27(8), 3998–4011 (2018)
Xing, B., Si, H., Chen, J., Ye, M., Shi, L.: Computational model for predicting user aesthetic preference for GUI using DCNNs. CCF Trans. Pervasive Comput. Interact. 3(2), 147–169 (2021). https://doi.org/10.1007/s42486-021-00064-4
Chen, J., et al.: Object detection for graphical user interface: old fashioned or deep learning or a combination? In: proceedings of the 28th ACM joint meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1202–1214 (2020)
Kamath, C.N., Bukhari, S.S., Dengel, A.: Comparative study between traditional machine learning and deep learning approaches for text classification. In: Proceedings of the ACM Symposium on Document Engineering 2018, pp. 1–11 (2018)
Asim, M.N., Ghani, M.U., Ibrahim, M.A., Mahmood, W., Dengel, A., Ahmed, S.: Benchmarking performance of machine and deep learning-based methodologies for Urdu text document classification. Neural Comput. Appl. 33(11), 5437–5469 (2020)
de Oliveira T. Souza, J., de Souza, A.D., Vasconcelos, L.G., Baldochi, L.A.: Usability Smells: A Systematic Review. In: Latifi, S. (ed.) ITNG 2021 18th International Conference on Information Technology-New Generations. AISC, vol. 1346, pp. 281–288. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-70416-2_36
Zen, M., Vanderdonckt, J.: Towards an evaluation of graphical user interfaces aesthetics based on metrics. In: 2014 IEEE Eighth International Conference on Research Challenges in Information Science, RCIS, pp. 1–12. IEEE (2014)
Yang, B. et al.: Don’t Do That! Hunting down visual design smells in complex UIs against design guidelines. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering, ICSE, pp. 761–772. IEEE (2021)
Michailidou, E., Eraslan, S., Yesilada, Y., Harper, S.: Automated prediction of visual complexity of web pages: Tools and evaluations. International Journal of Human-Computer Studies 145, 102523 (2021)
Bakaev, M., Heil, S., Khvorostov, V., Gaedke, M.: HCI vision for automated analysis and mining of web user interfaces. In: Mikkonen, T., Klamma, R., Hernández, J. (eds.) ICWE 2018. LNCS, vol. 10845, pp. 136–144. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91662-0_10
Boychuk, E., Bakaev, M.: Entropy and compression based analysis of web user interfaces. In: Bakaev, M., Frasincar, F., Ko, I.-Y. (eds.) ICWE 2019. LNCS, vol. 11496, pp. 253–261. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-19274-7_19
Elngar, A.A., et al.: Image classification based on CNN: a survey. J. Cybersecurity Inf. Manag. (JCIM) 6(1), 18–50 (2021)
Özgür, A., Nar, F.: Effect of dropout layer on classical regression problems. In: 2020 28th Signal Processing and Communications Applications Conference, SIU, pp. 1–4. IEEE (2020)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Bianco, S., Celona, L., Napoletano, P., Schettini, R.: Predicting image aesthetics with deep learning. In: Blanc-Talon, J., Distante, C., Philips, W., Popescu, D., Scheunders, P. (eds.) ACIVS 2016. LNCS, vol. 10016, pp. 117–125. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-48680-2_11
Acknowledgment
The reported study was funded by RFBR according to the research project No. 19–29-01017. The research was partially funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)—Project-ID 416228727—SFB 1410.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bakaev, M., Heil, S., Chirkov, L., Gaedke, M. (2022). Benchmarking Neural Networks-Based Approaches for Predicting Visual Perception of User Interfaces. In: Degen, H., Ntoa, S. (eds) Artificial Intelligence in HCI. HCII 2022. Lecture Notes in Computer Science(), vol 13336. Springer, Cham. https://doi.org/10.1007/978-3-031-05643-7_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-05643-7_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-05642-0
Online ISBN: 978-3-031-05643-7
eBook Packages: Computer ScienceComputer Science (R0)