Abstract
Technological innovations in artificial intelligence and machine learning enable business operators to engage with their customers 24/7 through chatbots. Many customers expect around-the-clock support which puts a strain on human resources; augmenting human resources with a chatbot can reduce costs for an organization and increase customer satisfaction. This work presents the HR Chatbot: a chatbot that answers general questions about human resource topics (i.e. payroll, benefits) for a private university. The research involves a collaboration between computer scientists, user experience researchers, and human resource administration. This work addresses two research questions: What are employees at a private university looking for from a chatbot for human resources?; and what are the appropriate methods to evaluate and measure the success of a chatbot for human resources? The HR Chatbot uses IBM Watson Assistant services, and an initial prototype was designed from a document of 31 frequently asked questions. Three rounds of user testing were conducted with employees of the university. The initial tests revealed that the chatbot was perceived as useful, but many were dissatisfied with the responses, specifically the lack of responses. Errors in the chatbot were classified into different categories; the most common being that the question was not in the content scope for the chatbot. Thus, data from the initial studies informed the scope of the chatbot; the number of unique questions grew to 157 and the total number of questions increased to 463. The HR Chatbot has 90% accuracy and an average sustained usability score of 69.5, surpassing the benchmark score. Following the initial tests, the HR Chatbot was deployed in real-time on the human resources website. This work describes how a chatbot was created, evaluated, and deployed online. We hope that this work inspires and informs others to explore similar use cases with chatbot technologies.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bot-ify your campus. https://ivy.ai/
Get more student leads and registrations automatically 24/7. https://www.virtualspirits.com/chatbot-for-university.aspx
AI-powered conversational strategy for student success, October 2019. https://www.admithub.com/
Adam, M., Wessel, M., Benlian, A.: AI-based chatbots in customer service and their effects on user compliance. Electr. Mark. 31, 1–19 (2020)
Brandtzaeg, P.B., Følstad, A., et al.: Why people use chatbots. In: Kompatsiaris, I. (ed.) INSCI 2017. LNCS, vol. 10673, pp. 377–392. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70284-1_30
Ho, R.C.: Chatbot for online customer service: customer engagement in the era of artificial intelligence. In: Impact of Globalization and Advanced Technologies on Online Business Models, pp. 16–31. IGI Global (2021)
Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., McTear, M.: Usability testing of a healthcare chatbot: can we use conventional methods to assess conversational user interfaces? In: Proceedings of the 31st European Conference on Cognitive Ergonomics, pp. 207–214 (2019)
Io, H.H.N., Lee, C.: Chatbots and conversational agents: a bibliometric analysis, pp. 215–219, December 2017. https://doi.org/10.1109/IEEM.2017.8289883
Jain, M., Kumar, P., Kota, R., Patel, S.N.: Evaluating and informing the design of chatbots. In: Proceedings of the 2018 Designing Interactive Systems Conference, pp. 895–906 (2018)
Jitgosol, Y., Kasemvilas, S., Boonchai, P.: Designing an HR chatbot to support human resource management, pp. 165, 170, December 2019
Jongerius, C.: Quantifying chatbot performance by using data analytics. Ph.D. thesis, Utrecht University (2018)
Lewis, J.R.: The system usability scale: past, present, and future. Int. J. Hum.-Comput. Inter. 34(7), 577–590 (2018)
Melián-González, S., Gutiérrez-Taño, D., Bulchand-Gidumal, J.: Predicting the intentions to use chatbots for travel and tourism. Curr. Issue Tour. 24(2), 192–210 (2021)
Madana Mohana, R., Pitty, N., Lalitha Surya Kumari, P.: Customer support chatbot using machine learning. In: Satapathy, S.C., Zhang, Y.-D., Bhateja, V., Majhi, R. (eds.) Intelligent Data Engineering and Analytics. AISC, vol. 1177, pp. 445–451. Springer, Singapore (2021). https://doi.org/10.1007/978-981-15-5679-1_42
Peras, D.: Chatbot evaluation metrics. In: Economic and Social Development: Book of Proceedings, pp. 89–97 (2018)
Radziwill, N.M., Benton, M.C.: Evaluating quality of chatbots and intelligent conversational agents. arXiv preprint arXiv:1704.04579 (2017)
Russell-Rose, T.: A framework for chatbot evaluation, February 2017. https://chatbotnewsdaily.com/a-framework-for-chatbot-evaluation-15d236557ad5
Sheth, B.: Want to use chatbots to automate the majority of HR services? June 2019. www.thebalancecareers.com/use-chatbots-to-automate-hr-many-services-4171964#::text=Chatbots%20enable%20HR%20teams%20to, need%20escalation%20for%20human%20engagement
Shumanov, M., Johnson, L.: Making conversations with chatbots more personalized. Comput. Hum. Behav. 117, 106627 (2021)
da Silva Oliveira, J., Espíndola, D.B., Barwaldt, R., Ribeiro, L.M., Pias, M.: IBM Watson application as FAQ assistant about moodle. In: 2019 IEEE Frontiers in Education Conference (FIE), pp. 1–8. IEEE (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Drozdal, J. et al. (2021). The Design and Evaluation of a Chatbot for Human Resources. In: Stephanidis, C., Antona, M., Ntoa, S. (eds) HCI International 2021 - Late Breaking Posters. HCII 2021. Communications in Computer and Information Science, vol 1498. Springer, Cham. https://doi.org/10.1007/978-3-030-90176-9_32
Download citation
DOI: https://doi.org/10.1007/978-3-030-90176-9_32
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-90175-2
Online ISBN: 978-3-030-90176-9
eBook Packages: Computer ScienceComputer Science (R0)