Abstract
Next activity prediction aims to forecast the future behavior of running process instances. Recent publications in this field predominantly employ deep learning techniques and evaluate their prediction performance using publicly available event logs. This paper presents empirical evidence that calls into question the effectiveness of these current evaluation approaches. We show that there is an enormous amount of example leakage in all of the commonly used event logs, so that rather trivial prediction approaches perform almost as well as ones that leverage deep learning. We further argue that designing robust evaluations requires a more profound conceptual engagement with the topic of next-activity prediction, and specifically with the notion of generalization to new data. To this end, we present various prediction scenarios that necessitate different types of generalization to guide future research.
L. Abb and P. Pfeiffer—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
A notable exception to this is [8], which focuses on process model structures.
References
Breuker, D., Matzner, M., Delfmann, P., Becker, J.: Comprehensible predictive models for business processes. MIS Q. 40(4), 1009–1034 (2016)
Brunk, J., Stottmeister, J., Weinzierl, S., Matzner, M., Becker, J.: Exploring the effect of context information on deep learning business process predictions. J. Decis. Syst. 29(sup1), 328–343 (2020)
Di Francescomarino, C., Ghidini, C.: Predictive process monitoring. In: van der Aalst, W.M.P., Carmona, J. (eds.) Process Mining Handbook. Lecture Notes in Business Information Processing, vol. 448, pp. 320–346. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-08848-3_10
Evermann, J., Rehse, J.R., Fettke, P.: Predicting process behaviour using deep learning. Decis. Support Syst. 100, 129–140 (2017)
Kaufman, S., Rosset, S., Perlich, C.: Leakage in data mining: formulation, detection, and avoidance. In: KDD Conference, vol. 6, pp. 556–563. ACM, New YOrk (2011)
Neu, D., Lahann, J., Fettke, P.: A systematic literature review on state-of-the-art deep learning methods for process prediction. Art. Int. Rev. 55, 1–27 (2022)
Pasquadibisceglie, V., Appice, A., Castellano, G., Malerba, D.: A multi-view deep learning approach for predictive business process monitoring. IEEE Trans. Serv. Comp. 15(04), 2382–2395 (2022)
Peeperkorn, J., Broucke, S.V., De Weerdt, J.: Can recurrent neural networks learn process model structure? J. Intell. Inf. Syst. 61, 1–25 (2022)
Pfeiffer, P., Lahann, J., Fettke, P.: Multivariate business process representation learning utilizing Gramian angular fields and convolutional neural networks. In: Polyvyanyy, A., Wynn, M.T., Van Looy, A., Reichert, M. (eds.) BPM 2021. LNCS, vol. 12875, pp. 327–344. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85469-0_21
Pfeiffer, P., Lahann, J., Fettke, P.: The label ambiguity problem in process prediction. In: Cabanillas, C., Garmann-Johnsen, N.F., Koschmider, A. (eds.) BPM 2022. LNBIP, vol. 460, pp. 37–44. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-25383-6_4
Rama-Maneiro, E., Vidal, J., Lama, M.: Deep learning for predictive business process monitoring: review and benchmark. IEEE Trans. Serv. Comp. 16(1) (2021)
Scheid, M., Rehse, J.R., Houy, C., Fettke, P.: Data set for MobIS challenge 2019 (2018). https://doi.org/10.13140/RG.2.2.11870.28487
Tax, N., Teinemaa, I., van Zelst, S.J.: An interdisciplinary comparison of sequence modeling methods for next-element prediction. Softw. Syst. Model. 19(6), 1345–1365 (2020)
Verenich, I.: Helpdesk event log. https://doi.org/10.17632/39bp3vv62t.1
Weytjens, H., De Weerdt, J.: Creating unbiased public benchmark datasets with data leakage prevention for predictive process monitoring. In: Marrella, A., Weber, B. (eds.) BPM 2021. LNBIP, vol. 436, pp. 18–29. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-94343-1_2
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Abb, L., Pfeiffer, P., Fettke, P., Rehse, JR. (2024). A Discussion on Generalization in Next-Activity Prediction. In: De Weerdt, J., Pufahl, L. (eds) Business Process Management Workshops. BPM 2023. Lecture Notes in Business Information Processing, vol 492. Springer, Cham. https://doi.org/10.1007/978-3-031-50974-2_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-50974-2_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-50973-5
Online ISBN: 978-3-031-50974-2
eBook Packages: Computer ScienceComputer Science (R0)