Sketch2Process: End-to-End BPMN Sketch Recognition Based on Neural Networks
Authors: 
Bernhard Schäfer, Han van der Aa, Henrik Leopold, Heiner StuckenschmidtAuthors Info & Claims
Pages 2621 - 2641
Abstract
Process models play an important role in various software engineering contexts. Among others, they are used to capture business-related requirements and provide the basis for the development of process-oriented applications in low-code/no-code settings. To support modelers in creating, checking, and maintaining process models, dedicated tools are available. While these tools are generally considered as indispensable to capture process models for their later use, the initial version of a process model is often sketched on a whiteboard or a piece of paper. This has been found to have great advantages, especially with respect to communication and collaboration. It, however, also creates the need to subsequently transform the model sketch into a digital counterpart that can be further processed by modeling and analysis tools. Therefore, to automate this task, various so-called sketch recognition approaches have been defined in the past. Yet, these existing approaches are too limited for use in practice, since they, for instance, require sketches to be created on a digital device or do not address the recognition of edges or textual labels. Against this background, we use this paper to introduce Sketch2Process, the first end-to-end sketch recognition approach for process models captured using BPMN. Sketch2Process uses a neural network-based architecture to recognize the shapes, edges, and textual labels of highly expressive process models, covering 25 types of BPMN elements. To train and evaluate our approach, we created a dataset consisting of 704 hand-drawn and manually annotated BPMN models. Our experiments demonstrate that our approach is highly accurate and consistently outperforms the state of the art.
References
[1]
C. Ouyang, M. Dumas, W. M. V. D. Aalst, A. H. T. Hofstede, and J. Mendling, “From business process models to process-oriented software systems,” ACM Trans. Softw. Eng. Methodol., vol. 19, no. 1, pp. 1–37, 2009.
[2]
H. Leopold, J. Mendling, and A. Polyvyanyy, “Supporting process model validation through natural language generation,” IEEE Trans. Softw. Eng., vol. 40, no. 8, pp. 818–840, Aug. 2014.
[3]
D. van der Linden, I. Hadar, and A. Zamansky, “What practitioners really want: Requirements for visual notations in conceptual modeling,” Softw. Syst. Model., vol. 18, no. 3, pp. 1813–1831, 2019.
[4]
R. Waszkowski, “Low-code platform for automating business processes in manufacturing,” IFAC-PapersOnLine, vol. 52, no. 10, pp. 376–381, 2019.
[5]
P. Vincent, K. Iijima, M. Driver, J. Wong, and Y. Natis, “Magic quadrant for enterprise low-code application platforms,” Gartner, Tech. Rep. G00361584, 2019.
[6]
T. Gorschek, E. Tempero, and L. Angelis, “On the use of software design models in software development practice: An empirical investigation,” J. Syst. Softw., vol. 95, pp. 176–193, 2014.
[7]
O. Badreddin, R. Khandoker, A. Forward, and T. Lethbridge, “The evolution of software design practices over a decade: A long term study of practitioners,” J. Object Technol., vol. 20, no. 2, pp. 1–1, 2021.
[8]
M. Cherubini, G. Venolia, R. DeLine, and A. J. Ko, “Let's go to the whiteboard: How and why software developers use drawings,” in Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 2007, pp. 557–566.
[9]
J. Walny, S. Carpendale, N. Henry Riche, G. Venolia, and P. Fawcett, “Visual thinking in action: Visualizations as used on whiteboards,” IEEE Trans. Vis. Comput. Graphics, vol. 17, no. 12, pp. 2508–2517, Dec. 2011.
[10]
C. Bartelt, M. Vogel, and T. Warnecke, “Collaborative creativity: From hand drawn sketches to formal domain specific models and back again,” in Proc. Workshop Models Their Role Collaboration, 2013, pp. 25–32.
[11]
P. Antunes, D. Simões, L. Carriço, and J. A. Pino, “An end-user approach to business process modeling,” J. Netw. Comput. Appl., vol. 36, no. 6, pp. 1466–1479, 2013.
[12]
R. Brown, J. Recker, and S. West, “Using virtual worlds for collaborative business process modeling,” Bus. Process Manage. J., vol. 17, pp. 546–564, 2011.
[13]
G. Casella, V. Deufemia, V. Mascardi, G. Costagliola, and M. Martelli, “An agent-based framework for sketched symbol interpretation,” J. Vis. Lang. Comput., vol. 19, no. 2, pp. 225–257, 2008.
[14]
M. Bresler, D. Průša, and V. Hlaváč, “Online recognition of sketched arrow-connected diagrams,” Int. J. Document Anal. Recognit., vol. 19, no. 3, pp. 253–267, 2016.
[15]
F. Brieler and M. Minas, “Recognition and processing of hand-drawn diagrams using syntactic and semantic analysis,” in Proc. Work. Conf. Adv. Vis. Interfaces, 2008, pp. 181–188.
[16]
B. Schäfer and H. Stuckenschmidt, “DiagramNet: Hand-drawn diagram recognition using visual arrow-relation detection,” in Proc. Int. Conf. Document Anal. Recognit., 2021, pp. 614–630.
[17]
M. Zapp, P. Fettke, and P. Loos, “Towards a software prototype supporting automatic recognition of sketched business process models,” in Proc. Int. Conf. Wirtschaftsinformatik, 2017, pp. 1283–1286.
[18]
C. Gonzalez Moyano, L. Pufahl, I. Weber, and J. Mendling, “Uses of business process modeling in agile software development projects,” Inf. Softw. Technol., vol. 152, 2022, Art. no.
[19]
A. Sahay, A. Indamutsa, D. Di Ruscio, and A. Pierantonio, “Supporting the understanding and comparison of low-code development platforms,” in Proc. 46th Euromicro Conf. Softw. Eng. Adv. Appl., 2020, pp. 171–178.
[20]
U. Frank, P. Maier, and A. Bock, “Low code platforms: Promises, concepts and prospects. a comparative study of ten systems,” ICB-Research Report, Tech. Rep. 70, 2021.
[21]
J. Wong, K. Iijima, A. Leow, A. Jain, and P. Vincent, “Magic quadrant for enterprise low-code application platforms,” Gartner, Tech. Rep. G00738236, 2021.
[22]
B. Schäfer, M. Keuper, and H. Stuckenschmidt, “Arrow R-CNN for handwritten diagram recognition,” Int. J. Document Anal. Recognit., vol. 24, pp. 3–17, Feb. 2021.
[23]
B. Schäfer, H. van der Aa, H. Leopold, and H. Stuckenschmidt, “Sketch2BPMN: Automatic recognition of hand-drawn BPMN models,” in Proc. Int. Conf. Adv. Inf. Syst. Eng., 2021, pp. 344–360.
[24]
B. Yu and S. Cai, “A domain-independent system for sketch recognition,” in PRoc. Int. Conf. Comput. Graph. Interactive Techn. Australasia South East Asia, 2003, pp. 141–146.
[25]
T. A. Hammond and R. Davis, “Recognizing interspersed sketches quickly,” Graph. Interface, vol. 2009, pp. 157–166, 2009.
[26]
T. E. Johnson, “Sketchpad III: A computer program for drawing in three dimensions,” in Proc. Spring Joint Comput. Conf., 1963, pp. 347–353.
[27]
Q. Chen, J. Grundy, and J. Hosking, “SUMLOW: Early design-stage sketching of uml diagrams on an E-whiteboard,” Softw. Pract. Experience, vol. 38, no. 9, pp. 961–994, 2008.
[28]
F. D. Julca-Aguilar and N. S. T. Hirata, “Symbol detection in online handwritten graphics using faster R-CNN,” in Proc. IAPR Int. Workshop Document Anal. Syst., 2018, pp. 151–156.
[29]
M. Schmidt and G. Weber, “Recognition of multi-touch drawn sketches,” in Proc. Int. Conf. Hum.-Comput. Interaction, 2013, pp. 479–490.
[30]
A. Coyette, S. Schimke, J. Vanderdonckt, and C. Vielhauer, “Trainable sketch recognizer for graphical user interface design,” in Proc. IFIP Conf. Hum.-Comput. Interaction, 2007, pp. 124–135.
[31]
T. Kurtoglu and T. F. Stahovich, “Interpreting schematic sketches using physical reasoning,” in Proc. AAAI Spring Symp. Sketch Understanding, 2002, pp. 78–85.
[32]
O. Bergig, N. Hagbi, J. El-Sana, and M. Billinghurst, “In-place 3D sketching for authoring and augmenting mechanical systems,” in Proc. Int. Symp. Mixed Augmented Reality, 2009, pp. 87–94.
[33]
B. Paulson and T. Hammond, “PaleoSketch: Accurate primitive sketch recognition and beautification,” in Proc. Int. Conf. Intell. User Interfaces, 2008, pp. 1–10.
[34]
F. Brieler and M. Minas, “A model-based recognition engine for sketched diagrams,” J. Vis. Lang. Comput., vol. 21, no. 2, pp. 81–97, 2010.
[35]
V. Deufemia, M. Risi, and G. Tortora, “Hand-drawn diagram recognition with hierarchical parsing: An experimental evaluation,” in Proc. Conf. Inf. Syst. Crossroads Org., Manage., Accounting Eng., 2012, pp. 217–225.
[36]
F. Julca-Aguilar, H. Mouchère, C. Viard-Gaudin, and N. S. T. Hirata, “A general framework for the recognition of online handwritten graphics,” Int. J. Document Anal. Recognit., vol. 23, pp. 143–160, 2020.
[37]
G. Costagliola, M. De Rosa, and V. Fuccella, “Local context-based recognition of sketched diagrams,” J. Vis. Lang. Comput., vol. 25, no. 6, pp. 955–962, 2014.
[38]
J. Wu, C. Wang, L. Zhang, and Y. Rui, “Offline sketch parsing via shapeness estimation,” in Proc. Int. Joint Conf. Artif. Intell., 2015, pp. 1200–1206.
[39]
M. Bresler, D. Průša, and V. Hlaváč, “Recognizing off-line flowcharts by reconstructing strokes and using on-line recognition techniques,” in Proc. Int. Conf. Front. Handwriting Recognit., 2016, pp. 48–53.
[40]
D. Doermann, J. Liang, and H. Li, “Progress in camera-based document image analysis,” in Proc. Int. Conf. Document Anal. Recognit., 2003, pp. 606–616.
[41]
S. Bhowmik, R. Sarkar, M. Nasipuri, and D. Doermann, “Text and non-text separation in offline document images: A survey,” Int. J. Document Anal. Recognit., vol. 21, no. 1/2, pp. 1–20, Jun. 2018.
[42]
S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in Proc. Conf. Neural Inf. Process. Syst., 2015, pp. 91–99.
[43]
B. Schäfer and H. Stuckenschmidt, “Arrow R-CNN for flowchart recognition,” in Proc. Int. Conf. Document Anal. Recognit. Workshops, 2019, pp. 7–13.
[44]
A. Antinori, R. Coltrinari, F. Corradini, F. Fornari, B. Re, and M. Scarpetta, “BPMN-redrawer: From images to BPMN models,” in Proc. Demonstration Res. Track, Best BPM Dissertation Award, Doctoral Consortium BPM, 2022, pp. 107–111.
[45]
Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2,” 2019. [online]. Available: https://github.com/facebookresearch/detectron2
[46]
F. Pittke, H. Leopold, and J. Mendling, “Automatic detection and resolution of lexical ambiguity in process models,” IEEE Trans. Softw. Eng., vol. 41, no. 6, pp. 526–544, Jun. 2015.
[47]
S. Chakraborty, S. Sarker, and S. Sarker, “An exploration into the process of requirements elicitation: A grounded approach,” J. Assoc. Inf. Syst., vol. 11, no. 4, 2010, Art. no.
[48]
H. Leopold, J. Mendling, and O. Günther, “Learning from quality issues of BPMN models from industry,” IEEE Softw., vol. 33, no. 4, pp. 26–33, Jul./Aug. 2016.
[49]
K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask R-CNN,” in Proc. Int. Conf. Comput. Vis., 2017, pp. 2961–2969.
[50]
J. Yang, J. Lu, S. Lee, D. Batra, and D. Parikh, “Graph R-CNN for scene graph generation,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 670–685.
[51]
J. Zhang, K. J. Shih, A. Elgammal, A. Tao, and B. Catanzaro, “Graphical contrastive losses for scene graph parsing,” in Proc. Conf. Comput. Vis. Pattern Recognit., 2019, pp. 11535–11543.
[52]
B. Davis, B. Morse, S. Cohen, B. Price, and C. Tensmeyer, “Deep visual template-free form parsing,” in Proc. Int. Conf. Document Anal. Recognit., 2019, pp. 134–141.
[53]
A. G. Howardet al., “MobileNets: Efficient convolutional neural networks for mobile vision applications,” 2017,.
[54]
D. H. Douglas and T. K. Peucker, “Algorithms for the reduction of the number of points required to represent a digitized line or its caricature,” Cartographica, vol. 10, no. 2, pp. 112–122, Dec. 1973.
[55]
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. Cambridge, MA, USA: MIT Press, 2016.
[56]
A. Buslaev, V. I. Iglovikov, E. Khvedchenya, A. Parinov, M. Druzhinin, and A. A. Kalinin, “Albumentations: Fast and flexible image augmentations,” Information, vol. 11, no. 2, 2020, Art. no.
[57]
P. Gervais, T. Deselaers, E. Aksan, and O. Hilliges, “The DIDI dataset: Digital ink diagram data,” 2020,.
[58]
A. Paszkeet al., “PyTorch: An imperative style, high-performance deep learning library,” in Proc. Adv. Neural Inf. Process. Syst., 2019, pp. 8024–8035.
[59]
T.-Y. Lin, P. Dollar, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in Proc. Conf. Comput. Vis. Pattern Recognit., 2017, pp. 2117–2125.
[60]
J. A. Sánchez, V. Romero, A. H. Toselli, M. Villegas, and E. Vidal, “ICDAR2017 competition on handwritten text recognition on the READ dataset,” in Proc. Int. Conf. Document Anal. Recognit., 2017, pp. 1383–1388.
[61]
M. Dumas, M. L. Rosa, J. Mendling, and H. A. Reijers, Fundamentals of Business Process Management, 2nd ed. Berlin, Germany: Springer, 2018.
Recommendations
Sketch2BPMN: Automatic Recognition of Hand-Drawn BPMN Models
Advanced Information Systems EngineeringAbstractDespite the widespread availability of process modeling tools, the first version of a process model is often drawn by hand on a piece of paper or whiteboard, especially when several people are involved in its elicitation. Though this has been ...
End-to-End Photo-Sketch Generation via Fully Convolutional Representation Learning
ICMR '15: Proceedings of the 5th ACM on International Conference on Multimedia RetrievalSketch-based face recognition is an interesting task in vision and multimedia research, yet it is quite challenging due to the great difference between face photos and sketches. In this paper, we propose a novel approach for photo-sketch generation, ...
e‐VOL BPMN: a technique to support the evolution and learning of BPMN diagrams
In the software maintenance, business process models help practitioners understanding of the software, because it represents the organisational process in which software is embedded. BPMN (Business Process Modelling and Notation) is the standard notation ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
0098-5589 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 01 April 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 13 Dec 2024