More Web Proxy on the site http://driver.im/

research-article

Open access

Operationalizing machine learning models: a systematic literature review

Authors:

Ask Berstad Kolltveit,

Jingyue LiAuthors Info & Claims

SE4RAI '22: Proceedings of the 1st Workshop on Software Engineering for Responsible AI

Pages 1 - 8

https://doi.org/10.1145/3526073.3527584

Published: 03 February 2023 Publication History

Abstract

Deploying machine learning (ML) models to production with the same level of rigor and automation as traditional software systems has shown itself to be a non-trivial task, requiring extra care and infrastructure to deal with the additional challenges. Although many studies focus on adapting ML software engineering (SE) approaches and techniques, few studies have summarized the status and challenges of operationalizing ML models. Model operationalization encompasses all steps after model training and evaluation, including packaging the model in a format appropriate for deployment, publishing to a model registry or storage, integrating the model into a broader software system, serving, and monitoring. This study is the first systematic literature review investigating the techniques, tools, and infrastructures to operationalize ML models. After reviewing 24 primary studies, the results show that there are a number of tools for most use cases to operationalize ML models and cloud deployment in particular. The review also revealed several research opportunities, such as dynamic model-switching, continuous model-monitoring, and efficient edge ML deployments.

References

[1]

2020. Oria Search Engine. http://oria.no/ Accessed 05.09.2020.

[2]

2020. The state of AI in 2020. https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/global-survey-the-state-of-ai-in-2020

[3]

2022. MLOps pipeline. https://ml-ops.org/content/mlops-principles.

[4]

Lucas Baier, Fabian Jöhren, and Stefan Seebacher. 2019. Challenges in the deployment and operation of machine learning in practice. In ECIS 2019 - 27th European Conference on Information Systems. https://www.researchgate.net/publication/332996647_CHALLENGES_IN_THE_DEPLOYMENT_AND_OPERATION_OF_MACHINE_LEARNING_IN_PRACTICE

[5]

Denis Baylor, Eric Breck, Heng-Tze Cheng, Noah Fiedel, Chuan Yu Foo, Zakaria Haque, Salem Haykal, Mustafa Ispir, Vihan Jain, Levent Koc, Chiu Yuen Koo, Lukasz Lew, Clemens Mewald, Akshay Naresh Modi, Neoklis Polyzotis, Sukriti Ramesh, Sudip Roy, Steven Euijong Whang, Martin Wicke, Jarek Wilkiewicz, Xin Zhang, and Martin Zinkevich. 2017. TFX. ACM.

Digital Library

[6]

Lucas Bernardi, Themistoklis Mavridis, and Pablo Estevez. 2019. 150 Successful Machine Learning Models. ACM.

Digital Library

[7]

Jan Bosch, Helena Holmström Olsson, and Ivica Crnkovic. 2021. Engineering AI Systems. IGI Global, 1--19.

[8]

Pierre Bourque. 2014. SWEBOK : guide to the software engineering body of knowledge. IEEE Computer Society, Los Alamitos, CA.

[9]

Dheeraj Chahal, Ravi Ojha, Sharod Roy Choudhury, and Manoj Nambiar. 2020. Migrating a Recommendation System to Cloud Using ML Workflow. ACM.

Digital Library

[10]

Andrew Chen, Andy Chow, Aaron Davidson, Arjun DCunha, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Clemens Mewald, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, Avesh Singh, Fen Xie, Matei Zaharia, Richard Zang, Juntai Zheng, and Corey Zumar. 2020. Developments in MLflow. In Proceedings of the Fourth International Workshop on Data Management for End-to-End Machine Learning. ACM.

Digital Library

[11]

Yujeong Choi, Yunseong Kim, and Minsoo Rhu. 2021. Lazy Batching: An SLA-aware Batching System for Cloud Machine Learning Inference. IEEE.

[12]

Daniel Crankshaw, Xin Wang, Guilio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. 2017. Clipper: A Low-Latency Online Prediction Serving System. In 14th USENIX Symposium on Networked Systems Design and Implementation (NSDI 17). USENIX Association, Boston, MA, 613--627. https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/crankshaw

[13]

Sweenor David, Hillion Steven, Rope Dan, Kannabiran Dev, Hill Thomas, O'Connell Michael, and Safari an O'Reilly Media Company. 2020. ML Ops : Operationalizing Data Science. https://go.oreilly.com/queensland-university-of-technology/library/view/-/9781492074663/?ar

[14]

Breno B. Nicolau de França, Helvio Jeronimo, and Guilherme Horta Travassos. 2016. Characterizing DevOps by Hearing Multiple Voices. ACM Press.

Digital Library

[15]

Christof Ebert, Gorka Gallardo, Josune Hernantes, and Nicolas Serrano. 2016. DevOps. 33, 3 (may 2016), 94--100.

Digital Library

[16]

Alvaro Lopez Garcia, Jesus Marco De Lucas, Marica Antonacci, Wolfgang Zu Castell, Mario David, Marcus Hardt, Lara Lloret Iglesias, Germen Molto, Marcin Plociennik, Viet Tran, Andy S. Alic, Miguel Caballer, Isabel Campos Plasencia, Alessandro Costantini, Stefan Dlugolinsky, Doina Cristina Duma, Giacinto Donvito, Jorge Gomes, Ignacio Heredia Cacha, Keiichi Ito, Valentin Y. Kozlov, Giang Nguyen, Pablo Orviz Fernandez, Zdenek Sustr, and Pawel Wolniewicz. 2020. A Cloud-Based Framework for Machine Learning Workloads and Applications. 8 (2020), 18681--18692.

[17]

Vahid Garousi, Michael Felderer, and Mika V. Mäntylä. 2016. The need for multivocal literature reviews in software engineering. In Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering. ACM.

Digital Library

[18]

Görkem Giray. 2021. A software engineering perspective on engineering machine learning systems: State of the art and challenges. Journal of Systems and Software 180 (oct 2021), 111031.

Digital Library

[19]

Tuomas Granlund, Vlad Stirbu, and Tommi Mikkonen. 2021. Towards Regulatory-Compliant MLOps: Oravizio's Journey from a Machine Learning Experiment to a Deployed Certified Medical Product. 2, 5 (jun 2021).

[20]

Nitu Gupta, Katpagavalli Anantharaj, and Karthikeyan Subramani. 2020. Containerized Architecture for Edge Computing in Smart Home : A consistent architecture for model deployment. IEEE.

[21]

Kim Hazelwood, Sarah Bird, David Brooks, Soumith Chintala, Utku Diril, Dmytro Dzhulgakov, Mohamed Fawzy, Bill Jia, Yangqing Jia, Aditya Kalro, James Law, Kevin Lee, Jason Lu, Pieter Noordhuis, Misha Smelyanskiy, Liang Xiong, and Xiaodong Wang. 2018. Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective. IEEE.

[22]

Waldemar Hummer, Vinod Muthusamy, Thomas Rausch, Parijat Dube, Kaoutar El Maghraoui, Anupama Murthi, and Punleuk Oum. 2019. ModelOps: Cloud-Based Lifecycle Management for Reliable and Trusted AI. IEEE.

[23]

Meenu Mary John, Helena Holmström Olsson, and Jan Bosch. 2021. Architecting AI Deployment: A Systematic Review of State-of-the-Art and State-of-Practice Literature. In Lecture Notes in Business Information Processing. Springer International Publishing, 14--29.

[24]

Meenu Mary John, Helena Holmstrom Olsson, and Jan Bosch. 2021. Towards MLOps: A Framework and Maturity Model. IEEE.

[25]

B. Kitchenham and S Charters. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering.

[26]

Rajalakshmi Krishnamurthi, Raghav Maheshwari, and Rishabh Gulati. 2019. Deploying Deep Learning Models via IOT Deployment Tools. IEEE.

[27]

Fumihiro Kumeno. 2020. Sofware engneering challenges for machine learning applications: A literature review. Intelligent Decision Technologies 13 (2020), 463--476.

[28]

Li Erran Li, Eric Chen, Jeremy Hermann, Pusheng Zhang, and Luming Wang. 2017. Scaling Machine Learning as a Service. In Proceedings of The 3rd International Conference on Predictive Applications and APIs (Proceedings of Machine Learning Research, Vol. 67), Claire Hardgrove, Louis Dorard, Keiran Thompson, and Florian Douetteau (Eds.). PMLR, 14--29. https://proceedings.mlr.press/v67/li17a.html

[29]

Yan Liu, Zhijing Ling, Boyu Huo, Boqian Wang, Tianen Chen, and Esma Mouine. 2020. Building A Platform for Machine Learning Operations from Open Source Frameworks. 53, 5 (2020), 704--709.

[30]

Chad Lochmiller. 2021. Conducting Thematic Analysis with Qualitative Data. The Qualitative Report (jun 2021).

[31]

Giuliano Lorenzoni, Paulo Alencar, Nathalia Nascimento, and Donald Cowan. 2021. Machine Learning Model Development from a Software Engineering Perspective: A Systematic Literature Review. (Feb. 2021). arXiv:2102.07574 [cs.SE]

[32]

Qinghua Lu, Liming Zhu, Xiwei Xu, Jon Whittle, David Douglas, and Conrad Sanderson. 2021. Software Engineering for Responsible AI: An Empirical Study and Operationalised Patterns. https://doi.org/abs/2111.09478

[33]

Lucy Ellen Lwakatare, Ivica Crnkovic, Ellinor Rånge, and Jan Bosch. 2020. From a Data Science Driven Process to a Continuous Delivery Process for Machine Learning Systems. In Product-Focused Software Process Improvement. Springer International Publishing, 185--201.

Digital Library

[34]

Lucy Ellen Lwakatare, Aiswarya Raj, Jan Bosch, Helena Holmström Olsson, and Ivica Crnkovic. 2019. A Taxonomy of Software Engineering Challenges for Machine Learning Systems: An Empirical Investigation. In Lecture Notes in Business Information Processing. Springer International Publishing, 227--243.

[35]

Lucy Ellen Lwakatare, Aiswarya Raj, Ivica Crnkovic, Jan Bosch, and Helena Holmström Olsson. 2020. Large-scale machine learning systems in real-world industrial settings: A review of challenges and solutions. Information and Software Technology 127 (nov 2020), 106368.

[36]

Silverio Martínez-Fernández, Justus Bogner, Xavier Franch, Marc Oriol, Julien Siebert, Adam Trendowicz, Anna Maria Vollmer, and Stefan Wagner. 2021. Software Engineering for AI-Based Systems: A Survey. (May 2021). arXiv:2105.01984 [cs.SE]

[37]

Elizamary Nascimento, Anh Nguyen-Duc, Ingrid Sundbø, and Tayana Conte. 2020. Software engineering for artificial intelligence and machine learning software: A systematic literature review. (Nov. 2020). arXiv:2011.03751 [cs.SE]

[38]

Andrei Paleyes, Raoul-Gabriel Urma, and Neil D. Lawrence. 2020. Challenges in Deploying Machine Learning: a Survey of Case Studies. The ML-Retrospectives, Surveys & Meta-Analyses Workshop, NeurIPS 2020, Article arXiv:2011.09926 (Nov. 2020). arXiv:2011.09926 [cs.LG] https://ui.adsabs.harvard.edu/abs/2020arXiv201109926P

[39]

Devon Peticolas, Russell Kirmayer, and Deepak Turaga. 2019. Mímir: Building and Deploying an ML Framework for Industrial IoT. IEEE.

[40]

Pekka Pääkkönen, Daniel Pakkala, Jussi Kiljander, and Roope Sarala. 2020. Architecture for Enabling Edge Inference via Model Transfer from Cloud Domain in a Kubernetes Environment. 13, 1 (dec 2020), 5.

[41]

Bin Qian, Jie Su, Zhenyu Wen, Devki Nandan Jha, Yinhao Li, Yu Guan, Deepak Puthal, Philip James, Renyu Yang, Albert Y. Zomaya, Omer Rana, Lizhe Wang, Maciej Koutny, and Rajiv Ranjan. 2020. Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey. ACM Comput. Surv. 53, 4, Article 82 (aug 2020), 47 pages.

Digital Library

[42]

Thomas Rausch and Schahram Dustdar. 2019. Edge Intelligence: The Convergence of Humans, Things, and AI. IEEE.

[43]

Thomas Rausch, Waldemar Hummer, Vinod Muthusamy, Alexander Rashed, and Schahram Dustdar. 2019. Towards a Serverless Platform for Edge AI. In 2nd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 19). USENIX Association, Renton, WA. https://www.usenix.org/conference/hotedge19/presentation/rausch

[44]

Daniel Richins, Dharmisha Doshi, Matthew Blackmore, Aswathy Thulaseedharan Nair, Neha Pathapati, Ankit Patel, Brainard Daguman, Daniel Dobrijalowski, Ramesh Illikkal, Kevin Long, David Zimmerman, and Vijay Janapa Reddi. 2020. Missing the Forest for the Trees: End-to-End AI Application Performance in Edge Data Centers. IEEE.

[45]

Philipp Ruf, Manav Madan, Christoph Reich, and Djaffar Ould-Abdeslam. 2021. Demystifying MLOps and Presenting a Recipe for the Selection of Open-Source Tools. 11, 19 (sep 2021), 8861.

[46]

Stephan Schlögl, Claudia Postulka, Reinhard Bernsteiner, and Christian Ploder. 2019. Artificial Intelligence Tool Penetration in Business: Adoption, Challenges and Fears. Springer International Publishing, 259--270.

[47]

D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, and Dan Dennison. 2015. Hidden Technical Debt in Machine Learning Systems. In Advances in Neural Information Processing Systems, C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28.

[48]

Alex Serban, Koen van der Blom, Holger Hoos, and Joost Visser. 2020. Adoption and Effects of Software Engineering Best Practices in Machine Learning. In Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). ACM.

Digital Library

[49]

Alex Serban and Joost Visser. 2021. An Empirical Study of Software Architecture for Machine Learning. (May 2021). arXiv:2105.12422 [cs.SE]

[50]

Julian Soh and Priyanshi Singh. 2020. Machine Learning Operations. In Data Science Solutions on Azure. Apress, 259--279.

[51]

Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies and a replication in software engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering - EASE '14. ACM Press.

Digital Library

[52]

Y. Xie, L. Cruz, P. Heck, and J. S. Rellermeyer. 2021. Systematic Mapping Study on the Machine Learning Lifecycle. In 2021 IEEE/ACM 1st Workshop on AI Engineering - Software Engineering for AI (WAIN). IEEE Computer Society, Los Alamitos, CA, USA, 70--73.

Digital Library

[53]

Neeraja J. Yadwadkar, Francisco Romero, Qian Li, and Christos Kozyrakis. 2019. A Case for Managed and Model-less Inference Serving. ACM.

Digital Library

[54]

Jeff Zhang, Sameh Elnikety, Shuayb Zarar, Atul Gupta, and Siddharth Garg. 2020. Model-Switching: Dealing with Fluctuating Workloads in Machine-Learning-asa-Service Systems. In 12th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 20). USENIX Association. https://www.usenix.org/conference/hotcloud20/presentation/zhang

Cited By

Conde JMunoz-Arcentales AAlonso ÁSalvachúa JHuecas G(2024)Enhanced FIWARE-Based Architecture for Cyberphysical Systems With Tiny Machine Learning and Machine Learning Operations: A Case Study on Urban Mobility SystemsIT Professional10.1109/MITP.2024.342196826:5(55-61)Online publication date: Sep-2024
https://doi.org/10.1109/MITP.2024.3421968
Ananthi SB LM SS S(2024)Framework for Platform Independent Machine Learning (ML) Model Execution2024 2nd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT)10.1109/IDCIoT59759.2024.10467931(728-732)Online publication date: 4-Jan-2024
https://doi.org/10.1109/IDCIoT59759.2024.10467931
Wewer CMahapatra HEsterle LLarsen P(2024)Using FactoryML for Deployment of Machine Learning Models in Industrial Production2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA)10.1109/ETFA61755.2024.10710938(1-8)Online publication date: 10-Sep-2024
https://doi.org/10.1109/ETFA61755.2024.10710938
Show More Cited By

Index Terms

Operationalizing machine learning models: a systematic literature review

Recommendations

Systematic literature review of machine learning based software development effort estimation models

Context: Software development effort estimation (SDEE) is the process of predicting the effort required to develop a software system. In order to improve estimation accuracy, many researchers have proposed machine learning (ML) based SDEE models (ML ...
Studying the Practices of Deploying Machine Learning Projects on Docker
EASE '22: Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering

Docker is a containerization service that allows for convenient deployment of websites, databases, applications’ APIs, and machine learning (ML) models with a few lines of code. Studies have recently explored the use of Docker for deploying general ...
A systematic review of machine learning techniques for software fault prediction

Reviews studies from 1991-2013 to assess application of ML techniques for SFP.Identifies seven categories of the ML techniques.Identifies 64 studies to answer the established research questions.Selects primary studies according to the quality assessment ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SE4RAI '22: Proceedings of the 1st Workshop on Software Engineering for Responsible AI

May 2022

57 pages

ISBN:9781450393195

DOI:10.1145/3526073

General Chairs:
Qinghua Lu
Data61, CSIRO, Australia
,
Xiwei Xu
Data61, CSIRO, Australia
,
Liming Zhu
Data61, CSIRO, Australia
,
John Grundy
Monash University, Australia

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 February 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '22

Sponsor:

SIGSOFT

ICSE '22: 44th International Conference on Software Engineering

May 19, 2022

Pennsylvania, Pittsburgh

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
1,269
Total Downloads

Downloads (Last 12 months)667
Downloads (Last 6 weeks)57

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Conde JMunoz-Arcentales AAlonso ÁSalvachúa JHuecas G(2024)Enhanced FIWARE-Based Architecture for Cyberphysical Systems With Tiny Machine Learning and Machine Learning Operations: A Case Study on Urban Mobility SystemsIT Professional10.1109/MITP.2024.342196826:5(55-61)Online publication date: Sep-2024
https://doi.org/10.1109/MITP.2024.3421968
Ananthi SB LM SS S(2024)Framework for Platform Independent Machine Learning (ML) Model Execution2024 2nd International Conference on Intelligent Data Communication Technologies and Internet of Things (IDCIoT)10.1109/IDCIoT59759.2024.10467931(728-732)Online publication date: 4-Jan-2024
https://doi.org/10.1109/IDCIoT59759.2024.10467931
Wewer CMahapatra HEsterle LLarsen P(2024)Using FactoryML for Deployment of Machine Learning Models in Industrial Production2024 IEEE 29th International Conference on Emerging Technologies and Factory Automation (ETFA)10.1109/ETFA61755.2024.10710938(1-8)Online publication date: 10-Sep-2024
https://doi.org/10.1109/ETFA61755.2024.10710938
Maier RSchlattl AGuess TMottok J(2024)CausalOps — Towards an industrial lifecycle for causal probabilistic graphical modelsInformation and Software Technology10.1016/j.infsof.2024.107520174:COnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107520
Rocha AMonteiro MMattos CDias MSoares JMagalhães RMacedo J(2024)Edge AI for Internet of Medical ThingsComputers and Electrical Engineering10.1016/j.compeleceng.2024.109202116:COnline publication date: 1-May-2024
https://dl.acm.org/doi/10.1016/j.compeleceng.2024.109202
Najafabadi F(2024)Reference Architecture of MLOps WorkflowsSoftware Architecture. ECSA 2024 Tracks and Workshops10.1007/978-3-031-71246-3_6(49-57)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-71246-3_6
Amou Najafabadi FBogner JGerostathopoulos ILago P(2024)An Analysis of MLOps Architectures: A Systematic Mapping StudySoftware Architecture10.1007/978-3-031-70797-1_5(69-85)Online publication date: 1-Sep-2024
https://doi.org/10.1007/978-3-031-70797-1_5
Haertel CStaegemann DDaase CPohl MNahhas ATurowski K(2023)MLOps in Data Science Projects: A Review2023 IEEE International Conference on Big Data (BigData)10.1109/BigData59044.2023.10386139(2396-2404)Online publication date: 15-Dec-2023
https://doi.org/10.1109/BigData59044.2023.10386139
Steidl MFelderer MRamler R(2023)The pipeline for the continuous development of artificial intelligence models—Current state of research and practiceJournal of Systems and Software10.1016/j.jss.2023.111615199:COnline publication date: 1-May-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111615
Ferrara CSellitto GFerrucci FPalomba FDe Lucia A(2023)Fairness-aware machine learning engineering: how far are we?Empirical Software Engineering10.1007/s10664-023-10402-y29:1Online publication date: 24-Nov-2023
https://dl.acm.org/doi/10.1007/s10664-023-10402-y
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents