Intelligent Optimization of Distributed Pipeline Execution in Serverless Platforms: A Predictive Model Approach
Pages 7 - 12
Abstract
Achieving efficient execution of distributed pipelines in serverless environments is essential to minimize both execution time and operational costs in cloud settings. This paper presents an approach to predict and optimize the duration of a serverless pipeline executed and parallelized with Lithops, using a geospatial water consumption analysis pipeline as a case study. The hyperparameters of the XGBoost model were optimized using Optuna, resulting in a 75.34% reduction in Mean Absolute Error (MAE) compared to a baseline model, and a 79.9% reduction in execution time compared to suboptimal configurations. Additionally, the model reduced the number of necessary pipeline executions by 30% compared to a full Design Space Analysis (DSA), leading to a 30% cost savings. These results highlight the model's ability to significantly improve both execution efficiency and cost-effectiveness, showcasing the benefits of using Lithops for serverless pipeline optimization.
References
[1]
L. Cloud, "Lithops - serverless cloud computing framework," 2023. [Online]. Available: https://lithops-cloud.github.io/
[2]
E. Jonas, Q. Pu, S. Venkataraman, I. Stoica, and B. Recht, "Occupy the cloud: Distributed computing for the 99%" in Proceedings of the 2017 Symposium on Cloud Computing. ACM, 2017, pp. 445--451. [Online]. Available: https://pywren.io
[3]
A. W. Services, "Aws lambda documentation," 2024. [Online]. Available: https://aws.amazon.com/lambda/
[4]
C. Project, "Water consumption geospatial use case," 2023. [Online]. Available: https://github.com/cloudbutton/geospatial-usecase/tree/main/water-consumption
[5]
Microsoft, "Azure functions documentation," 2024. [Online]. Available: https://docs.microsoft.com/en-us/azure/azure-functions/
[6]
G. Cloud, "Google cloud functions documentation," 2024. [Online]. Available: https://cloud.google.com/functions
[7]
A. Arjona, P. García-López, and D. Barcelona-Pons, "Dataplug: Unlocking extreme data analytics with on-the-fly dynamic partitioning of unstructured data," in Proceedings of the 24th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID'24). IEEE/ACM, May 2024, oral presentation.
[8]
H. Bannazadeh, M. Rabinovich, M. Chowdhury, and A. Wani, "Sizeless: Predicting the optimal size of serverless functions," in Proceedings of the 21st International Middleware Conference. ACM, 2020, pp. 56--70.
[9]
C. Harris, K. Millman, S. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. Smith et al., "Numpy: The fundamental package for array computing with python," Nature, vol. 585, pp. 357--362, 2020. [Online]. Available: https://numpy.org/
[10]
T. P. D. Team, "Pandas documentation," 2024. [Online]. Available: https://pandas.pydata.org/
[11]
Mapbox, "Rasterio documentation," 2024. [Online]. Available: https://rasterio.readthedocs.io/
[12]
S. Community, "Shapely documentation," 2024. [Online]. Available: https://shapely.readthedocs.io/
[13]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in python," Journal of Machine Learning Research, vol. 12, pp. 2825--2830, 2011. [Online]. Available: https://scikit-learn.org/stable/
[14]
O. Team, "Optuna: A hyperparameter optimization framework," 2024. [Online]. Available: https://optuna.org/
[15]
M. Corporation, "Lightgbm: A fast, distributed, high performance gradient boosting framework," https://github.com/microsoft/LightGBM, 2017, accessed: 2024-01-01.
Recommendations
Advancing Serverless Computing for Scalable AI Model Inference: Challenges and Opportunities
WoSC10 '24: Proceedings of the 10th International Workshop on Serverless ComputingArtificial Intelligence (AI) model inference has emerged as a crucial component across numerous applications. Serverless computing, known for its scalability, flexibility, and cost-efficiency, is an ideal paradigm for executing AI model inference tasks. ...
Dexter: A Performance-Cost Efficient Resource Allocation Manager for Serverless Data Analytics
Middleware '24: Proceedings of the 25th International Middleware ConferenceLeveraging serverless platforms for the efficient execution of distributed data analytics frameworks, such as Apache Spark [3], has gained substantial interest since early 2022. The elasticity, free-of-management, and on-demand scalability of serverless ...
Serverful Functions: Leveraging Servers in Complex Serverless Workflows (industry track)
Middleware Industrial Track '24: Proceedings of the 25th International Middleware Conference Industrial TrackThe scalability of cloud functions makes them a convenient backend for elastic data analytics pipelines where parallelism changes drastically from one stage to the next. However, cloud functions require intermediate storage systems for communication, ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2024
46 pages
ISBN:9798400713361
DOI:10.1145/3702634
This work is licensed under a Creative Commons Attribution International 4.0 License.
In-Cooperation
- IFIP
- Usenix
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 02 December 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
WoSC10 '24
WoSC10 '24: 10th International Workshop on Serverless Computing
December 2 - 6, 2024
Hong Kong, Hong Kong
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 104Total Downloads
- Downloads (Last 12 months)104
- Downloads (Last 6 weeks)36
Reflects downloads up to 03 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in