Intelligent Optimization of Distributed Pipeline Execution in Serverless Platforms: A Predictive Model Approach
Pages 7 - 12
Abstract
Achieving efficient execution of distributed pipelines in serverless environments is essential to minimize both execution time and operational costs in cloud settings. This paper presents an approach to predict and optimize the duration of a serverless pipeline executed and parallelized with Lithops, using a geospatial water consumption analysis pipeline as a case study. The hyperparameters of the XGBoost model were optimized using Optuna, resulting in a 75.34% reduction in Mean Absolute Error (MAE) compared to a baseline model, and a 79.9% reduction in execution time compared to suboptimal configurations. Additionally, the model reduced the number of necessary pipeline executions by 30% compared to a full Design Space Analysis (DSA), leading to a 30% cost savings. These results highlight the model's ability to significantly improve both execution efficiency and cost-effectiveness, showcasing the benefits of using Lithops for serverless pipeline optimization.
References
[1]
L. Cloud, "Lithops - serverless cloud computing framework," 2023. [Online]. Available: https://lithops-cloud.github.io/
[2]
E. Jonas, Q. Pu, S. Venkataraman, I. Stoica, and B. Recht, "Occupy the cloud: Distributed computing for the 99%" in Proceedings of the 2017 Symposium on Cloud Computing. ACM, 2017, pp. 445--451. [Online]. Available: https://pywren.io
[3]
A. W. Services, "Aws lambda documentation," 2024. [Online]. Available: https://aws.amazon.com/lambda/
[4]
C. Project, "Water consumption geospatial use case," 2023. [Online]. Available: https://github.com/cloudbutton/geospatial-usecase/tree/main/water-consumption
[5]
Microsoft, "Azure functions documentation," 2024. [Online]. Available: https://docs.microsoft.com/en-us/azure/azure-functions/
[6]
G. Cloud, "Google cloud functions documentation," 2024. [Online]. Available: https://cloud.google.com/functions
[7]
A. Arjona, P. García-López, and D. Barcelona-Pons, "Dataplug: Unlocking extreme data analytics with on-the-fly dynamic partitioning of unstructured data," in Proceedings of the 24th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID'24). IEEE/ACM, May 2024, oral presentation.
[8]
H. Bannazadeh, M. Rabinovich, M. Chowdhury, and A. Wani, "Sizeless: Predicting the optimal size of serverless functions," in Proceedings of the 21st International Middleware Conference. ACM, 2020, pp. 56--70.
[9]
C. Harris, K. Millman, S. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. Smith et al., "Numpy: The fundamental package for array computing with python," Nature, vol. 585, pp. 357--362, 2020. [Online]. Available: https://numpy.org/
[10]
T. P. D. Team, "Pandas documentation," 2024. [Online]. Available: https://pandas.pydata.org/
[11]
Mapbox, "Rasterio documentation," 2024. [Online]. Available: https://rasterio.readthedocs.io/
[12]
S. Community, "Shapely documentation," 2024. [Online]. Available: https://shapely.readthedocs.io/
[13]
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in python," Journal of Machine Learning Research, vol. 12, pp. 2825--2830, 2011. [Online]. Available: https://scikit-learn.org/stable/
[14]
O. Team, "Optuna: A hyperparameter optimization framework," 2024. [Online]. Available: https://optuna.org/
[15]
M. Corporation, "Lightgbm: A fast, distributed, high performance gradient boosting framework," https://github.com/microsoft/LightGBM, 2017, accessed: 2024-01-01.
Index Terms
- Intelligent Optimization of Distributed Pipeline Execution in Serverless Platforms: A Predictive Model Approach
Recommendations
Exploiting inherent elasticity of serverless in algorithms with unbalanced and irregular workloads
AbstractFunction-as-a-Service execution model in serverless computing has been successful in running large-scale computations like MapReduce, linear algebra, and machine learning. However, little attention has been given to executing highly-dynamic ...
Highlights- Serverless elasticity is ideally suited for irregular and unbalanced algorithms.
- Cost-performance analysis comparing serverless against cluster based solutions.
- Serverless can have better cost-performance than a Spark cluster of ...
Rescheduling serverless workloads across the cloud-to-edge continuum
AbstractServerless computing was a breakthrough in Cloud computing due to its high elasticity capabilities and fine-grained pay-per-use model offered by the main public Cloud providers. Meanwhile, open-source serverless platforms supporting the FaaS (...
Highlights- Two strategies are given to reschedule workloads in the cloud-to-edge continuum.
- The open-source OSCAR serverless platform has been extended to implement them.
- A use case on wildfire detection involving edge, cloud and FaaS is ...
Self-Provisioning Infrastructures for the Next Generation Serverless Computing
AbstractServerless computing has ushered in a transformative paradigm, with a promise to alleviate developers from the intricacies of infrastructure management. However, current serverless platforms predominantly offer only serverless compute ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
December 2024
46 pages
ISBN:9798400713361
DOI:10.1145/3702634
This work is licensed under a Creative Commons Attribution International 4.0 License.
In-Cooperation
- IFIP
- Usenix
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 02 December 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
WoSC10 '24
WoSC10 '24: 10th International Workshop on Serverless Computing
December 2 - 6, 2024
Hong Kong, Hong Kong
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 51Total Downloads
- Downloads (Last 12 months)51
- Downloads (Last 6 weeks)51
Reflects downloads up to 09 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in