Abstract
Minimizing energy consumption when executing Mapreduce jobs is a significant challenge for data centers; however, it traditionally conflicts with the system performance. This paper aims to address this problem by making a trade-off between energy consumption and performance. In this paper, we design an integer linear bi-objective optimization model and propose a two-phase heuristic allocation algorithm to find a high-quality feasible solution. By adopting Non-dominated Sorting Genetic Algorithm II with the feasible solution, we obtain a set of Pareto optimal solutions to minimize both energy consumption and makespan. Finally, we perform experiments on several real workloads to evaluate the solutions produced by our proposed algorithm and analyze the trade-off relationship between energy and makespan. The results show that the Pareto optimal solutions are close to the lower bound obtained by the relaxation of the integer linear bi-objective optimization model and can also assist system manager to make intelligent resource allocation decisions for Mapreduce applications based on the energy efficiency and performance needs of the system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)
Hadoop: http://Hadoop.apache.org/. Accessed 15 Nov 2017
Wirtz, T., Ge, R.: Improving Mapreduce energy efficiency for computation intensive workloads. In: Proceedings on the Green Computing Conference and Workshops, pp. 1–8. IEEE, Orlando (2011)
Goiri, Í., Le, K., Nguyen, T.D., Guitart, J., Torres, J., Bianchini, R.: GreenHadoop: leveraging green energy in data-processing frameworks. In: 7th ACM European conference on Computer Systems, pp. 57–70. ACM, New York (2012)
Johnson, C., Chiu, D.: Hadoop in flight: migrating live MapReduce jobs for power-shifting data centers. In: 9th International Conference on Cloud Computing, pp. 92–99. IEEE (2017)
Liao, B., Tao, Z., Yu, J., Yin, L.T., Guo, G., Guo, B.L.: Energy consumption modeling and optimization analysis for MapReduce. J. Comput. Res. Dev. 53(9), 2107–2131 (2016)
Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R.H., Stoica, I.: Improving MapReduce performance in heterogeneous environments. In: 8th USENIX Symposium on Operating Systems Design and Implementation, p. 7. USENIX, San Diego (2008)
Xiao, B., Wang, Z., Liu, Q., Liu, X.D.: SMK-means: an improved mini batch K-means algorithm based on Mapreduce with big data. Comput. Mater. Continua 56(3), 365–379 (2018)
Ren, Z.J., Wan, J., Shi, W.S., Xu, X.H., Zhou, M.: Workload analysis, implications, and optimization on a production hadoop cluster: a case study on taobao. IEEE Trans. Serv. Comput. 7(2), 307–321 (2014)
Zhang, J.X., Xie, N., Zhang, X.J., Yue, K., Li, W.D., Kumar, D.: Machine learning based resource allocation of cloud computing in auction. Comput. Mater. Continua 56(1), 123–135 (2018)
Chen, C., Xu, Y.F., Zhu, Y.Q., Sun, C.Y.: Online MapReduce scheduling problem of minimizing the makespan. J. Comb. Optim. 33(2), 590–608 (2017)
Friese, R., Brinks, T., Oliver, C., Siegel, H.J., Maciejewski, A.A.: Analyzing the trade-offs between minimizing makespan and minimizing energy consumption in a heterogeneous resource allocation problem. In: 2nd International Conference on Advanced Communications and Computation, pp. 81–89. IARIA XPS, Venice (2012)
Tarplee, K., Friese, R., Maciejewski, A., Siegel, H.J., Chong, E.K.P.: Energy and makespan tradeoffs in heterogeneous computing systems using efficient linear programming techniques. IEEE Trans. Parallel Distrib. Syst. 27(6), 1633–1646 (2016)
Lin, J.C., Leu, F.Y., Chen, Y.: Impact of MapReduce policies on job completion reliability and job energy consumption. IEEE Trans. Parallel Distrib. Syst. 26(5), 1364–1378 (2015)
Mashayekhy, L., Nejad, M.M., Grosu, D., Zhang, Q., Shi, W.S.: Energy-aware scheduling of Mapreduce jobs for big data applications. IEEE Trans. Parallel Distrib. Syst. 26(10), 2720–2733 (2015)
Zhang, X., Li, W., Liu, X., Zhang, X.: A profit-maximum resource allocation approach for Mapreduce in data centers. In: Au, M.H.A., Castiglione, A., Choo, K.-K.R., Palmieri, F., Li, K.-C. (eds.) GPC 2017. LNCS, vol. 10232, pp. 460–474. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-57186-7_34
Li, W.D., Liu, X., Zhang, X.J., Cai, X.B.: A Task-type-based algorithm for the energy-aware profit maximizing scheduling problem in heterogeneous computing systems. In: 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pp. 1107–1110. IEEE, Shenzhen (2015)
Li, W.D., Liu, X., Cai, X.B., Zhang, X.J.: Approximation algorithm for the energy-aware profit maximizing problem in heterogeneous computing systems. J. Parallel Distrib. Comput. 124, 70–77 (2019)
Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)
Graham, R.L.: Bounds on multiprocessing timing anomalies. SIAM J. Appl. Math. 17(2), 416–429 (1969)
Huang, S., Huang, J., Dai, J.Q., Xie, T., Huang, B.: The HiBench benchmark suite: characterization of the MapReduce-based data analysis. In: 26th International Conference on Data Engineering Workshops, pp. 41–51. IEEE, Long Beach (2010)
Braun, T.D., et al.: A comparison of eleven static heuristics for mapping a class of independent tasks onto heterogeneous distributed computing systems. J. Parallel Distrib. Comput. 61(6), 810–837 (2001)
Acknowledgments
This paper was supported by the National Natural Science Foundation of China (Nos. 61762091, 61662088), the Natural Science Foundation of Yunnan Province of China (No. 2017ZZX228), Program for Excellent Young Talents of Yunnan University, and Training Program of National Science Fund for Distinguished Young Scholars.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, X., Liu, X., Li, W., Zhang, X. (2019). Trade-off Between Energy Consumption and Makespan in the Mapreduce Resource Allocation Problem. In: Sun, X., Pan, Z., Bertino, E. (eds) Artificial Intelligence and Security. ICAIS 2019. Lecture Notes in Computer Science(), vol 11633. Springer, Cham. https://doi.org/10.1007/978-3-030-24265-7_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-24265-7_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-24264-0
Online ISBN: 978-3-030-24265-7
eBook Packages: Computer ScienceComputer Science (R0)