More Web Proxy on the site http://driver.im/

research-article

Towards rigorous validation of energy optimisation experiments

Authors:

Mahmoud A. Bokhari,

Brad Alexander,

Markus WagnerAuthors Info & Claims

GECCO '20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference

Pages 1232 - 1240

https://doi.org/10.1145/3377930.3390245

Published: 26 June 2020 Publication History

Abstract

The optimisation of software energy consumption is of growing importance across all scales of modern computing, i.e., from embedded systems to data-centres. Practitioners in the field of Search-Based Software Engineering and Genetic Improvement of Software acknowledge that optimising software energy consumption is difficult due to noisy and expensive fitness evaluations. However, it is apparent from results to date that more progress needs to be made in rigorously validating optimisation results. This problem is pressing because modern computing platforms have highly complex and variable behaviour with respect to energy consumption. To compare solutions fairly we propose in this paper a new validation approach called R3-validation which exercises software variants in a rotated-round-robin order. Using a case study, we present an in-depth analysis of the impacts of changing system states on software energy usage, and we show how R3-validation mitigates these. We compare it with current validation approaches across multiple devices and operating systems, and we show that it aligns best with actual platform behaviour.

References

[1]

Alameldeen, A. R., and Wood, D. A. Variability in architectural simulations of multi-threaded workloads. In Ninth International Symposium on High-Performance Computer Architecture (HPCA) (2003), IEEE, pp. 7--18.

[2]

Blackburn, S. M., Diwan, A., Hauswirth, M., Sweeney, P. F., Amaral, J. N., Brecht, T., Bulej, L., Click, C., Eeckhout, L., Fischmeister, S., et al. The truth, the whole truth, and nothing but the truth: A pragmatic guide to assessing empirical evaluations. ACM Transactions on Programming Languages and Systems (TOPLAS) 38, 4 (2016), 1--20.

Digital Library

[3]

Blackburn, S. M., Garner, R., Hoffmann, C., Khang, A. M., McKinley, K. S., Bentzur, R., Diwan, A., Feinberg, D., Frampton, D., Guyer, S. Z., et al. The dacapo benchmarks: Java benchmarking development and analysis. In ACM SIGPLAN Notices (2006), vol. 41, ACM, pp. 169--190.

Digital Library

[4]

Bokhari, M. A., Alexander, B., and Wagner, M. In-vivo and offline optimisation of energy use in the presence of small energy signals: A case study on a popular android library. In 15th EAI Int. Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous) (2018), ACM, pp. 207--215.

Digital Library

[5]

Bokhari, M. A., Bruce, B. R., Alexander, B., and Wagner, M. Deep Parameter Optimisation on Android Smartphones for Energy Minimisation: A Tale of Woe and a Proof-of-concept. In Genetic and Evolutionary Computation Conference Companion (2017), ACM, pp. 1501--1508.

Digital Library

[6]

Bokhari, M. A., Weng, L., Wagner, M., and Alexander, B. Mind the gap-a distributed framework for enabling energy optimisation on modern smart-phones in the presence of noise, drift, and statistical insignificance. In IEEE Congress on Evolutionary Computation (CEC) (2019), IEEE, pp. 1330--1337.

[7]

Bokhari, M. A., Xia, Y., Zhou, B., Alexander, B., and Wagner, M. Validation of internal meters of mobile android devices. CoRR abs/1701.07095 (2017).

[8]

Bruce, B. R., Petke, J., and Harman, M. Reducing energy consumption using genetic improvement. In Genetic and Evolutionary Computation Conference (GECCO) (2015), ACM, pp. 1327--1334.

Digital Library

[9]

Bruce, B. R., Petke, J., Harman, M., and Barr, E. T. Approximate oracles and synergy in software energy search spaces. IEEE Transactions on Software Engineering 45, 11 (Nov 2019), 1150--1169.

[10]

Burles, N., Bowles, E., Brownlee, A. E. I., Kocsis, Z. A., Swan, J., and Veerapen, N. Object-oriented genetic improvement for improved energy consumption in Google Guava. In Symposium on Search-Based Software Engineering (SSBSE) (2015), Springer, pp. 255--261.

[11]

Curtsinger, C., and Berger, E. D. Stabilizer: statistically sound performance evaluation. In ACM SIGPLAN Notices (2013), vol. 48, ACM, pp. 219--228.

Digital Library

[12]

Georges, A., Eeckhout, L., and Buytaert, D. Java performance evaluation through rigorous replay compilation. In ACM SIGPLAN Notices (2008), vol. 43, ACM, pp. 367--384.

Digital Library

[13]

Gordon, L. Z. M., and Tiwana, B. A power monitor for android-based mobile platforms. Application can be downloaded from: http://ziyang.eecs.umich.edu/projects/powertutor (2012).

[14]

Huang, X., Blackburn, S. M., McKinley, K. S., Moss, J. E. B., Wang, Z., and Cheng, P. The garbage collection advantage: improving program locality. ACM SIGPLAN Notices 39, 10 (2004), 69--80.

Digital Library

[15]

Integrated, M. MAX17047/MAX17050 ModelGauge m3 Fuel Gauge, 2016. Accessed 2 February 2020, https://datasheets.maximintegrated.com/en/ds/MAX17047-MAX17050.pdf.

[16]

Kalibera, T., Bulej, L., and Tuma, P. Automated detection of performance regressions: The mono experience. In 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS) (2005), IEEE, pp. 183--190.

Digital Library

[17]

Kalibera, T., Bulej, L., and Tuma, P. Benchmark precision and random initial state. In Proceedings of the 2005 International Symposium on Performance Evaluation of Computer and Telecommunications Systems (SPECTS) (2005), SCS, pp. 853--862.

[18]

Kalibera, T., and Jones, R. Rigorous benchmarking in reasonable time. ACM SIGPLAN Notices 48, 11 (2013), 63--74.

Digital Library

[19]

Li, D., Tran, A. H., and Halfond, W. G. Making web applications more energy efficient for oled smartphones. In Proceedings of the 36th International Conference on Software Engineering (2014), ACM, pp. 527--538.

Digital Library

[20]

Linares-Vásquez, M., Bavota, G., Cárdenas, C. E. B., Oliveto, R., Di Penta, M., and Poshyvanyk, D. Optimizing Energy Consumption of GUIs in Android Apps: A Multi-objective Approach. In 10th Foundations of Software Engineering (2015), ACM.

Digital Library

[21]

Mytkowicz, T., Diwan, A., Hauswirth, M., and Sweeney, P. F. Producing wrong data without doing anything obviously wrong! ACM SIGARCH Computer Architecture News 37, 1 (2009), 265--276.

[22]

Petke, J., Haraldsson, S. O., Harman, M., Langdon, W. B., White, D. R., and Woodward, J. R. Genetic improvement of software: A comprehensive survey. IEEE Transactions on Evolutionary Computation 22, 3 (June 2018), 415--432.

[23]

Pusukuri, K. K., Gupta, R., and Bhuyan, L. N. Thread tranquilizer: Dynamically reducing performance variation. ACM Transactions on Architecture and Code Optimization (TACO) 8, 4 (2012), 46.

Digital Library

[24]

Sachindran, N., and Moss, J. E. B. Mark-copy: Fast copying gc with less space overhead. In ACM SIGPLAN Notices (2003), vol. 38, ACM, pp. 326--343.

Digital Library

[25]

Schulte, E., Dorn, J., Harding, S., Forrest, S., and Weimer, W. Post-compiler software optimization for reducing energy. In 19th Int. Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) (2014), ACM, pp. 639--652.

Digital Library

[26]

Sillars, D. High Performance Android Apps: Improve ratings with speed, optimizations, and testing. O'Reilly Media, Inc., 2015.

[27]

Tarkoma, S., Siekkinen, M., Lagerspetz, E., and Xiao, Y. Smartphone energy consumption: modeling and optimization. Cambridge University Press, 2014.

[28]

Vargha, A., and Delaney, H. D. A critique and improvement of the cl common language effect size statistics of mcgraw and wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101--132.

[29]

White, D. R. Genetic programming for low-resource systems. PhD thesis, University of York, 2009.

[30]

Xu, F., Liu, Y., Li, Q., and Zhang, Y. V-edge: Fast self-constructive power modeling of smartphones based on battery voltage dynamics. In Presented as part of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI) (2013), pp. 43--55.

[31]

Zhang, L., Tiwana, B., Qian, Z., Wang, Z., Dick, R. P., Mao, Z. M., and Yang, L. Accurate online power estimation and automatic battery behavior based power model generation for smartphones. In Hardware/Software Codesign and System Synthesis (2010), ACM, pp. 105--114.

Cited By

Kuepper JWu DErbsen AGross JConoly OSun CTian SChlipala AChuengsatiansup CGenkin DWagner MYarom YGrundy J(2023)CryptOpt: Automatic Optimization of Straightline CodeProceedings of the 45th International Conference on Software Engineering: Companion Proceedings10.1109/ICSE-Companion58688.2023.00042(141-145)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE-Companion58688.2023.00042
Alofi ABokhari MBahsoon RHendley R(2022)Optimizing the Energy Consumption of Blockchain-Based Systems Using Evolutionary Algorithms: A New Problem FormulationIEEE Transactions on Sustainable Computing10.1109/TSUSC.2022.31604917:4(910-922)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TSUSC.2022.3160491
Hort MKechagia MSarro FHarman M(2022)A Survey of Performance Optimization for Mobile ApplicationsIEEE Transactions on Software Engineering10.1109/TSE.2021.307119348:8(2879-2904)Online publication date: 1-Aug-2022
https://doi.org/10.1109/TSE.2021.3071193
Show More Cited By

Index Terms

Towards rigorous validation of energy optimisation experiments
1. Hardware
  1. Power and energy
    1. Energy generation and storage
      1. Batteries
2. Software and its engineering
  1. Software creation and management
    1. Search-based software engineering
  2. Software notations and tools
    1. Software maintenance tools

Recommendations

In-vivo and offline optimisation of energy use in the presence of small energy signals: A case study on a popular Android library
MobiQuitous '18: Proceedings of the 15th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services

Energy demands of applications on mobile platforms are increasing. As a result, there has been a growing interest in optimising their energy efficiency. As mobile platforms are fast-changing, diverse and complex, the optimisation of energy use is a non-...
Energy Optimisation via Genetic Improvement: A SBSE technique for a new era in Software Development
GECCO Companion '15: Proceedings of the Companion Publication of the 2015 Annual Conference on Genetic and Evolutionary Computation

The discipline of Software Engineering has arisen during a time in which developers rarely concerned themselves with the energy efficiency of their software. Due to the growth in both mobile devices and large server clusters this period is undoubtedly ...
Genetic improvement of software efficiency: the curse of fitness estimation
GECCO '20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion

Many challenges arise in the application of Genetic Improvement (GI) of Software to improve non-functional requirements of software such as energy use and run-time. These challenges are mainly centred around the complexity of the search space and the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference

June 2020

1349 pages

ISBN:9781450371285

DOI:10.1145/3377930

General Chair:
Carlos Artemio Coello Coello
CINVESTAV-IPN

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

GECCO '20

Sponsor:

SIGEVO

GECCO '20: Genetic and Evolutionary Computation Conference

July 8 - 12, 2020

Cancún, Mexico

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
140
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)1

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kuepper JWu DErbsen AGross JConoly OSun CTian SChlipala AChuengsatiansup CGenkin DWagner MYarom YGrundy J(2023)CryptOpt: Automatic Optimization of Straightline CodeProceedings of the 45th International Conference on Software Engineering: Companion Proceedings10.1109/ICSE-Companion58688.2023.00042(141-145)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE-Companion58688.2023.00042
Alofi ABokhari MBahsoon RHendley R(2022)Optimizing the Energy Consumption of Blockchain-Based Systems Using Evolutionary Algorithms: A New Problem FormulationIEEE Transactions on Sustainable Computing10.1109/TSUSC.2022.31604917:4(910-922)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TSUSC.2022.3160491
Hort MKechagia MSarro FHarman M(2022)A Survey of Performance Optimization for Mobile ApplicationsIEEE Transactions on Software Engineering10.1109/TSE.2021.307119348:8(2879-2904)Online publication date: 1-Aug-2022
https://doi.org/10.1109/TSE.2021.3071193
Bokhari MAlexander B(2022)A Hybrid Distributed EA Approach for Energy Optimisation on SmartphonesEmpirical Software Engineering10.1007/s10664-022-10188-527:6Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10664-022-10188-5

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents