Abstract
End-to-end (E2E) testing, a technique employed to assure quality of web applications, is cost-effective only if the test suite is not flaky. Flaky test scripts produce non-deterministic results that undermine testers’ trust and thus the usefulness of the entire test suite.
Recently, we were involved in the refactoring of an existing automated flaky E2E test suite for a large Web application. In this paper, we report on our experience. During the refactoring, we have computed the effort made and formalized the procedure we followed in algorithmic way so that our experience can also be of help to other developers/testers. Our procedure allowed to reduce the flakiness to virtually zero w.r.t. the original flaky test suite. Moreover, as a positive side effect, the execution time of the test suite has been reduced by of the 57%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Computed using Locmetrics (https://www.cheonghyun.com/blog/120).
- 2.
Documentation for the ExpectedConditions class
- 3.
Jenkins website https://www.jenkins.io/.
References
Page Object Model. https://www.selenium.dev/documentation/en/guidelines_and_ recommendations/page_object_models/
Testng testing framework (2021). https://testng.org/doc/. Accessed 08 Apr 2021
Ahmad, A., Leifler, O., Sandahl, K.: Empirical analysis of factors and their effect on test flakiness - practitioners’ perceptions. arXiv:1906.00673 (2019)
Bell, J., Legunsen, O., Hilton, M., Eloussi, L., Yung, T., Marinov, D.: DeFlaker: automatically detecting flaky tests. In: 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), pp. 433–444 (2018). https://doi.org/10.1145/3180155.3180164
Eck, M., Palomba, F., Castelluccio, M., Bacchelli, A.: Understanding flaky tests: the developer’s perspective. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, pp. 830–840. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3338906.3338945
Garousi, V., Felderer, M.: Developing, verifying, and maintaining high-quality automated test scripts. IEEE Softw. 33, 68–75 (2016)
Leotta, M., Biagiola, M., Ricca, F., Ceccato, M., Tonella, P.: A family of experiments to assess the impact of page object pattern in web test suite development. In: Proceedings of 13th IEEE International Conference on Software Testing, Verification and Validation (ICST 2020), pp. 263–273. IEEE (2020). https://doi.org/10.1109/ICST46399.2020.00035
Leotta, M., Clerissi, D., Ricca, F., Spadaro, C.: Comparing the maintainability of selenium webdriver test suites employing different locators: a case study. In: Proceedings of 1st International Workshop on Joining AcadeMiA and Industry Contributions to testing Automation (ISSTA 2013 Workshops), pp. 53–58. ACM (2013). https://doi.org/10.1145/2489280.2489284
Leotta, M., Clerissi, D., Ricca, F., Spadaro, C.: Improving test suites maintainability with the page object pattern: an industrial case study. In: Proceedings of 6th International Conference on Software Testing, Verification and Validation Workshops (ICST 2013 Workshops), pp. 108–113. IEEE (2013). https://doi.org/10.1109/ICSTW.2013.19
Leotta, M., Clerissi, D., Ricca, F., Tonella, P.: Capture-replay vs. programmable web testing: an empirical assessment during test case evolution. In: Proceedings of 20th Working Conference on Reverse Engineering (WCRE 2013), pp. 272–281. IEEE (2013). https://doi.org/10.1109/WCRE.2013.6671302
Leotta, M., Clerissi, D., Ricca, F., Tonella, P.: Approaches and tools for automated end-to-end web testing. Adv. Comput. 101, 193–237 (2016). https://doi.org/10.1016/bs.adcom.2015.11.007
Leotta, M., Stocco, A., Ricca, F., Tonella, P.: PESTO: automated migration of DOM-based web tests towards the visual approach. J. Softw. Test. Verif. Reliabil. (STVR) 28(4), e1665 (2018). https://doi.org/10.1002/stvr.1665
Luo, Q., Hariri, F., Eloussi, L., Marinov, D.: An empirical analysis of flaky tests. In: Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, pp. 643–653. ACM (2014). https://doi.org/10.1145/2635868.2635920
Moran, J., Augusto Alonso, C., Bertolino, A., de la Riva, C., Tuya, J.: FlakyLoc: flakiness localization for reliable test suites in web applications. J. Web Eng. (2020). https://doi.org/10.13052/jwe1540-9589.1927
Presler-Marshall, K., Horton, E., Heckman, S., Stolee, K.T.: Wait wait. No, tell me: analyzing selenium configuration effects on test flakiness. In: Proceedings of the 14th International Workshop on Automation of Software Test, AST 2019, pp. 7–13. IEEE Press (2019). https://doi.org/10.1109/AST.2019.000-1
Rafi, D., Moses, K., Petersen, K., Mäntylä, M.: Benefits and limitations of automated software testing: systematic literature review and practitioner survey, pp. 36–42 (2012). https://doi.org/10.1109/IWAST.2012.6228988
Ricca, F., Stocco, A.: Web test automation: insights from the grey literature. In: Bureš, T., et al. (eds.) SOFSEM 2021. LNCS, vol. 12607, pp. 472–485. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67731-2_35
Romano, A., Song, Z., Grandhi, S., Yang, W., Wang, W.: An empirical analysis of UI-based flaky tests. arXiv:2103.02669 (2021)
SeleniumHQ web browser automation (2021). https://www.selenium.dev/. Accessed 08 Apr 2021
Shi, A., Lam, W., Oei, R., Xie, T., Marinov, D.: iFixFlakies: a framework for automatically fixing order-dependent flaky tests. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2019, pp. 545–555. Association for Computing Machinery, New York (2019). https://doi.org/10.1145/3338906.3338925
Stocco, A., Leotta, M., Ricca, F., Tonella, P.: APOGEN: automatic page object generator for web testing. Softw. Qual. J. (SQJ) 25(3), 1007–1039 (2017). https://doi.org/10.1007/s11219-016-9331-9
Zolfaghari, B., Parizi, R.M., Srivastava, G., Hailemariam, Y.: Root causing, detecting, and fixing flaky tests: state of the art and future roadmap. Softw.: Pract. Exp. 51, 851–867 (2020)
Acknowledgement
This work was carried out in the context of a collaboration between DIBRIS (University of Genova) and Printo (Gaslini Hospital). We want to show our gratitude to Dr. Nicolino Ruperto, Senior scientist of Printo for the support provided.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Olianas, D., Leotta, M., Ricca, F., Villa, L. (2021). Reducing Flakiness in End-to-End Test Suites: An Experience Report. In: Paiva, A.C.R., Cavalli, A.R., Ventura Martins, P., Pérez-Castillo, R. (eds) Quality of Information and Communications Technology. QUATIC 2021. Communications in Computer and Information Science, vol 1439. Springer, Cham. https://doi.org/10.1007/978-3-030-85347-1_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-85347-1_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-85346-4
Online ISBN: 978-3-030-85347-1
eBook Packages: Computer ScienceComputer Science (R0)