[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Automated testing of metamodels and code co-evolution

  • Regular Paper
  • Published:
Software and Systems Modeling Aims and scope Submit manuscript

Abstract

Metamodels are cornerstone in MDE. They define the different domain concepts and the relations between them. A metamodel is also used to generate concrete artifacts such as code. Developers then rely on the generated code to build their language services and tooling, e.g., editors, checkers. To check the behavior of their client code, developers write or generate unit tests. As metamodels evolve between releases, the generated code is automatically updated. As a consequence, the additional developers’ code is impacted and is co-evolved accordingly for each release. However, there is no guarantee that the co-evolution of the code is performed correctly. One way to do so is to rerun all the tests after each code co-evolution, which is expensive and time-consuming. This paper proposes an automatic solution for tracing impacted tests due to metamodel evolution. Thus, we end up matching metamodel changes with impacted code methods and their corresponding tests in both the original and evolved versions of a given project. After that, we map the two versions of the impacted tests and compare them to analyze the behavior of the code before and after its evolution due to the metamodel evolution. In particular, we implemented an Eclipse plug-in that allows tracing, mapping, execution, and reporting back the results to the developers for easier in-depth analysis of the effect of metamodel evolutions rather than analyzing the whole test suite. We first ran a user study experiment to gain evidence on the difficulty or not of the manual task of tracing impacted tests. We found that manually tracing the tests impacted by the evolution of the metamodel is a hard and error-prone task. Not only the participants could not trace all tests, but they even wrongly traced non-impacted tests. We then evaluated our approach on 18 Eclipse projects from OCL, Modisco, Papyrus, and EMF over several evolved versions of metamodels. For the 14 projects without manual tests, we generated a test suite for each release with the state-of-the-art tool EvoSuite. The results show that we successfully traced the impacted tests automatically by selecting 1608 out of 34,612 tests due to 473 metamodel changes. When running the traced tests before and after co-evolution, we observed cases indicating possibly both behaviorally correct and incorrect code co-evolution. Finally, we reached gains representing, on average, a reduction of 88% in the number of tests and 84% in the execution time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5

Similar content being viewed by others

Notes

  1. https://www.eclipse.org/modeling/mdt/downloads/?project=uml2.

  2. https://www.eclipse.org/bpmn2-modeler/.

  3. https://git.eclipse.org/r/plugins/gitiles/modisco/org.eclipse.modisco/+/refs/tags/0.12.1/org.eclipse.modisco.infra.discovery.benchmark/model/benchmark.ecore.

  4. Note that the knowledge about the generated code elements from the metamodel elements (e.g., getter/setter for EAttribute, class/interface for EClass, etc.) is so far hard-coded in the implementation of Algorithm 1. The mappings must be provided for our approach to be able to trace the tests..

  5. Eclipse Java development tools (JDT): https://www.eclipse.org/jdt/core/.

  6. https://junit.org/junit4/javadoc/4.13/org/junit/runner/Runner.html.

  7. https://figshare.com/s/b6251b9e47fa82983ce5.

  8. http://www-list.cea.fr/en/.

  9. Between ‘Useless–Little useful–Somewhat useful–Very useful–Extremely useful.’

  10. Between ‘Very unlikely–Unlikely–Somewhat likely–Likely–Very likely.’

  11. https://infinitest.github.io/doc/eclipse.

References

  1. Kent, S.: Model driven engineering. In: Butler, M., Petre, L., Sere, K. (eds.) Integrated Formal Methods, pp. 286–298. Springer, Berlin, Heidelberg (2002)

    Chapter  Google Scholar 

  2. Hutchinson, J., Whittle, J., Rouncefield, M., Kristoffersen, S.: Empirical assessment of mde in industry. In: Proceedings of the 33rd International Conference on Software Engineering, pp. 471–480 (2011). ACM

  3. Hutchinson, J., Rouncefield, M., Whittle, J.: Model-driven engineering practices in industry. In: Proceedings of the 33rd International Conference on Software Engineering, pp. 633–642 (2011). ACM

  4. Tolvanen, J.-P., Kelly, S.: Metaedit+: defining and using integrated domain-specific modeling languages. In: The 24th ACM SIGPLAN Conference Companion on OOPSLA, pp. 819–820 (2009)

  5. Cabot, J., Gogolla, M.: In: Bernardo, M., Cortellessa, V., Pierantonio, A. (eds.) Object Constraint Language (OCL): a definitive guide, pp. 58–90. Springer, Berlin, Heidelberg (2012). https://doi.org/10.1007/978-3-642-30982-3_3

  6. Steinberg, D., Budinsky, F., Merks, E., Paternostro, M.: EMF: eclipse modeling framework. pearson education (2008)

  7. Riedl-Ehrenleitner, M., Demuth, A., Egyed, A.: Towards model-and-code consistency checking. In: 2014 IEEE 38th Annual Computer Software and Applications Conference, pp. 85–90 (2014). IEEE

  8. Pham, V.C., Radermacher, A., Gerard, S., Li, S.: Bidirectional mapping between architecture model and code for synchronization. In: 2017 IEEE International Conference on Software Architecture (ICSA), pp. 239–242 (2017). IEEE

  9. Jongeling, R., Fredriksson, J., Ciccozzi, F., Cicchetti, A., Carlson, J.: Towards consistency checking between a system model and its implementation. In: Int. Conf. on Systems Modelling and Management, pp. 30–39 (2020). Springer

  10. Jongeling, R., Fredriksson, J., Ciccozzi, F., Carlson, J., Cicchetti, A.: Structural consistency between a system model and its implementation: a design science study in industry. In: ECMFA (2022)

  11. Zaheri, M., Famelis, M., Syriani, E.: Towards checking consistency-breaking updates between models and generated artifacts. In: 2021 ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C), pp. 400–409 (2021). IEEE

  12. Yu, Y., Lin, Y., Hu, Z., Hidaka, S., Kato, H., Montrieux, L.: Maintaining invariant traceability through bidirectional transformations. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 540–550 (2012). IEEE

  13. Khelladi, D.E., Combemale, B., Acher, M., Barais, O., Jézéquel, J.-M.: Co-evolving code with evolving metamodels. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. ICSE ’20, pp. 1496–1508 (2020)

  14. Khelladi, D.E., Combemale, B., Acher, M., Barais, O.: On the power of abstraction: a model-driven co-evolution approach of software code. In: 2020 IEEE/ACM 42st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER) (2020)

  15. Falleri, J.-R., Morandat, F., Blanc, X., Martinez, M., Monperrus, M.: Fine-grained and accurate source code differencing. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, pp. 313–324 (2014)

  16. Fraser, G., Arcuri, A.: Evosuite: automatic test suite generation for object-oriented software. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, pp. 416–419 (2011)

  17. Kebaili, Z.K., Khelladi, D.E., Acher, M., Barais, O.: Towards leveraging tests to identify impacts of metamodel and code co-evolution. In: Cabanillas, C., Pérez, F. (eds.) Intell. Inf. Syst., pp. 129–137. Springer, Cham (2023)

    Google Scholar 

  18. McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verif. Reliab. 14(2), 105–156 (2004)

    Article  Google Scholar 

  19. Beyer, D.: Advances in automatic software testing: Test-comp 2022. In: FASE, pp. 321–335 (2022)

  20. Richters, M., Gogolla, M.: Validating uml models and ocl constraints. In: International Conference on the Unified Modeling Language, pp. 265–277 (2000). Springer

  21. Leung, H.K., White, L.: Insights into regression testing (software testing). In: Proceedings. Conference on Software Maintenance-1989, pp. 60–69 (1989). IEEE

  22. Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab. 22(2), 67–120 (2012)

    Article  Google Scholar 

  23. Wong, W.E., Horgan, J.R., London, S., Agrawal, H.: A study of effective regression testing in practice. In: PROCEEDINGS The Eighth International Symposium On Software Reliability Engineering, pp. 264–274 (1997). IEEE

  24. Le Dilavrec, Q., Khelladi, D.E., Blouin, A., Jézéquel, J.-M.: Untangling spaghetti of evolutions in software histories to identify code and test co-evolutions. In: 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 206–216 (2021). IEEE

  25. Zaidman, A., Van Rompaey, B., Van Deursen, A., Demeyer, S.: Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Empir. Softw. Eng. 16, 325–364 (2011)

    Article  Google Scholar 

  26. Mens, T.: Introduction and Roadmap: History and Challenges of Software Evolution, pp. 1–11. Springer, Berlin, Heidelberg (2008). https://doi.org/10.1007/978-3-540-76440-3_1

  27. Herrmannsdoerfer, M., Vermolen, S.D., Wachsmuth, G.: An extensive catalog of operators for the coupled evolution of metamodels and models. Lecture Notes in Computer Science 6563 LNCS, 163–182 (2011)

  28. Vermolen, S.D., Wachsmuth, G., Visser, E.: Reconstructing complex metamodel evolution. In: Sloane, A., Aßmann, U. (eds.) Software Language Engineering, pp. 201–221. Springer, Berlin, Heidelberg (2012)

    Chapter  Google Scholar 

  29. Khelladi, D.E., Hebig, R., Bendraou, R., Robin, J., Gervais, M.-P.: Detecting complex changes during metamodel evolution. In: CAISE, pp. 263–278 (2015). Springer

  30. Alter, S.: Work system theory: A bridge between business and IT views of systems. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 9097, 520–521 (2015). https://doi.org/10.1007/978-3-319-19069-3

    Article  Google Scholar 

  31. Williams, J.R., Paige, R.F., Polack, F.A.: Searching for model migration strategies. In: Proceedings of the 6th International Workshop on Models and Evolution, pp. 39–44 (2012). ACM

  32. Cicchetti, A., Di Ruscio, D., Pierantonio, A.: Managing dependent changes in coupled evolution. In: International Conference on Theory and Practice of Model Transformations, pp. 35–51 (2009). Springer

  33. Langer, P., Wimmer, M., Brosch, P., Herrmannsdörfer, M., Seidl, M., Wieland, K., Kappel, G.: A posteriori operation detection in evolving software models. J. Syst. Softw. 86(2), 551–566 (2013)

    Article  Google Scholar 

  34. Khelladi, D.E., Hebig, R., Bendraou, R., Robin, J., Gervais, M.P.: Detecting complex changes and refactorings during (Meta)model evolution. Inf. Syst. 62, 220–241 (2016). https://doi.org/10.1016/j.is.2016.05.002

    Article  Google Scholar 

  35. Iovino, L., Pierantonio, A., Malavolta, I.: On the impact significance of metamodel evolution in MDE. J. Object Technol. 11(3), 1–3 (2012)

    Article  Google Scholar 

  36. MDT: Model Development Tools. Object Constraints Language (OCL). http://www.eclipse.org/modeling/mdt/?project=ocl (2015)

  37. MDT: Model Development Tools. MoDisco. http://www.eclipse.org/modeling/mdt/?project=modisco (2015)

  38. MDT: Model Development Tools. Papyrus. http://www.eclipse.org/modeling/mdt/?project=papyrus (2015)

  39. EMF, E.: Eclipse Modeling Framework (EMF). https://github.com/eclipse-emf/org.eclipse.emf (2020)

  40. Roslan, M.F., Rojas, J.M., McMinn, P.: An Empirical Comparison of EvoSuite and DSpot for Improving Developer-Written Test Suites with Respect to Mutation Score. In: Papadakis, M., Vergilio, S.R. (eds.) Search-Based Software Engineering, pp. 19–34. Springer, Cham (2022)

    Chapter  Google Scholar 

  41. Herculano, W.B.R., Alves, E.L.G., Mongiovi, M.: Generated tests in the context of maintenance tasks: A series of empirical studies. IEEE Access 10, 121418–121443 (2022). https://doi.org/10.1109/ACCESS.2022.3222803

    Article  Google Scholar 

  42. Roziere, B., Zhang, J.M., Charton, F., Harman, M., Synnaeve, G., Lample, G.: Leveraging automated unit tests for unsupervised code translation. arXiv preprint arXiv:2110.06773 (2021)

  43. Gruber, M., Roslan, M.F., Parry, O., Scharnböck, F., McMinn, P., Fraser, G.: Do automatic test generation tools generate flaky tests? (2023)

  44. Danglot, B., Vera-Perez, O., Yu, Z., Zaidman, A., Monperrus, M., Baudry, B.: A snowballing literature study on test amplification. Journal of Systems and Software 157, 110398 (2019) https://doi.org/10.1016/j.jss.2019.110398

  45. Rojas, J.M., Fraser, G., Arcuri, A.: Seeding strategies in search-based unit test generation. Softw. Test. Verif. Reliab. 26(5), 366–401 (2016). https://doi.org/10.1002/stvr.1601

    Article  Google Scholar 

  46. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in software engineering. Springer (2012)

  47. OMG: Object Management Group. Unified Modeling Language (UML). http://www.omg.org/spec/UML/ (2015)

  48. OMG: Object Management Group. Business Process Model And Notation (BPMN). https://www.omg.org/spec/BPMN/2.0/About-BPMN/ (2015)

  49. Kessentini, W., Alizadeh, V.: Interactive metamodel/model co-evolution using unsupervised learning and multi-objective search. In: Proceedings of the 23rd ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, pp. 68–78 (2020)

  50. Kessentini, W., Wimmer, M., Sahraoui, H.: Integrating the designer in-the-loop for metamodel/model co-evolution via interactive computational search. In: Proceedings of the 21th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, pp. 101–111 (2018). ACM

  51. Kessentini, W., Sahraoui, H., Wimmer, M.: Automated metamodel/model co-evolution: A search-based approach. Inf. Softw. Technol. 106, 49–67 (2019)

    Article  Google Scholar 

  52. Cicchetti, A., Di Ruscio, D., Eramo, R., Pierantonio, A.: Automating co-evolution in model-driven engineering. In: 2008 12th International IEEE Enterprise Distributed Object Computing Conference, pp. 222–231 (2008). IEEE

  53. Herrmannsdoerfer, M., Benz, S., Juergens, E.: Cope-automating coupled evolution of metamodels and models. In: ECOOP, vol. 9, pp. 52–76 (2009). Springer

  54. Garcés, K., Jouault, F., Cointe, P., Bézivin, J.: Managing model adaptation by precise detection of metamodel changes. In: Model Driven Architecture-Foundations and Applications: 5th European Conference, ECMDA-FA 2009, Enschede, The Netherlands, June 23-26, 2009. Proceedings 5, pp. 34–49 (2009). Springer

  55. Wachsmuth, G.: Metamodel adaptation and model co-adaptation. In: ECOOP, vol. 7, pp. 600–624 (2007). Springer

  56. Paige, R.F., Matragkas, N., Rose, L.M.: Evolving models in model-driven engineering: State-of-the-art and future challenges. J. Syst. Softw. 111, 272–280 (2016)

    Article  Google Scholar 

  57. Hebig, R., Khelladi, D.E., Bendraou, R.: Approaches to co-evolution of metamodels and models: A survey. IEEE Trans. Software Eng. 43(5), 396–414 (2016)

    Article  Google Scholar 

  58. Demuth, A., Riedl-Ehrenleitner, M., Lopez-Herrejon, R.E., Egyed, A.: Co-evolution of metamodels and models through consistent change propagation. J. Syst. Softw. 111, 281–297 (2016)

    Article  Google Scholar 

  59. Cherfa, E., Mesli-Kesraoui, S., Tibermacine, C., Sadou, S., Fleurquin, R.: Identifying metamodel inaccurate structures during metamodel/constraint co-evolution. In: 2021 ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems (MODELS), pp. 24–34 (2021). IEEE

  60. Batot, E., Kessentini, W., Sahraoui, H., Famelis, M.: Heuristic-based recommendation for metamodel—ocl coevolution. In: 2017 ACM/IEEE 20th International Conference on Model Driven Engineering Languages and Systems (MODELS), pp. 210–220 (2017). IEEE

  61. Khelladi, D.E., Hebig, R., Bendraou, R., Robin, J., Gervais, M.-P.: Metamodel and constraints co-evolution: A semi automatic maintenance of ocl constraints. In: International Conference on Software Reuse, pp. 333–349 (2016). Springer

  62. Khelladi, D.E., Bendraou, R., Hebig, R., Gervais, M.-P.: A semi-automatic maintenance and co-evolution of ocl constraints with (meta) model evolution. J. Syst. Softw. 134, 242–260 (2017)

    Article  Google Scholar 

  63. Correa, A., Werner, C.: Refactoring object constraint language specifications. Software & Systems Modeling 6(2), 113–138 (2007)

    Article  Google Scholar 

  64. Kusel, A., Etzlstorfer, J., Kapsammer, E., Retschitzegger, W., Schoenboeck, J., Schwinger, W., Wimmer, M.: Systematic co-evolution of ocl expressions. In: 11th APCCM 2015, vol. 27, p. 30 (2015)

  65. Kessentini, W., Sahraoui, H., Wimmer, M.: Automated co-evolution of metamodels and transformation rules: A search-based approach. In: International Symposium on Search Based Software Engineering, pp. 229–245 (2018). Springer

  66. Khelladi, D.E., Kretschmer, R., Egyed, A.: Change propagation-based and composition-based co-evolution of transformations with evolving metamodels. In: Proceedings of the 21th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems, pp. 404–414 (2018). ACM

  67. Garcés, K., Vara, J.M., Jouault, F., Marcos, E.: Adapting transformations to metamodel changes via external transformation composition. Softw. Syst. Model. 13(2), 789–806 (2014)

    Article  Google Scholar 

  68. García, J., Diaz, O., Azanza, M.: Model transformation co-evolution: A semi-automatic approach. In: Software Language Engineering: 5th International Conference, SLE 2012, Dresden, Germany, September 26-28, 2012, Revised Selected Papers 5, pp. 144–163 (2013). Springer

  69. Kusel, A., Etzlstorfer, J., Kapsammer, E., Retschitzegger, W., Schwinger, W., Schonbock, J.: Consistent co-evolution of models and transformations. In: ACM/IEEE 18th MODELS, pp. 116–125 (2015)

  70. Henkel, J., Diwan, A.: Catchup! capturing and replaying refactorings to support api evolution. In: Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005., pp. 274–283 (2005). IEEE

  71. Nguyen, H.A., Nguyen, T.T., Wilson, G., Jr., Nguyen, A.T., Kim, M., Nguyen, T.N.: A graph-based approach to API usage adaptation. ACM Sigplan Notices 45(10), 302–321 (2010)

    Article  Google Scholar 

  72. Dagenais, B., Robillard, M.P.: Recommending adaptive changes for framework evolution. ACM Trans. Softw. Eng. Methodol. (TOSEM) 20(4), 19 (2011)

    Article  Google Scholar 

  73. Andersen, J., Lawall, J.L.: Generic patch inference. Autom. Softw. Eng. 17(2), 119–148 (2010)

    Article  Google Scholar 

  74. Engström, E., Runeson, P., Skoglund, M.: A systematic review on regression test selection techniques. Inf. Softw. Technol. 52(1), 14–30 (2010)

    Article  Google Scholar 

  75. Mansour, P., El-Fakih, K.: Natural optimization algorithms for optimal regression testing. In: Proceedings Twenty-First Annual International Computer Software and Applications Conference (COMPSAC’97), pp. 511–514 (1997). IEEE

  76. Gupta, R., Harrold, M.J., Soffa, M.L.: An approach to regression testing using slicing. In: ICSM 92, 299–308 (1992)

    Google Scholar 

  77. Willmor, D., Embury, S.M.: A safe regression test selection technique for database-driven applications. In: 21st IEEE International Conference on Software Maintenance (ICSM’05), pp. 421–430 (2005). IEEE

  78. Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: A survey. Softw. Test. Verif. Reliab. 22(2), 67–120 (2012). https://doi.org/10.1002/stv.430

  79. Gligoric, M., Eloussi, L., Marinov, D.: Ekstazi: Lightweight test selection. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 2, pp. 713–716 (2015). https://doi.org/10.1109/ICSE.2015.230

  80. Ducasse, S., Lanza, M., Tichelaar, S.: Moose: an extensible language-independent environment for reengineering object-oriented systems. In: Proceedings of the Second International Symposium on Constructing Software Engineering Tools (CoSET 2000), vol. 4 (2000)

  81. Ge, X., Murphy-Hill, E.: Manual refactoring changes with automated refactoring validation. In: Proceedings of the 36th International Conference on Software Engineering, pp. 1095–1105 (2014)

Download references

Acknowledgements

The research leading to these results has received funding from the RENNES METROPOLE under Grant AIS no. 19C0330 and from ANR agency under Grant ANR JCJC MC-\(\hbox {EVO}^{2}\) 204687.

Author information

Author notes

  1. Djamel Eddine Khelladi, Mathieu Acher and Olivier Barais have contributed equally to this work.

Authors and Affiliations

  1. CNRS, Univ. Rennes 1, IRISA, INRIA, 35000, Rennes, France

    Zohra Kaouter Kebaili & Djamel Eddine Khelladi

  2. INSA, IUF, IRISA, Inria, 35000, Rennes, France

    Mathieu Acher

  3. Univ. Rennes 1, IRISA, INRIA, 35000, Rennes, France

    Olivier Barais

Authors
  1. Zohra Kaouter Kebaili
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Djamel Eddine Khelladi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mathieu Acher
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Olivier Barais
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zohra Kaouter Kebaili.

Additional information

Communicated by Alfonso Pierantonio.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kebaili, Z.K., Khelladi, D.E., Acher, M. et al. Automated testing of metamodels and code co-evolution. Softw Syst Model (2024). https://doi.org/10.1007/s10270-024-01245-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10270-024-01245-2

Keywords

Search

Navigation