An experiment on software test creation: can EvoSuite help test teaching?
Authors: 
Lucas Santos, Neto Sousa, Laís Gomes, Jannielly Garcia, Fischer FerreiraAuthors Info & Claims
Pages 281 - 290
Abstract
This paper presents the execution of a software testing experiment, where participants were tasked with identifying test cases from a provided software system and implementing the identified test cases. The objective of this study was to investigate students’ perception in test case creation. Participants were divided into two groups: a control group, which received the source code and documentation, and an experimental group, which received, in addition to the source code and documentation, a test suite generated by the EvoSuite tool. Furthermore, participants responded to three questionnaires: one about the project, another about their prior knowledge profile, and one providing feedback on their perception of the experiment and software testing education. Participants made it clear that the EvoSuite tool helped uncover new test scenarios. Additionally, it assisted in identifying new ways to test code segments that participants previously had no prior knowledge of how to access. Overall, the study suggests that the use of code generation tools can be a valuable addition to software testing education, aiding students in developing their testing skills and improving the quality of their tests. This work can benefit both teachers and students in computer science-related courses.
References
[1]
P. Ammann and J. Offutt. 2016. Introduction to Software Testing (2 ed.). Cambridge University Press. https://doi.org/10.1017/9781316771273
[2]
M. Aniche, F. Hermans, and A. van Deursen. 2019. Pragmatic Software Testing Education. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. https://doi.org/10.1145/3287324.3287461
[3]
V. Basili and H. Dieter Rombach. 1988. The TAME Project: Towards Improvement-oriented Software Environments. IEEE Transactions on Software Engineering (TSE) (1988), 758–773.
[4]
F. B. V. Benitti. 2018. A Methodology to Define Learning Objects Granularity: A Case Study in Software Testing. Informatics Educ. 17 (2018), 1–20.
[5]
L. Bijlsma, N. Doorn, H. Passier, H. Pootjes, and S. Stuurman. 2021. How Do Students Test Software Units?. In Proceedings of the 43rd International Conference on Software Engineering: Joint Track on Software Engineering Education and Training. https://doi.org/10.1109/ICSE-SEET52601.2021.00029
[6]
C. K. D. Corte, A. C. Riestin, M. A. G. Silva, E. F. Barbosa, and Jose Carlos Maldonado. 2007. PROGTEST: Ambiente para submissão e avaliação de trabalhos práticos. In XVIII Simpósio Brasileiro de Informática na Educação (SBIE 2007) Workshop sobre Ambientes de Apoio à Aprendizagem de Algoritmos e Programação. São Paulo, SP.
[7]
S. H. Edwards. 2004. Using Software Testing to Move Students from Trial-and-Error to Reflection-in-Action. SIGCSE Bull. 36, 1 (mar 2004), 26–30. https://doi.org/10.1145/1028174.971312
[8]
F. Ferreira, G. Vale, J. P. Diniz, and E. Figueiredo. 2020. On the Proposal and Evaluation of a Test-Enriched Dataset for Configurable Systems. In Proceedings of the 14th International Working Conference on Variability Modelling of Software-Intensive Systems (Magdeburg, Germany) (VaMoS ’20). Association for Computing Machinery, New York, NY, USA, Article 16, 10 pages. https://doi.org/10.1145/3377024.3377045
[9]
T. Ferreira, D. Viana, J. Fernandes, and R. Santos. 2018. Identifying Emerging Topics and Difficulties in Software Engineering Education in Brazil. In Proceedings of the XXXII Brazilian Symposium on Software Engineering. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3266237.3266247
[10]
G. Fraser and A. Arcuri. 2011. EvoSuite: Automatic Test Suite Generation for Object-Oriented Software. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (Szeged, Hungary) (ESEC/FSE ’11). Association for Computing Machinery, New York, NY, USA, 416–419. https://doi.org/10.1145/2025113.2025179
[11]
V. Garousi, M. Felderer, M. Kuhrmann, and K. Herkiloğlu. 2017. What Industry Wants from Academia in Software Testing? Hearing Practitioners’ Opinions. In Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering (Karlskrona, Sweden) (EASE’17). Association for Computing Machinery, New York, NY, USA, 65–69. https://doi.org/10.1145/3084226.3084264
[12]
V. Garousi, A. Rainer, P. Lauvås, and A. Arcuri. 2020. Software-testing education: A systematic literature mapping. Journal of Systems and Software 165 (2020), 110570. https://doi.org/10.1016/j.jss.2020.110570
[13]
A. Goffi, A. Gorla, M. D. Ernst, and M. Pezzè. 2016. Automatic Generation of Oracles for Exceptional Behaviors. In Proceedings of the 25th International Symposium on Software Testing and Analysis (Saarbrücken, Germany) (ISSTA 2016). Association for Computing Machinery, New York, NY, USA, 213–224. https://doi.org/10.1145/2931037.2931061
[14]
B. Gopal, S. Cooper, J. Olmanson, and R. Bockmon. 2021. Student Difficulties in Unit Testing, Integration Testing, and Continuous Integration: An Exploratory Pilot Qualitative Study. Psychology of Programming Interest Group (2021).
[15]
D. Huizinga and A. Kolawa. 2007. Automated Defect Prevention: Best Practices in Software Management.
[16]
A. Joshi, S. Kale, S. Chandel, and D. Pal. 2015. Likert Scale: Explored and Explained. British Journal of Applied Science & Technology 7 (01 2015), 396–403. https://doi.org/10.9734/BJAST/2015/14975
[17]
M. Leppänen, S. Mäkinen, M. Pagels, V.-P. Eloranta, J. Itkonen, M. V. Mäntylä, and T. Männistö. 2015. The highways and country roads to continuous deployment. IEEE Software 32, 2 (Mar 2015), 64–72. https://doi.org/10.1109/MS.2015.50
[18]
N. Lojo and A. Fox. 2022. Teaching Test-Writing As a Variably-Scaffolded Programming Pattern. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1 (Dublin, Ireland) (ITiCSE ’22). Association for Computing Machinery, New York, NY, USA, 498–504. https://doi.org/10.1145/3502718.3524789
[19]
J. Mendes, Y. Costa, K. Frazão, R. Santos, D. Santos, and L. Rivero. 2019. Identificação das Expectativas e Dificuldades de Alunos de Graduação no Ensino de Engenharia de Software. In Anais do XXVII Workshop sobre Educação em Computação (Belém). SBC, Porto Alegre, RS, Brasil, 334–347. https://doi.org/10.5753/wei.2019.6640
[20]
G. G. Moreira, S. Reinehr, A. Malucelli, and F. Van Amstel. 2023. ProTesters: A Board Game for Teaching the Testing Process. In Proceedings of the XXI Brazilian Symposium on Software Quality (Curitiba, Brazil) (SBQS ’22). Association for Computing Machinery, New York, NY, USA, Article 28, 9 pages. https://doi.org/10.1145/3571473.3571503
[21]
E. Oliveira Jr, T. Colanzi, A. Amaral, A. Cordeiro, J. C. Neto, and S. Souza. 2021. Ensino, Aprendizagem e Uso Profissional da UML em Maringá e Região. In Anais do XXIX Workshop sobre Educação em Computação (Evento Online). SBC, Porto Alegre, RS, Brasil, 328–337. https://doi.org/10.5753/wei.2021.15924
[22]
C. Pacheco and M. D. Ernst. 2007. Randoop: Feedback-directed Random Testing for Java. In OOPSLA 2007 Companion, Montreal, Canada. ACM.
[23]
L. Paschoal and S. Souza. 2018. A Survey on Software Testing Education in Brazil. https://doi.org/10.1145/3275245.3275289
[24]
G. Petrović, M. Ivanković, G. Fraser, and R. Just. 2022. Practical Mutation Testing at Scale: A view from Google. IEEE Transactions on Software Engineering 48, 10 (2022), 3900–3912. https://doi.org/10.1109/TSE.2021.3107634
[25]
N. Setiani, R. Ferdiana, P. I. Santosa, and R. Hartanto. 2019. Literature Review on Test Case Generation Approach. In Proceedings of the 2nd International Conference on Software Engineering and Information Management (Bali, Indonesia) (ICSIM 2019). Association for Computing Machinery, New York, NY, USA, 91–95. https://doi.org/10.1145/3305160.3305186
[26]
M. Shahin, M. Ali Babar, and L. Zhu. 2017. Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices. IEEE Access 5 (2017), 3909–3943. https://doi.org/10.1109/ACCESS.2017.2685629
[27]
M. Silva, T. Rocha, and M. Perkusich. 2021. O ensino de engenharia de software no nível superior: um mapeamento sistemático. Revista Principia - Divulgação Científica e Tecnológica do IFPB 1 (04 2021). https://doi.org/10.18265/1517-0306a2021id3874
[28]
D. Souza, J. C. Maldonado, and E. F. Barbosa. 2012. Aspectos de Desenvolvimento e Evolução de um Ambiente de Apoio ao Ensino de Programação e Teste de Software. Brazilian Symposium on Computers in Education (Simpósio Brasileiro de Informática na Educação - SBIE) 23, 1 (2012). https://doi.org/10.5753/cbie.sbie.2012.%p
[29]
A. B. Sánchez, P. Delgado-Pérez, I. Medina-Bulo, and S. Segura. 2022. Mutation Testing in the Wild: Findings from GitHub. Empirical Softw. Engg. 27, 6 (nov 2022), 35 pages. https://doi.org/10.1007/s10664-022-10177-8
[30]
P. H. Valle, E. Barbosa, and J. Maldonado. 2015. CS curricula of the most relevant universities in Brazil and abroad: Perspective of software testing education. 62–68. https://doi.org/10.1109/SIIE.2015.7451649
[31]
S. Vogl, S. Schweikl, G. Fraser, A. Arcuri, J. Campos, and A. Panichella. 2021. EVOSUITE at the SBST 2021 Tool Competition. In 2021 IEEE/ACM 14th International Workshop on Search-Based Software Testing (SBST). IEEE, 28–29.
[32]
Claes W.2014. Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering. In Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE). 1–10.
[33]
H. A. P. Zanetti, M. A. F. Borges, and I. L. M. Ricarte. 2023. ComFAPOO: Método de Ensino de Programação Orientada à Objetos Baseado em Aprendizagem Significativa e Computação Física. Revista Brasileira de Informática na Educação 31 (jan. 2023), 01–30. https://doi.org/10.5753/rbie.2023.2851
Index Terms
- An experiment on software test creation: can EvoSuite help test teaching?
Index terms have been assigned to the content through auto-classification.
Recommendations
A detailed investigation of the effectiveness of whole test suite generation
A common application of search-based software testing is to generate test cases for all goals defined by a coverage criterion (e.g., lines, branches, mutants). Rather than generating one test case at a time for each of these goals individually, whole ...
Empirical Study of Effectiveness of EvoSuite on the SBST 2020 Tool Competition Benchmark
Search-Based Software EngineeringAbstractEvoSuite is a state-of-the-art search-based software testing tool for Java programs and many researchers have applied EvoSuite to achieve high test coverage. However, due to high complexity of object-oriented programs, EvoSuite still suffers ...
Mock object creation for test factoring
PASTE '04: Proceedings of the 5th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineeringTest factoring creates fast, focused unit tests from slow system-wide tests; each new unit test exercises only a subset of the functionality exercised by the system tests. Augmenting a test suite with factored unit tests, and prioritizing the tests, ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2023
391 pages
ISBN:9798400707865
DOI:10.1145/3629479
- Editors:
- Edna Dias Canedo,
- Daniel de Paula Porto,
- Fábio Lúcio Lopes Mendonça,
- Rafael Timóteo de Sousa Júnior,
- Monalessa Perini Barcellos,
- Ismayle Sousa Santos,
- Sheila Reinehr,
- Sergio Soares,
- Uirá Kulesza,
- Érica Ferreira de Souza,
- Adriano Albuquerque,
- Carla Bezerra,
- Rodrigo Santos,
- Alessandro Garcia,
- Simone Dornelas Costa,
- Adolfo Gustavo Serra Seca Neto
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 06 December 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SBQS '23
Acceptance Rates
Overall Acceptance Rate 35 of 99 submissions, 35%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 33Total Downloads
- Downloads (Last 12 months)33
- Downloads (Last 6 weeks)5
Reflects downloads up to 21 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format