Static and Dynamic Comparison of Mutation Testing Tools for Python

Context: Mutation testing is a rigorous approach for assessing the quality of test suites by injecting faults (i.e., mutants) into software under test. Tools, such as CosmicRay and Mutpy, are examples of Mutation Testing tools for Python software programs. Problem: With different Python mutation testing tools, comparative analysis is lacking to evaluate their effectiveness in different usage scenarios. Furthermore, the evolution of these tools makes continuous evaluation of their functionalities and characteristics necessary. Method: In this work, we evaluate (statically and dynamically) four Python mutation testing tools, namely CosmicRay, MutPy, MutMut, and Mutatest. In static evaluation, we introduce a comparison framework, adapted from one previously applied to Java tools, and collected information from tool documentation and developer surveys. For dynamic evaluation, we use tests built based on those produced by Pynguin, which are improved through the application of Large Language Models (LLMs) and manual analyses. Then, the adequate test suites were cross-tested among different tools to evaluate their effectiveness in killing mutants each other. Results: Our findings reveal that CosmicRay offers superior functionalities and customization options for mutant generation compared to its counterparts. Although CosmicRay’s performance was slightly lower than MutPy in the dynamic tests, its recent updates and active community support highlight its potential for future enhancements. Cross-examination of the test suites further shows that mutation scores varied narrowly among tools, with a slight emphasis on MutPy as the most effective mutant fault model.