On Different Search Methods for Systematic Literature Reviews and Maps: Experiences from a Literature Search on Validation and Verification of Emergent Behavior

[Background] Systematic literature reviews and maps have become well-established research methods in software engineering research. Of the three commonly suggested and used search methods: manual search, database search, or snowball search; systematic literature reviews and maps typically employ one or a combination of two or three of those as their search strategy. As systematic literature reviews and maps raise a claim to result in a representative set of relevant papers for a certain area of investigation, it is of importance to understand the impact the search strategy has on achieving this goal. [Aim] This paper contributes a study to compare all three search methods. This study aims at providing evidence as to what advantages and disadvantages of these three search methods are. [Method] We conducted three systematic literature reviews on the same topic, which affects multiple software engineering related disciplines, using different search methods, while keeping other parameters like inclusion and exclusion criteria consistent among all three reviews. [Results] Our results show a similar effectiveness for snowball and database search and the highest efficiency for database searches. However, our literature reviews led to three barely overlapping sets of papers, which in turn led to distinct impressions of the same field. [Conclusion] Our results show that the use of a single search method can lead to a set of included papers, which misrepresents the research field under investigation. Hence, particularly when conducting literature reviews that affect different software engineering sub-disciplines and related disciplines, researchers should not just rely on the single most effective and/or efficient search method.