More Web Proxy on the site http://driver.im/

research-article

Systematic literature studies: database searches vs. backward snowballing

Authors:

Samireh Jalali,

Claes WohlinAuthors Info & Claims

ESEM '12: Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement

Pages 29 - 38

https://doi.org/10.1145/2372251.2372257

Published: 19 September 2012 Publication History

Abstract

Systematic studies of the literature can be done in different ways. In particular, different guidelines propose different first steps in their recommendations, e.g. start with search strings in different databases or start with the reference lists of a starting set of papers.

In software engineering, the main recommended first step is using search strings in a number of databases, while in information systems, snowballing has been recommended as the first step. This paper compares the two different search approaches for conducting literature review studies.

The comparison is conducted by searching for articles addressing "Agile practices in global software engineering". The focus of the paper is on evaluating the two different search approaches.

Despite the differences in the included papers, the conclusions and the patterns found in both studies are quite similar. The strengths and weaknesses of each first step are discussed separately and in comparison with each other.

It is concluded that none of the first steps is outperforming the other, and the choice of guideline to follow, and hence the first step, may be context-specific, i.e. depending on the area of study.

References

[1]

M. Ivarsson, T. Gorschek (2011): A method for Evaluating Rigor and Industrial Relevance of Technology Evaluations. Empirical Software Engineering 16(3): 365--395.

Digital Library

[2]

S. Jalali, C. Wohlin (2010): Agile Practices in Global Software Engineering - a Systematic Map. In 5th IEEE International Conference on Global Software Engineering, Princeton, USA, pp. 45--54.

Digital Library

[3]

H. Zhang, M. A. Babar, X. Bai, J. Li, L. Huang (2011): An Empirical Assessment of a Systematic Search Process for Systematic Reviews. In the Proceedings of the 15th International Conference on Evaluation and Assessment in Software Engineering, pp. 56--65.

[4]

J. Webster, R. T. Watson (2002.): Analyzing the Past to Prepare for the Future: Writing a Literature Review. MIS Quarterly 26(2): xiii-xxiii.

Digital Library

[5]

K. Petersen, C. Wohlin (2009): Context in Industrial Software Engineering Research. 3rd International Symposium on Empirical Software Engineering and Measurement, pp. 401--404.

Digital Library

[6]

T. Greenhalgh, R. Peacock (2005): Effectiveness and Efficiency of Search Methods in Systematic Reviews of Complex Evidence: Audit of Primary Sources. BMJ 331(7524): 1064--1065.

[7]

B. Kitchenham, T. Dyba, M. Jørgensen (2004): Evidence-based Software Engineering. In Proceeding of the 27th IEEE International Conference on Software Engineering, pp. 273--281, IEEE Computer Society.

Digital Library

[8]

T. Dyba, B. Kitchenham, M. Jørgensen (2005): Evidence-based Software Engineering for Practitioners. IEEE Software 22(1): 58--65.

Digital Library

[9]

S. Jalali, C. Wohlin (2011): Global Software Engineering and Agile Practices: A Systematic Review. Journal of Software: Evolution and Process, published online:

[10]

S. Jalali, C. Wohlin (2011): Global Software Engineering and Agile Practices: A Systematic Review through Snowballing, Technical Report, http://www.wohlin.eu/GS Search Agile and Global.pdf.

[11]

B. Kitchenham, S. Charters (2007): Guidelines for Performing Systematic Literature Reviews in Software Engineering. Version 2.3, Technical Report, Software Engineering Group, Keele University and Department of Computer Science, University of Durham.

[12]

S. MacDonell, M. Shepperd, B. Kitchenham, E. Mendes (2010): How Reliable are Systematic Reviews in Empirical Software Engineering?. IEEE Transactions on Software Engineering 36(5): 676--687.

Digital Library

[13]

H. Zhang, M. A. Babar, P. Tell (2011): Identifying Relevant Studies in Software Engineering. Information and Software Technology 53(6): 625--637.

Digital Library

[14]

P. Brereton, B. Kitchenham, D. Budgen, M. Turner, M. Khalil (2007): Lessons from Applying the Systematic Literature Review Process within the Software Engineering Domain. Journal of Systems and Software 80(4): 571 583.

Digital Library

[15]

M. Skoglund, P. Runeson (2009): Reference-based Search Strategies in Systematic Reviews. In the Proceedings of the 13th International Conference on Evaluation and Assessment in Software Engineering, Durham, England.

Digital Library

[16]

B. Kitchenham, P. Brereton, Z. Li, D. Budgen, A. Burn (2001): Repeatability of Systematic Literature Reviews. In the Proceedings of the 15th International Conference on Evaluation and Assessment in Software Engineering, pp. 46--55.

[17]

R. Wieringa, N. A. M. Maiden, N. R. Mead, C. Rolland (2006): Requirements Engineering Paper Classification and Evaluation Criteria: a Proposal and a Discussion. Journal of Requirements Engineering 11(1): 102--107.

Digital Library

[18]

M. A. Babar, H. Zhang (2009): Systematic Literature Reviews in Software Engineering: Preliminary Results from Interviews with Researchers. In the Proceedings of the 3rd International Symposium On Empirical Software Engineering And Measurement, pp. 346--355, IEEE Computer Society.

Digital Library

[19]

S. Jalali, C. Wohlin (2011): Systematic Literature Studies: Database Searches vs. Backward Snowballing. Technical Report, http://www.wohlin.eu/Database Snowballing.pdf.

[20]

K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson (2008): Systematic Mapping Studies in Software Engineering. In the Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering.

Digital Library

Cited By

Busch NZalewski A(2025)A Systematic Literature Review of Enterprise Architecture Evaluation MethodsACM Computing Surveys10.1145/370658257:5(1-36)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3706582
da Fontoura Vieira JEcheveste MTinoco MMarcon AMarcon ÉLermen F(2025)Enhancing value cocreation orientation in service innovation: a new service development and service design integrated processHumanities and Social Sciences Communications10.1057/s41599-024-04178-912:1Online publication date: 7-Jan-2025
https://doi.org/10.1057/s41599-024-04178-9
Santos IFelizardo KSteinmacher IGerosa M(2025)Software solutions for newcomers’ onboarding in software projects: A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2024.107568177(107568)Online publication date: Jan-2025
https://doi.org/10.1016/j.infsof.2024.107568
Show More Cited By

Index Terms

Systematic literature studies: database searches vs. backward snowballing
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. Project and people management
      2. Software management
2. Software and its engineering
  1. Software creation and management
    1. Software development process management

Recommendations

Guidelines for snowballing in systematic literature studies and a replication in software engineering
EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.

Objective: This paper presents guidelines for conducting literature reviews using ...
Experiences from using snowballing and database searches in systematic literature studies
EASE '15: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering

Background: Systematic literature studies are commonly used in software engineering. There are two main ways of conducting the searches for these type of studies; they are snowballing and database searches. In snowballing, the reference list (backward ...
Second-generation systematic literature studies using snowballing
EASE '16: Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

Systematic literature studies have become standard practice in software engineering to synthesize evidence in different areas of the discipline. As more such studies are published, there is also a need to extend previously published systematic ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '12: Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement

September 2012

338 pages

ISBN:9781450310567

DOI:10.1145/2372251

General Chair:
Per Runeson
Lund University, Sweden
,
Program Chairs:
Martin Höst
Lund University, Sweden
,
Emilia Mendes
Zayed University, United Arab Emirates
,
Anneliese Andrews
University of Denver, USA
,
Rachel Harrison
Oxford Brookes University, UK

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 September 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEM '12

Sponsor:

SIGSOFT

ESEM '12: 2012 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

September 19 - 20, 2012

Lund, Sweden

Acceptance Rates

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

428
Total Citations
View Citations
4,749
Total Downloads

Downloads (Last 12 months)902
Downloads (Last 6 weeks)97

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Busch NZalewski A(2025)A Systematic Literature Review of Enterprise Architecture Evaluation MethodsACM Computing Surveys10.1145/370658257:5(1-36)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3706582
da Fontoura Vieira JEcheveste MTinoco MMarcon AMarcon ÉLermen F(2025)Enhancing value cocreation orientation in service innovation: a new service development and service design integrated processHumanities and Social Sciences Communications10.1057/s41599-024-04178-912:1Online publication date: 7-Jan-2025
https://doi.org/10.1057/s41599-024-04178-9
Santos IFelizardo KSteinmacher IGerosa M(2025)Software solutions for newcomers’ onboarding in software projects: A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2024.107568177(107568)Online publication date: Jan-2025
https://doi.org/10.1016/j.infsof.2024.107568
Valle PTonon VGarcés LRezende SNakagawa E(2025)TASIS: A typology of architectural strategies for interoperability in software-intensive systemsComputer Standards & Interfaces10.1016/j.csi.2024.10387491(103874)Online publication date: Jan-2025
https://doi.org/10.1016/j.csi.2024.103874
Feil ATraesel E(2024)INDICADORES DE SUSTENTABILIDADE EMPREGADOS NA AVALIAÇÃO DO DESEMPENHO DA INDÚSTRIA DE BEBIDAS NO BRASILSUSTAINABILITY INDICATORS EMPLOYED IN THE PERFORMANCE EVALUATION OF THE BEVERAGE INDUSTRY IN BRAZILINDICADORES DE SOSTENIBILIDAD EMPLEADOS EN LA EVALUACIÓN DEL DESEMPEÑO DE LA INDUSTRIA DE BEBIDAS EN BRASILRevista de Estudos Interdisciplinares10.56579/rei.v6i1.6816:1(01-23)Online publication date: 26-May-2024
https://doi.org/10.56579/rei.v6i1.681
Ferreira JOliveira SPortela C(2024)Experiência do usuário e requisitos não funcionais de software no ensino interdisciplinar de interação humano-computador e Engenharia de Software: uma revisão sistemática da literaturaCaderno Pedagógico10.54033/cadpedv21n10-38421:10(e9887)Online publication date: 29-Oct-2024
https://doi.org/10.54033/cadpedv21n10-384
Kulawik J(2024)Global experiences with subsidising economic insurance in agriculture.Ubezpieczenia w Rolnictwie - Materiały i Studia10.48058/urms/81.2024.2(29-48)Online publication date: 28-Jun-2024
https://doi.org/10.48058/urms/81.2024.2
Kulawik J(2024)Światowe doświadczenia z subsydiowaniem ubezpieczeń gospodarczych w rolnictwieUbezpieczenia w Rolnictwie - Materiały i Studia10.48058/urms/81.2024.1(7-27)Online publication date: 28-Jun-2024
https://doi.org/10.48058/urms/81.2024.1
Lububu STwum-Darko M(2024)Identification of diagnostic methods for African swine fever: A systematic literature reviewInternational Journal of Business Ecosystem & Strategy (2687-2293)10.36096/ijbes.v6i6.6476:6(187-202)Online publication date: 17-Dec-2024
https://doi.org/10.36096/ijbes.v6i6.647
Boamah FTawiah BHoward E(2024)Biomimetic Design Concepts in African Textile Prints: Embracing Nature's Influence on Fashion and TextilesFashion and Textiles Review10.35738/ftr.v5.2024.085(22-39)Online publication date: 31-Dec-2024
https://doi.org/10.35738/ftr.v5.2024.08
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents