More Web Proxy on the site http://driver.im/

research-article

Quality assessment of systematic reviews in software engineering: a tertiary study

Authors:

Muhammad Ali Babar,

Hao TangAuthors Info & Claims

EASE '15: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering

Article No.: 14, Pages 1 - 14

https://doi.org/10.1145/2745802.2745815

Published: 27 April 2015 Publication History

Abstract

Context: The quality of an Systematic Literature Review (SLR) is as good as the quality of the reviewed papers. Hence, it is vital to rigorously assess the papers included in an SLR. There has been no tertiary study aimed at reporting the state of the practice of quality assessment used in SLRs in Software Engineering (SE).

Objective: We aimed to study the practices of quality assessment of the papers included in SLRs in SE.

Method: We conducted a tertiary study of the SLRs that have performed quality assessment of the reviewed papers.

Results: We identified and analyzed different aspects of the quality assessment of the papers included in 127 SLRs.

Conclusion: Researchers use a variety of strategies for quality assessment of the papers reviewed, but report little about the justification for the used criteria. The focus is creditability but not relevance aspect of the papers. Appropriate guidelines are required for devising quality assessment strategies.

References

[1]

J. Biolchini, P. G. Mian, A. C. C. Natali, and G. H. Travassos. Systematic review in software engineering. System Engineering and Computer Science Department COPPE/UFRJ, Technical Report ES, 679(05): 45, 2005.

[2]

D. S. Cruzes and T. Dybå. Research synthesis in software engineering: A tertiary study. Information and Software Technology, 53(5): 440--455, 2011.

Digital Library

[3]

F. Q. Da Silva, A. L. Santos, S. Soares, A. C. C. França, C. V. Monteiro, and F. F. Maciel. Six years of systematic literature reviews in software engineering: An updated tertiary study. Information and Software Technology, 53(9): 899--913, 2011.

Digital Library

[4]

F. Q. da Silva, A. L. Santos, S. C. Soares, A. C. C. França, and C. V. Monteiro. A critical appraisal of systematic reviews in software engineering from the perspective of the research questions asked in the reviews. In Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, page 33. ACM, 2010.

Digital Library

[5]

T. Dybå and T. Dingsøyr. Strength of evidence in systematic reviews in software engineering. In Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, pages 178--187. ACM, 2008.

Digital Library

[6]

T. Dyba, T. Dingsoyr, and G. K. Hanssen. Applying systematic reviews to diverse study types: An experience report. In Empirical Software Engineering and Measurement, 2007. ESEM 2007. First International Symposium on, pages 225--234. IEEE, 2007.

Digital Library

[7]

T. Dyba, B. A. Kitchenham, and M. Jorgensen. Evidence-based software engineering for practitioners. Software, IEEE, 22(1): 58--65, 2005.

Digital Library

[8]

T. Greenhalgh and R. Taylor. How to read a paper. BMJ Publishing Group London, 2002.

[9]

P. H. R. U. in Oxford. Critical appraisal skills programme. http://www.casp-uk.net/, 2013.

[10]

A. R. Jadad, R. A. Moore, D. Carroll, C. Jenkinson, D. J. M. Reynolds, D. J. Gavaghan, and H. J. McQuay. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Controlled clinical trials, 17(1): 1--12, 1996.

[11]

S. Keele. Guidelines for performing systematic literature reviews in software engineering. Technical report, Technical report, EBSE Technical Report EBSE-2007-01, 2007.

[12]

B. Kitchenham. Procedures for performing systematic reviews. Keele, UK, Keele University, 33: 2004, 2004.

[13]

B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman. Systematic literature reviews in software engineering--a systematic literature review. Information and software technology, 51(1): 7--15, 2009.

Digital Library

[14]

B. Kitchenham, R. Pretorius, D. Budgen, O. Pearl Brereton, M. Turner, M. Niazi, and S. Linkman. Systematic literature reviews in software engineering--a tertiary study. Information and Software Technology, 52(8): 792--805, 2010.

Digital Library

[15]

B. A. Kitchenham, T. Dyba, and M. Jorgensen. Evidence-based software engineering. In Software Engineering, 2004. ICSE 2004. Proceedings. 26th International Conference on, pages 273--281. IEEE, 2004.

Digital Library

[16]

B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. El Emam, and J. Rosenberg. Preliminary guidelines for empirical research in software engineering. Software Engineering, IEEE Transactions on, 28(8): 721--734, 2002.

Digital Library

[17]

A. D. Oxman, G. W. Group, et al. Grading quality of evidence and strength of recommendations. Bmj, 328(19): 1490--4, 2004.

[18]

K. Petersen and N. B. Ali. Identifying strategies for study selection in systematic reviews and maps. Empirical Software Engineering, 2011.

Digital Library

[19]

M. Riaz. Systematic review of software maintainability prediction and metrics. http://chao.stat.nthu.edu.tw/, 2011.

[20]

D. I. Sjoberg, T. Dyba, and M. Jorgensen. The future of empirical methods in software engineering research. In Future of Software Engineering, 2007. FOSE'07, pages 358--378. IEEE, 2007.

Digital Library

[21]

H. Zhang and M. Ali Babar. On searching relevant studies in software engineering. In Proceedings of the 14th international conference on evaluation and assessment in software engineering (EASE), 2010.

Digital Library

[22]

H. Zhang and M. Ali Babar. Systematic reviews in software engineering: An empirical investigation. Information and Software Technology, 55(7): 1341--1354, 2013.

[23]

H. Zhang, M. A. Babar, and P. Tell. Identifying relevant studies in software engineering. Information and Software Technology, 53(6): 625--637, 2011.

Digital Library

Cited By

Sepúlveda SCravero AFonseca GAntonelli L(2024)Systematic Review on Requirements Engineering in Quantum Computing: Insights and Future DirectionsElectronics10.3390/electronics1315298913:15(2989)Online publication date: 29-Jul-2024
https://doi.org/10.3390/electronics13152989
Lin HWang CSun Y(2024)How big five personality traits influence information sharing on social media: A meta analysisPLOS ONE10.1371/journal.pone.030377019:6(e0303770)Online publication date: 12-Jun-2024
https://doi.org/10.1371/journal.pone.0303770
Okesola MOkesola JOgunlana OAfolabi I(2024)Quality assessment of systematic literature on uterine fibroids: a systematic reviewF1000Research10.12688/f1000research.124879.211(1050)Online publication date: 8-Mar-2024
https://doi.org/10.12688/f1000research.124879.2
Show More Cited By

Index Terms

Quality assessment of systematic reviews in software engineering: a tertiary study
1. Software and its engineering

Recommendations

Automated Selection and Quality Assessment of Primary Studies: A Systematic Literature Review
ON THE HORIZON, CHALLENGE PAPER, REGULAR PAPERS, and EXPERIENCE PAPER

Researchers use systematic literature reviews (SLRs) to synthesize existing evidence regarding a research topic. While being an important means to condense knowledge, conducting an SLR requires a large amount of time and effort. Consequently, ...
Systematic review in software engineering: where we are and where we should be going
EAST '12: Proceedings of the 2nd international workshop on Evidential assessment of software technologies

In 2004 Kitchenham et al. first proposed the idea of evidence-based software engineering (EBSE). EBSE requires a systematic and unbiased method of aggregating empirical studies and has encouraged software engineering researches to undertake systematic ...
A critical appraisal tool for systematic literature reviews in software engineering
Abstract
Context: Methodological research on systematic literature reviews (SLRs) in Software Engineering (SE) has so far focused on developing and evaluating guidelines for conducting systematic reviews. However, the support for quality assessment of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '15: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering

April 2015

305 pages

ISBN:9781450333504

DOI:10.1145/2745802

General Chair:
Jian Lv
Nanjing University, China
,
Program Chairs:
He (Jason) Zhang
Nanjing University, China
,
Muhammad Ali Babar
University of Adelaide, Australia

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

NJU: Nanjing University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 April 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

EASE '15

Sponsor:

NJU

EASE '15: 19th International Conference on Evaluation and Assessment in Software Engineering

April 27 - 29, 2015

Nanjing, China

Acceptance Rates

EASE '15 Paper Acceptance Rate 20 of 65 submissions, 31%;

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

44
Total Citations
View Citations
1,321
Total Downloads

Downloads (Last 12 months)160
Downloads (Last 6 weeks)9

Reflects downloads up to 28 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sepúlveda SCravero AFonseca GAntonelli L(2024)Systematic Review on Requirements Engineering in Quantum Computing: Insights and Future DirectionsElectronics10.3390/electronics1315298913:15(2989)Online publication date: 29-Jul-2024
https://doi.org/10.3390/electronics13152989
Lin HWang CSun Y(2024)How big five personality traits influence information sharing on social media: A meta analysisPLOS ONE10.1371/journal.pone.030377019:6(e0303770)Online publication date: 12-Jun-2024
https://doi.org/10.1371/journal.pone.0303770
Okesola MOkesola JOgunlana OAfolabi I(2024)Quality assessment of systematic literature on uterine fibroids: a systematic reviewF1000Research10.12688/f1000research.124879.211(1050)Online publication date: 8-Mar-2024
https://doi.org/10.12688/f1000research.124879.2
Imtiaz SImtiaz SAlmadhor AKulhanek R(2024)A Tertiary Study on Open-Source Software ResearchIEEE Access10.1109/ACCESS.2024.351407512(189952-189993)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3514075
Costa AChoren RPereira DTerra ACosta IJunior CSantos MGomes CMoreira M(2024)Integrating multicriteria decision making and principal component analysis: a systematic literature reviewCogent Engineering10.1080/23311916.2024.237494411:1Online publication date: 16-Jul-2024
https://doi.org/10.1080/23311916.2024.2374944
dos Santos VIwazaki AFelizardo Kde Souza ÉNakagawa E(2024)Sustainable systematic literature reviewsInformation and Software Technology10.1016/j.infsof.2024.107551(107551)Online publication date: Aug-2024
https://doi.org/10.1016/j.infsof.2024.107551
de Andrade AJackson VPrikladnicki Rvan der Hoek A(2024)On meetings involving remote software teamsInformation and Software Technology10.1016/j.infsof.2024.107541175:COnline publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107541
Tripathi NHietala HXu YLiyanage R(2024)Stakeholders collaborations, challenges and emerging concepts in digital twin ecosystemsInformation and Software Technology10.1016/j.infsof.2024.107424169(107424)Online publication date: May-2024
https://doi.org/10.1016/j.infsof.2024.107424
Hannousse AYahiouche SNait-Hamoud M(2024)Twenty-two years since revealing cross-site scripting attacksComputer Science Review10.1016/j.cosrev.2024.10063452:COnline publication date: 18-Jul-2024
https://dl.acm.org/doi/10.1016/j.cosrev.2024.100634
Orošnjak MŠtrbac BVulanović SRunje BHorvatić Novak ARazumić A(2024)RCE (rationale–cogency–extent) criterion unravels features affecting citation impact of top-ranked systematic literature reviews: leaving the impression…is all you needScientometrics10.1007/s11192-024-04935-2129:3(1891-1947)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s11192-024-04935-2
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents