More Web Proxy on the site http://driver.im/

Article

Free access

The impact of limited search procedures for systematic literature reviews A participant-observer case study

Authors:

Barbara Kitchenham,

Pearl Brereton,

Stephen Linkman,

Rialette Pretorius,

David BudgenAuthors Info & Claims

ESEM '09: Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement

Pages 336 - 345

https://doi.org/10.1109/ESEM.2009.5314238

Published: 15 October 2009 Publication History

Abstract

This study aims to compare the use of targeted manual searches with broad automated searches, and to assess the importance of grey literature and breadth of search on the outcomes of SLRs. We used a participant-observer multi-case embedded case study. Our two cases were a tertiary study of systematic literature reviews published between January 2004 and June 2007 based on a manual search of selected journals and conferences and a replication of that study based on a broad automated search. Broad searches find more papers than restricted searches, but the papers may be of poor quality. Researchers undertaking SLRs may be justified in using targeted manual searches if they intend to omit low quality papers; if publication bias is not an issue; or if they are assessing research trends in research methodologies.

References

[1]

Brereton, O. P., Kitchenham, B.A. 2007 The Scope of EPIC Case Studies. EPIC technical Report EPIC-2007-04.

[2]

Khan, Khalid, S., Kunz, Regina, Kleijnen, Jos and Antes, Gerd. 2003 Systematic Reviews to Support Evidencebased Medicine, The Royal Society of Medicine Press Ltd.

[3]

Kitchenham, B.A. and Charters, S. 2007. Guidelines for performing Systematic Literature Reviews in Software Engineering Technical Report EBSE/EPIC-2007-01, 2007.

[4]

Petticrew, Mark and Helen Roberts. 2005 Systematic Reviews in the Social Sciences: A Practical Guide, Blackwell Publishing.

[5]

Kitchenham, B.A., Brereton, O.P., Budgen, D., Turner, M., Bailey, J., and Linkman, S.G. 2009 Systematic Literature reviews in Software Engineering - A systematic Literature review. IST, 51, pp 7-15.

[6]

Juristo, N., Moreno, A.M. Vegas, S. and Solari, M. 2006 In Search of What We Experimentally Know about Unit Testing, IEEE Software, 23 (6), pp.72-80.

[7]

Sjøberg, D.I.K., Hannay, J.E., Hansen, O., Kampenes, V.B., Karahasanovic, A., Liborg, N.K. and Rekdal, A.C. 2005 A survey of controlled experiments in software engineering. IEEE Transactions on SE, 31 (9), pp.733-753.

[8]

Jørgensen, M., and Shepperd, M. 2007 A Systematic Review of Software Development Cost Estimation Studies, IEEE Transactions on SE, 33(1), pp. 33-53.

[9]

Yin, Robert K. 2003 Case Study Research: Design and Methods, 3rd Edition, Sage Publications.

[10]

Kitchenham, B.A., Brereton, O.P., and Turner, M. 2008 EPIC Case Study 2 - Extension of a Tertiary Study. EPIC Technical Report, EPIC-2009-007.

[11]

Centre for Reviews and Dissemination. 2007 What are the criteria for the inclusion of reviews on DARE? Available at http://www.york.ac.uk/inst/crd/faq4.htm (accessed 24 July 2007).

[12]

Kitchenham, B.A., Brereton, O.P., and Budgen, D. 2008 Protocol for extending a Tertiary Study of Systematic Literature Reviews in Software Engineering. EPIC Technical Report, EBSE-2008-006, June 2008.

[13]

Grimstad, S., Jorgensen, M. and Møløkken-østvold, K. 2005 The Clients' Impact on Effort Estimation Accuracy in Software Development Projects, 11th IEEE International Software Metrics Symposium (METRICS'05), pp. 3.

[14]

Jørgensen, M. 2005 Evidence-Based Guidelines for Assessment of Software Development Cost Uncertainty, IEEE Transactions on Software Engineering, November 2005, pp. 942-954.

[15]

Davis, A., Dieste, O. Hickey, A., Juristo, N. and Moreno, A.M. 2006 Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review, 14th IEEE International Requirements Engineering Conference (RE'06), pp. 179-188.

[16]

Shepperd, M. 2007 Software project economics: a roadmap, FOSE'07.

[17]

Mair, M., Shepperd, M. and Jørgensen, M. 2005 An analysis of data sets used to train and validate cost prediction systems, PROMISE'05 Workshop.

[18]

Yalaho, A. 2006 A Conceptual Model of ICT-Supported Unified Process of International Outsourcing of Software Production, 10th International Enterprise Distributed Object Computing Conference Workshops (EDOCW'06).

[19]

Kagdi, H., Collard, M.L. and Maletic, J.I. 2006 A survey and taxonomy of approaches for mining software repositories in the context of software evolution. Journal of Software Maintenance and Evolution: Research and Practice, 19(2), pp 77-131.

[20]

Segal, J., Grinyer, A., Sharp, H. 2005 The type of evidence produced by empirical software engineers. REBSE'05.

[21]

Höst M., Wohlin, C. and Thelin, T. 2005 Experimental context classification: incentives and experience of subjects, ICSE'05, Proceedings of the 27th international conference on Software engineering, ACM.

[22]

Hosbond, J.H. and Nielsen, P.A. 2005 Mobile Systems Development - A literature review, Proceedings of IFIP 8.2 Annual Conference.

[23]

Shaw, M. and Clements, P. 2006 The Golden Age of Software Architecture: A Comprehensive Survey. Technical Report CMU-ISRI-06-101, Software Engineering Institute, Carnegie Mellon University.

[24]

Davis, A., Hickey, A., Dieste, O., Juristo, N. and Moreno, A.M. 2007 A Quantitative Assessment of Requirements Engineering Publications - 1963-2006, LNCS 4542/2007, Requirements Engineering: Foundation for Software Quality, pp 129-143.

[25]

Höfer, A. and Tichy, W.F. 2007 Status of Empirical Research in Software Engineering, in V. Basili et al., (Eds) Empirical Software Engineering Issues, Springer-Verlag LNCS 4336, pp 10-19.

[26]

Feller, J. Finnegan, P., Kelly, D. and MacNamara, M. 2006 Developing Open Source Software: A Community-Based Analysis of Research, IFIP International Federation for Information Processing, 208/2006, Social Inclusion: Societal and Organizational Implications for Information Systems, pp 261-278.

[27]

Dybå, T., Kitchenham, B. and Jørgensen, M. 2005 Evidence-based Software Engineering for Practitioners, IEEE Software, Volume 22 (1) January, 2005, pp 58-65.

[28]

Kitchenham, B., Dybå, T. and Jørgensen, M. 2004 Evidence-based Software Engineering. Proceedings of the 26th International Conference on Software Engineering, (ICSE '04), IEEE Computer Society, Washington DC, USA, pp 273-281.

[29]

Kitchenham, B.A. Procedures for Undertaking Systematic Reviews, Joint Technical Report, Computer Science Department, 2004, Keele University (TR/SE-0401) and National ICT Australia Ltd (0400011T.1).

[30]

Barcelos, R.F., and Travassos, G.H. 2006 Evaluation approaches for Software Architectural Documents: A systematic Review, Ibero-American Workshop on Requirements Engineering and Software Environments (IDEAS). La Plata, Argentina.

[31]

Jørgensen, M. 2007 Estimation of Software Development Work Effort: Evidence on Expert Judgement and Formal Models, International Journal of Forecasting, 3(3), pp 449- 462.

[32]

Torchiano, M. and Morisio, M. 2004 Overlooked Aspects of COTS-Based Development. IEEE Software, pp 88-93.

[33]

Jørgensen, M. 2004 A review of studies on expert estimation of software development effort, Journal of Systems and Software, 70 (1-2), pp 37-60.

[34]

Ramesh, V., Glass, R. L.; Vessey, I. 2004 Research in computer science: an empirical study, Journal of Systems and Software, 70(1-2), pp 165-176.

[35]

Tichy, W., Lukowicz, P., Prechelt, L., and Heinze, E. 1995 Experimental Evaluation in Computer Science: A Quantitative Study, Journal Systems and Software, 28(9), pp 9-18.

[36]

Glass, R.L., Vessey, I., and Ramesh, V. 2002 Research in software engineering: an analysis of the literature. Information and Software Technology, 44, pp 491-506.

[37]

Greenhalgh, Trisha. 2000 How to read a paper: The Basics of Evidence-Based Medicine. BMJ Books.

Cited By

Khanna DChristensen EGosu SWang XPaasivaara MGraziotin DNolte A(2024)Hybrid Work meets Agile Software Development: A Systematic Mapping StudyProceedings of the 2024 IEEE/ACM 17th International Conference on Cooperative and Human Aspects of Software Engineering10.1145/3641822.3641863(57-67)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3641822.3641863
Theunissen Tvan Heesch UAvgeriou P(2022)A mapping study on documentation in Continuous Software DevelopmentInformation and Software Technology10.1016/j.infsof.2021.106733142:COnline publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1016/j.infsof.2021.106733
Al-Ali RBulej LKofroň JBureš T(2022)A guide to design uncertainty-aware self-adaptive components in Cyber–Physical SystemsFuture Generation Computer Systems10.1016/j.future.2021.10.027128:C(466-489)Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1016/j.future.2021.10.027
Show More Cited By

Index Terms

The impact of limited search procedures for systematic literature reviews A participant-observer case study
1. Information systems
  1. Information retrieval
2. Software and its engineering

Recommendations

Refining the systematic literature review process--two participant-observer case studies

Systematic literature reviews (SLRs) are a major tool for supporting evidence-based software engineering. Adapting the procedures involved in such a review to meet the needs of software engineering and its literature remains an ongoing process. As part ...
Automation of systematic literature reviews: A systematic literature review
Abstract Context
Systematic Literature Review (SLR) studies aim to identify relevant primary papers, extract the required data, analyze, and synthesize results to gain further and broader insight into the investigated domain. ...
On Different Search Methods for Systematic Literature Reviews and Maps: Experiences from a Literature Search on Validation and Verification of Emergent Behavior
EASE '18: Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018

[Background] Systematic literature reviews and maps have become well-established research methods in software engineering research. Of the three commonly suggested and used search methods: manual search, database search, or snowball search; systematic ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

ESEM '09: Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement

October 2009

601 pages

ISBN:9781424448425

Publisher

IEEE Computer Society

United States

Publication History

Published: 15 October 2009

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
369
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Khanna DChristensen EGosu SWang XPaasivaara MGraziotin DNolte A(2024)Hybrid Work meets Agile Software Development: A Systematic Mapping StudyProceedings of the 2024 IEEE/ACM 17th International Conference on Cooperative and Human Aspects of Software Engineering10.1145/3641822.3641863(57-67)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3641822.3641863
Theunissen Tvan Heesch UAvgeriou P(2022)A mapping study on documentation in Continuous Software DevelopmentInformation and Software Technology10.1016/j.infsof.2021.106733142:COnline publication date: 1-Feb-2022
https://dl.acm.org/doi/10.1016/j.infsof.2021.106733
Al-Ali RBulej LKofroň JBureš T(2022)A guide to design uncertainty-aware self-adaptive components in Cyber–Physical SystemsFuture Generation Computer Systems10.1016/j.future.2021.10.027128:C(466-489)Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1016/j.future.2021.10.027
Kamei FPinto GWiese IRibeiro MSoares SLanubile F(2021)What Evidence We Would Miss If We Do Not Use Grey Literature?Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3475716.3475777(1-11)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1145/3475716.3475777
Kamei FWiese IPinto GRibeiro MSoares SCavalcante EDantas FBatista T(2020)On the Use of Grey LiteratureProceedings of the XXXIV Brazilian Symposium on Software Engineering10.1145/3422392.3422442(183-192)Online publication date: 21-Oct-2020
https://dl.acm.org/doi/10.1145/3422392.3422442
Barros-Justo JBenitti FTiwari S(2019)The impact of Use Cases in real-world software development projectsComputer Standards & Interfaces10.1016/j.csi.2019.10336266:COnline publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1016/j.csi.2019.103362
Brings JDaun MKempe MWeyer TRainer AMacDonell SKeung J(2018)On Different Search Methods for Systematic Literature Reviews and MapsProceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 201810.1145/3210459.3210463(35-45)Online publication date: 28-Jun-2018
https://dl.acm.org/doi/10.1145/3210459.3210463
Butt AAsif MAhmad SImdad U(2018)An Empirical Study for Adopting Social Computing in Global Software DevelopmentProceedings of the 2018 7th International Conference on Software and Computer Applications10.1145/3185089.3185105(31-35)Online publication date: 8-Feb-2018
https://dl.acm.org/doi/10.1145/3185089.3185105
Stevanetic SZdun ULv JZhang HBabar M(2015)Software metrics for measuring the understandability of architectural structuresProceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering10.1145/2745802.2745822(1-14)Online publication date: 27-Apr-2015
https://dl.acm.org/doi/10.1145/2745802.2745822
Felizardo KNakagawa EMacDonell SMaldonado JShepperd MHall TMyrtveit I(2014)A visual analysis approach to update systematic reviewsProceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering10.1145/2601248.2601252(1-10)Online publication date: 13-May-2014
https://dl.acm.org/doi/10.1145/2601248.2601252
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents