More Web Proxy on the site http://driver.im/

research-article

An evidence-based inquiry into the use of grey literature in software engineering

Authors:

Muhammad Ali BabarAuthors Info & Claims

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

Pages 1422 - 1434

https://doi.org/10.1145/3377811.3380336

Published: 01 October 2020 Publication History

Abstract

Context: Following on other scientific disciplines, such as health sciences, the use of Grey Literature (GL) has become widespread in Software Engineering (SE) research. Whilst the number of papers incorporating GL in SE is increasing, there is little empirically known about different aspects of the use of GL in SE research.

Method: We used a mixed-methods approach for this research. We carried out a Systematic Literature Review (SLR) of the use of GL in SE, and surveyed the authors of the selected papers included in the SLR (as GL users) and the invited experts in SE community on the use of GL in SE research. Results: We systematically selected and reviewed 102 SE secondary studies that incorporate GL in SE research, from which we identified two groups based on their reporting: 1) 76 reviews only claim their use of GL; 2) 26 reviews report the results by including GL. We also obtained 20 replies from the GL users and 24 replies from the invited SE experts. Conclusion: There is no common understanding of the meaning of GL in SE. Researchers define the scopes and the definitions of GL in a variety of ways. We found five main reasons of using GL in SE research. The findings have enabled us to propose a conceptual model for how GL works in SE research lifecycle. There is an apparent need for research to develop guidelines for using GL in SE and for assessing quality of GL. The current work can provide a panorama of the state-of-the-art of using GL in SE for the follow-up research, as to determine the important position of GL in SE research.

References

[1]

F. Henry Abanda, C. Vidalakis, Akponanabofa Henry Oti, and Joseph H.M. Tah. 2015. A critical analysis of Building Information Modelling systems used in construction projects. Advances in Engineering Software 90 (2015), 183--201.

Digital Library

[2]

Jean Adams, Frances C. Hillier-Brown, Helen J. Moore, Amelia A. Lake, Vera Araujo-Soares, Martin White, and Carolyn Summerbell. 2016. Searching and synthesising 'grey literature' and 'grey information' in public health: critical reflections on three case studies. Systematic Reviews 5, 1 (2016), 164.

[3]

Richard J. Adams, Palie Smart, and Anne Sigismund Huff. 2017. Shades of grey: guidelines for working with the grey literature in systematic reviews for management and organizational studies. International Journal of Management Reviews 19, 4 (2017), 432--454.

[4]

Osama Al-Baik and James Miller. 2015. The kanban approach, between agility and leanness: a systematic review. Empirical Software Engineering 20, 6 (2015), 1861--1897.

Digital Library

[5]

Waleed Alabbas, Haider M. Al-Khateeb, and Ali Mansour. 2016. Arabic text classification methods: Systematic literature review of primary studies. In Proceedings of the 4th IEEE International Colloquium on Information Science and Technology. IEEE, 361--367.

[6]

Vilma Alberani, Paola de Castro Pietrangeli, and Anna Maria Rossi Mazza. 1990. The use of grey literature in health sciences: a preliminary survey. Bulletin of the Medical Library Association 78, 4 (1990), 358.

[7]

Marcus Banks. 2009. Blog posts and tweets: the next frontier for grey literature. In Grey literature in library and information studies. De Gruyter.

[8]

Marcus A. Banks. 2006. Towards a continuum of scholarship: the eventual collapse of the distinction between grey and non-grey literature. Publishing Research Quarterly 22, 1 (2006), 4--11.

[9]

Sarah P. Bellefontaine and Catherine M. Lee. 2014. Between Black and White: Examining Grey Literature in Meta-analyses of Psychological Research. Journal of Child and Family Studies 23, 8 (2014), 1378--1388.

[10]

Sally C. Brailsford, Paul Robert Harper, Brijesh Patel, and Martin Pitt. 2009. An analysis of the academic literature on simulation and modelling in health care. Journal of Simulation 3, 3 (2009), 130--140.

[11]

Gleison Brito, Thais Mombach, and Marco Tulio Valente. 2019. Migrating to GraphQL: A Practical Assessment. In Proceedings of the 26th IEEE International Conference on Software Analysis, Evolution and Reengineering. IEEE, 140--150.

[12]

Marcus Ciolkowski. 2009. What do we know about perspective-based reading? An approach for quantitative aggregationin software engineering. In Proceedings of the Third International Symposium on Empirical Software Engineering and Measurement, ESEM 2009, October 15-16, 2009, Lake Buena Vista, Florida, USA. 133--144.

Digital Library

[13]

Vicki S. Conn, Jeffrey C. Valentine, Harris M. Cooper, and Marilyn J. Rantz. 2003. Grey literature in meta-analyses. Nursing Research 52, 4 (2003), 256--261.

[14]

Alison M. Cook, Ilora G. Finlay, Adrian G.K. Edwards, Kerenza Hood, Irene J. Higginson, Danielle M. Goodwin, Charles E. Normand, and Hannah-Rose Douglas. 2001. Efficiency of searching the grey literature in palliative care. Journal of Pain and Symptom Management 22, 3 (2001), 797--801.

[15]

Ruth Cordes. 2004. Is grey literature ever used? Using citation analysis to measure the impact of GESAMP, an international marine scientific advisory body. Canadian Journal of Information and Library Science 28, 1 (2004), 49--69.

[16]

Paulo Anselmo da Mota Silveira Neto, Ivan do Carmo Machado, John D. McGregor, Eduardo Santana de Almeida, and Silvio Romero de Lemos Meira. 2011. A systematic mapping study of software product lines testing. Information & Software Technology 53, 5 (2011), 407--423.

Digital Library

[17]

Rosa Di Cesare and Roberta Ruggieri. 2006. The evaluation of grey literature using bibliometric indicators: A methodological proposal. (2006), 57--63.

[18]

Oscar Díaz, Jeremías P. Contell, and John R. Venable. 2017. Strategic Reading in Design Science: Let Root-Cause Analysis Guide Your Readings. In Proceedings of the 12th International Conference on Designing the Digital Transformation. Springer, 231--246.

[19]

Kathel Dunn. 2005. Impact of the inclusion of grey literature on the scholarly communication patterns of an interdisciplinary specialty. Grey journal 1, 1 (2005), 25--30.

[20]

Dominic Farace and Joachim Schöpfel. 2010. Grey literature in library and information studies. Walter de Gruyter.

[21]

Vahid Garousi, Michael Felderer, and Mika V. Mäntylä. 2016. The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature. In Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering. ACM, 1--6.

[22]

Vahid Garousi, Michael Felderer, and Mika V Mäntylä. 2019. Guidelines for including grey literature and conducting multivocal literature reviews in software engineering. Information and Software Technology 106 (2019), 101--121.

[23]

Julia Gelfand. 1998. Teaching and exposing grey literature: what the information profession needs to know-examples from the sciences. Collection building 17, 4 (1998), 159--166.

[24]

Dean Giustini. 2012. Finding the hard to finds: Searching for grey literature. (2012).

[25]

Dean Giustini and Diane Thompson. 2006. Finding the Hard to Finds: Searching for Grey (Gray) Literature. Vancouver, UBC: University of British Columbia.

[26]

Robert L. Glass. 2006. Software Creativity 2.0. Developer.* Books.

[27]

Katelyn Godin, Jackie Stapleton, Sharon I. Kirkpatrick, Rhona M. Hanning, and Scott T. Leatherdale. 2015. Applying systematic review search methods to the grey literature: a case study examining guidelines for school-based breakfast programs in Canada. Systematic Reviews 4, 1 (2015), 138.

[28]

Neal R. Haddaway. 2015. The use of web-scraping software in searching for grey literature. Grey Journal 11, 3 (2015), 186--190.

[29]

Neal Robert Haddaway, Alexandra Mary Collins, Deborah Coughlin, and Stuart Kirk. 2015. The role of Google Scholar in evidence reviews and its applicability to grey literature searching. PloS One 10, 9 (2015).

[30]

Michele Harms. 2004. The Survey Research Handbook.

[31]

Ali Idri, Sofia Ouhbi, Jose Luis Fernandezaleman, and Ambrosio Toval. 2012. A survey of requirements engineering education. In Proceedings of the Global Engineering Education Conference.

[32]

Nicholas A. John. 2013. The social logics of sharing. The Communication Review 16, 3 (2013), 113--131.

[33]

Sven Kepes, George C. Banks, Michael McDaniel, and Deborah L. Whetzel. 2012. Publication bias in the organizational sciences. Organizational Research Methods 15, 4 (2012), 624--662.

[34]

Barbara Kitchenham and Stuart Charters. 2007. Guidelines for Performing Systematic Literature Reviews in Software Engineering (version 2.3). Technical Report. Keele University and University of Durham.

[35]

Carol Lefebvre, Eric Manheimer, and Julie Glanville. 2008. Searching for studies. Cochrane handbook for systematic reviews of interventions: Cochrane book series (2008), 95--150.

[36]

Lech Madeyski, Wojciech Orzeszyna, Richard Torkar, and Mariusz Jozala. 2014. Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order Mutation. IEEE Transactions on Software Engineering 40, 1 (2014), 23--42.

Digital Library

[37]

Quenby Mahood, Dwayne Van Eerd, and Emma Irvin. 2014. Searching for grey literature for systematic reviews: challenges and benefits. Research Synthesis Methods 5, 3 (2014), 221--234.

[38]

Sue Mallett, Sally Hopewell, and Mike J. Clarke. 2002. Grey literature in systematic reviews: The first 1000 Cochrane systematic reviews. In Proceedings of the 4th Symposium on Systematic Reviews: Pushing the Boundaries. 2--4.

[39]

José Luis R. Martin, Víctor Pérez, Montse Sacristán, and Enric Álvarez. 2005. Is grey literature essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia. European Psychiatry 20, 8 (2005), 550--553.

[40]

Santiago Matalonga, Felyppe Rodrigues, and Guilherme Horta Travassos. 2017. Characterizing testing methods for context-aware software systems: Results from a quasi-systematic literature review. Journal of Systems & Software 131 (2017), 1--21.

Digital Library

[41]

Laura McAuley, Peter Tugwell, and David Moher. 2000. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? The Lancet 356, 9237 (2000), 1228--1231.

[42]

Yuko McGrath, Mark Bellis, Keith Edmonds, Jim McVeigh, and Harry Sumnall. 2006. Review of grey literature on drug prevention among young people. NICE London.

[43]

Martin McHugh, Oisín Cawley, Fergal McCaffery, Ita Richardson, and Xiaofeng Wang. 2013. An agile v-model for medical device software development to overcome the challenges with plan-driven software development lifecycles. In Proceedings of the 5th International Workshop on Software Engineering in Health Care. IEEE Press, 12--19.

Digital Library

[44]

Vesh Modi, Nandita Murukutla, World Lung Foudation, Paresh Dave, Tahir Turk, and World Lung Foundation. 2012. Impact Evaluation of Smoke-Free Mass Media Campaign On Knowledge, Attitude and Behavior of the Target Audience In India. In Proceedings of the 2012 National Conference on Health Communication Marketing and Media.

[45]

Håvard Myrbakken and Ricardo Colomo Palacios. 2017. DevSecOps: A Multivocal Literature Review. In Proceedings of the 17th International Conference on Software Process Improvement and Capability Determination. 17--29.

[46]

Indira Nurdiani, Jürgen Börstlerrstler, Samuel Fricker, Kai Petersen, and Panagiota Chatzipetrou. 2019. Understanding the order of agile practice introduction: Comparing agile maturity models and practitioners' experience. Journal of Systems & Software 156 (2019), 1 -- 20.

Digital Library

[47]

Rodney T. Ogawa and Betty Malen. 1991. A Response to Commentaries on "Towards Rigor in Reviews of Multivocal Literatures...". Review of Educational Research 61, 3 (1991), 307--313.

[48]

Rodney T. Ogawa and Betty Malen. 1991. Towards rigor in reviews of multivocal literatures: Applying the exploratory case study method. Review of Educational Research 61, 3 (1991), 265--286.

[49]

Anthony Petrosino, Robert F. Boruch, Cath Rounding, Steve McDonald, and Iain Chalmers. 2000. The Campbell Collaboration Social, Psychological, Educational and Criminological Trials Register (C2-SPECTR) to Facilitate the Preparation and Maintenance of Systematic Reviews of Social and Educational Interventions. Evaluation & Research in Education 14, 3--4 (2000), 206--219.

[50]

N.W. Posnett. 1984. Factors affecting the accessibility of nonconventional literature for use in the United Kingdom and some possible solutions. Library Acquisitions: Practice & Theory 8, 4 (1984), 275--285.

[51]

Abheeshta Putta, Maria Paasivaara, and Casper Lassenius. 2018. Benefits and Challenges of Adopting the Scaled Agile Framework (SAFe): Preliminary Results from a Multivocal Literature Review. In Proceedings of the 19th International Conference on Product-Focused Software Process Improvement. 334--351.

[52]

Austen Rainer and Ashley Williams. 2019. Heuristics for improving the rigour and relevance of grey literature searches for software engineering research. Information and Software Technology 106 (2019), 231--233.

[53]

Hannah R. Rothstein, Alexander J. Sutton, and Michael Borenstein. 2005. Publication bias in meta-analysis. John Wiley & Sons.

[54]

Ahlam A. Saleh, Melissa A. Ratajeski, and Marnie Bertolet. 2014. Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized. Evidence based Library and Information Practice 9, 3 (2014), 28--50.

[55]

Norsaremah Salleh, Emilia Mendes, and John Grundy. 2010. Empirical studies of pair programming for CS/SE teaching in higher education: A systematic literature review. IEEE Transactions on Software Engineering 37, 4 (2010), 509--525.

Digital Library

[56]

Joachim Schöpfel. 2006. Observations on the future of grey literature. The Grey Journal 2, 2 (2006), 67--76.

[57]

Jacopo Soldani, Damian Andrew Tamburri, and Willem-Jan van den Heuvel. 2018. The pains and gains of microservices: A Systematic grey literature review. Journal of Systems and Software 146 (2018), 215--232.

[58]

Nirnaya Tripathi, Pertti Seppänen, Ganesh Boominathan, Markku Oivo, and Kari Liukkunen. 2019. Insights into startup ecosystems through exploration of multi-vocal literature. Information & Software Technology 105 (2019), 56--77.

[59]

K.T.L. Vaughan, Barrie E. Hayes, Rachel C. Lerner, Karen R. McElfresh, Laura Pavlech, David Romito, Laurie H. Reeves, and Erin N. Morris. 2013. Development of the research lifecycle model for library services. Journal of the Medical Library Association 101, 4 (2013), 310--314.

[60]

Affan Yasin and Muhammad Ijlal Hasnain. 2012. On the quality of grey literature and its use in information synthesis during systematic literature reviews. (2012).

[61]

Robert K. Yin. 1991. Advancing Rigorous Methodologies: A Review of "Towards Rigor in Reviews of Multivocal Literatures...". Review of Educational Research 61, 3 (1991), 299--305.

Cited By

Kalam Azad MIqbal NHassan FRoy P(2023)An Empirical Study of High Performance Computing (HPC) Performance Bugs2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)10.1109/MSR59073.2023.00037(194-206)Online publication date: May-2023
https://doi.org/10.1109/MSR59073.2023.00037
Pobaru MGokulapriya R(2023)Node Overlapping Detection for Draggable Node-Based Applications2023 IEEE International Conference on Contemporary Computing and Communications (InC4)10.1109/InC457730.2023.10262840(1-5)Online publication date: 21-Apr-2023
https://doi.org/10.1109/InC457730.2023.10262840
Gazzinelli Cruz VRocha HValente M(2023)Snapshot testing in practiceJournal of Systems and Software10.1016/j.jss.2023.111797204:COnline publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111797
Show More Cited By

Recommendations

On the Use of Grey Literature: A Survey with the Brazilian Software Engineering Research Community
SBES '20: Proceedings of the XXXIV Brazilian Symposium on Software Engineering

Background: The use of Grey Literature (GL) has been investigated in diverse research areas. In Software Engineering (SE), this topic has an increasing interest over the last years. Problem: Even with the increase of GL published in diverse sources, the ...
What Evidence We Would Miss If We Do Not Use Grey Literature?
ESEM '21: Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

Context: Multivocal Literature Reviews (MLR) search for evidence in both Traditional Literature (TL) and Grey Literature (GL). Despite the growing interest in MLR-based studies, the literature assessing how GL has contributed to MLR studies is still ...
The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature
EASE '16: Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

Systematic Literature Reviews (SLR) may not provide insight into the "state of the practice" in SE, as they do not typically include the "grey" (non-published) literature. A Multivocal Literature Review (MLR) is a form of a SLR which includes grey ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '20: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

June 2020

1640 pages

ISBN:9781450371216

DOI:10.1145/3377811

General Chairs:
Gregg Rothermel
North Carolina State University
,
Doo-Hwan Bae
KAIST, South Korea

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

KIISE: Korean Institute of Information Scientists and Engineers
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

ICSE '20

Sponsor:

SIGSOFT

ICSE '20: 42nd International Conference on Software Engineering

June 27 - July 19, 2020

Seoul, South Korea

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
319
Total Downloads

Downloads (Last 12 months)49
Downloads (Last 6 weeks)3

Reflects downloads up to 28 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kalam Azad MIqbal NHassan FRoy P(2023)An Empirical Study of High Performance Computing (HPC) Performance Bugs2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)10.1109/MSR59073.2023.00037(194-206)Online publication date: May-2023
https://doi.org/10.1109/MSR59073.2023.00037
Pobaru MGokulapriya R(2023)Node Overlapping Detection for Draggable Node-Based Applications2023 IEEE International Conference on Contemporary Computing and Communications (InC4)10.1109/InC457730.2023.10262840(1-5)Online publication date: 21-Apr-2023
https://doi.org/10.1109/InC457730.2023.10262840
Gazzinelli Cruz VRocha HValente M(2023)Snapshot testing in practiceJournal of Systems and Software10.1016/j.jss.2023.111797204:COnline publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111797
Ferreira FValente M(2023)Detecting code smells in React-based Web appsInformation and Software Technology10.1016/j.infsof.2022.107111155:COnline publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1016/j.infsof.2022.107111
Francisco da Matta Vegi LValente M(2023)Understanding code smells in Elixir functional languageEmpirical Software Engineering10.1007/s10664-023-10343-628:4Online publication date: 7-Jul-2023
https://dl.acm.org/doi/10.1007/s10664-023-10343-6
Vegi LValente MRastogi ATufano RBavota GArnaoudova VHaiduc S(2022)Code smells in ElixirProceedings of the 30th IEEE/ACM International Conference on Program Comprehension10.1145/3524610.3527881(580-584)Online publication date: 16-May-2022
https://dl.acm.org/doi/10.1145/3524610.3527881
Grbin LNichols PRussell FFuller-Tyszkiewicz MOlsson C(2022)The Development of a Living Knowledge System and Implications for Future Systematic SearchingJournal of the Australian Library and Information Association10.1080/24750158.2022.208795471:3(275-292)Online publication date: 10-Jul-2022
https://doi.org/10.1080/24750158.2022.2087954
Yasin AFatima RLiu LAli Khan JAli RWang J(2022)On the utilization of non‐quality assessed literature in software engineering researchJournal of Software: Evolution and Process10.1002/smr.246434:7Online publication date: 24-May-2022
https://doi.org/10.1002/smr.2464
Kamei FPinto GWiese IRibeiro MSoares SLanubile F(2021)What Evidence We Would Miss If We Do Not Use Grey Literature?Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3475716.3475777(1-11)Online publication date: 11-Oct-2021
https://dl.acm.org/doi/10.1145/3475716.3475777
De Angelis GLonetti F(2021)About the Assessment of Grey Literature in Software EngineeringProceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering10.1145/3463274.3463362(373-378)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3463274.3463362
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents