More Web Proxy on the site http://driver.im/

research-article

A note on rigour and replicability

Authors:

Panos Louridas,

Georgios GousiosAuthors Info & Claims

ACM SIGSOFT Software Engineering Notes, Volume 37, Issue 5

Pages 1 - 4

https://doi.org/10.1145/2347696.2347706

Published: 02 September 2012 Publication History

Abstract

As any empirical science, Software Engineering research should strive towards better research practices. Replication is regrettably not a priority for Software Engineering researchers and, moreover, not afforded by many published studies. Here we report our experience from our encounter with a recent paper in a agship Software Engineering conference. Our experience shows that current publication requirements do not guarantee replicability.

References

[1]

A. Brooks, M. Roper, M. Wood, J. Daly, and J. Miller. Replication's role in software engineering. In Forrest Shull, Janice Singer, and Dag I. K. Sjøberg, editors, Guide to Advanced Empirical Software Engineering, pages 365--379. Springer London, 2008. 10.1007/978-1-84800-044-5 14.

[2]

H.L. Dreyfus and S.E. Dreyfus. Mind over machine. Free Press, 1988.

[3]

Marius Eriksen. Effective Sscala. http://twitter.github.com/effectivescala, 2012.

[4]

Myles Hollander and Douglas A. Wolfe. Nonparametric Statistical Methods. John Wiley & Sons, Inc., 2nd edition, 1999.

[5]

Cay S. Horstmann. Scala for the Impatient. Addison-Wesley, 2012.

Digital Library

[6]

Darrel C. Ince, Leslie Hatton, and John Graham-Cumming. The case for open computer programs. Nature, 482:485--488, 23 February 2012.

[7]

Barbara R. Jasny, Gilbert Chin, Lisa Chong, and Sacha Vignieri. Again, and again, and again.... Science, 334:1225, 2 December 2011.

[8]

Peter Norvig. Teach yourself programming in ten years. http://norvig.com/21-days.html, 2001.

[9]

Martin Odersky. Scala levels: beginner to expert, application programmer to library designer. http://www.scala-lang.org/node/8610, 2011.

[10]

Martin Odersky. The Sscala language specification version 2.9. Technical report, Programming Methods Laboratory, EPFL, Switzerland, May 24 2011.

[11]

Victor Pankratius, Felix Schmidt, and Gilda Garretón. Combining functional and imperative programming for multicore software: An empirical study evaluating Scala and Java. In Proceedings of te 34th International Conference on Software Engineering, Zurich, June 6-9 2012.

Digital Library

[12]

Roger D. Peng. Reproducible research in computational science. Science, 334:1226--1227, 2 December 2011.

[13]

Victor R.Basili, Forrest Shull, and Filippo Lanubile. Building knowledge through families of experiments. IEEE Transactions on Software Engineering, 25(04):456--473, 1999.

Digital Library

[14]

M. Shaw. Writing good software engineering research papers. In Proceedings of the 25th International Conference on Software Engineering, 2003., pages 726--736, May 2003.

Digital Library

[15]

Forrest Shull, Jeffrey Carver, Sira Vegas, and Natalia Juristo. The role of replications in empirical software engineering. Empirical Software Engineering, 13:211--218, 2008. 10.1007/s10664-008-9060-1.

Digital Library

[16]

Dag I.K Sjøberg, J.E. Hannay, O. Hansen, V.B. Kampenes, A. Karahasanovic, N.-K. Liborg, and A.C. Rekdal. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering, 31(9):733--753, Sept. 2005.

Digital Library

[17]

Bjarne Stroustrup. Posting to comp.lang.c++. http://www2.research.att.com/~bs/blast.html, December 1994.

[18]

Dean Wampler and Alex Payne. Programming Scala. O'Reilly, 2009.

[19]

Leon E. Winslow. Programming pedagogy a psychological overview. SIGCSE Bull., 28(3):17--22, September 1996.

Digital Library

[20]

Carmen Zannier, Grigori Melnik, and Frank Maurer. On the success of empirical studies in the International Conference on Software Engineering. In ICSE '06: Proceedings of the 28th international conference on Software engineering, pages 341--350, New York, NY, USA, 2006. ACM.

Digital Library

Cited By

Nong YSharma RHamou-Lhadj ALuo XCai H(2023)Open Science in Software Engineering: A Study on Deep Learning-Based Vulnerability DetectionIEEE Transactions on Software Engineering10.1109/TSE.2022.320714949:4(1983-2005)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3207149
Garousi VCoşkunçay ADemirörs OYazici A(2019)Cross-factor analysis of software engineering practices versus practitioner demographicsJournal of Systems and Software10.1016/j.jss.2015.09.013111:C(49-73)Online publication date: 3-Jan-2019
https://dl.acm.org/doi/10.1016/j.jss.2015.09.013
Kelling NWard CMalin DBuras WHetherington S(2017)The Use of Human Factors to Address Medical Research Replicability Through the Development of Software Based SolutionProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/154193121360163361:1(597-601)Online publication date: 28-Sep-2017
https://doi.org/10.1177/1541931213601633
Show More Cited By

Index Terms

A note on rigour and replicability
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics
2. Software and its engineering

Recommendations

Engaging the net generation with evidence-based software engineering through a community-driven web database

Software engineering faculty face the challenge of educating future researchers and industry practitioners regarding the generation of empirical software engineering studies and their use in evidence-based software engineering. In order to engage the ...
Communication Metrics for Software Development

We present empirical evidence that metrics on communication artifacts generated by groupware tools can be used to gain significant insight into the development process that produced them. We describe a test-bed for developing and testing communication ...
Evidence-Based Software Engineering for Practitioners

Software engineers might make incorrect decisions about adopting new techniques if they donýt consider scientific evidence about the techniquesý efficacy. Procedures used for evidence-based medicine can also apply to software engineering. Such evidence-...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM SIGSOFT Software Engineering Notes

ACM SIGSOFT Software Engineering Notes Volume 37, Issue 5

September 2012

129 pages

ISSN:0163-5948

DOI:10.1145/2347696

Issue’s Table of Contents

Copyright © 2012 Authors.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 September 2012

Published in SIGSOFT Volume 37, Issue 5

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
170
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Nong YSharma RHamou-Lhadj ALuo XCai H(2023)Open Science in Software Engineering: A Study on Deep Learning-Based Vulnerability DetectionIEEE Transactions on Software Engineering10.1109/TSE.2022.320714949:4(1983-2005)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3207149
Garousi VCoşkunçay ADemirörs OYazici A(2019)Cross-factor analysis of software engineering practices versus practitioner demographicsJournal of Systems and Software10.1016/j.jss.2015.09.013111:C(49-73)Online publication date: 3-Jan-2019
https://dl.acm.org/doi/10.1016/j.jss.2015.09.013
Kelling NWard CMalin DBuras WHetherington S(2017)The Use of Human Factors to Address Medical Research Replicability Through the Development of Software Based SolutionProceedings of the Human Factors and Ergonomics Society Annual Meeting10.1177/154193121360163361:1(597-601)Online publication date: 28-Sep-2017
https://doi.org/10.1177/1541931213601633
Branco ACohen KVossen PIde NCalzolari N(2017)Replicability and reproducibility of research results for human language technologyLanguage Resources and Evaluation10.1007/s10579-017-9380-051:1(1-5)Online publication date: 1-Mar-2017
https://dl.acm.org/doi/10.1007/s10579-017-9380-0

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents