More Web Proxy on the site http://driver.im/

research-article

Replications types in experimental disciplines

Authors:

Omar S. Gómez,

Natalia Juristo,

Sira VegasAuthors Info & Claims

ESEM '10: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

Article No.: 3, Pages 1 - 10

https://doi.org/10.1145/1852786.1852790

Published: 16 September 2010 Publication History

Abstract

Experiment replication is a key component of the scientific paradigm. The purpose of replication is to verify previously observed findings. Although some Software Engineering (SE) experiments have been replicated, yet, there is still disagreement about how replications should be run in our field. With the aim of gaining a better understanding of how replications are carried out, this paper examines different replication types in other scientific disciplines. We believe that by analysing the replication types proposed in other disciplines it is possible to clarify some of the question marks still hanging over experimental SE replication.

References

[1]

J. P. F. Almqvist. Replication of controlled experiments in empirical software engineering - a survey. Master's thesis, Department of Computer Science, Faculty of Science, Lund University, Sweden, 2006.

[2]

H. M. Bahr, T. Caplow, and B. A. Chadwick. Middletown iii: Problems of replication, longitudinal measurement, and triangulation. Annu. Rev. Sociol., 9(1):243--264, Aug. 1983.

[3]

V. Basili, F. Shull, and F. Lanubile. Building knowledge through families of experiments. IEEE Transactions on Software Engineering, 25(4):456--473, Jul/Aug 1999.

Digital Library

[4]

A. Brooks, J. Daly, J. Miller, M. Roper, and M. Wood. Replication of experimental results in software engineering. Technical Report ISERN-96-10, Univ. of Strathclyde, Scotland, 1996.

[5]

D. T. Campbell and J. C. Stanley. Experimental and Quasi-Experimental Designs for Research. Houghton Mifflin Company, 1963.

[6]

T. Cook and D. Campbell. The design and conduct of quasi-experiments and true experiments in field settings. Rand McNally, Chicago, 1976.

[7]

R. Easley, C. Madden, and M. Dunn. Conducting marketing science: The role of replication in the research process. Journal of Business Research, 48(1):83--92, April 2000.

[8]

H. Evanschitzky and J. S. Armstrong. Replications of forecasting research. International Journal of Forecasting, In Press, Corrected Proof, 2009.

[9]

B. Finifter. The generation of confidence: Evaluating research findings by random subsample replication. Sociological Methodology, 4:112--175, 1972.

[10]

J. Hannay, T. Dybå, E. Arisholm, and D. Sjøberg. The effectiveness of pair programming: A meta-analysis. Information and Software Technology, 51(7):1110--1122, July 2009.

Digital Library

[11]

W. Hayes. Research synthesis in software engineering: A case for meta-analysis. In Proc. of the 6th International Symposium on Software Metrics (METRICS '99), page 143, Washington, DC, USA, 1999.

Digital Library

[12]

C. Hendrick. Replications, strict replications, and conceptual replications: Are they important? In J. W. Neuliep, editor, Handbook of Replication Studies in the Social and Behavioral Sciences. Sage, Newbury Park, California, 1990.

[13]

J. Hunter. The desperate need for replications. Journal of Consumer Research, 28(1):149--158, 2001.

[14]

B. H. Kantowitz, H. L. Roediger III, and D. G. Elmes. Experimental Psychology. Wadsworth Publishing, second ed. edition, 1984.

[15]

C. Kelly, L. Chase, and R. Tucker. Replication in experimental communication research: an analysis. Human Communication Research, 5(4):338--342, 1979.

[16]

B. Kitchenham. The role of replications in empirical software engineering-a word of warning. Empirical Softw. Engg., 13(2):219--221, 2008.

Digital Library

[17]

M. A. La Sorte. Replication as a verification technique in survey research: A paradigm. The Sociological Quarterly, 13(2):218--227, 1972.

[18]

R. Leone and R. Schultz. A study of marketing generalizations. The Journal of Marketing, 44(1):10--18, 1980.

[19]

R. M. Lindsay and A. S. C. Ehrenberg. The design of replicated studies. The American Statistician, 47(3):217--228, 1993.

[20]

J. Lung, J. Aranda, S. Easterbrook, and G. Wilson. On the difficulty of replicating human subjects studies in software engineering. In Proc. of the 30th international conference on Software engineering (ICSE '08), pages 191--200, New York, NY, USA, 2008.

Digital Library

[21]

D. T. Lykken. Statistical significance in psychological research. Psychol Bull, 70(3):151--159, 1968.

[22]

M. Mendonça, J. Maldonado, M. de Oliveira, J. Carver, S. Fabbri, F. Shull, G. Travassos, E. Hőhn, and V. Basili. A framework for software engineering experimental replications. In Proc. of the 13th IEEE International Conference on Engineering of Complex Computer Systems (ICECCS '08), pages 203--212, Washington, DC, USA, 2008.

Digital Library

[23]

J. Miller. Can results from software engineering experiments be safely combined? In Proc. of the 6th International Symposium on Software Metrics (METRICS '99), page 152, Washington, DC, USA, 1999.

Digital Library

[24]

J. Miller. Applying meta-analytical procedures to software engineering experiments. J. Syst. Softw., 54(1):29--39, 2000.

Digital Library

[25]

J. Miller. Replicating software engineering experiments: a poisoned chalice or the holy grail. Information and Software Technology, 47(4):233--244, Mar. 2005.

Digital Library

[26]

J. Miller. Triangulation as a basis for knowledge discovery in software engineering. Empirical Softw. Engg., 13(2):223--228, 2008.

Digital Library

[27]

R. Mittelstaedt and T. Zorn. Econometric replication: Lessons from the experimental sciences. Quarterly Journal of Business & Economics, 23(1), 1984.

[28]

K. B. Monroe. Front matter. The Journal of Consumer Research, 19(1):i--iv, 1992.

[29]

H. Radder. Experimental reproducibility and the experimenters' regress. PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association, 1:63--73, 1992.

[30]

R. Rosenthal. Replication in behavioral research. Journal of Social Behavior and Personality, 4(4):1--30, 1990.

[31]

C. L. Sargent. The repeatability of significance and the significance of repeatability. European Journal of Parapsychology, 3:423--433, 1981.

[32]

S. Schmidt. Shall we really do it again? the powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2):90--100, 2009.

[33]

K. L. Shewmaker and C. W. Berenda. Science and the problem of psi. Philosophy of Science, 29(2):195--203, 1962.

[34]

F. Shull, J. Carver, S. Vegas, and N. Juristo. The role of replications in empirical software engineering. Empirical Softw. Engg., 13(2):211--218, 2008.

Digital Library

[35]

K. Singh, S. H. Ang, and S. M. Leong. Increasing replication for knowledge accumulation in strategy research. Journal of Management, 29(4):533--549, 2003.

[36]

E. Tsang and K.-M. Kwan. Replication and theory development in organizational science: A critical realist perspective. The Academy of Management Review, 24(4):759--780, 1999.

[37]

M. H. Van IJzendoorn. A process model of replication studies: on the relation between different types of replication. Leiden University Library, 1994.

Cited By

Kedron PHoller JBardin S(2024)A Survey of Researcher Perceptions of Replication in GeographyAnnals of the American Association of Geographers10.1080/24694452.2024.2415695(1-21)Online publication date: 6-Nov-2024
https://doi.org/10.1080/24694452.2024.2415695
Frattini JFucci DTorkar RMontgomery LUnterkalmsteiner MFischbach JMendez D(2024)Applying bayesian data analysis for causal inference about requirements quality: a controlled experimentEmpirical Software Engineering10.1007/s10664-024-10582-130:1Online publication date: 22-Nov-2024
https://doi.org/10.1007/s10664-024-10582-1
Wohlin CRuneson PHöst MOhlsson MRegnell BWesslén AWohlin CRuneson PHöst MOhlsson MRegnell BWesslén A(2024)Essential Areas in Empirical ResearchExperimentation in Software Engineering10.1007/978-3-662-69306-3_3(27-49)Online publication date: 8-May-2024
https://doi.org/10.1007/978-3-662-69306-3_3
Show More Cited By

Index Terms

Replications types in experimental disciplines
1. Software and its engineering

Recommendations

A Framework for Software Engineering Experimental Replications
ICECCS '08: Proceedings of the 13th IEEE International Conference on on Engineering of Complex Computer Systems

Experimental replications are very important to the advancement of empirical software engineering. Replications are one of the key mechanisms to confirm previous experimental findings. They are also used to transfer experimental knowledge, to train ...
Knowledge-Sharing Issues in Experimental Software Engineering

Recently the awareness of the importance of replicating studies has been growing in the empirical software engineering community. The results of any one study cannot simply be extrapolated to all environments because there are many uncontrollable ...
Investigations about replication of empirical studies in software engineering: preliminary findings from a mapping study
EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Context. A recent mapping study intended to verify the current state of replication of empirical studies in Software Engineering (SE) identified two sets of studies: the empirical studies reporting actually replications and a second group of studies ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '10: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

September 2010

423 pages

ISBN:9781450300391

DOI:10.1145/1852786

General Chair:
Giancarlo Succi
Free University of Bolzano, Italy
,
Program Chairs:
Maurizio Morisio
Politecnico di Torino, Italy
,
Nachi Nagappan
Microsoft Research

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 September 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

ESEM '10

Sponsor:

SIGSOFT

ESEM '10: 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

September 16 - 17, 2010

Bolzano-Bozen, Italy

Acceptance Rates

ESEM '10 Paper Acceptance Rate 30 of 102 submissions, 29%;

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

73
Total Citations
View Citations
641
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)2

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kedron PHoller JBardin S(2024)A Survey of Researcher Perceptions of Replication in GeographyAnnals of the American Association of Geographers10.1080/24694452.2024.2415695(1-21)Online publication date: 6-Nov-2024
https://doi.org/10.1080/24694452.2024.2415695
Frattini JFucci DTorkar RMontgomery LUnterkalmsteiner MFischbach JMendez D(2024)Applying bayesian data analysis for causal inference about requirements quality: a controlled experimentEmpirical Software Engineering10.1007/s10664-024-10582-130:1Online publication date: 22-Nov-2024
https://doi.org/10.1007/s10664-024-10582-1
Wohlin CRuneson PHöst MOhlsson MRegnell BWesslén AWohlin CRuneson PHöst MOhlsson MRegnell BWesslén A(2024)Essential Areas in Empirical ResearchExperimentation in Software Engineering10.1007/978-3-662-69306-3_3(27-49)Online publication date: 8-May-2024
https://doi.org/10.1007/978-3-662-69306-3_3
Nong YSharma RHamou-Lhadj ALuo XCai H(2023)Open Science in Software Engineering: A Study on Deep Learning-Based Vulnerability DetectionIEEE Transactions on Software Engineering10.1109/TSE.2022.320714949:4(1983-2005)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3207149
Gheran BVillarreal-Narvaez SVatavu RVanderdonckt JBottoni PPanizzi E(2022)RepliGES and GEStory: Visual Tools for Systematizing and Consolidating Knowledge on User-Defined GesturesProceedings of the 2022 International Conference on Advanced Visual Interfaces10.1145/3531073.3531112(1-9)Online publication date: 6-Jun-2022
https://dl.acm.org/doi/10.1145/3531073.3531112
Brendel AMuntermann J(2022)Replication of design theories: Reflections on function, outcome, and impactInformation Systems Journal10.1111/isj.1238732:6(1158-1176)Online publication date: 21-Apr-2022
https://doi.org/10.1111/isj.12387
Masapanta-Carrion SVelazquez-Iturbide J(2022)Replication of an Evaluation of Teacher Training in the Classification of Programming Exercises Using Bloom’s Taxonomy2022 IEEE Global Engineering Education Conference (EDUCON)10.1109/EDUCON52537.2022.9766707(1839-1848)Online publication date: 28-Mar-2022
https://doi.org/10.1109/EDUCON52537.2022.9766707
Morrison K(2022)Conceptual replications, research, and the “what works” agenda in educationEducational Research and Evaluation10.1080/13803611.2021.2022314(1-26)Online publication date: 31-Jan-2022
https://doi.org/10.1080/13803611.2021.2022314
Cruz MBernárdez BDurán AGuevara-Vega CRuiz-Cortés A(2022)A model-based approach for specifying changes in replications of empirical studies in computer ScienceComputing10.1007/s00607-022-01133-x105:6(1189-1213)Online publication date: 3-Dec-2022
https://doi.org/10.1007/s00607-022-01133-x
Brendel ALembcke TMuntermann JKolbe L(2021)Toward replication study types for design science researchJournal of Information Technology10.1177/0268396221100642936:3(198-215)Online publication date: 28-Apr-2021
https://doi.org/10.1177/02683962211006429
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents