More Web Proxy on the site http://driver.im/

research-article

Replication types: towards a shared taxonomy

Authors:

Maria Teresa Baldassarre,

Jeffrey Carver,

Natalia JuristoAuthors Info & Claims

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Article No.: 18, Pages 1 - 4

https://doi.org/10.1145/2601248.2601299

Published: 13 May 2014 Publication History

Abstract

Context: The software engineering community is becoming more aware of the need for experimental replications. In spite of the importance of this topic, there is still much inconsistency in the terminology used to describe replications.

Objective: Understand the perspectives of empirical researchers about various terms used to characterize replications and propose a consistent taxonomy of terms.

Method: A survey followed by plenary discussion during the 2013 International Software Engineering Research Network meeting.

Results: We propose a taxonomy which consolidates the disparate terminology. This taxonomy had a high level of agreement among workshop attendees.

Conclusion: Consistent terminology is important for any field to progress. This work is the first step in that direction. Additional study and discussion is still necessary.

References

[1]

Almqvist J. P. F, Replication of controlled experiments in empirical software engineering -- a survey. Master thesis, Department of computer science, faculty of science, Lund University, Sweden, 2006.

[2]

Victor R. Basili, Forrest J. Shull, and Filippo Lanubile. Building knowledge through families of experiments. IEEE Transactions on Software Engineering, 25(4):456--473, July/August 1999.

Digital Library

[3]

Brooks A., Daly J. W., Miller J., Roper M., Wood M. Replication's role in experimental computer science. Technical Report EFoCS-5-941 {RR/94/172}, Department of Computer Science, University of Strathclyde, Glasgow, Scotland, UK, 1995.

[4]

Carver J, Towards reporting guidelines for experimental replications: a proposal. In: RESER'2010: proceedings of the 1st international workshop on replication in empirical software engineering research. 2010

[5]

Carver J, Juristo N., Baldassarre M. T., Vegas S., Replications of Software Engineering Experiments - Editorial, Empirical Software Engineering Journal, DOI 10.1007/s10664-013-9290-8, online 05.12.2013

Digital Library

[6]

Gomez OS, Juristo N, Vegas S, Replications types in experimental disciplines. In: Proceedings of the 4th ACM-IEEE international symposium on empirical software engineering and measurement. ACM, New York, pp 3:1--3:10, 2010

Digital Library

[7]

International Software Engineering Research Network - ISERN, http://isern.iese.de/Portal/

[8]

Kitchenham B, The role of replications in empirical software engineering--a word of warning. EmpirSoftw Eng 13(2):219--221, 2008

Digital Library

[9]

Krein JL, Knutson CD, A case for replication: synthesizing research methodologies in software engineering. In: RESER2010: proceedings of the 1st international workshop on replication in empirical software engineering research. 2010

[10]

Lindsay R. M., Ehrenberg A. S. C. The design of replicated studies. American Statistician, 47(3):217--228, August 1993

[11]

RESER Workshop series, http://sequoia.cs.byu.edu/lab/?page=reser2013

[12]

Silva FQ, Suassuna M, Frana ACC, Grubb AM, Gouveia TB, Monteiro CV, Santos IE (2012) Replication of empirical studies in software engineering research: a systematic mapping study. Empirical Software Engineering Journal. DOI 10.1007/s10664-012-9227-7, pp 1--57.

Digital Library

[13]

Shull F, Carver J, Vegas S, Juristo N. The role of replications in empirical software engineering. Empirical software engineering 13(2) pp.211--218, 2008.

Digital Library

[14]

de Magalhaes C. V, da Silva FQ, Towards a taxonomy of replications in empirical software engineering research: a research proposal, In: proceedings of the 3rd international workshop on replication in empirical software engineering research. 2013, pp.50--55.

Digital Library

[15]

de Magalhães CV, da Silva FQ, Santos R. A preliminary analysis of conceptual studies about replication of empirical studies in software engineering, to appear in Proceedings of the 18^th international conference on Evaluation and Assessment in Software Engineering, London UK, 2014.

Cited By

Frattini JFucci DTorkar RMontgomery LUnterkalmsteiner MFischbach JMendez D(2024)Applying bayesian data analysis for causal inference about requirements quality: a controlled experimentEmpirical Software Engineering10.1007/s10664-024-10582-130:1Online publication date: 22-Nov-2024
https://doi.org/10.1007/s10664-024-10582-1
Kosar TKovačević ŽMernik MSlivnik B(2023)The Impact of Code Bloat on Genetic Program Comprehension: Replication of a Controlled Experiment on Semantic InferenceMathematics10.3390/math1117374411:17(3744)Online publication date: 31-Aug-2023
https://doi.org/10.3390/math11173744
Furia CTorkar RFeldt R(2022)Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code QualityACM Transactions on Software Engineering and Methodology10.1145/349095331:3(1-38)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.1145/3490953
Show More Cited By

Index Terms

Replication types: towards a shared taxonomy
1. Software and its engineering

Recommendations

Replication of Empirical Studies in Software Engineering: Preliminary Findings from a Systematic Mapping Study
RESER '11: Proceedings of the 2011 Second International Workshop on Replication in Empirical Software Engineering Research

Our goal in this study is to review the research related to the replication of empirical studies in software engineering in terms of replications of empirical studies and conceptual or theoretical work about replications. In this article we present the ...
Plug-in software engineering case studies
CESI '16: Proceedings of the 4th International Workshop on Conducting Empirical Studies in Industry

Empirical software engineering is a growing research area. Industrial experience gathered by systematic empirical case studies is extremely important for further evolution of the software engineering discipline. Scientific theory cannot provide ...
Replication of empirical studies in software engineering research: a systematic mapping study

In this article, we present a systematic mapping study of replications in software engineering. The goal is to plot the landscape of current published replications of empirical studies in software engineering research. We applied the systematic review ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

May 2014

486 pages

ISBN:9781450324762

DOI:10.1145/2601248

General Chair:
Martin Shepperd
Brunel University, UK
,
Program Chairs:
Tracy Hall
Brunel University
,
Ingunn Myrtveit
The Norwegian Business School

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Brunel University: Brunel University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

EASE '14

Sponsor:

Brunel University

EASE '14: 18th International Conference on Evaluation and Assessment in Software Engineering

May 13 - 14, 2014

England, London, United Kingdom

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

42
Total Citations
View Citations
232
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)2

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Frattini JFucci DTorkar RMontgomery LUnterkalmsteiner MFischbach JMendez D(2024)Applying bayesian data analysis for causal inference about requirements quality: a controlled experimentEmpirical Software Engineering10.1007/s10664-024-10582-130:1Online publication date: 22-Nov-2024
https://doi.org/10.1007/s10664-024-10582-1
Kosar TKovačević ŽMernik MSlivnik B(2023)The Impact of Code Bloat on Genetic Program Comprehension: Replication of a Controlled Experiment on Semantic InferenceMathematics10.3390/math1117374411:17(3744)Online publication date: 31-Aug-2023
https://doi.org/10.3390/math11173744
Furia CTorkar RFeldt R(2022)Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code QualityACM Transactions on Software Engineering and Methodology10.1145/349095331:3(1-38)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.1145/3490953
dos Santos Dde Almeida EAhmed I(2022)Investigating replication challenges through multiple replications of an experimentInformation and Software Technology10.1016/j.infsof.2022.106870147:COnline publication date: 1-Jul-2022
https://dl.acm.org/doi/10.1016/j.infsof.2022.106870
Cruz MBernárdez BDurán AGuevara-Vega CRuiz-Cortés A(2022)A model-based approach for specifying changes in replications of empirical studies in computer ScienceComputing10.1007/s00607-022-01133-x105:6(1189-1213)Online publication date: 3-Dec-2022
https://doi.org/10.1007/s00607-022-01133-x
Revoredo KDjurica DMendling J(2021)A study into the practice of reporting software engineering experimentsEmpirical Software Engineering10.1007/s10664-021-10007-326:6Online publication date: 18-Aug-2021
https://doi.org/10.1007/s10664-021-10007-3
Gren LBerntsson Svensson R(2021)Is it possible to disregard obsolete requirements? a family of experiments in software effort estimationRequirements Engineering10.1007/s00766-021-00351-7Online publication date: 12-Apr-2021
https://doi.org/10.1007/s00766-021-00351-7
Vilela JFerrari A(2021)SaPeer Approach for Training Requirements Analysts: An Application Tailored to a Low-resource ContextRequirements Engineering: Foundation for Software Quality10.1007/978-3-030-73128-1_14(191-207)Online publication date: 2-Apr-2021
https://doi.org/10.1007/978-3-030-73128-1_14
Rivera MFahey TLópez-Taylor JMartínez J(2020)The Association of Aquaporin-1 Gene with Marathon Running Performance Level: a Confirmatory Study Conducted in Male Hispanic Marathon RunnersSports Medicine - Open10.1186/s40798-020-00243-06:1Online publication date: 20-Mar-2020
https://doi.org/10.1186/s40798-020-00243-0
Romano SScanniello GBaldassarre MFucci D(2020)On the Effect of Noise on Software Engineers’ Performance: Results from Two Replicated Experiments2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)10.1109/SEAA51224.2020.00062(334-341)Online publication date: Aug-2020
https://doi.org/10.1109/SEAA51224.2020.00062
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents