More Web Proxy on the site http://driver.im/

research-article

The effect of the number of inspectors on the defect estimates produced by capture-recapture models

Authors:

Gursimran Singh Walia,

Jeffrey C. Carver,

Nachiappan NagappanAuthors Info & Claims

ICSE '08: Proceedings of the 30th international conference on Software engineering

Pages 331 - 340

https://doi.org/10.1145/1368088.1368134

Published: 10 May 2008 Publication History

Abstract

Inspections can be made more cost-effective by using capture-recapture methods to estimate post-inspection defects. Previous capture-recapture studies of inspections used relatively small data sets compared with those used in biology and wildlife research (the origin of the models). A common belief is that capture-recapture models underestimate the number of defects but their performance can be improved with data from more inspectors. This increase has not been evaluated in detail. This paper evaluates new estimators from biology not been previously applied to inspections. Using a data from seventy-three inspectors, we analyze the effect of the number of inspectors on the quality of estimates. Contrary to previous findings indicating that Jackknife is the best estimator, our results show that the SC estimators are better suited to software inspections. Our results also provide a detailed analysis of the number of inspectors necessary to obtain estimates within 5% to 20% of the actual.

References

[1]

Ackerman, A., Buchwald, L., and Lewski, F., "Software Inspections: An Effective Verification Process." IEEE Software, 1989. 6(3): 31--36.

Digital Library

[2]

Briand, L.C., Emam, K.E., and B.G.Freimut. "A Comparison and Integration of Capture-Recapture ". In Proceedings of the 9th International Symposium on Software Reliability Engineering. 1998. Paderborn, Germany: 32--41

Digital Library

[3]

Briand, L.C., Emam, K.E., Freimut, B.G., and Laitenberger, O., "A Comprehensive Evaluation of Capture Recapture Models for Estimating Software Defect Content." IEEE Transactions on Software Engineering, 2000. 26(6): 518--539.

Digital Library

[4]

Burnham, K.P. and Overtom, W.S., "Estimation of the Size of a Closed Population When Capture Probabilities Vary Among Animals." Biometrica, 1978. 65: 625--633.

[5]

Chao, A., "Estimation the population Size for Capture-Recapture Data with Unequal Catchability." Biometrics, 1987. 43(4): 783--791.

[6]

Chao, A., "Estimating Animal Abundance with Capture Frequency Data." Journal of Wildlife Management, 1988. 52(2): 295--300.

[7]

Chao, A. and Yeng, H.C., Program CARE-2 (for Capture-Recapture Part.2), http://chao.stat.nthu.edu.tw, 2003.

[8]

Darroch, J.N., "The Multiple-Recapture Conensus 1: Estimation of a Closed Population." Biometrika, 1958. 45: 343--359.

[9]

Ebrahimi, N.B., "On the Statisctical Analysis of the Number of Errors Remaining in a Software Design Document after Inspection." IEEE Transactions on Software Engineering, 1997. 23(8): 529--532.

Digital Library

[10]

Eick, S., Loader, C., Long, M., Votta, L., and Weil, S.V. "Estimating Software Fault Content Before Coding". In Proceedings of the 14th International Conference on Software Engineering. 1992. Melbourne, Australia: ACM Press: 59--65

Digital Library

[11]

Eick, S., Loader, C., Weil, S.V., and Votta, L. "How Many Errors Remain in a Software Design after Inspection". In Proceedings of the 25th Symposium on the Interface. 1993:

[12]

El-Emam, K., Laitenberger, O., and Harbrich, T., "The Application of Subjective Estimates of Effectiveness to Controlling Software Inspections " Journal of Systems and Software, 2000. 54(2): 119--136.

Digital Library

[13]

El-Emam, K. and Laitenberger, O., "Evaluating Capture-Recapture Models with Two Inspectors." IEEE Transactions on Software Engineering, 2001. 27(9): 851--864.

Digital Library

[14]

Lee, S.M. and Chao, A., "Estimating Population Size via Sample Coverage for Closed Capture-Recapture Models." Biometrics, 1994. 50: 88--97.

[15]

Miller, J., "Estimating the Number of Remaining Defects after Inspection." Software Testing, Verification and Reliability, 1999. 9(3): 167--189.

[16]

Otis, D., Burnham, K., White, G., and Anderson, D., "Statistical Inference from Capture Data on Closed Animal Population." Wildlife Monograph, 1978. 64: 1--135.

[17]

Petersson, H., Thelin, T., Runeson, P., and Wohlin, C., "Capture-Recapture in Software Inspections after 10 Years Research - Theory, Evaluation and Application." Journal of Systems and Software, 2003.

[18]

Runeson, P. and Wohlin, C., "An Experimental Evaluation of an Experience-Based Capture-Recapture Method in Software Code Inspections." Empirical Software Engineering: An International Journal, 1998. 3(4): 381--406.

Digital Library

[19]

Shull, F., Carver, J., and Travassos, G. "An Empirical Methodology for Introducing Software Processes". In In Proceedings of Joint 8th European Software Engineering Conference and 9th ACM SIGSOFT Foundations of Software Engineering. 2001. Vienna, Austria: 288--296

Digital Library

[20]

Thelin, T. and Runeson, P. "Capture-Recapture Estimators for perspective Based Reading - A Simulating Experiment". In Proceedings of the International Conference on Product Focused Software Process Improvement. 1999:

[21]

Thelin, T., Petersson, P., and Runeson, P., "Confidence Intervals for Capture-Recapture Estimations in Software Inspections." Journal of Information and Software Technology, 2002. 44(12): 683--702.

[22]

Weil, S.V. and Votta, L., "Assessing Software Designs Using Capture-Recapture Methoda." IEEE Transactions on Software Engineering, 1993. 19(11): 1045--1054.

Digital Library

[23]

White, G.C., Anderson, D.R., Burnham, K.p., and Otis, D.l., Capture-Recapture and Removal Methods for Sampling Closed Populations, Los Alomos National Laboratory, 1982.

[24]

Wohlin, C., Runeson, P., and Brantestam, J., "An Experimental Evaluation of Capture-Recapture in Software Inspections." Software Testing, Verification and Reliability, 1995. 5(4): 213--232.

[25]

Wohlin, C. and Runeson, P. "Defect Content Estimation from Review Data". In Proceedings of the 20th International Conference on Software Engineering. 1998. Kyoto, Japan: IEEE Computer Society Press: 400--409

Digital Library

[26]

Yip, P.S.F., "A Martingale Estimating Equation for a Capture-Recapture Experiment in Discrete Time." Biometrics, 1991. 47: 1081--1088.

Cited By

Zhu XNiu BWhitehead ESun Z(2018)An empirical study of software change classification with imbalance data‐handling methodsSoftware: Practice and Experience10.1002/spe.260648:11(1968-1999)Online publication date: 29-Jun-2018
https://doi.org/10.1002/spe.2606
Rong GLiu BZhang HZhang QShao DMendes ECounsell SPetersen K(2017)Towards Confidence with Capture-recapture EstimationProceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering10.1145/3084226.3084250(242-251)Online publication date: 15-Jun-2017
https://dl.acm.org/doi/10.1145/3084226.3084250
Zhang QRong GZhang HBeecham SKitchenham BMacDonell S(2016)An empirical study on independence-driven data selection for improving capture-recapture estimationProceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering10.1145/2915970.2915991(1-10)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1145/2915970.2915991
Show More Cited By

Index Terms

Recommendations

A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content

An important requirement to control the inspection of software artifacts is to be able to decide, based on more objective information, whether the inspection can stop or whether it should continue to achieve a suitable level of artifact quality. A ...
Evaluating Capture-Recapture Models with Two Inspectors

Capture-recapture (CR) models have been proposed as an objective method for controlling software inspections. CR models were originally developed to estimate the size of animal populations. In software, they have been used to estimate the number of ...
Evaluation of capture-recapture models for estimating the abundance of naturally-occurring defects
ESEM '08: Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement

Project managers can use capture-recapture models to manage the inspection process by estimating the number of defects present in an artifact and determining whether a reinspection is necessary. Researchers have previously evaluated capture-recapture ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '08: Proceedings of the 30th international conference on Software engineering

May 2008

558 pages

ISBN:9781605580791

DOI:10.1145/1368088

General Chair:
Wilhelm Schäfer
University of Paderborn
,
Program Chairs:
Matthew B. Dwyer
University of Nebraska
,
Volker Gruhn
University of Leipzig

Copyright © 2008 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 May 2008

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '08

Sponsor:

ICSE '08: International Conference on Software Engineering

May 10 - 18, 2008

Leipzig, Germany

Acceptance Rates

ICSE '08 Paper Acceptance Rate 56 of 370 submissions, 15%;

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
455
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhu XNiu BWhitehead ESun Z(2018)An empirical study of software change classification with imbalance data‐handling methodsSoftware: Practice and Experience10.1002/spe.260648:11(1968-1999)Online publication date: 29-Jun-2018
https://doi.org/10.1002/spe.2606
Rong GLiu BZhang HZhang QShao DMendes ECounsell SPetersen K(2017)Towards Confidence with Capture-recapture EstimationProceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering10.1145/3084226.3084250(242-251)Online publication date: 15-Jun-2017
https://dl.acm.org/doi/10.1145/3084226.3084250
Zhang QRong GZhang HBeecham SKitchenham BMacDonell S(2016)An empirical study on independence-driven data selection for improving capture-recapture estimationProceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering10.1145/2915970.2915991(1-10)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1145/2915970.2915991
Sauro JLewis J(2016)What sample sizes do we need? Part 2: formative studiesQuantifying the User Experience10.1016/B978-0-12-802308-2.00007-2(143-183)Online publication date: 2016
https://doi.org/10.1016/B978-0-12-802308-2.00007-2
Rong GZhang HShao D(2015)Does Detecting more Defects Contribute to Better Estimation? An Empirical Investigation to the Capture-Recapture MethodProceedings of the 2015 24th Australasian Software Engineering Conference (ASWEC)10.1109/ASWEC.2015.35(233-241)Online publication date: 28-Sep-2015
https://dl.acm.org/doi/10.1109/ASWEC.2015.35
Shan QRong GZhang HLiu GShao D(2015)An Empirical Evaluation of Capture-Recapture Estimators in Software InspectionProceedings of the 2015 24th Australasian Software Engineering Conference (ASWEC)10.1109/ASWEC.2015.17(58-67)Online publication date: 28-Sep-2015
https://dl.acm.org/doi/10.1109/ASWEC.2015.17
Mäntylä MPetersen KLehtinen TLassenius CJalote PBriand LHoek A(2014)Time pressure: a controlled experiment of test case development and requirements reviewProceedings of the 36th International Conference on Software Engineering10.1145/2568225.2568245(83-94)Online publication date: 31-May-2014
https://dl.acm.org/doi/10.1145/2568225.2568245
Walia GCarver J(2013)Using capture-recapture models to make objective post-inspection decisions2013 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)10.1109/ISSREW.2013.6688868(54-54)Online publication date: Nov-2013
https://doi.org/10.1109/ISSREW.2013.6688868
Mäntylä MItkonen J(2013)More testers - The effect of crowd size and time restriction in software testingInformation and Software Technology10.1016/j.infsof.2012.12.00455:6(986-1003)Online publication date: 1-Jun-2013
https://dl.acm.org/doi/10.1016/j.infsof.2012.12.004
Kläs MElberzhager FMünch JHartjes Kvon Graevemeyer OKramer JBishop JDevanbu PUchitel S(2010)Transparent combination of expert and measurement data for defect predictionProceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 210.1145/1810295.1810313(119-128)Online publication date: 1-May-2010
https://dl.acm.org/doi/10.1145/1810295.1810313
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents