More Web Proxy on the site http://driver.im/

research-article

Guidelines for benchmarking automated software traceability techniques

Authors:

Jane Huffman Hayes,

Jane Cleland-HuangAuthors Info & Claims

SST '15: Proceedings of the 8th International Symposium on Software and Systems Traceability

Pages 61 - 67

Published: 16 May 2015 Publication History

Abstract

To comparatively evaluate automated traceability solutions, we need to develop standardized benchmarks. However there is currently no consensus on how a benchmark should be constructed and used to evaluate competing techniques. In this paper we discuss recurring problems in evaluating traceability techniques, identify essential properties that evaluation methods should possess, and provide guidelines for benchmarking software traceability techniques. We illustrate the properties and guidelines using empirical evaluation of three software traceability techniques on nine data sets.

References

[1]

TEFSE challenge, The 6th International Workshop on Traceability in Emerging Forms of Software Engineering, 2011.

[2]

1st workshop on the next 5 years of text analysis in software maintenance. https://dibt.unimol.it/TAinSM2012/pages/program.html, 2012.

[3]

G. Antoniol, G. Canfora, G. Casazza, A. D. Lucia, and E. Merlo. Recovering traceability links between code and documentation. IEEE Trans. on Software Eng., 28(10):970--983, 2002.

Digital Library

[4]

A. Arcuri and L. C. Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proc. of Intn'l Conf. on Software Eng., ICSE '11, pages 1--10, 2011.

Digital Library

[5]

M. Borg, P. Runeson, and A. Ard. Recovering from a decade: a systematic mapping of information retrieval approaches to software traceability. Empirical Software Engineering, 19(6):1565--1616, 2014.

Digital Library

[6]

L. Briand, S. Morasca, and V. Basili. Property-based software engineering measurement. IEEE Trans. on Software Eng., 22(1):68--86, January 1996.

Digital Library

[7]

D. Cuddeback, A. Dekhtyar, and J. Hayes. Automated requirements traceability: The study of human analysts. In Proc. of Requirements Eng. Conf., pages 231--240, Los Alamitos, CA, USA, 2010.

Digital Library

[8]

J. Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7(Jan):1--30, 2006.

Digital Library

[9]

M. Gethers, R. Oliveto, D. Poshyvanyk, and A. D. Lucia. On integrating orthogonal information retrieval methods to improve traceability recovery. In Proc. of Intn'l Conf. on Software Maintenance, 2011.

Digital Library

[10]

J. H. Hayes, A. Dekhtyar, and S. K. Sundaram. Advancing candidate link generation for requirements tracing: The study of methods. IEEE Trans. on Software Eng., 32(1):4--19, 2006.

Digital Library

[11]

J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl. Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst., 22:5--53, January 2004.

Digital Library

[12]

E. A. Holbrook, J. Huffman Hayes, and A. Dekhtyar. Toward automating requirements satisfaction assessment. In Proc. of Requirements Eng. Conf., pages 149--158, 2009.

Digital Library

[13]

Keenan et al. Tracelab: An experimental workbench for equipping researchers to innovate, synthesize, and comparatively evaluate traceability solutions. In Tool Demo: Int'l Conf. on S/W Eng., 2012.

Digital Library

[14]

B. Kitchenham and S. Charters. Guidelines for performing systematic literature reviews in software engineering. Keele University and Durham University Joint Report, 2007.

[15]

Y. Ma and B. Cukic. Adequate and precise evaluation of quality models in software engineering studies. In Workshop on Predictor Models in Software Eng., PROMISE '07, pages 1--, Washington, DC, USA, 2007.

Digital Library

[16]

C. D. Manning, P. Raghavan, and H. Schuätze. Introduction to Information Retrieval. Cambridge University Press, NY, USA, 2008.

Digital Library

[17]

A. Marcus and J. Maletic. Recovering documentation-to-source-code traceability links using latent semantic indexing. In Proc. of Intn'l Conf. on Software Eng., pages 125--135, May 2003.

Digital Library

[18]

Y. Shin and J. Cleland-Huang. A comparative evaluation of two user feedback techniques for requirements trace retrieval. In Symposium on Applied Computing, 2012.

Digital Library

[19]

Y. Shin, J. Hayes, and J. Cleland-Huang. A framework for evaluating traceability benchmark metrics. March 2012.

[20]

S. E. Sim, S. Easterbrook, and R. C. Holt. Using benchmarking to advance research: a challenge to software engineering. In Proc. of Intn'l Conf. on Software Eng., ICSE '03, pages 74--83, 2003.

Digital Library

[21]

S. K. Sundaram, J. H. Hayes, A. Dekhtyar, and E. A. Holbrook. Assessing traceability of software engineering artifacts. Requir. Eng., 15:313--335, September 2010.

Digital Library

[22]

E. Weyuker. Evaluating software complexity measures. IEEE Trans. on S/W Eng., 14(9):1357--1365, 9/1988.

Digital Library

[23]

E. J. Weyuker, R. M. Bell, and T. J. Ostrand. We're finding most of the bugs, but what are we missing? In Proc. of the 3rd Intn'l Conf. on Software Testing, Verification and Validation, pages 313--322, Washington, DC, USA, 2010.

Digital Library

Cited By

Deshpande GRuhe G(2020)Beyond AccuracyProceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3382494.3422159(1-6)Online publication date: 5-Oct-2020
https://dl.acm.org/doi/10.1145/3382494.3422159
Liu YLin JCleland-Huang J(2020)Traceability Support for Multi-Lingual Software ProjectsProceedings of the 17th International Conference on Mining Software Repositories10.1145/3379597.3387440(443-454)Online publication date: 29-Jun-2020
https://dl.acm.org/doi/10.1145/3379597.3387440
Rath MRendall JGuo JCleland-Huang JMäder PChaudron MCrnkovic IChechik MHarman M(2018)Traceability in the wildProceedings of the 40th International Conference on Software Engineering10.1145/3180155.3180207(834-845)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3180155.3180207
Show More Cited By

Index Terms

Guidelines for benchmarking automated software traceability techniques
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics
2. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Documentation
  2. Software notations and tools

Recommendations

Guidelines for Benchmarking Automated Software Traceability Techniques
SST '15: Proceedings of the 2015 IEEE/ACM 8th International Symposium on Software and Systems Traceability

To comparatively evaluate automated trace ability solutions, we need to develop standardized benchmarks. However there is currently no consensus on how a benchmark should be constructed and used to evaluate competing techniques. In this paper we discuss ...
Toward improved traceability of non-functional requirements
TEFSE '05: Proceedings of the 3rd international workshop on Traceability in emerging forms of software engineering

This position paper examines current practices and challenges for tracing non-functional requirements (NFRs). Anecdotal evidence suggests that many organizations do not effectively trace NFRs and that functional changes are often implemented with very ...
Towards a benchmark for traceability
IWPSE-EVOL '11: Proceedings of the 12th International Workshop on Principles of Software Evolution and the 7th annual ERCIM Workshop on Software Evolution

Rigorously evaluating and comparing traceability link generation techniques is a challenging task. In fact, traceability is still expensive to implement and it is therefore difficult to find a complete case study that includes both a rich set of ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SST '15: Proceedings of the 8th International Symposium on Software and Systems Traceability

May 2015

78 pages

General Chair:
Patrick Mäder
Technische Universität Ilmenau, Germany
,
Program Chair:
Rocco Oliveto
University of Molise, Italy

Sponsors

ACM: Association for Computing Machinery
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS\DATC: IEEE Computer Society
TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

IEEE Press

Publication History

Published: 16 May 2015

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '15

Sponsor:

ACM
SIGSOFT
IEEE-CS\DATC
TCSE

ICSE '15: 37th International Conference on Software Engineering

May 16 - 24, 2015

Florence, Italy

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
114
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)1

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Deshpande GRuhe G(2020)Beyond AccuracyProceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1145/3382494.3422159(1-6)Online publication date: 5-Oct-2020
https://dl.acm.org/doi/10.1145/3382494.3422159
Liu YLin JCleland-Huang J(2020)Traceability Support for Multi-Lingual Software ProjectsProceedings of the 17th International Conference on Mining Software Repositories10.1145/3379597.3387440(443-454)Online publication date: 29-Jun-2020
https://dl.acm.org/doi/10.1145/3379597.3387440
Rath MRendall JGuo JCleland-Huang JMäder PChaudron MCrnkovic IChechik MHarman M(2018)Traceability in the wildProceedings of the 40th International Conference on Software Engineering10.1145/3180155.3180207(834-845)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3180155.3180207
Rahimi MCleland-Huang J(2018)Evolving software trace links between requirements and source codeEmpirical Software Engineering10.1007/s10664-017-9561-x23:4(2198-2231)Online publication date: 1-Aug-2018
https://dl.acm.org/doi/10.1007/s10664-017-9561-x
Shi LChen CWang QLi SBoehm BRosu GDi Penta MNguyen T(2017)Understanding feature requests by leveraging fuzzy method and linguistic analysisProceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering10.5555/3155562.3155619(440-450)Online publication date: 30-Oct-2017
https://dl.acm.org/doi/10.5555/3155562.3155619
Guo JRahimi MCleland-Huang JRasin AHayes JVierhauser MKim MRobbes RBird C(2016)Cold-start software analyticsProceedings of the 13th International Conference on Mining Software Repositories10.1145/2901739.2901740(142-153)Online publication date: 14-May-2016
https://dl.acm.org/doi/10.1145/2901739.2901740

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents