More Web Proxy on the site http://driver.im/

short-paper

AutoSMP: an evaluation platform for sampling algorithms

Authors:

Sebastian Krieter,

Ina SchaeferAuthors Info & Claims

SPLC '21: Proceedings of the 25th ACM International Systems and Software Product Line Conference - Volume B

Pages 41 - 44

https://doi.org/10.1145/3461002.3473073

Published: 06 September 2021 Publication History

Abstract

Testing configurable systems is a challenging task due to the combinatorial explosion problem. Sampling is a promising approach to reduce the testing effort for product-based systems by finding a small but still representative subset (i.e., a sample) of all configurations for testing. The quality of a generated sample wrt. evaluation criteria such as run time of sample generation, feature coverage, sample size, and sampling stability depends on the subject systems and the sampling algorithm. Choosing the right sampling algorithm for practical applications is challenging because each sampling algorithm fulfills the evaluation criteria to a different degree. Researchers keep developing new sampling algorithms with improved performance or unique properties to satisfy application-specific requirements. Comparing sampling algorithms is therefore a necessary task for researchers. However, this task needs a lot of effort because of missing accessibility of existing algorithm implementations and benchmarks. Our platform AutoSMP eases practitioners and researchers lifes by automatically executing sampling algorithms on predefined benchmarks and evaluating the sampling results wrt. specific user requirements. In this paper, we introduce the open-source application of AutoSMP and a set of predefined benchmarks as well as a set of T-wise sampling algorithms as examples.

References

[1]

M. Acher, G. Perrouin, and M. Cordy. 2021. BURST: A BenchmarkingPlatform for Uniform Random Sampling Techniques. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM.

Digital Library

[2]

Mustafa Al-Hajjaji, Sebastian Krieter, Thomas Thüm, Malte Lochau, and Gunter Saake. 2016. IncLing: Efficient Product-line Testing Using Incremental Pairwise Sampling. In Proc. Int'l Conf. on Generative Programming: Concepts & Experiences (GPCE). ACM, 144--155.

Digital Library

[3]

Sven Apel, Alexander von Rhein, Philipp Wendler, Armin Größlinger, and Dirk Beyer. 2013. Strategies for Product-Line Verification: Case Studies and Experiments. In Proc. Int'l Conf. on Software Engineering (ICSE). IEEE, 482--491.

Digital Library

[4]

Thorsten Berger, Ralf Rublack, Divya Nair, Joanne M. Atlee, Martin Becker, Krzysztof Czarnecki, and Andrzej Wąsowski. 2013. A Survey of Variability Modeling in Industrial Practice. In Proc. Int'l Workshop on Variability Modelling of Software-Intensive Systems (VaMoS). ACM, 7:1--7:8.

Digital Library

[5]

Lianping Chen, Muhammad Ali Babar, and Nour Ali. 2009. Variability management in software product lines: a systematic review. In 13th International Software Product Line Conference. ACM.

Digital Library

[6]

Vasek Chvatal. 1979. A Greedy Heuristic for the Set-Covering Problem. Mathematics of operations research 4, 3 (1979), 233--235.

Digital Library

[7]

Thiago N. Ferreira, Jackson A. Prado Lima, Andrei Strickler, Josiel N. Kuk, Silvia R. Vergilio, and Aurora Pozo. 2017. Hyper-Heuristic Based Product Selection for Software Product Line Testing. Comp. Intell. Mag. (CIM) 12, 2 (2017), 34--45.

Digital Library

[8]

Thiago Nascimento Ferreira, Silvia Regina Vergilio, and Mauroane Kessentini. 2020. Many-objective Search-based Selection of Software Product Line Test Products with Nautilus. In SPLC. 1--4.

Digital Library

[9]

Christopher Henard, Mike Papadakis, and Yves Le Traon. 2014. Mutation-Based Generation of Software Product Line Test Configurations. In Search-Based Software Engineering, Claire Le Goues and Shin Yoo (Eds.). Lecture Notes in Computer Science, Vol. 8636. Springer International Publishing, 92--106.

[10]

Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, and Yves Le Traon. 2013. Multi-Objective Test Generation for Software Product Lines. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM, 62--71.

Digital Library

[11]

Martin Fagereng Johansen, Øystein Haugen, and Franck Fleurey. 2011. Properties of Realistic Feature Models Make Combinatorial Testing of Product Lines Feasible. In Proc. Int'l Conf. on Model Driven Engineering Languages and Systems (MODELS). Springer, 638--652.

Digital Library

[12]

Martin Fagereng Johansen, Øystein Haugen, Franck Fleurey, Anne Grete Eldegard, and Torbjørn Syversen. 2012. Generating Better Partial Covering Arrays by Modeling Weights on Sub-Product Lines. In Proc. Int'l Conf. on Model Driven Engineering Languages and Systems (MODELS). Springer, 269--284.

Digital Library

[13]

Kyo C. Kang, Sholom G. Cohen, James A. Hess, William E. Novak, and A. Spencer Peterson. 1990. Feature-Oriented Domain Analysis (FODA) Feasibility Study. Technical Report CMU/SEI-90-TR-21.

[14]

Sebastian Krieter, Thomas Thüm, Sandro Schulze, Gunter Saake, and Thomas Leich. 2020. YASA: Yet Another Sampling Algorithm. In Proc. Int'l Working Conf. on Variability Modelling of Software-Intensive Systems (VaMoS). ACM, Article 4, 10 pages.

Digital Library

[15]

Jihyun Lee, Sungwon Kang, and Danhyung Lee. 2012. A Survey on Software Product Line Testing. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM, 31--40.

Digital Library

[16]

Flávio Medeiros, Christian Kästner, Márcio Ribeiro, Sarah Nadi, and Rohit Gheyi. 2015. The Love/Hate Relationship with the C Preprocessor: An Interview Study. In 29th European Conference on Object-Oriented Programming (ECOOP 2015). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 495--518.

[17]

Changhai Nie and Hareton Leung. 2011. A Survey of Combinatorial Testing. Comput. Surveys 43, 2, Article 11 (2011), 11:1--11:29 pages.

Digital Library

[18]

José A Parejo, Ana B Sánchez, Sergio Segura, Antonio Ruiz-Cortés, Roberto E Lopez-Herrejon, and Alexander Egyed. 2016. Multi-objective test case prioritization in highly configurable systems: A case study. Journal of Systems and Software 122 (2016), 287--310.

Digital Library

[19]

Tobias Pett, Sebastian Krieter, Tobias Runge, Thomas Thüm, Malte Lochau, and Ina Schaefer. 2021. Stability of Product-Line Samplingin Continuous Integration. In 15th International Working Conference on Variability Modelling of Software-Intensive Systems. 1--9.

Digital Library

[20]

Tobias Pett, Thomas Thüm, Tobias Runge, Sebastian Krieter, Malte Lochau, and Ina Schaefer. 2019. Product Sampling for Product Lines: The Scalability Challenge. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM, 78--83.

Digital Library

[21]

Martin Potthast, Tim Gollub, Matti Wiegmann, and Benno Stein. 2019. TIRA integrated research architecture. In Information Retrieval Evaluation in a Changing World. Springer, 123--160.

[22]

Reimar Schröter, Sebastian Krieter, Thomas Thüm, Fabian Benduhn, and Gunter Saake. 2016. Feature-Model Interfaces: The Highway to Compositional Analyses of Highly-Configurable Systems. In Proc. Int'l Conf. on Software Engineering (ICSE). ACM, 667--678.

Digital Library

[23]

Joshua Sprey. 2020. Automated Comparison of Product Sampling Algorithms: Master's Thesis. (2020).

[24]

Mahsa Varshosaz, Mustafa Al-Hajjaji, Thomas Thüm, Tobias Runge, Mohammad Reza Mousavi, and Ina Schaefer. 2018. A Classification of Product Sampling for Software Product Lines. In Proc. Int'l Systems and Software Product Line Conf. (SPLC). ACM, 1--13.

Digital Library

Cited By

Sundermann CRaab HHeß TThüm TSchaefer I(2024)Reusing d-DNNFs for Efficient Feature-Model CountingACM Transactions on Software Engineering and Methodology10.1145/368046533:8(1-32)Online publication date: 30-Jul-2024
https://dl.acm.org/doi/10.1145/3680465
Sundermann CKuiter EHeß TRaab HKrieter SThüm T(2023)On the benefits of knowledge compilation for feature-model analysesAnnals of Mathematics and Artificial Intelligence10.1007/s10472-023-09906-692:5(1013-1050)Online publication date: 6-Nov-2023
https://dl.acm.org/doi/10.1007/s10472-023-09906-6
Heradio RFernandez-Amoros DGalindo JBenavides DBatory D(2022)Uniform and scalable sampling of highly configurable systemsEmpirical Software Engineering10.1007/s10664-021-10102-527:2Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1007/s10664-021-10102-5
Show More Cited By

Index Terms

AutoSMP: an evaluation platform for sampling algorithms
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Reusability
        Software product lines
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Stability of Product-Line Samplingin Continuous Integration
VaMoS '21: Proceedings of the 15th International Working Conference on Variability Modelling of Software-Intensive Systems

Companies strive to implement continuous integration into their development process to ensure the quality of their systems. Regression testing within the CI process considers the efficient re-test of systems after changes. However, even with regression ...
Line segment sampling with blue-noise properties

Line segment sampling has recently been adopted in many rendering algorithms for better handling of a wide range of effects such as motion blur, defocus blur and scattering media. A question naturally raised is how to generate line segment samples with ...
Sampling in software engineering research: a critical review and guidelines
Abstract
Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SPLC '21: Proceedings of the 25th ACM International Systems and Software Product Line Conference - Volume B

September 2021

148 pages

ISBN:9781450384704

DOI:10.1145/3461002

General Chairs:
Mohammad Reza Mousavi
King's College London, UK
,
Pierre-Yves Schobbens
University of Namur, Belgium

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 September 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Zentrum für digitale Innovation Niedersachsen

Conference

SPLC '21

Sponsor:

SIGSOFT

SPLC '21: 25th ACM International Systems and Software Product Line Conference

September 6 - 11, 2021

Leicester, United Kindom

Acceptance Rates

Overall Acceptance Rate 167 of 463 submissions, 36%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
119
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)2

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sundermann CRaab HHeß TThüm TSchaefer I(2024)Reusing d-DNNFs for Efficient Feature-Model CountingACM Transactions on Software Engineering and Methodology10.1145/368046533:8(1-32)Online publication date: 30-Jul-2024
https://dl.acm.org/doi/10.1145/3680465
Sundermann CKuiter EHeß TRaab HKrieter SThüm T(2023)On the benefits of knowledge compilation for feature-model analysesAnnals of Mathematics and Artificial Intelligence10.1007/s10472-023-09906-692:5(1013-1050)Online publication date: 6-Nov-2023
https://dl.acm.org/doi/10.1007/s10472-023-09906-6
Heradio RFernandez-Amoros DGalindo JBenavides DBatory D(2022)Uniform and scalable sampling of highly configurable systemsEmpirical Software Engineering10.1007/s10664-021-10102-527:2Online publication date: 1-Mar-2022
https://dl.acm.org/doi/10.1007/s10664-021-10102-5
Acher MPerrouin GCordy MMousavi MSchobbens P(2021)BURSTProceedings of the 25th ACM International Systems and Software Product Line Conference - Volume B10.1145/3461002.3473070(36-40)Online publication date: 6-Sep-2021
https://dl.acm.org/doi/10.1145/3461002.3473070

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents