[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
10.1145/3643664.3648213acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article
Open access

Apples, Oranges, and Software Engineering: Study Selection Challenges for Secondary Research on Latent Variables

Published: 09 August 2024 Publication History

Abstract

Software engineering (SE) is full of abstract concepts that are crucial for both researchers and practitioners, such as programming experience, team productivity, code comprehension, and system security. Secondary studies aimed at summarizing research on the influences and consequences of such concepts would therefore be of great value.
However, the inability to measure abstract concepts directly poses a challenge for secondary studies: primary studies in SE can operationalize such concepts in many ways. Standardized measurement instruments are rarely available, and even if they are, many researchers do not use them or do not even provide a definition for the studied concept. SE researchers conducting secondary studies therefore have to decide a) which primary studies intended to measure the same construct, and b) how to compare and aggregate vastly different measurements for the same construct.
In this experience report, we discuss the challenge of study selection in SE secondary research on latent variables. We report on two instances where we found it particularly challenging to decide which primary studies should be included for comparison and synthesis, so as not to end up comparing apples with oranges. Our report aims to spark a conversation about developing strategies to address this issue systematically and pave the way for more efficient and rigorous secondary studies in software engineering.

References

[1]
Luis Anunciacao. 2018. An Overview of the History and Methodological Aspects of Psychometrics: History and Methodological Aspects of Psychometrics. Journal for ReAttach Therapy and Developmental Diversities 1, 1 (2018), 44--58.
[2]
Pearl Brereton, Barbara A. Kitchenham, David Budgen, Mark Turner, and Mohamed Khalil. 2007. Lessons from Applying the Systematic Literature Review Process within the Software Engineering Domain. Journal of systems and software 80, 4 (2007), 571--583.
[3]
Lionel Briand, Khaled El Emam, and Sandro Morasca. 1996. On the Application of Measurement Theory in Software Engineering. Empirical Software Engineering 1, 1 (1996), 61--88.
[4]
Timothy A. Brown. 2015. Confirmatory Factor Analysis for Applied Research. Guilford publications.
[5]
Teresa Busjahn, Roman Bednarik, Andrew Begel, Martha Crosby, James H Paterson, Carsten Schulte, Bonita Sharif, and Sascha Tamm. 2015. Eye movements in code reading: Relaxing the linear order. In 2015 IEEE 23rd International Conference on Program Comprehension. IEEE, 255--265.
[6]
John B Carroll. 2005. The three-stratum theory of cognitive abilities. Contemporary intellectual assessment: theories, tests, and issues (2005).
[7]
Jeffrey C. Carver, Natalia Juristo, Maria Teresa Baldassarre, and Sira Vegas. 2014. Replications of Software Engineering Experiments. Empirical Software Engineering 19, 2 (April 2014), 267--276.
[8]
Denis Cousineau. 2005. The Rise of Quantitative Methods in Psychology. Tutorials in Quantitative Methods for Psychology 1, 1 (2005), 1--3.
[9]
Lee J. Cronbach and Paul E. Meehl. 1955. Construct Validity in Psychological Tests. Psychological bulletin 52, 4 (1955), 281.
[10]
Daniela S. Cruzes and Tore Dybå. 2011. Research Synthesis in Software Engineering: A Tertiary Study. Information and Software Technology 53, 5 (2011), 440--455.
[11]
Fabio QB Da Silva, Andre LM Santos, Sérgio Soares, A. César C. França, Cleviton VF Monteiro, and Felipe Farias Maciel. 2011. Six Years of Systematic Literature Reviews in Software Engineering: An Updated Tertiary Study. Information and Software Technology 53, 9 (2011), 899--913.
[12]
Ray Dawson, Phil Bones, Briony J. Oates, Pearl Brereton, Motoei Azuma, and Mary Lou Jackson. 2003. Empirical Methodologies in Software Engineering. In Eleventh Annual International Workshop on Software Technology and Engineering Practice. IEEE, 52--58.
[13]
Rebecca DerSimonian and Nan Laird. 1986. Meta-Analysis in Clinical Trials. Controlled clinical trials 7, 3 (1986), 177--188.
[14]
Rebecca DerSimonian and Nan Laird. 2015. Meta-Analysis in Clinical Trials Revisited. Contemporary clinical trials 45 (2015), 139--145.
[15]
Robert F. DeVellis. 2006. Classical Test Theory. Medical care (2006), S50--S59. jstor:41219505
[16]
Susan E. Embretson and Steven P. Reise. 2013. Item Response Theory. Psychology Press.
[17]
R. Michael Furr. 2021. Psychometrics: An Introduction. SAGE publications.
[18]
Daniel Graziotin, Per Lenberg, Robert Feldt, and Stefan Wagner. 2021. Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines. ACM Trans. Softw. Eng. Methodol. 31, 1, Article 7 (sep 2021), 36 pages.
[19]
Thom Holwerda. 2008. WTFs/m. https://web.archive.org/web/20231122170548/ https://www.osnews.com/story/19266/wtfsm/
[20]
Seyedrebvar Hosseini, Burak Turhan, and Dimuthu Gunarathna. 2017. A Systematic Literature Review and Meta-Analysis on Cross Project Defect Prediction. IEEE Transactions on Software Engineering 45, 2 (2017), 111--147.
[21]
Letizia Jaccheri, Zamira Kholmatova, and Giancarlo Succi. 2021. Systematizing the Meta-Analytical Process in Software Engineering. In 2021 2nd European Symposium on Software Engineering. ACM, Larissa Greece, 1--5.
[22]
Andreas Jedlitschka and Marcus Ciolkowski. 2004. Towards Evidence in Software Engineering. In Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE'04. IEEE, 261--270.
[23]
Vigdis By Kampenes, Tore Dybå, Jo E. Hannay, and Dag IK Sjøberg. 2007. A Systematic Review of Effect Size in Software Engineering Experiments. Information and Software Technology 49, 11-12 (2007), 1073--1086.
[24]
Barbara Kitchenham. 2008. The Role of Replications in Empirical Software Engineering---a Word of Warning. Empirical Software Engineering 13, 2 (April 2008), 219--221.
[25]
Barbara Kitchenham, Lech Madeyski, and David Budgen. 2022. SEGRESS: Software Engineering Guidelines for Reporting Secondary Studies. IEEE Transactions on Software Engineering 49, 3 (2022), 1273--1298.
[26]
Barbara Kitchenham, Rialette Pretorius, David Budgen, O. Pearl Brereton, Mark Turner, Mahmood Niazi, and Stephen Linkman. 2010. Systematic Literature Reviews in Software Engineering-a Tertiary Study. Information and software technology 52, 8 (2010), 792--805.
[27]
Barbara Ann Kitchenham. 2015. Evidence-Based Software Engineering and Systematic Reviews. Chapman & Hall / CRC Innovations in Software Engineering and Software Development Series, Vol. v.4. CRC Press, Boca Raton.
[28]
Ian McChesney and Raymond Bond. 2019. Eye tracking analysis ofcomputer program comprehension in programmers with dyslexia. Empirical Software Engineering 24, 3 (2019), 1109--1154.
[29]
Robert R. McCrae. 2020. The Five-Factor Model of personality traits: consensus and controversy. In The Cambridge Handbook of Personality Psychology (2nd ed.). Cambridge University Press.
[30]
James Miller. 2000. Applying Meta-Analytical Procedures to Software Engineering Experiments. Journal of Systems and Software 54, 1 (Jan. 2000), 29--39.
[31]
David Moher, Alison Jones, Leah Lepage, and for the CONSORT Group. 2001. Use of the CONSORT Statement and Quality of Reports of Randomized TrialsA Comparative Before-and-After Evaluation. JAMA 285, 15 (April 2001), 1992--1995.
[32]
Marvin Muñoz Barón, Marvin Wyrich, and Stefan Wagner. 2020. An empirical validation of cognitive complexity as a measure of source code understandability. In Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) (Bari, Italy). Association for Computing Machinery, New York, NY, USA, Article 5, 12 pages.
[33]
P David Pearson and Gina N Cervetti. 2015. Fifty years of reading comprehension theory and practice. Research-based practices for teaching Common Core literacy (2015), 1--24.
[34]
Norman Peitek, Janet Siegmund, and Sven Apel. 2020. What Drives the Reading Order of Programmers? An Eye Tracking Study. In Proceedings of the 28th International Conference on Program Comprehension (Seoul, Republic of Korea) (ICPC '20). Association for Computing Machinery, New York, NY, USA, 342--353.
[35]
Kai Petersen, Robert Feldt, Shahid Mujtaba, and Michael Mattsson. 2008. Systematic Mapping Studies in Software Engineering. In 12th International Conference on Evaluation and Assessment in Software Engineering (EASE) 12. 1--10.
[36]
Paul Ralph and Sebastian Baltes. 2022. Paving the Way for Mature Secondary Research: The Seven Types of Literature Review. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore) (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 1632--1636.
[37]
Paul Ralph and Ewan Tempero. 2018. Construct Validity in Software Engineering Research and Software Metrics. In Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018 (Christchurch, New Zealand) (EASE '18). Association for Computing Machinery, New York, NY, USA, 13--23.
[38]
Martin Schmettow and Wolfgang Vietze. 2008. Introducing Item Response Theory for Measuring Usability Inspection Processes. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Florence Italy, 893--902.
[39]
Janet Siegmund. 2022. MIP Talk: Measuring Programming Experience: 2012 vs. 2022. https://www.youtube.com/watch?v=_OwQiUyVp-M
[40]
Janet Siegmund, Christian Kästner, Jörg Liebig, Sven Apel, and Stefan Hanenberg. 2014. Measuring and modeling programming experience. Empirical Software Engineering 19 (2014), 1299--1334.
[41]
I. Simera, D. Moher, J. Hoey, K. F. Schulz, and D. G. Altman. 2010. A Catalogue of Reporting Guidelines for Health Research. European Journal of Clinical Investigation 40, 1 (Jan. 2010), 35--53.
[42]
Dag IK Sjøberg and Gunnar R. Bergersen. 2023. Improving the Reporting of Threats to Construct Validity. arXiv preprint arXiv:2306.05336 (2023). arXiv:2306.05336
[43]
Dag I.K. Sjøberg and Gunnar R. Bergersen. 2022. Construct Validity in Software Engineering. IEEE Transactions on Software Engineering 49, 3 (2022), 1374--1396.
[44]
Gregory T. Smith. 2005. On Construct Validity: Issues of Method and Measurement. Psychological assessment 17, 4 (2005), 396.
[45]
Ewan Tempero and Paul Ralph. 2016. A Model for Defining Coupling Metrics. In 2016 23rd Asia-Pacific Software Engineering Conference (APSEC). IEEE, 145--152.
[46]
Drew Westen and Robert Rosenthal. 2003. Quantifying Construct Validity: Two Simple Measures. Journal of personality and social psychology 84, 3 (2003), 608.
[47]
Marvin Wyrich. 2023. Source Code Comprehension: A Contemporary Definition and Conceptual Model for Empirical Investigation. arXiv:2310.11301 [cs.SE]
[48]
Marvin Wyrich, Justus Bogner, and Stefan Wagner. 2023. 40 Years of Designing Code Comprehension Experiments: A Systematic Mapping Study. ACM Comput. Surv. (2023), 42 pages.

Index Terms

  1. Apples, Oranges, and Software Engineering: Study Selection Challenges for Secondary Research on Latent Variables

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    WSESE '24: Proceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering
    April 2024
    87 pages
    ISBN:9798400705670
    DOI:10.1145/3643664
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Sponsors

    In-Cooperation

    • Faculty of Engineering of University of Porto

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 09 August 2024

    Check for updates

    Author Tags

    1. secondary research
    2. concepts
    3. constructs
    4. unobserved variables
    5. experience report

    Qualifiers

    • Research-article

    Conference

    WSESE '24
    Sponsor:

    Upcoming Conference

    ICSE 2025

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 55
      Total Downloads
    • Downloads (Last 12 months)55
    • Downloads (Last 6 weeks)19
    Reflects downloads up to 14 Dec 2024

    Other Metrics

    Citations

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media