Abstract
Programming experience is an important confounding parameter in controlled experiments regarding program comprehension. In literature, ways to measure or control programming experience vary. Often, researchers neglect it or do not specify how they controlled for it. We set out to find a well-defined understanding of programming experience and a way to measure it. From published comprehension experiments, we extracted questions that assess programming experience. In a controlled experiment, we compare the answers of computer-science students to these questions with their performance in solving program-comprehension tasks. We found that self estimation seems to be a reliable way to measure programming experience. Furthermore, we applied exploratory and confirmatory factor analyses to extract and evaluate a model of programming experience. With our analysis, we initiate a path toward validly and reliably measuring and describing programming experience to better understand and control its influence in program-comprehension experiments.
Similar content being viewed by others
Notes
ICPC was a workshop until 2005 (IWPC), which we also included.
ESEM first took place in 2007.
The German system allows students to take courses in a flexible order and timing.
Small: ±0.1 to ±0.3; medium: ±0.3 to ±0.5; strong: ±0.5 to ±1 (Cohen 1988).
At German universities, semester starts mid October.
http://www-01.ibm.com/software/analytics/spss/products/statistics/amos/. The script of the analysis is available at the project’s website.
References
Anderson T, Finn J (1996) The new statistical analysis of data. Springer
Anderson T, Rubin H (1956) Statistical inference in factor analysis. In: Proc. Berkeley symposium on mathematical statistics and probability, vol 5. University of California Press, pp 111–150
Arisholm E, Gallis H, Dybå T, Sjøberg D (2007) Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Trans Softw Eng 33(2):65–86
Askar P, Davenport D (2009) An investigation of factors related to self-efficacy for Java programming among engineering students. TOJET 8(1):26–32
Bagozzi R, Yi Y (2012) Specification, evaluation, and interpretation of structural equation models. J Acad Mark Sci 40(1):8–34
Bartholomew D, Martin Knott IM (2011) Latent variable models and factor analysis: a unified approach. Wiley, New York
Biffl S, Grossmann W (2001) Evaluating the accuracy of defect estimation models based on inspection data from two inspection cycles. In: Proc. int’l conf. software engineering (ICSE). IEEE CS, pp 145–154
Bornat R, Dehnadi S, Simon (2008) Mental models, consistency and programming aptitude. In: Proc. conf. on Australasian computing education, vol 78. Australian Computer Society, Inc., pp 53–61
Bunse C (2006) Using patterns for the refinement and translation of UML models: a controlled experiment. Empir Software Eng 11(2):227–267
Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Routledge Academic Press
Cohen J, Cohen P (1983) Applied multiple regression: correlation analysis for the behavioral sciences, 2nd edn. Addison-Wesley, Reading, MA
Costello A, Osborne J (2005) Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation (PARE) 10(7):173–178
Easterbrook S, Singer J, Storey M-A, Damian D (2008) Selecting empirical methods for software engineering research. In: Guide to advanced empirical software engineering. Springer, pp 285–311
Erdogmus H, Morisio M, Torchiano M (2005) On the effectiveness of the test-first approach to programming. IEEE Trans Softw Eng 31(3):226–237
Feigenspan J, Apel S, Liebig J, Kästner C (2011a) Exploring software measures to assess program comprehension. In: Proc. int’l symposium empirical software engineering and measurement (ESEM). IEEE CS, paper 3, pp 1–10
Feigenspan J, Schulze M, Papendieck M, Kästner C, Dachselt R, Köppen, V, Frisch M (2011b) Using background colors to support program comprehension in software product lines. In: Proc. int’l conf. evaluation and assessment in software engineering (EASE). Institution of Engineering and Technology, pp 66–75
Feigenspan J, Siegmund N (2012) Supporting comprehension experiments with human subjects. In: Proc. int’l conf. program comprehension (ICPC). IEEE CS, Tool demo, pp 244–246
Feigenspan J, Kästner C, Liebig J, Apel S, Hanenberg S (2012) Measuring programming experience. In: Proc. int’l conf. program comprehension (ICPC). IEEE CS, pp 73–82
Feigenspan J, Kästner C, Apel S, Liebig J, Schulze M, Dachselt R, Papendieck M, Leich T, Saake G (2013) Do background colors improve program comprehension in the #ifdef hell?. Empir Software Eng 18(4):699–745
Figueiredo E, Cacho N, Monteiro M, Kulesza U, Garcia R, Soares S, Ferrari F, Khan S, Filho F, Dantas F (2008) Evolving software product lines with aspects: an empirical study on design stability. In: Proc. int’l conf. software engineering (ICSE). ACM Press, pp 261–270
Goodwin J (1999) Research in psychology: methods and design, 2nd edn. Wiley, New York
Hanenberg S, Kleinschmager S, Josupeit-Walter M (2009) Does aspect-oriented programming increase the development speed for crosscutting code? An empirical study. In: Proc. int’l symposium empirical software engineering and measurement (ESEM). IEEE CS, pp 156–167
Hannay J, Arisholm E, Engvik H, Sjøberg D (2010) Effects of personality on pair programming. IEEE Trans Softw Eng 36(1):61–80
Höst M, Regnell B, Wohlin C (2000) Using students as subjects: a comparative study of students and professionals in lead-time impact assessment. Empir Software Eng 5(3):201–214
Hu L, Bentler PM (1999) Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model 6(1):1–55
Hu L, Bentler PM (1998) Fit indexes in covariance structure modeling: sensitivity to underparameterized model misspecification. Psychol Methods 3(4):424–453
Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Guide to advanced empirical software engineering. Springer, pp 201–228
Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-001, Keele University and Durham University Joint Report
Kleinschmager S, Hanenberg S (2011) How to rate programming skills in programming experiments? A preliminary, exploratory, study based on university marks, pretests, and self-estimation. In: Proc. ACM SIGPLAN workshop on evaluation and usability of programming languages and tools. ACM Press, pp 15–24
Lewis-Beck M (1980) Applied regression: an introduction. Sage Pubications
Müller M (2004) Are reviews an alternative to pair programming? Empir Software Eng 9(4):335–351
Peterson R (2000) Constructing effective questionnaires. Sage Publications
Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M (2007) The role of experience and ability in comprehension tasks supported by UML stereotypes. In: Proc. int’l conf. software engineering (ICSE). IEEE CS, pp 375–384
Siegmund J, Kästner C, Liebig J, Apel S (2012) Comparing program comprehension of physically and virtually separated concerns. In: Proc. int’l workshop on feature-oriented software development (FOSD). ACM Press, pp 17–24
Sillito J, Murphy G, De Volder K (2008) Asking and answering questions during a programming change task. IEEE Trans Softw Eng 34(4):434–451
Steiger JH, Lind J (1980) Statistically-based tests for the number of common factors. Presented at the annual meeting of the Psychometric Society
Tichy W (2000) Hints for reviewing empirical work in software engineering. Empir Software Eng 5(4):309–312
von Mayrhauser A, Vans M (1995) Program comprehension during software maintenance and evolution. Computer 28(8):44–55
Westland C (2010) Lower bounds on sample size in structural equation modeling. Electron Commer Res Appl 9(6):476–487
Acknowledgements
We thank all the reviewers for their constructive feedback. We thank Jana Schumann for her support in the literature study and all experimenters for their support in setting up and conducting the experiment. Thanks to Veit Köppen for his support in the analysis of data. Siegmund’s work is supported by BMBF project 01IM10002B, Kästner’s work by ERC grant #203099, and Apel’s work by DFG projects AP 206/2, AP 206/4, and AP 206/5.
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by: Michael Godfrey and Arie van Deursen
Appendix
Appendix
We show the results of an exploratory factor analysis, including question s.PE, which loads on factor experience with mainstream languages.
Rights and permissions
About this article
Cite this article
Siegmund, J., Kästner, C., Liebig, J. et al. Measuring and modeling programming experience . Empir Software Eng 19, 1299–1334 (2014). https://doi.org/10.1007/s10664-013-9286-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-013-9286-4