Testing a model of program quality

Reviewer: David W. Ballew

This paper presents the results of an experiment testing the validity of a model of program quality (the Oviedo/Van Verth model) that was the context of two PhD theses at SUNY/Buffalo [1,2]; the model is very briefly discussed, but not presented, in this paper. The paper describes an experiment in which 11 “experts” (read ten graduate students who had experience grading programs and one professor who had never graded such programs) graded Pascal programs and assessed their quality and measured program complexity; their results are compared to a computerized version of the above named model which also assesses quality and measures complexity. The object of the experiment is to demonstrate that the model does as well as, or better than, the experts. The statistical results presented are mixed, but the author concludes that at the .5 significance level there is no difference between the experts and the model. This conclusion is followed by considerable discussion and analysis of the items and individual grading methods that caused divergence. The paper seems to assume that there is an accepted and correct style of programming and that there is a correct way to interpret this to give a correct method of grading and assessing program quality, but the method and style are not presented. The algorithm and the experts are often compared to the “correct” style. To the best of the reviewer's knowledge, there is no consensus in the software community on style and measures of complexity, so it is premature to make the above mentioned comparisons.