In July 1981, thirteen and a half years after the Sackman paper, Proceedings of the IEEE published a little-known response from Thomas Dickey.
In it, he points out that the now oft-quoted 28:1 productivity difference is an inaccurate reading of the data. The CACM article usually referenced was only a summary of the full paper, excluding the actual data, so Dickey returned to the original sources and discovered that the ratios cited are misleading, as they do not differentiate between the impact of:
- programmers on the time-sharing system versus those on the batch system
- those who programmed in JTS, an ALGOL variant (one of whom learnt the language in order to do the experiment), and those who programmed in machine code
- the programmers who had poor, or in some cases no, knowledge of the time-sharing system.
In this case, the 28:1 figure (which, we should remember, only applies to debugging time), was the difference between the 6 hours taken by one programmer to debug his JTS solution on a time-share platform, versus 170 hours taken to debug a machine-language solution in a batch environment!
After accounting for the differences in the classes, only a range of 5:1 can be attributed to programmer variability. The casual researcher, in encountering Sackman’s paper, seizes on the 28:1 figure primarily to support arguments to the effect that programmer variability is “orders of magnitude” larger than effects due to language or system differences.
Dickey also goes on to show how the figure made it into common use: The CACM paper was cited at the NATO Conference on Software Engineering, 1968, which in turn provoked an article in Infosystems: “The Mongolian Hordes Versus Superprogrammer” (J L Ogdin, December 1973), bringing the number to the wider industry, to be picked up and used by Yourdon, Boehm, Brooks, Constantine, Weinberg et al, often mutating in the process.
Strangely neither this paper nor its conclusions seem to have made much of an impact on the popular view.