The Final Verdict on Pair vs. Solo Programming: A Summary

Table of Links

Abstract and 1. Introduction

2. Experiment Definition

3. Experiment Design and Conduct

3.1 Latin Square Designs

3.2 Subjects, Tasks and Objects

3.3 Conduct

3.4 Measures

4. Data Analysis

4.1 Model Assumptions

4.2 Analysis of Variance (ANOVA)

4.3 Treatment Comparisons

4.4 Effect Size and Power Analysis

5. Experiment Limitations and 5.1 Threats to the Conclusion Validity

5.2 Threats to Internal Validity

5.3 Threats to Construct Validity

5.4 Threats to External Validity

6. Discussion and 6.1 Duration

6.2 Effort

7. Conclusions and Further Work, and References

7. Conclusions and Further Work

This paper presented a controlled experiment that was run as part of a university course in DOE. The aim of the experiment was to evaluate pair versus solo programming with respect to duration and effort. Subjects who jointly wrote the program assignments took less time (28%) than subjects who worked individually. Conversely subjects grouped in pairs spent more effort (30%) than those who worked individually. These results are very close to those reported in [24].

\ With the aiming of striving towards better research practices in SE [18] we reported all the collected measures. This data will help other researchers to verify or re-analyze [14] the experiment results presented in this work. This data can also be used to accumulate and consolidate a body of knowledge about pair programing.

\ We are planning to conduct future replications of this experiment to get more insight about the effect of pair programming. Although we did not observe interactions between treatment and blocks, we plan to use another experimental design to assess possible interactions.

References

[1] E. Arisholm, H. Gallis, T. Dybå, and D. I. Sjøberg. Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Transactions on Software Engineering, 33(2):65–86, 2007.

\ [2] V. Basili, G. Caldiera, and H. Rombach. Goal question metric paradigm. Encyclopedia of Software Eng, pages 528–532, 1994. John Wiley & Sons.

\ [3] K. Beck. Embracing change with extreme programming. Computer, 32(10):70–77, 1999.

\ [4] K. Beck. Extreme programming explained: embrace change . Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 2000.

\ [5] M. Borenstein. The handbook of research synthesis and meta analysis. Chapter: Effect sizes for continuous data, pages 279–293. Russell Sage Foundation, New York, USA, 2009.

\ [6] G. E. P. Box, W. G. Hunter, J. S. Hunter, and W. G. Hunter. Statistics for Experimenters: An Introduction to Design, Data Analysis, and Model Building. John Wiley & Sons, June 1978.

\ [7] G. Canfora, A. Cimitile, F. Garcia, M. Piattini, and C. A. Visaggio. Evaluating performances of pair designing in industry. Journal of Systems and Software, 80(8):1317 – 1327, 2007.

\ [8] J. Carver, L. Jaccheri, S. Morasca, and F. Shull. Issues in using students in empirical studies in software engineering education. In METRICS ’03: Proceedings of the 9th International Symposium on Software Metrics, page 239, Washington, DC, USA, 2003. IEEE Computer Society.

\ [9] S. Champely. pwr: Basic functions for power analysis , 2012. R package version 1.1.1.

\ [10] J. Cohen. Statistical power analysis for the behavioral sciences . L. Erlbaum Associates, Hillsdale, NJ, 1988.

\ [11] T. Cook and D. Campbell. The design and conduct of quasiexperiments and true experiments in field settings. Rand McNally, Chicago, 1976.

\ [12] H. L. Dreyfus and S. Dreyfus. Mind over Machine. The Power of Human Intuition and Expertise in the Era of the Computer . Basil Blackwell, New York, 1986.

\ [13] R. A. Fisher. The Design of Experiments. Oliver & Boyd, Edimburgh, 1935.

\ [14] O. S. Gómez, N. Juristo, and S. Vegas. Replication, reproduction and re-analysis: Three ways for verifying experimental findings. In International Workshop on Replication in Empirical Software Engineering Research (RESER’2010) , Cape Town, South Africa, May 2010.

\ [15] A. N. Kolmogorov. Sulla determinazione empirica di una legge di distribuzione. Giornale dell’Istituto Italiano degli Attuari, 4:83–91, 1933.

\ [16] R. Kuehl. Design of Experiments: Statistical Principles of Research Design and Analysis. Duxbury Thomson Learning, California, USA. second ed. edition, 2000.

\ [17] H. Levene. Robust tests for equality of variances. In I. Olkin, editor, Contributions to probability and statistics . Stanford Univ. Press. Palo Alto, CA, 1960.

\ [18] P. Louridas and G. Gousios. A note on rigour and replicability. SIGSOFT Softw. Eng. Notes, 37(5):1–4, Sept. 2012.

\ [19] K. M. Lui and K. C. C. Chan. When does a pair outperform two individuals? In Proceedings of the 4th international conference on Extreme programming and agile processes in software engineering, XP’03, pages 225–233, Berlin, Heidelberg, 2003. Springer-Verlag.

\ [20] K. M. Lui, K. C. C. Chan, and J. Nosek. The effect of pairs in program design tasks. IEEE Trans. Softw. Eng., 34(2):197–211, Mar. 2008.

\ [21] C. McDowell, L. Werner, H. E. Bullock, and J. Fernald. The impact of pair programming on student performance, perception and persistence. In Proceedings of the 25th International Conference on Software Engineering , ICSE ’03, pages 602–607, Washington, DC, USA, 2003. IEEE Computer Society.

\ [22] M. M. Müller. Two controlled experiments concerning the comparison of pair programming to peer review. Journal of Systems and Software, 78(2):166 – 179, 2005.

\ [23] J. Nawrocki and A. Wojciechowski. Experimental evaluation of pair programming. In Proceedings of the 12th European Software Control and Metrics Conference, pages 269–276, London, April 2001.

\ [24] J. T. Nosek. The case for collaborative programming. Commun. ACM , 41(3):105–108, Mar. 1998.

\ [25] H. Scheffé. A method for judging all contrasts in the analysis of variance. Biometrika , 40(1/2):87–104, 1953.

\ [26] N. V. Smirnov. Table for estimating the goodness of fit of empirical distributions. Ann. Math. Stat., 19:279–281, 1948.

\ [27] J. W. Tukey. One degree of freedom for non-additivity. Biometrics, 5(3):pp. 232–242, 1949.

\ [28] L. Williams, R. Kessler, W. Cunningham, and R. Jeffries. Strengthening the case for pair programming. Software, IEEE, 17(4):19 –25, jul/aug 2000.

\ Omar S. Gómez received a BS degree in Computing from the University of Guadalajara (UdG), and a MS degree in Software Engineering from the Center for Mathematical Research (CIMAT), both in Mexico. Recently, he received a PhD degree in Software and Systems from the Technical University of Madrid (UPM). Currently he is a full time professor of Software Engineering at Mathematics Faculty of the Autonomous University of Yucatan (UADY). His main research interests include: Experimentation in software engineering, software process improvement and software architectures.

\ José L. Batún received a BS degree in Mathematics from the Autonomous University of Yucatan (UADY). He received a MS degree and a PhD degree in Probability and Statistics, both, from the Center for Mathematical Research (CIMAT) in Guanajuato, Mexico. He is currently full time professor of Statistics at Mathematics Faculty of the Autonomous University of Yucatan (UADY). His research interests include: Multivariate statistical models, copulas, survival analysis, time series and their applications.

\ Raúl A. Aguilar was born in Telchac Pueblo, Mexico, in 1971. He received the BS degree in Computer Science from the Autonomous University of Yucatan (UADY) and a PhD degree (PhD European mention) at the Technical University of Madrid (UPM), Spain. Currently he is full time professor of software engineering at Mathematics Faculty of the Autonomous University of Yucatan (UADY). His main research interests include: Software engineering and computer science applied to education.

\

:::info Authors:

(1) Omar S. Gómez, full time professor of Software Engineering at Mathematics Faculty of the Autonomous University of Yucatan (UADY);

(2) José L. Batún, full time professor of Statistics at Mathematics Faculty of the Autonomous University of Yucatan (UADY);

(3) Raúl A. Aguilar, Faculty of Mathematics, Autonomous University of Yucatan Merida, Yucatan 97119, Mexico.

:::

---

:::info This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license.

:::

\