Software Testing Education: A Systematic Literature Review

Muhammad Shoaib Farooq, Rabia Tehseen, Uzma Omer, Shamyla Riaz, Saba Tahir


Software Testing is the core part of computer science & engineering curriculum. It has been observed that software testing has been taught much in Computer science & engineering disciplines at undergraduate level. Software Testing Education (STE) involves time, cost, risk, quality, integration, communication, human resource, and procurement management skills. The STE at undergraduate and graduate level is a challenging task because it also requires knowledge and experience. This article aims to investigate and synthesize the state-of-the-art research in STE  for the improvement of STE curriculum, pedagogical tools and techniques, cognitive, empirical and assessments methods. STE research approaches has been categorized into five categories including empirical type, research approaches, software testing education processes, key areas, and curricula. The ninety-seven articles for the area of STE have been chosen after rigorous systematic screening process published in during 2004 to 2021. Furthermore, tools and techniques, testing processes, pedagogy and student performance are the frequently addressed in STE; whereas assessment methods, gamification, curriculum and exemplary program development appeared as mostly ignored areas. Lastly, research gaps and challenges relate to STE has been presented as future directions for the faculty, software industry, and researchers.

Full Text:



Taipale, O., Smolander, K., & Kälviäinen, H. (2005, November). Finding and ranking research directions for software testing. In European Conference on Software Process Improvement (pp. 39-48). Springer, Berlin, Heidelberg.

Pech, J. P. U., Vera, R. A. A., & Gómez, O. S. (2017, October). Software testing education through a collaborative virtual approach. In International Conference on Software Process Improvement (pp. 231-240). Springer, Cham.

Lemos, O. A. L., Silveira, F. F., Ferrari, F. C., & Garcia, A. (2018). The impact of Software Testing education on code reliability: An empirical assessment. Journal of Systems and Software, 137, 497-511.

Buckley, I. A., & Buckley, W. S. (2017). Teaching software testing using data structures. International Journal of Advanced Computer Science and Applications, 8(4), 1-4.

Clegg, B. S., Rojas, J. M., & Fraser, G. (2017, May). Teaching software testing concepts using a mutation testing game. In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering Education and Training Track (ICSE-SEET) (pp. 33-36). IEEE.

Kasurinen, J. (2017). Experiences from a web-based course in software testing and quality assurance. International Journal of Computer Applications, 166(2).

Rojas, J. M., & Fraser, G. (2016, April). Code defenders: a mutation testing game. In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 162-167). IEEE.

Smith, R., Tang, T., Warren, J., & Rixner, S. (2017, June). An automated system for interactively learning software testing. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (pp. 98-103).

Soska, A., Mottok, J., & Wolff, C. (2016, April). An experimental card game for software testing: Development, design and evaluation of a physical card game to deepen the knowledge of students in academic software testing education. In 2016 IEEE Global Engineering Education Conference (EDUCON) (pp. 576-584). IEEE.

Towey, D., Chen, T. Y., Kuo, F. C., Liu, H., & Zhou, Z. Q. (2016, December). Metamorphic testing: A new student engagement approach for a new software testing paradigm. In 2016 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE) (pp. 218-225). IEEE.

Ouhbi, S., Idri, A., Fernández-Alemán, J. L., & Toval, A. (2015). Requirements engineering education: a systematic mapping study. Requirements Engineering, 20(2), 119-138.

Abid, A., Farooq, M. S., Raza, I., Farooq, U., & Abid, K. (2015). Variants of Teaching First Course in Database Systems. Bulletin of Education and Research, 37(2), 9-25.

Mehmood, E., Abid, A., Farooq, M. S., & Nawaz, N. A. (2020). Curriculum, teaching and learning, and assessments for introductory programming course. IEEE Access, 8, 125961-125981.

Hassan, B., Farooq, M. S., Abid, A., & Sabir, N. (2015). Pakistan Sign Language: computer vision analysis & recommendations. VFAST Transactions on Software Engineering, 9(1), 1-6.

Farooq, M. S., Khan, S. A., & Abid, A. (2012). A framework for the assessment of a first programming language. Journal of Basic and Applied Scientific Research, 2(8), 8144-8149.

Valle, P. H. D., Barbosa, E. F., & Maldonado, J. C. (2015, November). CS curricula of the most relevant universities in Brazil and abroad: Perspective of software testing education. In 2015 International Symposium on Computers in Education (SIIE) (pp. 62-68). IEEE.

Alazzam, I., & Akour, M. (2015). Improving software testing course experience with pair testing pattern. International Journal of Teaching and Case Studies, 6(3), 244-250.

Allison, M., & Joo, S. F. (2015, July). An adaptive delivery strategy for teaching software testing and maintenance. In 2015 10th International Conference on Computer Science & Education (ICCSE) (pp. 237-242). IEEE.

Nascimento, D. M., Almeida Bittencourt, R., & Chavez, C. (2015). Open source projects in software engineering education: a mapping study. Computer Science Education, 25(1), 67-114.

Krutz, D. E., Malachowsky, S. A., & Reichlmayr, T. (2014, March). Using a real world project in a software testing course. In Proceedings of the 45th ACM technical symposium on Computer science education (pp. 49-54).

de Souza, D. M., Oliveira, B. H., Maldonado, J. C., Souza, S. R., & Barbosa, E. F. (2014, October). Towards the use of an automatic assessment system in the teaching of software testing. In 2014 IEEE Frontiers in Education Conference (FIE) Proceedings (pp. 1-8). IEEE.

Allison, M. A., & Joo, S. F. (2014, August). Revisiting Polya's approach to foster problem solving skill development in software engineers. In 2014 9th International Conference on Computer Science & Education (pp. 379-384). IEEE.

Buffardi, K., & Edwards, S. H. (2014, June). Responses to adaptive feedback for software testing. In Proceedings of the 2014 conference on Innovation & technology in computer science education (pp. 165-170).

Clarke, P. J., Davis, D., King, T. M., Pava, J., & Jones, E. L. (2014). Integrating testing into software engineering courses supported by a collaborative learning environment. ACM Transactions on Computing Education (TOCE), 14(3), 1-33.

Fu, Y., Clarke, P. J., & Barnes, N. (2016, June). Integrating Software Testing to CS Curriculum Using WRESTT-CyLE. In 2016 ASEE Annual Conference & Exposition.

Alelaiwi, A. (2014). Indirect assessment of student learning in a software testing course. Life Science Journal, 11(6s).

Banfield, J., & Wilkerson, B. (2014). Increasing student intrinsic motivation and self-efficacy through gamification pedagogy. Contemporary Issues in Education Research (CIER), 7(4), 291-298.

Bin, Z., & Shiming, Z. (2014, August). Experiment teaching reform for software testing course based on CDIO. In 2014 9th International Conference on Computer Science & Education (pp. 488-491). IEEE

Lemos, O. A. L., Ferrari, F. C., Silveira, F. F., & Garcia, A. (2015, November). Experience report: Can software testing education lead to more reliable code?. In 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE) (pp. 359-369). IEEE.

Chen, Z., Memon, A., & Luo, B. (2014, March). Combining research and education of software testing: a preliminary study. In Proceedings of the 29th Annual ACM Symposium on Applied Computing (pp. 1179-1180).

Zhang, P., White, J., & Schmidt, D. C. (2016, May). Holicow: Automatically breaking team-based software projects to motivate student testing. In 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C) (pp. 436-439). IEEE.

Garousi, V., Giray, G., Tuzun, E., Catal, C., & Felderer, M. (2019). Closing the gap between software engineering education and industrial needs. IEEE Software, 37(2), 68-77.

Bin, Z., & Shiming, Z. (2013). Curriculum reform and practice of software testing. In International Conference on Education Technology and Information System (ICETIS 2013) (pp. 841-844).

Smith, J., Tessler, J., Kramer, E., & Lin, C. (2012, September). Using peer review to teach software testing. In Proceedings of the ninth annual international conference on International computing education research (pp. 93-98).

Wong, W. Eric, et al. "Teaching software testing: Experiences, lessons learned and the path forward." 2011 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T). IEEE, 2011.

Blanch-Hartigan, Danielle. "Medical students’ self-assessment of performance: results from three meta-analyses." Patient education and counseling 84.1 (2011): 3-9.

Wang, Yan-Qing, et al. "Research and practice on education of SQA at source code level." International Journal of Engineering Education 27.1 (2011): 70.

Nivas, Tuli, and ChristophCsallner. "Managing performance testing with release certification and data correlation." Proceedings of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering. 2011.

Garousi, Vahid. "Incorporating real-world industrial testing projects in software testing courses: Opportunities, challenges, and lessons learned." 2011 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T). IEEE, 2011.

Yinnan, Zhang, and W. A. N. G. Xiaochi. "Implementation of Software Testing Course Based on CDIO." 2011 6th International Conference on Computer Science & Education (ICCSE). IEEE, 2011.

Muşlu, Kivanç, Bilge Soran, and JochenWuttke. "Finding bugs by isolating unit tests." Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. ACM, 2011.

Sampath, Premalatha. "The Emerging Role of Software Testing in Curriculum." Poster presented at the Computing and Information Technology Research and Education New Zealand (CITRENZ), Queenstown, New Zealand. CITRENZ (2015).

Harrison, Neil B. "Teaching software testing from two viewpoints." Journal of Computing Sciences in Colleges 26.2 (2010): 55-62.

Aaltonen, Kalle, Petri Ihantola, and Otto Seppälä. "Mutation analysis vs. code coverage in automated assessment of students' testing skills." Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion. ACM, 2010.

Burguillo, Juan C. "Using game theory and competition-based learning to stimulate student motivation and performance." Computers & education 55.2 (2010): 566-575.

Fung, Kam Hay, Graham Low, and P. K. Bay. "Embracing dynamic evolution in distributed systems." IEEE software 21.2 (2004): 49-55.

Mao, Chengying. "Towards a question-driven teaching method for software testing course." 2008 International Conference on Computer Science and Software Engineering. Vol. 5. IEEE, 2008.

Hauswirth, Matthias, et al. "The JavaFest: a collaborative learning technique for Java programming courses." Proceedings of the 6th international symposium on Principles and practice of programming in Java. ACM, 2008.

Jones, Edward L., and Christy L. Chatmon. "A perspective on teaching software testing." Journal of Computing Sciences in Colleges. Vol. 16. No. 3. Consortium for Computing Sciences in Colleges, 2001.

CemKaner, J. D., SowmyaPadmanabhan, and B. Eng. "Practice and Transfer of Learning in the Teaching of Software Testing."

Jaccheri, Letizia, and Thomas Osterlie. "Open source software: A source of possibilities for software engineering education and empirical software engineering." First International Workshop on Emerging Trends in FLOSS Research and Development (FLOSS'07: ICSE Workshops 2007). IEEE, 2007.

Elbaum, Sebastian, et al. "Bug hunt: Making early software testing lessons engaging and affordable." Proceedings of the 29th international conference on Software Engineering. IEEE Computer Society, 2007.

Lethbridge, Timothy C., et al. "Improving software practice through education: Challenges and future trends." Future of Software Engineering (FOSE'07). IEEE, 2007.

Cowling, Anthony J. "Stages in teaching software design." 20th Conference on Software Engineering Education & Training (CSEET'07). IEEE, 2007.

Cassel, Lillian Boots, et al. "The current crisis in computing: what are the real issues?." ACM SIGCSE Bulletin. Vol. 39. No. 1. ACM, 2007.



  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.