Software Testing Education: A Systematic Literature Review

Authors

  • Muhammad Shoaib Farooq University of Management and Technology, Lahore, Pakistan
  • Rabia Tehseen University of Management and Technology, Lahore, Pakistan
  • Uzma Omer University of Management and Technology, Lahore, Pakistan
  • Shamyla Riaz University of Management and Technology, Lahore, Pakistan
  • Saba Tahir University of Management and Technology, Lahore, Pakistan

DOI:

https://doi.org/10.21015/vtse.v9i4.953

Abstract

Software Testing is the core part of computer science & engineering curriculum. It has been observed that software testing has been taught much in Computer science & engineering disciplines at undergraduate level. Software Testing Education (STE) involves time, cost, risk, quality, integration, communication, human resource, and procurement management skills. The STE at undergraduate and graduate level is a challenging task because it also requires knowledge and experience. This article aims to investigate and synthesize the state-of-the-art research in STE  for the improvement of STE curriculum, pedagogical tools and techniques, cognitive, empirical and assessments methods. STE research approaches has been categorized into five categories including empirical type, research approaches, software testing education processes, key areas, and curricula. The ninety-seven articles for the area of STE have been chosen after rigorous systematic screening process published in during 2004 to 2021. Furthermore, tools and techniques, testing processes, pedagogy and student performance are the frequently addressed in STE; whereas assessment methods, gamification, curriculum and exemplary program development appeared as mostly ignored areas. Lastly, research gaps and challenges relate to STE has been presented as future directions for the faculty, software industry, and researchers.

References

Taipale, O., Smolander, K., & Kälviäinen, H. (2005, November). Finding and ranking research directions for software testing. In European Conference on Software Process Improvement (pp. 39-48). Springer, Berlin, Heidelberg. DOI: https://doi.org/10.1007/11586012_5

Pech, J. P. U., Vera, R. A. A., & Gómez, O. S. (2017, October). Software testing education through a collaborative virtual approach. In International Conference on Software Process Improvement (pp. 231-240). Springer, Cham. DOI: https://doi.org/10.1007/978-3-319-69341-5_21

Lemos, O. A. L., Silveira, F. F., Ferrari, F. C., & Garcia, A. (2018). The impact of Software Testing education on code reliability: An empirical assessment. Journal of Systems and Software, 137, 497-511. DOI: https://doi.org/10.1016/j.jss.2017.02.042

Buckley, I. A., & Buckley, W. S. (2017). Teaching software testing using data structures. International Journal of Advanced Computer Science and Applications, 8(4), 1-4. DOI: https://doi.org/10.14569/IJACSA.2017.080401

Clegg, B. S., Rojas, J. M., & Fraser, G. (2017, May). Teaching software testing concepts using a mutation testing game. In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering Education and Training Track (ICSE-SEET) (pp. 33-36). IEEE. DOI: https://doi.org/10.1109/ICSE-SEET.2017.1

Kasurinen, J. (2017). Experiences from a web-based course in software testing and quality assurance. International Journal of Computer Applications, 166(2). DOI: https://doi.org/10.5120/ijca2017913918

Rojas, J. M., & Fraser, G. (2016, April). Code defenders: a mutation testing game. In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 162-167). IEEE. DOI: https://doi.org/10.1109/ICSTW.2016.43

Smith, R., Tang, T., Warren, J., & Rixner, S. (2017, June). An automated system for interactively learning software testing. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (pp. 98-103). DOI: https://doi.org/10.1145/3059009.3059022

Soska, A., Mottok, J., & Wolff, C. (2016, April). An experimental card game for software testing: Development, design and evaluation of a physical card game to deepen the knowledge of students in academic software testing education. In 2016 IEEE Global Engineering Education Conference (EDUCON) (pp. 576-584). IEEE. DOI: https://doi.org/10.1109/EDUCON.2016.7474609

Towey, D., Chen, T. Y., Kuo, F. C., Liu, H., & Zhou, Z. Q. (2016, December). Metamorphic testing: A new student engagement approach for a new software testing paradigm. In 2016 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE) (pp. 218-225). IEEE. DOI: https://doi.org/10.1109/TALE.2016.7851797

Ouhbi, S., Idri, A., Fernández-Alemán, J. L., & Toval, A. (2015). Requirements engineering education: a systematic mapping study. Requirements Engineering, 20(2), 119-138. DOI: https://doi.org/10.1007/s00766-013-0192-5

Abid, A., Farooq, M. S., Raza, I., Farooq, U., & Abid, K. (2015). Variants of Teaching First Course in Database Systems. Bulletin of Education and Research, 37(2), 9-25.

Mehmood, E., Abid, A., Farooq, M. S., & Nawaz, N. A. (2020). Curriculum, teaching and learning, and assessments for introductory programming course. IEEE Access, 8, 125961-125981.

Hassan, B., Farooq, M. S., Abid, A., & Sabir, N. (2015). Pakistan Sign Language: computer vision analysis & recommendations. VFAST Transactions on Software Engineering, 9(1), 1-6. DOI: https://doi.org/10.21015/vtse.v9i1.386

Farooq, M. S., Khan, S. A., & Abid, A. (2012). A framework for the assessment of a first programming language. Journal of Basic and Applied Scientific Research, 2(8), 8144-8149.

Valle, P. H. D., Barbosa, E. F., & Maldonado, J. C. (2015, November). CS curricula of the most relevant universities in Brazil and abroad: Perspective of software testing education. In 2015 International Symposium on Computers in Education (SIIE) (pp. 62-68). IEEE. DOI: https://doi.org/10.1109/SIIE.2015.7451649

Alazzam, I., & Akour, M. (2015). Improving software testing course experience with pair testing pattern. International Journal of Teaching and Case Studies, 6(3), 244-250. DOI: https://doi.org/10.1504/IJTCS.2015.072632

Allison, M., & Joo, S. F. (2015, July). An adaptive delivery strategy for teaching software testing and maintenance. In 2015 10th International Conference on Computer Science & Education (ICCSE) (pp. 237-242). IEEE. DOI: https://doi.org/10.1109/ICCSE.2015.7250249

Nascimento, D. M., Almeida Bittencourt, R., & Chavez, C. (2015). Open source projects in software engineering education: a mapping study. Computer Science Education, 25(1), 67-114. DOI: https://doi.org/10.1080/08993408.2015.1033159

Krutz, D. E., Malachowsky, S. A., & Reichlmayr, T. (2014, March). Using a real world project in a software testing course. In Proceedings of the 45th ACM technical symposium on Computer science education (pp. 49-54). DOI: https://doi.org/10.1145/2538862.2538955

de Souza, D. M., Oliveira, B. H., Maldonado, J. C., Souza, S. R., & Barbosa, E. F. (2014, October). Towards the use of an automatic assessment system in the teaching of software testing. In 2014 IEEE Frontiers in Education Conference (FIE) Proceedings (pp. 1-8). IEEE. DOI: https://doi.org/10.1109/FIE.2014.7044375

Allison, M. A., & Joo, S. F. (2014, August). Revisiting Polya's approach to foster problem solving skill development in software engineers. In 2014 9th International Conference on Computer Science & Education (pp. 379-384). IEEE. DOI: https://doi.org/10.1109/ICCSE.2014.6926489

Buffardi, K., & Edwards, S. H. (2014, June). Responses to adaptive feedback for software testing. In Proceedings of the 2014 conference on Innovation & technology in computer science education (pp. 165-170). DOI: https://doi.org/10.1145/2591708.2591756

Clarke, P. J., Davis, D., King, T. M., Pava, J., & Jones, E. L. (2014). Integrating testing into software engineering courses supported by a collaborative learning environment. ACM Transactions on Computing Education (TOCE), 14(3), 1-33. DOI: https://doi.org/10.1145/2648787

Fu, Y., Clarke, P. J., & Barnes, N. (2016, June). Integrating Software Testing to CS Curriculum Using WRESTT-CyLE. In 2016 ASEE Annual Conference & Exposition.

Alelaiwi, A. (2014). Indirect assessment of student learning in a software testing course. Life Science Journal, 11(6s).

Banfield, J., & Wilkerson, B. (2014). Increasing student intrinsic motivation and self-efficacy through gamification pedagogy. Contemporary Issues in Education Research (CIER), 7(4), 291-298. DOI: https://doi.org/10.19030/cier.v7i4.8843

Bin, Z., & Shiming, Z. (2014, August). Experiment teaching reform for software testing course based on CDIO. In 2014 9th International Conference on Computer Science & Education (pp. 488-491). IEEE

Lemos, O. A. L., Ferrari, F. C., Silveira, F. F., & Garcia, A. (2015, November). Experience report: Can software testing education lead to more reliable code?. In 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE) (pp. 359-369). IEEE.

Chen, Z., Memon, A., & Luo, B. (2014, March). Combining research and education of software testing: a preliminary study. In Proceedings of the 29th Annual ACM Symposium on Applied Computing (pp. 1179-1180). DOI: https://doi.org/10.1145/2554850.2555124

Zhang, P., White, J., & Schmidt, D. C. (2016, May). Holicow: Automatically breaking team-based software projects to motivate student testing. In 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C) (pp. 436-439). IEEE. DOI: https://doi.org/10.1145/2889160.2889197

Garousi, V., Giray, G., Tuzun, E., Catal, C., & Felderer, M. (2019). Closing the gap between software engineering education and industrial needs. IEEE Software, 37(2), 68-77.

Bin, Z., & Shiming, Z. (2013). Curriculum reform and practice of software testing. In International Conference on Education Technology and Information System (ICETIS 2013) (pp. 841-844).

Smith, J., Tessler, J., Kramer, E., & Lin, C. (2012, September). Using peer review to teach software testing. In Proceedings of the ninth annual international conference on International computing education research (pp. 93-98). DOI: https://doi.org/10.1145/2361276.2361295

Wong, W. Eric, et al. "Teaching software testing: Experiences, lessons learned and the path forward." 2011 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T). IEEE, 2011. DOI: https://doi.org/10.1109/CSEET.2011.5876141

Blanch-Hartigan, Danielle. "Medical students’ self-assessment of performance: results from three meta-analyses." Patient education and counseling 84.1 (2011): 3-9. DOI: https://doi.org/10.1016/j.pec.2010.06.037

Wang, Yan-Qing, et al. "Research and practice on education of SQA at source code level." International Journal of Engineering Education 27.1 (2011): 70.

Nivas, Tuli, and ChristophCsallner. "Managing performance testing with release certification and data correlation." Proceedings of the European Software Engineering Conference and the Symposium on the Foundations of Software Engineering. 2011.

Garousi, Vahid. "Incorporating real-world industrial testing projects in software testing courses: Opportunities, challenges, and lessons learned." 2011 24th IEEE-CS Conference on Software Engineering Education and Training (CSEE&T). IEEE, 2011. DOI: https://doi.org/10.1109/CSEET.2011.5876112

Yinnan, Zhang, and W. A. N. G. Xiaochi. "Implementation of Software Testing Course Based on CDIO." 2011 6th International Conference on Computer Science & Education (ICCSE). IEEE, 2011. DOI: https://doi.org/10.1109/ICCSE.2011.6028595

Muşlu, Kivanç, Bilge Soran, and JochenWuttke. "Finding bugs by isolating unit tests." Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. ACM, 2011. DOI: https://doi.org/10.1145/2025113.2025202

Sampath, Premalatha. "The Emerging Role of Software Testing in Curriculum." Poster presented at the Computing and Information Technology Research and Education New Zealand (CITRENZ), Queenstown, New Zealand. CITRENZ (2015).

Harrison, Neil B. "Teaching software testing from two viewpoints." Journal of Computing Sciences in Colleges 26.2 (2010): 55-62.

Aaltonen, Kalle, Petri Ihantola, and Otto Seppälä. "Mutation analysis vs. code coverage in automated assessment of students' testing skills." Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion. ACM, 2010. DOI: https://doi.org/10.1145/1869542.1869567

Burguillo, Juan C. "Using game theory and competition-based learning to stimulate student motivation and performance." Computers & education 55.2 (2010): 566-575. DOI: https://doi.org/10.1016/j.compedu.2010.02.018

Fung, Kam Hay, Graham Low, and P. K. Bay. "Embracing dynamic evolution in distributed systems." IEEE software 21.2 (2004): 49-55. DOI: https://doi.org/10.1109/MS.2004.1270762

Mao, Chengying. "Towards a question-driven teaching method for software testing course." 2008 International Conference on Computer Science and Software Engineering. Vol. 5. IEEE, 2008. DOI: https://doi.org/10.1109/CSSE.2008.1258

Hauswirth, Matthias, et al. "The JavaFest: a collaborative learning technique for Java programming courses." Proceedings of the 6th international symposium on Principles and practice of programming in Java. ACM, 2008. DOI: https://doi.org/10.1145/1411732.1411734

Jones, Edward L., and Christy L. Chatmon. "A perspective on teaching software testing." Journal of Computing Sciences in Colleges. Vol. 16. No. 3. Consortium for Computing Sciences in Colleges, 2001.

CemKaner, J. D., SowmyaPadmanabhan, and B. Eng. "Practice and Transfer of Learning in the Teaching of Software Testing."

Jaccheri, Letizia, and Thomas Osterlie. "Open source software: A source of possibilities for software engineering education and empirical software engineering." First International Workshop on Emerging Trends in FLOSS Research and Development (FLOSS'07: ICSE Workshops 2007). IEEE, 2007. DOI: https://doi.org/10.1109/FLOSS.2007.12

Elbaum, Sebastian, et al. "Bug hunt: Making early software testing lessons engaging and affordable." Proceedings of the 29th international conference on Software Engineering. IEEE Computer Society, 2007. DOI: https://doi.org/10.1109/ICSE.2007.23

Lethbridge, Timothy C., et al. "Improving software practice through education: Challenges and future trends." Future of Software Engineering (FOSE'07). IEEE, 2007. DOI: https://doi.org/10.1109/FOSE.2007.13

Cowling, Anthony J. "Stages in teaching software design." 20th Conference on Software Engineering Education & Training (CSEET'07). IEEE, 2007. DOI: https://doi.org/10.1109/CSEET.2007.47

Cassel, Lillian Boots, et al. "The current crisis in computing: what are the real issues?." ACM SIGCSE Bulletin. Vol. 39. No. 1. ACM, 2007.

Downloads

Published

2021-12-31

How to Cite

Farooq, M. S., Tehseen, R., Omer, U., Riaz, S., & Tahir, S. (2021). Software Testing Education: A Systematic Literature Review. VFAST Transactions on Software Engineering, 9(4), 109–125. https://doi.org/10.21015/vtse.v9i4.953