Evaluation Tools for Digital Educational Games: A Systematic Review
Abstract
This systematic review investigates various evaluation tools for digital educational games and answers the question of which evaluation tools could be used to evaluate digital educational games. A systematic review of studies, by searching for related keywords in the title, abstract, and keywords of studies in the scientific databases EMBASE, Web of Science, Scopus, and PubMed, was launched without time-limited on November 2, 2021. The same checklist was used to extract data such as reference, first author's name, year of publication, tool name, type of tool, instructional strategy, and evaluation factors. A total of 3516 articles were extracted and finally, an analysis of the included studies gave us 22 different approaches to the systematic evaluation of educational games. The same study developed some proprietary evaluation tools exclusively for game evaluation. However, some tools evaluated games in different dimensions, most of which did not consider the tool's validity. In the same sense, we have five prominent evaluation guidelines, including E-GESS, MEEGA+, EGameFlow, HEP, and Kato evaluation guideline, all of which have been developed by explicitly decomposing the evaluation objectives into criteria and using a questionnaire assessed through a collection of case studies. Our systematic review showed the need to identify more consistent and uniform patterns in different dimensions for the systematic evaluation of digital educational games to achieve valid results that can be used as a basis for deciding on the use of digital educational games.
1. Sudarmilah E, Siregar RMP. The usability of ‘keepin’collect the trash: Virtual reality educational game in android smartphone for children. Int J Eng Adv Technol 2019;8:944-7.
2. Palee P, Wongta N, Khwanngern K, Jitmun W, Choosri N. Serious game for teaching undergraduate medical students in cleft lip and palate treatment protocol. Int J Med Inform 2020;141:104166.
3. Pesántez-Cabrera P, Acosta MI, Jimbo V, Sinchi P, Cedillo P. Towards an evaluation method of how accessible serious games are to older adults. 2020 IEEE 8th International Conference on Serious Games and Applications for Health (SeGAH). IEEE, 2020:1-8.
4. Savi R, von Wangenheim CG, Borgatto AF. A model for the evaluation of educational games for teaching software engineering. 2011 25th Brazilian Symposium on Software Engineering (IEEE). IEEE, 2011:194-203.
5. Kato PM. Evaluating efficacy and validating games for health. Games Health J 2012;1:74-6.
6. Omari K, Moussetad M, Labriji E, Harchi S. Proposal for a New Tool to Evaluate a Serious Game. Int J Emerg Technol Learn 2020;15:238-51.
7. Eseryel D, Guo Y, Law V. Interactivity3 design and assessment framework for educational games to promote motivation and complex problem-solving skills. In: Ifenthaler D, Eseryel D, Ge X, eds. Assessment in Game-Based Learning: Foundations, Innovations, and Perspectives. New York: Springer, 2012:257-85.
8. McFarlane DA. Evaluating training programs: The four levels. J Appl Manag Entrep. Author manuscript 2006;11:96.
9. de Carvalho CV. Is game-based learning suitable for engineering education? Proceedings of the 2012 IEEE Global Engineering Education Conference (EDUCON). IEEE, 2012:1-8.
10. De Freitas S, Oliver M. How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Comput Educ 2006;46:249-64.
11. Connolly T, Stansfield M, Boyle L. Games-Based Learning Advancements for Multi-Sensory Human Computer Interfaces: Techniques and Effective Practices. Hershey, Pennsylvania, USA: IGI Global; 2009.
12. Escudeiro P, Escudeiro N. Evaluation of serious games in mobile platforms with qef: Qef (quantitative evaluation framework). 2012 March. 27. IEEE Seventh International Conference on Wireless, Mobile and Ubiquitous Technology in Education. IEEE, 2012:268-71.
13. Altanis I, Retalis S. A multifaceted students’ performance assessment framework for motion-based game-making projects with Scratch. EMI Educ Media Int 2019;56:201-17.
14. Geerts D, Nouwen M, van Beek E, Slegers K, Miranda FC, Bleumers L. Using the SGDA Framework to Design and Evaluate Research Games. Simul Gaming 2019;50:272-301.
15. Eseryel D, Ifenthaler D, Ge X. Validation study of a method for assessing complex ill-structured problem solving by using causal representations. Educ Technol Res Dev 2013;61:443-63.
16. De Grove F, Cauberghe V, Van Looy J. Development and Validation of an Instrument for Measuring Individual Motives for Playing Digital Games. Media Psychol 2016;19:101-25.
17. Hong JC, Cheng CL, Hwang MY, Lee CK, Chang HY. Assessing the educational values of digital games. J Comput Assist Learn 2009;25:423-37.
18. Borji YEL, Khaldi M. Comparative study to develop a tool for the quality assessment of serious games intended to be used in education. Int J Emerg Technol Learn 2014;9:50-5.
19. Fu FL, Su RC, Yu SC. EGameFlow: A scale to measure learners’ enjoyment of e-learning games. Comput Educ 2009;52:101-12.
20. Ak O. A game scale to evaluate educational computer games. Procedia-Social Behav Sci 2012;46:2477-81.
21. da Silveira AC, Martins RX, Vieira EA. E-Guess: Usability evaluation for educational games. RIED 2021;24:245-63.
22. Mayer I, Bekebrede G, Harteveld C, Warmelink H, Zhou Q, Van Ruijven T, et al. The research and evaluation of serious games: Toward a comprehensive methodology. Br J Educ Technol 2014;45:502-27.
23. Petri G, von Wangenheim CG, Borgatto AF. MEEGA+: an evolution of a model for the evaluation of educational games. Eval Educ Games Comput Educ 2016;3:1-40.
24. Alhuhud G, Altamimi W. Quality evaluation of mobile game: Miftah Alfasaha. Mob Inf Syst 2016;1:1-8.
25. Oprins E, Boer-Visschedijk GC, Roozeboom MB, Dankbaar M, Trooster W, Schuit SC. The game-based learning evaluation model (GEM): measuring the effectiveness of serious games using a standardised method. Int J Technol Enhanc Learn 2015;7:326-45.
26. Desurvire H, Caplan M, Toth JA, editors. Using heuristics to evaluate the playability of games. CHI'04 extended abstracts on Human factors in computing systems. New York, NY, United States: Association for Computing Machinery, 2004:1509-12.
27. Frazão K, Costa J, Viana D, Rivero L, editors. Analyzing App Store Comments and Quality Attributes for Defining an Inspection Checklist for Mobile Educational Games. Proceedings of the 34th Brazilian Symposium on Software Engineering. York, NY, United States: Association for Computing Machinery, 2020:854-9.
28. Escudeiro P, Escudeiro N. Evaluating educational games in mobile platforms. Int J Mobile Learn Organ 2013;7:14-28.
29. Mitgutsch K, Alvarado N, editors. Purposeful by design? A serious game design assessment framework. Proceedings of the International Conference on the foundations of digital games. New York, NY, USA: Association for Computing Machinery (ACM), 2012:121-8.
30. Mayer I. Towards a comprehensive methodology for the research and evaluation of serious games. Procedia Comput Sci 2012;15:233-47.
31. Steffen D, Muhm M, Christmann C, Bleser G, editors. A usability evaluation of a mobile exergame for ankle joint exercises. 2018 IEEE 6th International Conference on Serious Games and Applications for Health (SeGAH). New York, NY, USA: IEEE, 2018:1-8.
32. Löffler A, Levkovskyi B, Prifti L, Kienegger H, Krcmar H, editors. Teaching the digital transformation of business processes: design of a simulation game for information systems education. Wirtschaftsinformatik 2019;7:1-15.
33. Mousavi Baigi SF, Sarbaz M, Marouzi P, Kimiafar K. Evaluating the impact of digital game on learning medical terminology of paramedical students: Protocol for a randomized controlled trial. Stud Health Technol Inform 2022;295:51-4.
34. Mousavi Baigi SF, Moradi F, Vasseifard F, Mohammad Abadi F, Mazaheri Habibi MR. The Effect of Nutrition Training on Knowledge of Students at University of Medical Sciences. Top Clin Nutr 2022;37:236-41.
35. Sarbaz M, Baigi SF, Marouzi P, Hasani SM, Kimiafar K. Type and Number of Errors of the Iranian Electronic Health Record (SEPAS) in Hospitals Affiliated with Mashhad University of Medical Sciences. Stud Health Technol Inform 2022;295:354-7.
36. Sarbaz M, Monazah FM, Eslami S, Kimiafar K, Baigi SF. Effect of Mobile Health Interventions for Side Effects Management in Patients Undergoing Chemotherapy: A Systematic Review. Health Policy Technol 2022;11:100680.
37. Özsezer G, Mermer G. Using Artificial Intelligence in the COVID-19 Pandemic: A Systematic Review. Acta Med Iran 2022;60:387-97.
38. Garavand A, Samadbeik M, Aslani N. The Applications of Machine Learning Algorithms in Multiple Sclerosis: A Systematic Review. Acta Med Iran 2022;60:259-69.
39. Raeesi S, Hashemi R, Vahabi Z, Abdolahi M, Sedighiyan M. Is Percutaneous Endoscopic Gastrostomy (PEG) tube feeding Beneficial for Improving Survival in Patients with Dementia? A systematic review and meta-analysis of current evidences. Acta Med Iran 2022;60:5-17.
40. Kiani Feizabadi M, Mafakherian AM, Goudarzi A, Asadzandi S, Ahmadi M, Bigdeli S. Gamification in Radiology: A Systematic Review. Acta Med Iran 2020;57:605-13.
2. Palee P, Wongta N, Khwanngern K, Jitmun W, Choosri N. Serious game for teaching undergraduate medical students in cleft lip and palate treatment protocol. Int J Med Inform 2020;141:104166.
3. Pesántez-Cabrera P, Acosta MI, Jimbo V, Sinchi P, Cedillo P. Towards an evaluation method of how accessible serious games are to older adults. 2020 IEEE 8th International Conference on Serious Games and Applications for Health (SeGAH). IEEE, 2020:1-8.
4. Savi R, von Wangenheim CG, Borgatto AF. A model for the evaluation of educational games for teaching software engineering. 2011 25th Brazilian Symposium on Software Engineering (IEEE). IEEE, 2011:194-203.
5. Kato PM. Evaluating efficacy and validating games for health. Games Health J 2012;1:74-6.
6. Omari K, Moussetad M, Labriji E, Harchi S. Proposal for a New Tool to Evaluate a Serious Game. Int J Emerg Technol Learn 2020;15:238-51.
7. Eseryel D, Guo Y, Law V. Interactivity3 design and assessment framework for educational games to promote motivation and complex problem-solving skills. In: Ifenthaler D, Eseryel D, Ge X, eds. Assessment in Game-Based Learning: Foundations, Innovations, and Perspectives. New York: Springer, 2012:257-85.
8. McFarlane DA. Evaluating training programs: The four levels. J Appl Manag Entrep. Author manuscript 2006;11:96.
9. de Carvalho CV. Is game-based learning suitable for engineering education? Proceedings of the 2012 IEEE Global Engineering Education Conference (EDUCON). IEEE, 2012:1-8.
10. De Freitas S, Oliver M. How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Comput Educ 2006;46:249-64.
11. Connolly T, Stansfield M, Boyle L. Games-Based Learning Advancements for Multi-Sensory Human Computer Interfaces: Techniques and Effective Practices. Hershey, Pennsylvania, USA: IGI Global; 2009.
12. Escudeiro P, Escudeiro N. Evaluation of serious games in mobile platforms with qef: Qef (quantitative evaluation framework). 2012 March. 27. IEEE Seventh International Conference on Wireless, Mobile and Ubiquitous Technology in Education. IEEE, 2012:268-71.
13. Altanis I, Retalis S. A multifaceted students’ performance assessment framework for motion-based game-making projects with Scratch. EMI Educ Media Int 2019;56:201-17.
14. Geerts D, Nouwen M, van Beek E, Slegers K, Miranda FC, Bleumers L. Using the SGDA Framework to Design and Evaluate Research Games. Simul Gaming 2019;50:272-301.
15. Eseryel D, Ifenthaler D, Ge X. Validation study of a method for assessing complex ill-structured problem solving by using causal representations. Educ Technol Res Dev 2013;61:443-63.
16. De Grove F, Cauberghe V, Van Looy J. Development and Validation of an Instrument for Measuring Individual Motives for Playing Digital Games. Media Psychol 2016;19:101-25.
17. Hong JC, Cheng CL, Hwang MY, Lee CK, Chang HY. Assessing the educational values of digital games. J Comput Assist Learn 2009;25:423-37.
18. Borji YEL, Khaldi M. Comparative study to develop a tool for the quality assessment of serious games intended to be used in education. Int J Emerg Technol Learn 2014;9:50-5.
19. Fu FL, Su RC, Yu SC. EGameFlow: A scale to measure learners’ enjoyment of e-learning games. Comput Educ 2009;52:101-12.
20. Ak O. A game scale to evaluate educational computer games. Procedia-Social Behav Sci 2012;46:2477-81.
21. da Silveira AC, Martins RX, Vieira EA. E-Guess: Usability evaluation for educational games. RIED 2021;24:245-63.
22. Mayer I, Bekebrede G, Harteveld C, Warmelink H, Zhou Q, Van Ruijven T, et al. The research and evaluation of serious games: Toward a comprehensive methodology. Br J Educ Technol 2014;45:502-27.
23. Petri G, von Wangenheim CG, Borgatto AF. MEEGA+: an evolution of a model for the evaluation of educational games. Eval Educ Games Comput Educ 2016;3:1-40.
24. Alhuhud G, Altamimi W. Quality evaluation of mobile game: Miftah Alfasaha. Mob Inf Syst 2016;1:1-8.
25. Oprins E, Boer-Visschedijk GC, Roozeboom MB, Dankbaar M, Trooster W, Schuit SC. The game-based learning evaluation model (GEM): measuring the effectiveness of serious games using a standardised method. Int J Technol Enhanc Learn 2015;7:326-45.
26. Desurvire H, Caplan M, Toth JA, editors. Using heuristics to evaluate the playability of games. CHI'04 extended abstracts on Human factors in computing systems. New York, NY, United States: Association for Computing Machinery, 2004:1509-12.
27. Frazão K, Costa J, Viana D, Rivero L, editors. Analyzing App Store Comments and Quality Attributes for Defining an Inspection Checklist for Mobile Educational Games. Proceedings of the 34th Brazilian Symposium on Software Engineering. York, NY, United States: Association for Computing Machinery, 2020:854-9.
28. Escudeiro P, Escudeiro N. Evaluating educational games in mobile platforms. Int J Mobile Learn Organ 2013;7:14-28.
29. Mitgutsch K, Alvarado N, editors. Purposeful by design? A serious game design assessment framework. Proceedings of the International Conference on the foundations of digital games. New York, NY, USA: Association for Computing Machinery (ACM), 2012:121-8.
30. Mayer I. Towards a comprehensive methodology for the research and evaluation of serious games. Procedia Comput Sci 2012;15:233-47.
31. Steffen D, Muhm M, Christmann C, Bleser G, editors. A usability evaluation of a mobile exergame for ankle joint exercises. 2018 IEEE 6th International Conference on Serious Games and Applications for Health (SeGAH). New York, NY, USA: IEEE, 2018:1-8.
32. Löffler A, Levkovskyi B, Prifti L, Kienegger H, Krcmar H, editors. Teaching the digital transformation of business processes: design of a simulation game for information systems education. Wirtschaftsinformatik 2019;7:1-15.
33. Mousavi Baigi SF, Sarbaz M, Marouzi P, Kimiafar K. Evaluating the impact of digital game on learning medical terminology of paramedical students: Protocol for a randomized controlled trial. Stud Health Technol Inform 2022;295:51-4.
34. Mousavi Baigi SF, Moradi F, Vasseifard F, Mohammad Abadi F, Mazaheri Habibi MR. The Effect of Nutrition Training on Knowledge of Students at University of Medical Sciences. Top Clin Nutr 2022;37:236-41.
35. Sarbaz M, Baigi SF, Marouzi P, Hasani SM, Kimiafar K. Type and Number of Errors of the Iranian Electronic Health Record (SEPAS) in Hospitals Affiliated with Mashhad University of Medical Sciences. Stud Health Technol Inform 2022;295:354-7.
36. Sarbaz M, Monazah FM, Eslami S, Kimiafar K, Baigi SF. Effect of Mobile Health Interventions for Side Effects Management in Patients Undergoing Chemotherapy: A Systematic Review. Health Policy Technol 2022;11:100680.
37. Özsezer G, Mermer G. Using Artificial Intelligence in the COVID-19 Pandemic: A Systematic Review. Acta Med Iran 2022;60:387-97.
38. Garavand A, Samadbeik M, Aslani N. The Applications of Machine Learning Algorithms in Multiple Sclerosis: A Systematic Review. Acta Med Iran 2022;60:259-69.
39. Raeesi S, Hashemi R, Vahabi Z, Abdolahi M, Sedighiyan M. Is Percutaneous Endoscopic Gastrostomy (PEG) tube feeding Beneficial for Improving Survival in Patients with Dementia? A systematic review and meta-analysis of current evidences. Acta Med Iran 2022;60:5-17.
40. Kiani Feizabadi M, Mafakherian AM, Goudarzi A, Asadzandi S, Ahmadi M, Bigdeli S. Gamification in Radiology: A Systematic Review. Acta Med Iran 2020;57:605-13.
| Files | ||
| Issue | Vol 60 No 8 (2022) | |
| Section | Review Article(s) | |
| DOI | https://doi.org/10.18502/acta.v60i8.10835 | |
| Keywords | ||
| Digital educational game Evaluation Evaluation tool | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
How to Cite
1.
Mousavi Baigi SF, Norouzi Aval R, Sarbaz M, Kimiafar K. Evaluation Tools for Digital Educational Games: A Systematic Review. Acta Med Iran. 2022;60(8):508-512.
