When I started my Ph.D. project funded by the Flemish Agency by Innovation and Technology (IWT) at iMinds-MICT-Ghent University, aiming to develop a standardized procedure for assessing the effectiveness of digital game-based learning (DGBL), my supervisors and I stumbled upon an issue. When we tried to find a clear definition of effectiveness in this context, we were not able to find one. So, it was not surprising that when we conducted a systematic literature review on how effectiveness of DGBL is currently being assessed, we saw that researchers were assessing different types of outcomes as indicators of effectiveness.
Since the concept of ‘effectiveness’ is essential in my Ph.D. project, we needed to explore this concept in more detail and look for a way to operationalize it, make it usable. So, I asked myself, why is it important to assess effectiveness? Well, common sense could firstly help me answer this question: if we are making claims about the potential benefits of games as instructional tools, we need to provide empirical evidence for this. Also, games are expensive to develop, so if we decide to invest in them, we better have a good reason for it. But it was scientific literature -of course- that brought light in my darkness. A proven effectiveness also leads to a higher probability that the target group will adopt or implement DGBL, which has proven to be a big issue for the industry. It was this statement that guided us towards a theoretical framework to help us develop an operationalization of effectiveness, enter Social Cognitive Theory. Why? Because Bandura’s concept of Agency links effectiveness evaluation to human behavior. More specifically, people evaluate their own behavior by using the desired outcomes of that behavior as a benchmark. So, we concluded that in order to define effectiveness of DGBL we needed to know what outcomes people desire to attain with its implementation.
For this purpose, we conducted focus groups with 3 relevant stakeholders: DGBL researchers and developers, potential adopters of DGBL and stakeholders on a governmental level. An overview of our results can be found in the table below. Our results revealed the existence of three categories of outcomes that should be considered when assessing DGBL effectiveness: learning, motivational and efficiency outcomes. For these outcomes, several indicators can be used. This does not imply that all outcomes need to be assessed. This depends on the sector in which the game will be implemented and the type of content treated in the game.
So, we hope that the table below will be used as a communication tool between DGBL developers, DGBL researchers and clients in order to tune game development and research to the needs of the client ordering the development of DGBL. By doing this, implementation of DGBL is stimulated considering that -following to social cognitive theory- the more outcomes are desired for the client(s), the higher the likelihood of implementing DGBL.
| Learning outcomes | Motivational outcomes | Efficiency outcomes |
| Situational interest
DGBL stimulates interest in the content matter discussed in the game. |
Enjoyment
DGBL succeeds in creating an enjoyable game experience. |
Time management
DGBL succeeds in reducing the timeframe required to teach a certain content matter. This is a judgment of relative worth compared to other instructional methods.
|
| Performance
DGBL succeeds achieving learning goals as defined by the game developer/the client who ordered the development of the game.
|
Motivation towards DGBL
Learning with the game-based method is motivating. This is a judgment of relative worth, compared to other instructional methods. |
Cost-effectiveness
DGBL succeeds in reducing the cost of the intervention with regard to: a) the number of learners that can be reached and b) the time required to teach the target group certain content. This is a judgment of relative worth, compared to other instructional methods. |
| Transfer
DGBL stimulates application of learned content matter in the game to real world situations. |
Results of this study are discussed in:
All., A. Nuñez Castellar, E.P. & Van Looy, J. (2015). Towards a conceptual framework for assessing the effectiveness of digital game-based learning. Computers & Education, In press.
If you are interested in staying up to date with date of publication, please follow me on Twitter: @AnissaAll
For more information about this Ph.D. project here
Connect with me on LinkedIn or contact me at: Anissa.All@UGent.be