The research presented in this dissertation utilised a quasi-experimental design, involving a control (n = 15) and experimental group (n = 15), in order to answer the research question posed and consequently evaluate in empirical terms, the effect of a game-informed approach to assessment on academic achievement of Form 5 male students in Malta.

This section will discuss these research findings by interpreting and critically evaluating the data and results obtained from both the questionnaire and post-intervention test, in view of the present study and scholarly literature. The implications of the findings will be then assessed and explored on both a practical and theoretical dimension.

The use of digital technologies for assessment purposes

The initial results from the questionnaire point towards an extremely limited use of digital technologies in assessment procedures prior to the research, in both Physics and other subjects. The findings also confirm that all the students were at par in the previous use of digital assessment technologies and thus ethically, no participating group was at a disadvantage. Seventy-five percent of the technologies listed in the questionnaire (including wikis, clickers and web-based assessment systems), were never used by the students for assessment purposes in Physics. Since computer labs with internet access are available throughout the school, such finding may be likely to be related to the reluctance of traditional educators ‘to make substantial changes in assessment processes’ (Russell et al 2006, p 475) and adopt such technologies, in view of the ever-increasing institutional demands for reliability and validity (Knight 2001). No statistically significant difference was found between the 2 groups, for the frequency of use of interactive whiteboards and online quizzes in assessment during the past Physics lessons, which was however still limited from ‘weekly to never’. All the students indicated the use of clickers during assessment activities, yet again sporadically (from ‘every term to rarely’), in ICT lessons. This further reinforces the research design in terms of controlling over a possible extraneous variable (Robson 2011) and establishing equity between groups, as the use of digital technologies for assessment purposes represented an innovation for all participants. As such, any reactivity to the intervention from the participants, resulting from the awareness of being observed, known as Hawthorne effect (Hedges 2012), is to be equally attributed across both groups (Hedges 2012).

Although the adjusted post-intervention mean score measuring academic achievement was higher for the experimental (51.5%) than the control group (45.3%), such variation represents a non-statistically significant difference in the students’ overall achievement (F(1,27) = 2.663, p > 0.05, partial η² = 0.090). This suggests that a game-informed approach to assessment has an equal overall effect to its traditional counterpart, on students’ academic achievement, operationally defined over the different cognitive processes set by Bloom’s taxonomy (Bloom 1956) and measured using a conventional objectively scored achievement test. To an extent, this is in contrast to the meta-analysis performed by Black and William (1998) and more recent empirical research (Araceli Ruiz-Primo and Furtak 2006, Wang 2007, Wiliam et al 2004, Wininger 2005), suggesting that students who are formatively assessed are more inclined to obtain higher scores on summative tests. However, as already discussed in the Review of Literature, such findings should be interpreted with caution in their insufficiency to conclusively demonstrate the effect of formative assessment on students’ academic achievement (Dunn and Mulvenon 2009). It is also worth noting, that even though the difference in means between the game-informed and traditional assessment groups is not statistically significant, the probability that such variation has occurred by chance rather than due to the educational intervention is low (11.4%). Furthermore, the effect size of the proportion of total variance associated with the game-informed approach rather than to other variables is medium (η² = 0.090) (Richardson 2011). These findings are even more encouraging given that the game-informed assessment group, having a traditional assessment background, managed to obtain a higher (although not statistically significant) overall score than students following a traditional assessment, in a traditional context and on a traditional achievement test. This lack of constructive alignment between teaching/learning activities and assessment (Biggs 2003) represented a further challenge for the experimental group.

However, on analysing the scores obtained in terms of the cognitive processes involved, the academic achievement of the experimental group was statistically significantly superior to that of the control group at the higher levels of the taxonomy (Bloom 1956). In fact, the traditional assessment group performance at the knowledge, comprehension and application levels matched that of the experimental group. This supports literature, which suggests that traditional modes of assessment work equally well at the lower cognitive levels (Biggs 1996, Ramsden 1997), while a game-informed approach becomes more effective at the more complex and abstract levels of the cumulative hierarchy represented by Bloom’s taxonomy (Bloom 1956). This also implies that traditional assessment, in the form of summative examinations, which is predominantly testing students at lower cognitive levels (Zheng et al 2008), requires adequate consideration and due diligence in preparing students at the higher levels of the cognitive domain. Such findings are in line with previous research, which reported an improved academic achievement at the higher levels of the cognitive domain for students who followed various formative assessment and learning strategies (Chang and Mao 1999, Crooks 1988, Tenenbaum 1986). In turn, the observed correlation may be explained in terms of the intrinsic metacognitive properties possessed by the game-informed intervention, during which the experimental group was continuously being exposed to metacognitive knowledge, along with the more traditional factual, conceptual and procedural types (Anderson and Krathwohl 2001). As such, operating metacognitive knowledge at the higher cognitive processes facilitates and promotes meaningful rather than rote learning (Mayer 2002).

The metacognitive skills practiced during the intervention by the game-informed assessment group, allowed the students to pay conscious awareness to the learning and assessment process (Pintrich 2002). This was achieved by creating a learning and assessment context that sustained multiple ‘feedback loops’ (Sadler 1989, p 120) and a cumulative approach to both coursework and learning (Hounsell et al 2007); exemplified in the 2-staged submission of practical work write-ups. Students’ self- and peer-assessment in the PeerWise activity allowed for an increased participation in assessment (Carless 2007) and provided greater access to the teacher’s and peers’ expertise and experience (Taras 2001). Student-generated assessment criteria and nominated test questions in the wiki exercise, together with the collaborative nature of the task, favoured self-regulation strategies (Nicol 2014, Pintrich 2002) and encouraged teacher and peer dialogue around learning (Nicol and Macfarlane-Dick 2006). Prompt and just-in-time feedback (Chickering and Gamson 1999) during the online multiple-choice questions on in2fiziks.com, the computer adaptive test on TokToL and offline quizzes using clickers facilitated the development of self-assessment and reflection (Nicol and Macfarlane-Dick 2006). Feedforward on these early low-stakes tasks contributed qualitatively to and fed directly into later work (Hounsell et al 2007), which resulted in a formal post-intervention test score measuring academic achievement. This constructive interplay between the formative and summative dimensions of assessment, in terms of an enhanced metacognitive activity, has significantly increased the quality of learning for the experimental group (Hounsell et al 2007). As a result, students who followed a game-informed assessment outperformed their peers in the control group on the post-intervention test, in breaking the problems posed into basic components, noting and explaining existing relationships between different parts so as to form a new complete whole and evaluating evidence through informed judgements (Bloom 1956).