Systematic literature review about gamification in MOOCsMarta Jarnac de Freitasand Miguel Mira da SilvaUniversidade De Lisboa Instituto Superior Técnico, Lisbon, PortugalABSTRACTOne of the main problems of Massive Open Online Courses (MOOCs) is the low retention rate of students and a high no-show rate. Meanwhile gamification has been gaining notoriety in the last decade within the education field. This systematic literature review explores the impact gamification has had in MOOCs, how their success is being measured, what are the theories normally asso-ciated with these gamified learning environments, and what game design elements are used as well as their implementation and outcome. We’ve reviewed 22 papers dating from 2014 to July 2019. Our findings are positive in terms of the outcomes, with a general increase in participation and retention on gamified MOOCs.KEYWORDS MOOC; gamification; motivation; student retention1.IntroductionGamification as the use of game design elements in non-game contexts (Deterding et al., 2011) has been researched in several domains such as Health/Exercise, Crowdsourcing and most notably Education/Learning. The latter corresponding to 46.7% and 35.4% of empirical and non-empirical papers, respectively, on gamification (Koivisto & Hamari, 2019). This rapid rise of academic literature on gamification started in 2011 (Koivisto & Hamari, 2019) and gamification is becoming more mainstream and could be a potential strategy to apply to MOOCs and raise retention.MOOCs’ low retention rates, percentage of users having completed the course, are a predominant problem with rates of less than 10% (Jordan, 2014). This further highlights the potential of gamification in MOOCs as a way to increase retention, such as the success of Vaibhav and Gupta (2014) when applying game elements in their MOOC. This high-lights our first motivator for this study, the impact that gamification has had on education already.By their very nature MOOCs offer distance learning for all sorts of individuals, being especially beneficial for those who do not have access to a traditional higher education setting. Butcher and Rose-Adams (2015) found that in their study sample of part-time distance learners this type of distance education was at times their only option, making clear the need for further research in the area.CONTACT Marta Jarnac de Freitas marta.freitas@tecnico.ulisboa.pt Universidade De Lisboa Instituto Superior Técnico, Lisboa 1049-001, PortugalOPEN LEARNING: THE JOURNAL OF OPEN, DISTANCE AND E-LEARNING 2023, VOL. 38, NO. 1, 73–95 https://doi.org/10.1080/02680513.2020.1798221© 2020 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group. This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any med-ium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. The terms on which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent.
We’ve found one existing literature review (Khalil et al., 2018) and one systematic literature review (Ortega-Arranz et al., 2017), although neither was extensive and they analysed a low number of empirical studies. The limitation of existing literature is our second motivator to conduct this systematic literature review, in order to provide a concise summary of the existing literature on this growing field and to examine how it’s being studied.With engagement in courses so low our goal is to see how gamification is implemented in MOOCs, and how its impact is being measured. We’ll look into the motivational theory used to support the choice of game design elements, to the objective behind its imple-mentation and later the effects it all had on users.1.1.Research goal and questionsThe main goals of this research are to identify the impact that gamification has in MOOCs, as well as the different success metrics being used. Our research questions are:RQ1 What’s the impact of gamification on MOOCs?●What type of MOOCs are used?●What identifiers of performance are used to evaluate gamified MOOCs?●Why does it have impact?●What gamification elements are used?●What effect does gamification have in MOOCs?2.Background2.1.MOOCs and gamification in educationMOOCs emerged in the last decade, with 2012 being called by The New York Times ‘the year of the MOOCs’ (Pappano, 2012). They are Massive, meaning the number of people who can enrol is typically unlimited. Open, meaning anyone can participate and it’s usually free. Online, meaning anyone anywhere with internet access can participate and get an education even if unable to move around. These three qualities present a great potential to bring education to anyone with access to the internet.Despite these promising qualities, Jordan (2014) who analysed the initial trends in MOOCs showed average completion rates of 9.8%, ranging from 1.4% to 50.1%, with average enrolments of 42,844 users.Meanwhile gamification as the use of game design elements in non-game contexts (Deterding et al., 2011) has been researched in several domains such as Health/Exercise, Crowdsourcing and most notably Education/Learning. The latter corresponding to 46.7% and 35.4% of empirical and non-empirical papers, respectively, on gamification (Koivisto & Hamari, 2019). This rapid rise of academic literature on gamification started in 2011 (Koivisto & Hamari, 2019) and gamification is becoming more mainstream and could be a potential strategy to apply to MOOCs.Domínguez et al. (2013) concluded that gamification in e-learning platforms has the potential to increase student motivation, when designed and implemented with thought. Dicheva et al. (2015) came to the same conclusion. When conducting a literature review they found that most authors agreed gamification has the potential to improve learning, 74M. JARNAC DE FREITAS AND M. MIRA DA SILVA
however they further explained that empirical evaluation is still scarce. But most impor-tantly they emphasised the difficult task of correctly implementing gamification to increase motivation.In a subsequent critical review, Dichev and Dicheva (2017) reported that the use of gamification in education studies lacks a theoretical background. This is in line with our own findings in Section 4.3. Landers (2015) too pointed out the need for a theoretical model that relates the game design elements to its learning outcomes.2.2.Self-determination theorySelf-determination theory (SDT) was originally developed by Deci and Ryan, and is an empirically derived theory of human motivation and personality in social contexts (Deci & Ryan, 2012).SDT distinguishes between intrinsic and extrinsic motivation. An intrinsically motivated individual will do an activity because it is inherently interesting or enjoyable, not because of an external source. While an extrinsically motivated individual will perform a task for an external outcome different from the task itself.For Niemiec and Ryan (2009) intrinsic motivation satisfies the need for competence, autonomy and relatedness. Students are autonomous when choosing to spend their time and energy on a class, and competent when they are able to do their work. However it is necessary to be both autonomous and competent to be intrinsically motivated, if students are competent but not autonomous the state of intrinsic motivation will not be main-tained (Niemiec & Ryan, 2009).Ryan and Deci showed also that the more students were externally regulated the less they showed interest, value, or effort. These findings agree with those of Landers, Bauer, Callan, and Armstrong who state that if users in areas like education only find external rewards to motivate them, their genuine interest in the activities may not be present. In fact an individual may shift intrinsic instincts to extrinsic ones if rewards are present (Landers et al., 2015). For students many tasks are inherently interesting or enjoyable, and knowing how to promote more active (self-determined) forms of extrinsic motivation becomes an essential strategy for successful teaching (Ryan & Deci, 2000).2.3.Flow theoryCsikszentmihalyi (2000) defined flow as a state of absorption in one’s work characterised by intense concentration, loss of self-awareness, a feeling of being perfectly challenged and a sense that time is flying, it is an optimal state of intrinsic motivation (Hansch et al., 2015). A person in the flow is so involved in an activity that nothing else seems important (Csikszentmihalyi, 2010).Aparicio et al. (2019) say flow won’t help gamification. It can however be used to plan difficulty of badges (Bustamante & Jiménez, 2019), meaning to plan the difficulty of small goals. When reviewed alongside SDT, Flow Theory can help us to know when to set rewards, and what those rewards should be.OPEN LEARNING: THE JOURNAL OF OPEN, DISTANCE AND E-LEARNING75
3.Review methodTo perform a Systematic Literature Review (SLR) we followed the guidelines of Keele et al. (2007). There are three phases in this process: planning, conducting and reporting the review.In the initial phase, planning the review (Section 3.1), we analysed the need for an SLR, formulated the research questions, developed and later evaluated a review protocol. In the second phase, conducting the review (Section 3.2), we identified the research available, did a study selection and quality assessment, and conducted a data extraction and synthesis. The third and final phase consisted of reporting of the review (Section 4).3.1.Planning the reviewIn the introduction we discussed the need for this SLR stemming from the lack of a concise yet thorough review of the literature. We formulated the research questions (Section 1.1) and developed a review protocol.To do this we set the search term to ((MOOC OR MOOCs) AND gamification) and selected the search engines with which we would conduct the search: ACM Digital Library; IEEE Digital Library; dblp; Scopus; Google Scholar.Lastly we set our selection criteria. Inclusion criteria: empirical papers; related to the main question; languages spoken by us en/pt/es/fr. Exclusion criteria: languages all but en/pt/es/fr; conceptual papers; papers on Serious Games; papers on Playful Design; no access to full paper; low quality journal; unknown journal quality. Also a limited number (n = 3) of conceptual papers were included.3.2.Conducting the reviewThe initial search returned 169 papers, excluding 209 duplicates. After that an initial selection took place, based on the title and abstract of each paper and its relatedness to our research questions and based on the quality of the journal. This first selection rejected 45 papers.A final selection took place based on the introduction and conclusion of the papers resulting in 102 rejected papers. And so we were left with 22 papers to analyse (see Table 1).4.DiscussionIn this section we will present the data collected and summarised in order to answer our research questions.Table 1. Paper selection.Accepted2nd Rejection1st RejectionDuplicatedTotalACM Digital Library0311721dblp3201419IEEE Digital Library3531425Scopus127131125239Google Scholar421103974Total221024520937876M. JARNAC DE FREITAS AND M. MIRA DA SILVA
4.1.What type of MOOCs are used?MOOC types vary greatly based on the objectives and theory used to develop them. These are the types of MOOCs that appeared in our literary review:●SPOC (Small Private Open Course): supplement classroom teaching as opposed to replacing it (Piccioni et al., 2014);●cMOOC (Connectivist MOOC): the concept comprehends a connected and sharing digital context, which follows a philosophy of collectivism (Aparicio et al., 2019);●xMOOC (eXtended MOOC): based on a behaviourist pedagogical approach and focused on existing courses in universities (Aparicio et al., 2019);●cooperative MOOC: this combines features both of the xMOOC and cMOOC, defined by three layers: technological, training, behavioural (Fidalgo-Blanco et al., 2013);●gcMOOC (gamification cooperative MOOC): adds a new gamification layer to the cooperative MOOC.Our search analysed papers from 2014 to July 2019, see Figure 1.In Figure 2 we added only papers that self-identified as a certain type, hence only 5 papers were included. For a more extensive list of MOOC types, that did not appear in our literary review, see Bustamante and Jiménez (2019).Figure 1.: Number of papers by year of publication.OPEN LEARNING: THE JOURNAL OF OPEN, DISTANCE AND E-LEARNING77
4.2.What identifiers of performance are used to evaluate gamified MOOCs?In the following section we’ll look into the different ways in which papers define success in MOOCs, meaning what identifiers of performance are used to evaluate gamified MOOCs and their overall performance.4.2.1.EngagementThe most common goal of research into MOOCs is to increase engagement in users, however this is not reported as often as an effect of gamification in MOOCs (see Figures 3 and 6).Engagement is often used as a means to increase other effects like retention rates, participation or motivation. It should be decided at the planning stage exactly how the level of engagement will be determined at the end of a study.4.2.2.Retention rateThe increment of retention rates, sometimes referred to as completion rate, is the second most reported goal. However its definition is not always the same with some authors calculating it with equation 1 and others with equation 2, giving vastly different results.Retention rate is usually defined as the number of students who completed the course (ncomp) divided by the number of students who enrolled (nenrolled). Figure 2.: Number of papers by type of MOOC.78M. JARNAC DE FREITAS AND M. MIRA DA SILVA
4.2.3.Net retention rateHowever some papers define it not by the number of enrolled students, but by the number of students who never participated in any MOOC activity (nno_show). This is because there can be a big discrepancy between the number of enrolled (nenrolled) and the number of initially active people (nenrolled – nno_show). To solve this problem some authors like Bustamante and Jiménez (2019) calculate their retention rate like so: In Table 3 we compared three of the papers which had enough information to calculate Retention Rate (RR), Net Retention Rate (NRR) and the rate of enrolled users who never participated in any MOOC activity, no_show rate. The latter is defined as: We can see in Table 3, the NRR is always higher than RR, and as we can deduce NRR will always be higher or equal to RR since the best possible outcome is to have a case with nno_show = 0, which would lead to NRR = RR.Staubitz et al. also argue that no-show rates effect results, but they went a step further proposing to look only at users who enrolled before the middle of the course when considering the completion rate. This reasoning stands because, depending on the evaluation method, a user who entered half-way through a course doesn’t have enough time to get a certificate and some activities may have already ended. Instead of regarding the lack of a certificate gained by these users as a failure, they can be considered simply as those who could never have finished the MOOC since they enrolled too late.Figure 3.: Number of papers by goal.OPEN LEARNING: THE JOURNAL OF OPEN, DISTANCE AND E-LEARNING79
Some confusion may arise because some papers don’t specify exactly which equation they use referring simply to retention or completion rates. NRR (equation 2) enhances results when compared to RR (equation 1).4.2.4.Overall goal achievement rateThe retention rate focuses on completing a MOOC and earning a certificate, but that’s not every user’s goal.We cannot assume everyone intends to finish a MOOC, users have their own personal goals, from earning a certificate, to auditing the course or simply browsing. Ortega-Arranz et al. (2019) sent an initial questionnaire to the students of their study and found only 58.64% planned to actively participate in the course.Antonaci et al. (2017a) consider that MOOCs do not need to be completed to be considered successful, and so they propose a new way of calculating the completion rate, keeping in mind the users’ personal goals:●User Intention Ratio (UIR): percentage of the course intended to be completed by the user●Personal Completion Rate (PCR): percentage of the course completed by the user.The new measure for personal success becomes: And so the new measure for overall success becomes: Even with these ways of calculating overall success we should keep in mind that Krause et al. (2015) stress that even students with a personal goal to complete the course may struggle to achieve it.Wilkowski et al. (2014) were also interested by personal individual goals and asked students of their MOOC to complete a questionnaire about their intent when enrolling in the MOOC. These goals could vary from completing the whole course, to just one specific part. They concluded that 42.4% either met or exceeded the goals they set out to achieve when enrolling.Furthermore, after the MOOC was over Wilkowski et al. (2014) sent a follow up survey and found that:●90.8% of the students who completed the course and answered the survey agreed that they met their personal goal●51.8% of the students who did not complete the course but answered the survey agreed that they met their goal.These new ways of rethinking retention rates can show that MOOCs are more successful than previously thought. Thinking about the objective of the users and not the point of view of instructors helps teachers to design better MOOCs.80M. JARNAC DE FREITAS AND M. MIRA DA SILVA
4.2.5.MotivationThree papers identified motivation as one of their goals in the MOOC and all three were successful in their goal, indicating motivation as an outcome of their MOOC (see Table 2). Staubitz et al. (2017) reported based on SDT that users were intrinsically motivated to perform self-tests and early submissions, and proposed they should decrease gamification on these activities and focus on forum activity (see Section 4.3.2 Self-Determination Theory for more on intrinsic/extrinsic motivation).Motivation can be important to lead a user to complete a task and participate (Muntean, 2011), which in turn can lead to an increase in retention rates (Romero- Rodríguez et al., 2019). But as with engagement, motivation is not directly measurable and it should be decided beforehand with a base theory how exactly it should be measured.For more detail on these three papers, see: Aparicio et al. (2019), Ortega-Arranz et al. (2019).4.2.6.OthersSeveral identifiers of performance are used, but we should call to attention that these goals set are not always analysed extensively at the end of studies. Just as mentioned in Section 4.2.1 engagement identifiers of performance should be planned before the study and later properly reported so as to have a clearer idea of what makes a MOOC successful.4.3.Why does it have impact?In this section we will look into the theories used when developing a gamified MOOC. Many papers did not mention motivational nor education theoretical references as a stepping stone for their work. As a result the most mentioned theories (see Figure 4), Self-Determination Theory and Flow Theory, are two decades old and don’t reflect the current boom that has appeared in the last few years around gamification.4.3.1.Intrinsic and extrinsic motivationAll papers mentioning extrinsic motivation also mentioned intrinsic motivation (see Figure 4 and Table 4). These concepts were further explained in Section 2.2.To prevent gamification from lowering intrinsic motivation some authors gamified only optional parts of the course (Ortega-Arranz, Bote-Lorenzo et al., 2019; Ortega-Arranz et al., 2017). Some used curiosity and novelty to try and increase intrinsic motivation (Khalil et al., 2017).Staubitz et al. (2017) used intrinsic and extrinsic motivation to review user types, based on the user types defined by Marczewski (2015), and which game design element could have the effect they wanted on each type. They tried to define three user types (socialisers, achievers, explorers) with each type being intrinsically motivated by a slightly different thing. For example the socialisers being motivated by relatedness.4.3.2.Self-determination theorySelf-Determination Theory (SDT), as explained in Section 2.2, has been at least partly mentioned by four papers out of the 22 analysed (see Figure 4). SDT is also indirectly mentioned twice by papers talking about intrinsic motivation (see Table 4).OPEN LEARNING: THE JOURNAL OF OPEN, DISTANCE AND E-LEARNING81
Table 2. Data collected.ReferencesGoalsTheoriesElementsEffectsExtraAparicio et al. (2019)- Success- Flow theory - Information Systems (IS) success theory - Gamification theory- Points - Peer-review - Time constraints - Clear goals - Increasing difficulty- Participation - Success - Enjoyment - Satisfaction - Challenge- xMOOC - IS success modelStaubitz et al. (2017)- Retention Rate (RR) - Motivation- Self-determination theory - Intrinsic/extrinsic motivation - Drive theory - RAMP (Relatedness, Autonomy, Mastery, Purpose) framework- Badges - Points - Forums - Progress bar- Participation - Motivation - Performance- User type - Analyse several MOOCs - 26 weeksVaibhav and Gupta (2014)- Engagement - Retention Rate - Compare non-gamified with gamified MOOC- None- Flash cards - Others (speller, space race scatter)- Increased RR - Participation - Enjoyment - Feeling challenged - Reduced failure rate - Improved learning100 participants - Evaluating: one final testSaraguro-Bravo et al. (2016)- Satisfaction - Increase Participation- Intrinsic/extrinsic motivation - Game Thinking- Badges- Increased motivation - Increased engagement - Satisfaction− 100 active students - 4 weeksOrtega-Arranz et al. (2019)- Engagement - Motivation- Flow theory- Badges - Leaderboard - Forums - Peer-review- Participation - Increased motivation- User types - 8 weeks - nenrolled = 1031 - nno_show = 342 - ncomp = 117Chang and Wei (2016)- Engagement - Identifying game elements- None− 40 gamification elements- Identified game elements- nparticipants = 5020 –40 gamification elements analysed(Continued)82M. JARNAC DE FREITAS AND M. MIRA DA SILVA