Personalised learning paths for information literacy using Canvas MasteryPaths

This project report describes using MasteryPaths in the Canvas Virtual Learning Environment as a method of helping improve the information literacy (IL) competence for undergraduate science students studying in their first year at university. The MasteryPaths incorporated a series of formative quiz assessments on referencing and finding and evaluating information, which depending on the students’ initial score directed them to further enrichment or support materials. Four degree courses (Biology, Forensics, Biomedicine and Sport) each included the MasteryPaths in a first-year module in 2020/21, which were available in Canvas following face-to-face IL sessions. Focus groups were conducted with students, and interviews were carried out with two module leaders to explore perceptions of the MasteryPaths design and effectiveness for IL skills. The article provides insight into how online, self-paced, IL formative quiz assessments can be developed in a way to motivate and engage students in their learning.


Introduction
The importance of information Literacy (IL) has been recognised as a factor for progression and retention in Higher Education (Nieuwoudt & Pedler, 2021), and is a crucial life and employability skill. Specifically, in science disciplines the ability to find, evaluate, process, and use information is critical for researchers and practitioners operating in a rapidly changing, information rich field (American Library Association, 2006). Similarly, the Quality Assurance Agency (QAA) recognise IL in subject benchmark statements (QAA, 2019), and the ability is stated in course requirements for accrediting bodies such as the Institute of Biomedical Science (2020) and the Royal Society of Biology (2019). However, these skills are often not addressed directly within the taught curriculum (Weetman, 2005) and studies show teaching staff perceive students as lacking appropriate IL skills (Stebbing et al., 2019).
Often IL may be side-lined through lack of curriculum time, taken as implicit by experienced lecturers (Mabee & Francher, 2020), or subsumed within course learning objectives which focus on subject knowledge and higher order skills (Howard, 2012). IL skills are often taught by librarians as one-off sessions, either delivered in parallel to a course (McGuinness, 2007), or via inductions at the start of the year (Parramore, 2019;Willson, 2012). In these scenarios, librarians need to be flexible and consider student needs on a session-by-session basis, particularly for cohorts from non-traditional academic backgrounds (Mabee & Fancher, 2020).
As a result, IL frameworks are used as a pragmatic way for librarians to establish a set of baseline competencies (Reedy & Goodfellow, 2014;Ward & Hockey, 2007). In this way, librarians can progress and scaffold IL skills through the student's course. Even so, the fact remains that IL activities are often non-credit bearing, meaning students simply have less incentive to take part compared to course-focused activities (Jacobson, 2012).
To help students take ownership of their own IL development, librarians can provide cocurricular resources and tools, such as self-paced learning tutorials. The benefits of self-paced tutorials are that students can study independently at their own pace, have a choice of when and where to learn, and the content is perpetually available (Thornes, 2012). These tutorials extend the opportunities for one-off IL, though librarians need to consider how to make the selfpaced learning engaging and motivate students to participate. For example, accentuating choice in how students work through the material is preferable to a linear design with a single route through the content because students use a variety of learning strategies (Ramsden, 2003). One way of emphasising choice is to provide a range of short activities where students can self-select what is relevant to them (Villagrá, 2015). In this way, librarians can facilitate students' metacognitive skill development and ability to work as independent learners (Biggs & Tang, 2011;Humrickhouse, 2021).
Contextualised, discipline-based, IL is more relevant, and therefore more engaging to students, although Farrell and Badke (2015) suggest that collaboration with academic staff is key to achieving this alignment as they are best placed to identify course specific needs and IL gaps. Virtual Learning Environments (VLE) like Canvas can be a useful way to deliver credit-bearing, contextualised, asynchronous IL courses (Idleman, 2022;Lomness et al., 2021;Wegener, 2022). Embedding tutorials into the course VLE is a way for students see them as an integral part of their study, which can lead to more students engaging with the material (Porter et al., 2010;Tsunekage et al., 2020). Rivers-Latham et al. (2020) used the Canvas VLE to develop a subject-agnostic IL tutorial in Canvas that could be embedded in a variety of courses, though this did not yield course credit for students upon completion. Subject-agnostic tutorials that use generic examples may save development time, but students could struggle to apply the examples to their specific discipline, so may be less motivated to engage or participate.
Another way to incentivize students with online, self-paced learning, is to incorporate direct feedback and accountability (Piskurich, 2015). Lierman and Santiago (2019) for example, found that student engagement increased with course-focused tutorials combined with a librarian-led face-to-face session, compared with online tutorials alone. Evidence also suggests that IL competence is correlated with increased librarian contact (Bonnet et al., 2018). In these instances, pedagogical use cases, like flipped learning materials (Shen, 2018), or post-learning activities, are both ways in which tutorials can be used in conjunction with librarian contact.
When designing tutorial activities, active modes of learning such as quizzes, rather than passive activities like reading text or watching videos (Carr et al., 2015) are most effective for helping to increase engagement and student learning. Active learning is defined as "anything that 'involves students in doing things and thinking about the things they are doing.'" (Bonwell & Eison, 1991, p. 19). When developing quizzes, for instance, question types such as drag and drop, text entry, or multiple-choice can test knowledge comprehension. This variety of questions gives multiple opportunities for engagement (Lange et al., 2011) and the immediate feedback and the automatic grading can inspire reflection, which is notably suitable for students to learn procedural skills such as referencing. Nevertheless, quiz questions should cover different concepts to encourage multi-level thinking (Goodsett, 2018). Explorative tasks are also an element of active learning (Carr et al., 2015), and are ideal for complex skills like information evaluation.
Students report greater control over their learning, and better understanding of the topic in selfpaced tutorials that utilise or are based on mastery learning (Liu et al., 2017;Oerther, 2022;Reddy et al., 2013;Simon-Campbell & Phelan, 2016). Mastery learning consists of providing students a diagnostic assessment for a topic to formatively assess their knowledge. Depending how students perform, they are directed to further enrichment or support materials, making the learning personalised, and providing students an equal opportunity to learn the topic whatever their starting knowledge.
MasteryPaths in Canvas (Instructure, 2022a) provide a convenient way to develop simple mastery learning activities with quizzes and discussions. Similar features can be found in other VLEs such as Desire to Learn and Moodle (Padayachee et al., 2018). To date there is lack of studies either evaluating MasteryPaths in Canvas or using mastery learning for IL skills development. This study aimed to develop a set of activities, which were structured as MasteryPaths. The MasteryPaths were intended to develop IL competence for first year Science degree students at Anglia Ruskin University (ARU) in referencing, evaluation and finding articles. The study objective was to evaluate how the MasteryPaths could be improved via focus groups with students.

Methodology
Canvas MasteryPaths (Instructure, 2022a) was used to structure a set of quizzes to develop students' referencing, finding information and evaluating information skills. The MasteryPaths were designed to be embedded in the Canvas courses of four 15-credit first year degree modules in Biomedicine, Sport Science, Biology, and Forensics courses. The lead author (Learning Technologist) researched the pedagogical approach and tested the technical functionality of MasteryPaths, while the co-author (Subject Librarian) wrote and developed the quizzes in Canvas. The MasteryPaths structure is shown in Figure 1 and the full list of quizzes is outlined in Table 1. The overall learning objectives of the MasteryPaths were constructively aligned with the assignment and marking rubrics for these modules and used the Open University's Digital and Information Literacy Framework (Open University, 2019; Reedy & Goodfellow, 2014) to map the IL skills required at university. Although the quiz questions were the same for each module, the literature and website examples used were tailored towards each specific discipline. Feedback on the MasteryPath quiz questions was sought from module leaders and library colleagues but no formal pilot was conducted with students beforehand. A diagnostic quiz is followed either by enrichment or supporting material and a further quiz, depending on the score achieved. Students scoring above 80% were deemed to have met the quiz learning objectives which could be achieved either after the first or second quiz. A score below 80% unlocked supporting material and extra practice questions.
Quiz questions were a mixture of multiple choice, fill in the blanks (with and without drop-down options), and matching (see figures 2 and 3 for example questions), and used fictional students, basic scenarios, and vignettes which were designed to be meaningful yet depersonalised (Benedetti et al., 2018). The MasteryPaths were imported into the Canvas course for each of the four modules by week 3 of the trimester and promoted in face-to-face sessions with students by the Subject Librarian. The nature and level of Subject Librarian contact varied between the subjects, ranging from lecture slots to practical workshops. An incentive was offered for successful completion of one of the diagnostic quizzes and subsequent participation in a focus group. The incentive was Personal and Professional Development (PPD) points which contributed to a compulsory pass/fail course module.

Data collection
Ethics approval was obtained from the School of Education and Social Care (ESC-SREP-18-328) to conduct research with students and staff. Focus groups were undertaken with students to gather qualitative feedback and were chosen as the most appropriate method to obtain rich data from multiple viewpoints (Savin-Baden & Howell-Major, 2013). The focus groups were advertised to 490 students across the four modules containing the MasteryPaths and participants were recruited using convenience sampling. Two focus groups were conducted in person in Trimester 1, one with 6 students, the other 3. In Trimester 2, an additional 2 students were interviewed, both of whom had started their course in January. Each focus group or interview lasted between 30-40 minutes, and a student research assistant was employed to lead the discussions with the lead author. Both the focus groups and interviews were audio recorded and transcribed later by the research assistant.
Focus group questions were designed to explore what students liked, disliked, what they would change, their emotional response, and any perceived impact on their IL skill level. Feedback was sought on the questions from a range of representatives prior to the focus group questions to check they were inclusive and appropriate for the audience.
In addition, two module leaders took part in one-to-one interviews, each lasting 15 minutes. Interview questions focused on the module leader's view of the embedded quizzes and student learning. The interview and focus group questions can be found in the appendix. The data that support the findings of this study are available in FigShare at 10.25411/aru.19903267, the private link to view the data is: https://figshare.com/s/b2ba19f5a893b5235b4c.

Results
Trimester 1 MasteryPath engagement are summarised in Table 1, with 165 attempts of the quizzes in total. There were half as many attempts at the supporting quizzes which contained additional questions. Take-up of the MasteryPaths introduced in Trimester 2 was low and statistics are not reported here. A third of students taking the diagnostic quizzes achieved above the 80% threshold score.
Biomedical Science received the most hands-on contact time with the librarian, with that group constituting the highest MasteryPath uptake. Students from Sport and Exercise Sciences and Forensic Science made the fewest quiz attempts, despite receiving similar contact time. *Students were allowed repeated attempts to achieve above 80% score.

Focus group findings
The interview and focus group transcripts were analysed using content analysis using an inductive coding method. The content analysis was completed by the lead author using the Nvivo software (QSR International, 2022), whilst the co-author completed regular checks for coding consistency. The first phase identified transcript text that described how the MasteryPath quizzes could be improved, student engagement increased, or helped with student learning. The highlighted sections of the transcript text were checked in isolation and assigned codes to form the initial code base. These codes were categorised into seven themes which were Cognizance of the quizzes, Future development, Improved areas of learning, Independent learning and study methods, Motivation to do the quizzes, Quiz design and content, and Usability. In the second phase, duplicated codes were removed or organised into more appropriate themes, and the meaning of the themes was further clarified. In the final phase, the seven themes were reduced to four main themes by merging similar themes and re-coding where necessary. The final four main themes were labelled Motivation, Cognizance, Design and Development, and Learning, each of which had their own sub-themes, which are described in more detail below. The section for each theme includes a table with representative quotes from the focus groups and interviews with lecturers. Quotes were chosen to present a balanced view, both positive and negative.

Theme 1 -Motivation
Factors included under this theme were those that motivated students to attempt and complete the MasteryPaths. Opinions on what would incentivise students from future cohorts to complete the activities were also included. The two sub-themes identified were credit and prior experience (see Table 2 for representative quotes). Suggestions to encourage greater take-up included making the MasteryPaths mandatory or optional, or giving more substantial credit, such as counting directly towards the summative assessment. Such comments suggested the quizzes were viewed favourably. However, some students expressed confusion about the difference between the MasteryPaths and quizzes which had been introduced by the lecturer. Self-motivation and conscientious learning behaviour were other driving factorsstudents wanted to do well in their course and take-up relevant development opportunities. Some students said they had completed the Extended Project Qualification (EPQ) and already appreciated the importance of skills like referencing to achieve a higher grade. From the lecturers' perspective, one interviewee stated that MasteryPaths could be a pre-requisite of submitting an appropriate assignment, and the reason to learn the skills, alongside the relevant areas of the marking criteria, should be explicitly highlighted.

Theme 2 -Cognizance
Students' awareness of the MasteryPaths and the ways they found them were categorised in this theme (see Table 3 with representative quotes). Ideas voiced around better ways to present or promote future quizzes were also included. Students in the focus groups said they had not discussed the MasteryPaths with peers in their classes, nor could they recall them being promoted by the Subject Librarian in lectures. Suggestions to help promotion included lecturers recommending the quizzes at the end of relevant classes and reminders via the VLE. Students also expressed frustration at being unable to find the MasteryPaths, and some students recalled discovering them by accident.

Theme 3 -Design and development
Comments on question design and writing along with references to the usability and functionality in the VLE were included in this theme (see Table 4 for representative quotes). Students and lecturers thought the examples used in the quizzes were relevant to the subject area and the number of questions was suitable. However, the scoring and weighting of the questions could have been clearer and fairer. Although the difficulty level was considered appropriate, it was felt an easier quiz could have provided a better entry point for some students, especially any students less familiar with the English language.
Some questions had text-heavy instructions, and students suggested condensing these descriptions, using videos instead, or making instructions available as a downloadable PDF cheat-sheet. Some of the specific instructions to locate databases were described as subjective and students had other ways of performing the same task. Students liked the fact that they could re-take the quiz to check and practice their skills, but a frequent recommendation was to allow a variation in questions on subsequent attempts.

Theme 4 -Learning
Students' comments on improvements in their own skills or knowledge because of taking the MasteryPaths were included in this theme along with any other comments related to what or how they learnt (see representative quotes in table 5). Students said that the MasteryPaths helped them to identify how to use library resources, such as the website, Google Scholar and how to start evaluating online sources. Some new ways of finding sources were identified. The questions on referencing provided a way to identify the correct referencing style. A lecturer interviewee also highlighted that plagiarism referrals were reduced for the relevant module, although no causal link can be made to quiz participation. Students also provided insight into where they completed the quizzes, and how using a mobile device gave more opportunities for learning at opportune moments.

Discussion
The study aimed to evaluate MasteryPath quizzes in Canvas to teach IL skills for Sport, Biomedical Science, Biology and Forensics. The MasteryPaths were introduced to students via face-to-face IL sessions with a Subject Librarian and were then available in the VLE course for each module for students to complete independently. The study was limited by the small sample in both number of students who took the MasteryPaths and participated in the focus groups, so the findings may not generalise to other subjects or student cohorts.
In terms of the number of quiz attempts, the PDP points incentive may not have been substantial enough to motivate the non-quiz takers. Many studies such as Williams (2013) and Mahmood (2016) highlight that some students are overconfident in their information searching capability, and the students in this study may not have perceived themselves as needing to improve their skills by taking the MasteryPaths. Conversely, a lack of confidence and difficulties in coping with the many aspects of transition to Higher Education could reduce students' willingness to engage with non-credit bearing activities (Mabee & Fancher, 2020). Moreover, students with prior experience of Higher Education, through the EPQ or a foundation year, saw the importance of taking the quizzes to orientate themselves to such factors such as referencing. This demonstrates that recognising the value of the IL skills for educational progression, employability, and life skills, may be a more important motivating factor for students than credit. However, when asked how take-up could be improved, focus group participants said the quizzes should be credit-bearing rather than voluntary. One barrier to implementing this strategy is the time and resources required from both the librarian and the module leader. Changing the summative assessment would require module re-validation for example, which is beyond the scope of this study.
The most pertinent finding was the favourable way the MasteryPaths were viewed by the students in the focus groups. Students liked getting the chance to repeat the questions and improve their scores, commented they wanted more opportunities to practice, and retake the quizzes with the same questions or different questions. The challenge of completing the quiz and not achieving the score they wanted may have motivated students to try again, and the immediate available feedback from the multiple-choice questions was a reward which reinforced the learning process. Also, participants valued the flexibility of asynchronous learning, which provided an opportunity to learn during spare moments like commuting. Students also commented that they benefited from learning new skills, such as citation searching and information evaluation, by answering questions which encouraged multi-level thinking through exploration and self-discovery.
Despite participants valuing the MasteryPaths for their own learning, of the many students who completed the first quiz, only a few continued to the second level of supplementary quizzes. Some students, therefore, may have thought the first quiz was sufficient for their skill development. Moreover, competing demands on students' time can mitigate against participation with learning materials (Liu et al., 2017). Students commented that the text-heavy nature of the questions looked intimidating, and simpler presentation, or a more straightforward quiz, may improve student engagement. In addition, students noted that the questions did not always award partial points in line with their expectations, which could have compounded the intimidation factor. It is important to account for the limitations or functionality in the VLE when designing learning materials (Lierman & Santiago, 2019). In this case, making it clear that points were only awarded for a full correct answer would have made the scoring more transparent and appear fairer.
However, a wider concern was the comment by a student that they discovered the MasteryPaths by accident. Other students suggested that clearer signposting to the material was needed. Although the MasteryPaths were placed prominently in each module's Canvas course, it is easy for some items on the VLE to go unnoticed by students due to the sheer volume of material (Rothera, 2015). In addition, VLEs may be difficult to use (Abuhlfaia & De Quincey, 2019), which can make it harder to find content. Features such as full-text search, suggested content, or notification functionality might not be present in VLEs, placing the onus on the student to locate relevant pages manually and bookmark key content for future reference. However, even if a group of students are taught how to find relevant material, the fact remains that some students often forget key messages, and the only way to counter this is through constant repetition and reminders (Rothera, 2015). Closer collaboration with academic staff, rather than co-ordination and co-operation (Montiel-Overall, 2006), could also have enabled greater prominence for the quizzes and potentially increased engagement.

Conclusion
In conclusion, the study found that students viewed the MasteryPaths positively, and the active learning approach was one factor in this. A small sample of students engaged with the MasteryPaths, and not all students who took the diagnostic quiz took the supplementary quizzes. In future, the MasteryPaths can be improved by making it easier to find them on the VLE, providing an easier point of entry, and increasing the number of questions available. One approach for future development we would suggest is to redesign the initial diagnostic quiz. The diagnostic quiz could focus on the students' existing practical experience of IL tools, techniques and concepts, and use initial questions that require a Yes or No response. This quiz could serve as a directional tool, rather than a mechanism for detailed skills development and selfassessment. This would help students to realise the value of the content covered by the MasteryPaths. Supplementary material could then focus on the areas of IL in more detail. The quizzes could also use the question banks feature, also available in Canvas (Instructure, 2022b). Question banks allow a quiz to show a selection of questions to the student, which are automatically taken from a wide selection of questions. Each time the student takes the quiz, a random, but distinct set of questions is displayed, meaning the students can re-test themselves. Additionally, the MasteryPaths were closely tailored to specific course assignments with sources selected for each of the subjects (Sport, Biomedicine, Forensics or Biology). A further avenue of research would be to create a version of the MasteryPaths with more generally relevant scientific examples to investigate if a similar proportion of students engaged with the MasteryPaths. One benefit of using the same examples in all the MasteryPaths would be to save quiz development time and enable easier redevelopment for other subjects.