Example of a Moodle Workshop activity dashboard

Update Dec 2015: there are now three posts on our refinements to this peer feedback activity: one, two, and three.

Working with E-Learning Environments, UCL Arena Teaching Associate Programme leaders in CALT have been trialling Workshop (Moodle’s peer assessment tool) to run a peer review activity with participants. We’re now on the second iteration – here are some opportunities and lessons learned. One group is using Moodle Forum, so in a month or so we’ll be able to compare the two platforms – but for now I’ll focus on the Workshop.

The scenario

Participants write a 500 word case study about an aspects of learning, teaching and assessment mapped to aspects of the UK Professional Services Framework, and review three others. The review takes the form of summary comments (i.e. no numeric marks, no rubric, no structured questions to answer). They have roughly a week to prepare the submission and a week to carry out the assessments. Participation is strongly encouraged but not compulsory.

From the evaluation of the first iteration

36 participants gave feedback at the end of 2014. 29 participants found the experience of giving assessment positive (fine, good or excellent, 12, 14 and 3 respectively) while 7 found it unsatisfactory or poor (5 and 2 respectively). Receiving assessment was less positive (fine, good or excellent, 6, 3 and 0 respectively) while 4 found it unsatisfactory and 3, poor.

The gist was that the concept was good and the software worked fine but the management needed some attention. The first problem was one of disorientation – “finding my feedback was not straightforward”. We addressed this in the next iteration by using the instructions (in the Workshop settings) and announcements in person and via the News Forum. The second and related problem was to do with lack of notification – “it wasn’t very clear how to use the system and no emails were received”; “working fine but it needs to be improved – notification; instructions”; “I did not receive any alert or instructions on how to check if the feedback from my colleague was in”. We addressed this by putting diary entries for each group leader to notify, remind and instruct participants about what to do at each stage. The third problem was that several participants didn’t receive any reviews – this was because the activity was grouped with a consequently smaller pool or reviewers for each submission, coupled with the fact that it wasn’t a compulsory activity, and exacerbated by the fact that Moodle doesn’t send out alerts when the phases switch e.g. from submission to assessment. We straightforwardly addressed this by removing the groups setting and undertaking to notify students about what to expect and when.

Decisions, decisions – settings and reasons

Below are some of the less obvious settings, our rationale, and implications.

• Grading strategy: Comments – this gives a single field into which participants type or paste summary comments.
• Grades: none; neither for the submission nor the peer assessment.
• Instructions for submission: as briefly as possible what participants need to do to make a successful submission.
• Submissions after the deadline: we left this set to No (unchecked) because rather than manually allocating submissions to reviewers we wanted Moodle to handle this with a scheduled allocation at the submission deadline. Workshop (unlike Turnitin Peermark) does this once only, which means that unless somebody was prepared to go into the Workshop and manually make allocations for late submissions, those late submissions would go unreviewed. Disallowing late submissions would give a very hard cut-off for submissions but greatly reduce the admin burden. This is what we ultimately decided to do, hoping that we could increase participation through good instructions and some scheduled reminders.
• Instructions for assessment: since the activity required reviewers to leave just a single summary comment, all we did here was direct attention to the guidance on relating the case study to the Professional Services Framework, and remind about the lack of autosave in Moodle form fields.
• Students may assess their own work:  we left this set to No (unchecked) since one aim of the activity was to share practice.
• Overall feedback mode: this is the setting that gives a text field for the summary comments; we set it to Enabled And Required.
• Maximum number of feedback files: set to zero, since we wanted the experience of reading the feedback to be as seamless as possible.
• Use examples: for this low stakes peer review we didn’t require participants to assess examples.
• Open for assessment / open for submission: we set the the assessment phase to begin directly as the submission phase closed; this meant that we’d also to set up Scheduled Allocation to run at that time.
• Switch to the next phase after the submissions deadline: we set this to Yes (checked); in combination with Scheduled Allocation this would reduce the amount of active supervision required on the part of staff.
• Group mode: we left this set to No Groups. Groups of four (learning sets which we call Quartets) had been set up on Moodle but the previous iteration had shown that when applied to a Workshop (set not to allow self-assessment) the would diminish the pool of possible submissions and possible reviewers, and was vulnerable to non-participation.
• Grouping: constrasting with Groups, this allows a given activity or resource to be hidden from everyone except the chosen grouping. We’d set up Groupings in the Moodle area corresponding to UCL schools, because the sessions (and therefore the deadlines) for them happen at different times. So we set up Moodle workshops which were duplicates in every respect except the dates.
• Scheduled allocations: these can be set up via a link from the dashboard.
• Enable scheduled allocations: Yes (checked) for the reasons above. This would happen once at the end of the Submission Phase.
• Number of reviews: we set three per submission but if (rather than focusing on ensuring that each submission got three reviews) we wanted to shift the emphasis onto the reviewing process we could have set three per reviewer.
• Participants can assess without having submitted anything: we left this set to No (unchecked) reasoning that participants were more likely to receive reviews if we kept the pool of reviewers limited to those who were actively participating. (That said, we could do with finding out more about how Workshop allocates reviews if they are set to allocate to reviewers rather than to submissions.)

Dates for diaries (it doesn’t run itself…)

That said, where participants are unfamiliar with the process any peer review activity needs quite active supervision. For this reason, CALT staff (who have many other commitments) put dates in their diaries to monitor participation and send reminders, as well as to maintain awareness of which phases the activity was in. Of particular note, to release the feedback to participants a staff member needs to actively close the activity by clicking something in the Workshop dashboard.

What happened next?

It went well – see this March 2015 follow-up post including evaluation.

Improve your Moodle skills and enhance your online/blended teaching by taking part in UCL Arena Digital.

UCL Arena Digital is a free online course for all staff at UCL.

The course is fully online and will take only 2-3 hours of your week. The course is made up of three Units. Each unit will last 2 weeks and there will be breaks in between Units. Each fortnight will end with a live online webinar where you can share your experiences with your colleagues on the course.

The course is designed so you can take all three Units, or simply pop in for the Units that especially interest you.

• Unit 1: multimedia – find out how to create and embed media and interactive tools in Moodle to enliven the online environment for your students.
• Unit 2: communication – discover ways of using tools inside and outside of Moodle you can use to communicate with students and support their collaboration with each other.
• Unit 3: assessment and feedback – explore ways of using the online environment to create new kinds of assessment and give feedback to students.

Unit 1 starts Monday 2nd March 2015 and lasts for 2 weeks.

Enrol now at https://moodle.ucl.ac.uk/course/view.php?id=29477

Log on using your UCL username and password

Unit 2 will launch in early April 2015 – look out for further announcements.

Box of Broadcast

BoB (Box of Broadcasts) National is an innovative shared online off-air TV and radio recording service for UK higher and further education institutions.

Staff and students can record programmes from 65+ TV and radio channels.  The recorded programmes are kept indefinitely in an media archive, which currently stores over 2 million programmes and are shared by users across all subscribing institutions. The archive also includes searchable transcripts and one click citation referencing.
The recordings can be set before or after the broadcast (30 day recording buffer). The programmes can be edited into clips and shared with others. They can also be embedded into Moodle.
To start using BoB, log in with your UCL user details http://bobnational.net/

Clive Young (ELE) and Stefanie Anyadi (UCL Division of Psychology and Language Sciences) led an ALT webinar today on work UCL has been doing with our community of teaching administrators (TAs).

We described the now-completed JISC Digital Department project that supported these staff in developing their digital literacies and in working more systematically and strategically with them as change agents. This had led directly to the establishment of our supported programme leading to the Certified Membership of the Association for Learning Technology (CMALT). We also introduced the E-Learning Champions initiative and explained why we had included TAs to work in partnership with academics and ELE staff. Although very much a work in progress this has proved effective and has already helped benchmark e-learning activity, develop local plans across two of our schools and has led to the emergence of active faculty-level e-learning groups.

The slides and recording are available on the ALT Repository at http://repository.alt.ac.uk/2351/

The UCL E-Learning Champions initiative is actually now a bit over a year old – we launched on Valentine’s Day in 2013. At the Summits & Horizons event on Monday we took the chance to review progress so far.

We have certainly grown over the last year, we now have over 130 members. Nearly all UCL departments and divisions are represented and although there is work to be done (not everyone knows who their Champion is!) in some areas the initiative seems remarkably well established.

I opened the session by reminding the sizeable audience that the Champions were key to our ambitious institutional target to develop departmental e-learning plans. Pairs of departmental E-Learning Champions – one academic and one teaching administrator (TA) – had been identified to help develop local e-learning statements articulating departments’ expectations and priorities, highlighting good practice, and identifying support needs.

Fourteen months into the implementation plan, this type of engagement with the champions has been excellent. Champions are well embedded in UCL’s learning technology strategic and support initiatives and are now beginning to be regarded as an important group of change agents, with the support of ELE.

This approach has already helped ELE benchmark e-learning activity, develop local plans across two of our schools and has led to the emergence of faculty-level e‑learning Champion groups. During the event our three E-Learning Facilitators, Jess Gramp (BEAMS), Natasa Perovic (SLMS) and Mira Vogel (SLASH) explained how the Champions had helped with a wide range of local initiatives and projects, in areas such as Turnitin, multimedia group work and enhancing Moodle provision.

The Champions’ role is also evolving. Mira reported that SLASH Joint Faculty Departmental Teaching Committee Chairs had been asked to ensure that their departmental E-Learning Champions were either members of the DTC or had a direct reporting link to it. SLASH has also convened an E-Learning Forum to discuss the issues of the day, identify support needs and plan future directions. Likewise in BEAMS there are now regular E-Learning Champions Faculty meetings in MAPS & Engineering. In SLMS Natasa has been meeting with meetings with all faculty tutors and now divisions. She reported the E-learning needs in SLMS  are also related to collaborative tools (for content delivery and learning activities), media rich interactive resources and ‘beyond the baseline’ Moodle activities.

We asked attendees if the Champions’ role should be more ‘formally ‘described but the audience was split. As one participant said, it was the informality of the group that made it successful.

We believe UCL is at the forefront of recognising that the complexity of e-learning provision in a modern university requires the development of a digitally literate community comprising a wide range of colleagues. We therefore discussed the personal development of Champions and Rosalind Duhs from CALT explained how the UCL Arena Fellowship programme could be an excellent route. Being a Champion and involvement in local projects would provide a sound basis for the development of portfolio case studies.

The session was concluded with an engaging presentation from Dr Adrien Desjardins who explained his role as a Champion in Medical Physics and Bioengineering. This was followed by further contributions from the audience and a lively discussion. We are planning a more formal review of the programme and this was a great start. Many thanks to all who contributed.

Congratulations to all the winners of the 2012-13 Provost Teaching Awards.

In this video the winners of this year’s awards describe the initiatives they use to support students in their learning at UCL. Read more on the CALT website…

(more…)