Update Dec 2015: there are now three posts on our refinements to this peer feedback activity: one, two, and three.

Readers have been begging for news of how it went with the Moodle Workshop activity from this post.

Workshop is an activity in Moodle which allows staff to set up a peer assessment or (in our case) peer review. Workshop collects student work, automatically allocates reviewers, allows the review to be scaffolded with questions, imposes deadlines on the submission and assessment phase, provides a dashboard so staff can follow progress, and allows staff to assess the reviews/assessments as well as the submissions.

However, except for some intrepid pioneers, it is almost never seen in the wild.

The reason for that is partly to do with daunting number and nature of the settings – there are several pitfalls to avoid which aren’t obvious on first pass – but also the fact that because it is a process you can’t easily see a demo and running a test instance is pretty time consuming. If people try once and it doesn’t work well they rarely try again.

Well look no further – CALT and ELE have it working well now and can support you with your own peer review.

What happened?

Students on the UCL Arena Teaching Associate Programme reviewed each others’ case studies. 22 then completed a short evaluation questionnaire in which they rated their experience of giving and receiving feedback on a five-point scale and commented on their responses. The students were from two groups with different tutors running the peer review activity. A third group leader chose to run the peer review on Moodle Forum since it would allow students to easily see each others’ case studies and feedback.

The students reported that giving feedback went well (21 respondents):

Satisfaction with reviewing work – click to enlarge

This indicates that the measures we took – see previous post – to address disorientation and participation were successful. In particular we were better aware of where the description, instructions for submission, instructions for assessment, and concluding comments would display, and put the relevant information into each.

Receiving feedback also went well (22 respondents) though with a slightly bigger spread in both directions:

Satisfaction with receiving reviews – click to enlarge

Students appreciated:

• Feedback on their work.
• Insights about their own work from considering others’ work.
• Being able to edit their submission in advance of the deadline.
• The improved instructions letting them know what to do, when and where.

Staff appreciated:

This hasn’t been formally evaluated, but from informal conversations I know that the two group leaders appreciate Moodle taking on the grunt work of allocation. However, this depends on setting a hard deadline with no late submissions (otherwise staff have to keep checking for late submissions and allocating those manually) and one of the leaders was less comfortable with this than the other. Neither found it too onerous to write diary notes to send reminders and alerts to students to move the activity along – in any case this manual messaging will hopefully become unnecessary with the arrival of Moodle Events in the coming upgrade.

For next time:

• Improve signposting from the Moodle course area front page, and maybe the title of the Workshop itself, so students know what to do and when.
• Instructions: let students know how many reviews they are expected to do; let them know if they should expect variety in how the submissions display – in our case some were attachments while others were typed directly into Moodle (we may want to set attachments to zero); include word count guidance in the instructions for submission and assessment.
• Consider including an example case study & review for reference (Workshop allows this).
• Address the issue that, due to some non-participation during the Assessment phase, some students gave more feedback than they received.
• We originally had a single comments field but will now structure the peer review with some questions aligned to the relevant parts of the criteria.
• Decide about anonymity – should both submissions and reviews be anonymous, or one or the other, or neither? These can be configured via the Workshop’s Permissions. Let students know who can see what.
• Also to consider – we could also change Permissions after it’s complete (or even while it’s running) to allow students to access the dashboard and see all the case studies and all the feedback.

Have you had a good experience with Moodle Workshop? What made it work for you?

Over 200 UCL colleagues have already joined UCL Arena Digital, our free online course to help improve Moodle skills and enhance your online/blended learning.

We are in Week 2 but you are still welcome to join.

The course is fully online and will take only 2-3 hours of your week. The course is made up of three Units. Each unit will last 2 weeks and there will be breaks in between Units. Each fortnight will end with a live online webinar where you can share your experiences with your colleagues on the course.

The course is designed so you can take all three Units, or simply pop in for the Units that especially interest you.

• Unit 1: multimedia – the current one – find out how to create and embed media and interactive tools in Moodle to enliven the online environment for your students.
• Unit 2: communication – discover ways of using tools inside and outside of Moodle you can use to communicate with students and support their collaboration with each other.
• Unit 3: assessment and feedback – explore ways of using the online environment to create new kinds of assessment and give feedback to students.

Unit 1 started last week and will continue to Thursday 12 March, when we will conclude with a webinar.

Even if you missed last week there is still time to get involved and all the materials will also be available afterwards.

You can enrol at https://moodle.ucl.ac.uk/course/view.php?id=29477

Log on using your UCL username and password

Unit 2 will launch in early April 2015 – look out for further announcements.

Example of a Moodle Workshop activity dashboard

Update Dec 2015: there are now three posts on our refinements to this peer feedback activity: one, two, and three.

Working with E-Learning Environments, UCL Arena Teaching Associate Programme leaders in CALT have been trialling Workshop (Moodle’s peer assessment tool) to run a peer review activity with participants. We’re now on the second iteration – here are some opportunities and lessons learned. One group is using Moodle Forum, so in a month or so we’ll be able to compare the two platforms – but for now I’ll focus on the Workshop.

The scenario

Participants write a 500 word case study about an aspects of learning, teaching and assessment mapped to aspects of the UK Professional Services Framework, and review three others. The review takes the form of summary comments (i.e. no numeric marks, no rubric, no structured questions to answer). They have roughly a week to prepare the submission and a week to carry out the assessments. Participation is strongly encouraged but not compulsory.

From the evaluation of the first iteration

36 participants gave feedback at the end of 2014. 29 participants found the experience of giving assessment positive (fine, good or excellent, 12, 14 and 3 respectively) while 7 found it unsatisfactory or poor (5 and 2 respectively). Receiving assessment was less positive (fine, good or excellent, 6, 3 and 0 respectively) while 4 found it unsatisfactory and 3, poor.

The gist was that the concept was good and the software worked fine but the management needed some attention. The first problem was one of disorientation – “finding my feedback was not straightforward”. We addressed this in the next iteration by using the instructions (in the Workshop settings) and announcements in person and via the News Forum. The second and related problem was to do with lack of notification – “it wasn’t very clear how to use the system and no emails were received”; “working fine but it needs to be improved – notification; instructions”; “I did not receive any alert or instructions on how to check if the feedback from my colleague was in”. We addressed this by putting diary entries for each group leader to notify, remind and instruct participants about what to do at each stage. The third problem was that several participants didn’t receive any reviews – this was because the activity was grouped with a consequently smaller pool or reviewers for each submission, coupled with the fact that it wasn’t a compulsory activity, and exacerbated by the fact that Moodle doesn’t send out alerts when the phases switch e.g. from submission to assessment. We straightforwardly addressed this by removing the groups setting and undertaking to notify students about what to expect and when.

Decisions, decisions – settings and reasons

Below are some of the less obvious settings, our rationale, and implications.

• Grading strategy: Comments – this gives a single field into which participants type or paste summary comments.
• Grades: none; neither for the submission nor the peer assessment.
• Instructions for submission: as briefly as possible what participants need to do to make a successful submission.
• Submissions after the deadline: we left this set to No (unchecked) because rather than manually allocating submissions to reviewers we wanted Moodle to handle this with a scheduled allocation at the submission deadline. Workshop (unlike Turnitin Peermark) does this once only, which means that unless somebody was prepared to go into the Workshop and manually make allocations for late submissions, those late submissions would go unreviewed. Disallowing late submissions would give a very hard cut-off for submissions but greatly reduce the admin burden. This is what we ultimately decided to do, hoping that we could increase participation through good instructions and some scheduled reminders.
• Instructions for assessment: since the activity required reviewers to leave just a single summary comment, all we did here was direct attention to the guidance on relating the case study to the Professional Services Framework, and remind about the lack of autosave in Moodle form fields.
• Students may assess their own work:  we left this set to No (unchecked) since one aim of the activity was to share practice.
• Overall feedback mode: this is the setting that gives a text field for the summary comments; we set it to Enabled And Required.
• Maximum number of feedback files: set to zero, since we wanted the experience of reading the feedback to be as seamless as possible.
• Use examples: for this low stakes peer review we didn’t require participants to assess examples.
• Open for assessment / open for submission: we set the the assessment phase to begin directly as the submission phase closed; this meant that we’d also to set up Scheduled Allocation to run at that time.
• Switch to the next phase after the submissions deadline: we set this to Yes (checked); in combination with Scheduled Allocation this would reduce the amount of active supervision required on the part of staff.
• Group mode: we left this set to No Groups. Groups of four (learning sets which we call Quartets) had been set up on Moodle but the previous iteration had shown that when applied to a Workshop (set not to allow self-assessment) the would diminish the pool of possible submissions and possible reviewers, and was vulnerable to non-participation.
• Grouping: constrasting with Groups, this allows a given activity or resource to be hidden from everyone except the chosen grouping. We’d set up Groupings in the Moodle area corresponding to UCL schools, because the sessions (and therefore the deadlines) for them happen at different times. So we set up Moodle workshops which were duplicates in every respect except the dates.
• Scheduled allocations: these can be set up via a link from the dashboard.
• Enable scheduled allocations: Yes (checked) for the reasons above. This would happen once at the end of the Submission Phase.
• Number of reviews: we set three per submission but if (rather than focusing on ensuring that each submission got three reviews) we wanted to shift the emphasis onto the reviewing process we could have set three per reviewer.
• Participants can assess without having submitted anything: we left this set to No (unchecked) reasoning that participants were more likely to receive reviews if we kept the pool of reviewers limited to those who were actively participating. (That said, we could do with finding out more about how Workshop allocates reviews if they are set to allocate to reviewers rather than to submissions.)

Dates for diaries (it doesn’t run itself…)

That said, where participants are unfamiliar with the process any peer review activity needs quite active supervision. For this reason, CALT staff (who have many other commitments) put dates in their diaries to monitor participation and send reminders, as well as to maintain awareness of which phases the activity was in. Of particular note, to release the feedback to participants a staff member needs to actively close the activity by clicking something in the Workshop dashboard.

What happened next?

It went well – see this March 2015 follow-up post including evaluation.

Improve your Moodle skills and enhance your online/blended teaching by taking part in UCL Arena Digital.

UCL Arena Digital is a free online course for all staff at UCL.

The course is fully online and will take only 2-3 hours of your week. The course is made up of three Units. Each unit will last 2 weeks and there will be breaks in between Units. Each fortnight will end with a live online webinar where you can share your experiences with your colleagues on the course.

The course is designed so you can take all three Units, or simply pop in for the Units that especially interest you.

• Unit 1: multimedia – find out how to create and embed media and interactive tools in Moodle to enliven the online environment for your students.
• Unit 2: communication – discover ways of using tools inside and outside of Moodle you can use to communicate with students and support their collaboration with each other.
• Unit 3: assessment and feedback – explore ways of using the online environment to create new kinds of assessment and give feedback to students.

Unit 1 starts Monday 2nd March 2015 and lasts for 2 weeks.

Enrol now at https://moodle.ucl.ac.uk/course/view.php?id=29477

Log on using your UCL username and password

Unit 2 will launch in early April 2015 – look out for further announcements.

Do you use Moodle, Turnitin, Lecturecast, Opinio, My Portfolio, the UCL Wiki? Do you use Email, text, Facebook to contact students, do you read or contribute to blogs or Twitter etc. or use other technologies to support the student learning experience?

If so, why not try UCL UCL’s professional portfolio in e-learning? 

Now in its fourth year at UCL, CMALT is a chance to learn about, share and implement good practice in the wide range of technologies that support our students’ teaching and learning.

“Working together with colleagues from across UCL was helpful in terms of discovering and developing good practice”.

CMALT is a national peer-based professional accreditation scheme developed by the Association for Learning Technology (ALT) and an opportunity to certify your growing skills and experience in learning technology.

Why not join us and work towards a CMALT certificate?

Please see our  flyer for more information.

There will be a lunch time meeting with Liz Wyatt from ALT to provide further information for prospective participants on Wednesday 21st January 2015 from 12pm-1pm in Chandler House, room 118.

All staff are welcome, although we recommend that UCL CMALT candidates have been working for one academic year, or have relevant previous experience. Please email a.gilry@ucl.ac.uk

If you are interested in CMALT but unable to attend this meeting please contact a.gilry@ucl.ac.uk

All UCL Faculty of Brain Sciences Masters students were invited to submit a two-minute video that summarised their research. The aim of the competition was to showcase the high quality research being conducted by Masters students and to provide an opportunity for students to develop the necessary skills to make their research accessible to the public.

Students were instructed to answer the following:

What is your research question?
What have you found in relation to your question?
Why is it important?

The entries were of a  high standard and demonstrated the excellent work taking place across the faculty. Four students particularly impressed the panel of judges with their ability to communicate their message in a clear and engaging manner.

The winners of our first Masters Video Competition are:

1st place:

Tara Brah (MSc Biology of Vision, supervised by Prof Shin-ichi Ohnuma). How do we make the third eye?

2nd place:

Giulia Borghini (MSc Cognitive Neuroscience, supervised by Prof Vince Walsh and Dr Marinella Cappaletti). Alpha stimulation effects on working memory and inhibitory abilities in elderly

Highly Commended:

Nathan Hayes (MSc Developmental Neuroscience and Psychopathology, supervised by Dr Helena Rutherford). The Impact of Maternal Substance Use on Neural Processing of Social and Non-social Feedback

Highly Commended:

Seray Ibrahim (MRes Speech, Language and Cognition, supervised by Dr Michael Clarke and Dr Duncan Brumby).   Involving child communication aid users in the development of communication aids
https://www.youtube.com/watch?v=tEwtQCcJpPU

More information:  Poster about the competition http://www.ucl.ac.uk/slms/education/education-domain/documents/posters/video_competition_showcasing_research.pdf by Dr Jennifer Rodd and Dr Alex Standen