This blog post has been re-made for exams in 2021 – visit the Marking centrally managed exams in 2021 blog post.

---

Please note this post is being regularly updated with additional resources.
+ New on Friday 8 May: Guide to online marking from Mary Richardson
+ New on Friday 8 May: Microsoft Drawboard PDF demo from Dewi Lewis, UCL Chemistry
+ New on Friday 15 May: Updated details on Microsoft Drawboard PDF

The move to online 24 hour assessments that replace traditional exams leads to a challenge for those that have to grade and mark the work.

We start from a place of two knowns:

• Students are submitting work to Turnitin in Moodle during the 24 hour window; and
• Final grades need to be stored in Portico, our student records system.

But in between those two endpoints, there are many different workflows by which marking can take place. These are set out by the UCL’s Academic Manual but encompasses a range of choices, particularly in how second marking is completed. One key difference between regular courseworks is that this is not about providing feedback to students, but instead about supporting the marking process, the communication between markers and the required record of the marking process. At the end of the marking process departments will need to ensure that scripts are stored securely but can be accessed by relevant staff as required, much in line with requirements for paper versions over previous years.

Neither SRS nor Digital Education mandate any particular way that marking should take place and there is considerable flexibility for departments to use processes that work best for them. So we are suggesting a menu of options which provide a basis for departments to build on if they so choose. We are also running daily training sessions which as listed at the foot of this document.

The menu options are:

• Markers review the scripts and mark or annotate them using Turnitin Feedback Studio
• Digital Education will provide PDF copies of scripts for departments to annotate using PDF annotation software on a computer or tablet device.
• Markers review the scripts using Turnitin Feedback Studio, but keep a ‘marker file’ or notes and comments on the marking process.
• Markers print the scripts and mark them, then scan them for storage or keep them for return to the department on paper.

The rest of this post goes into these options in more detail.

---

Turnitin Feedback Studio

Turnitin Feedback Studio provides a web-based interface where comments can be overlaid on a student’s work. QuickMarks provide a bank of comments that occur regularly that can be just drag and dropped onto the work. In addition, the traditional Turnitin Similarity Report is also available. This method probably works best for text-based documents like essays and reports. Turnitin is integrated into Moodle and set up for you as part of the exam process for students to submit their work, but it’s your choice if you wish to use the marking tools available after the work has been submitted. The Turnitin submission boxes have been set up for you, and we ask that you don’t change the settings or set up any grading templates to mark student work before submission, as this could prevent students from submitting.

You can also allocate marks using grading forms or rubrics.  On the whole we think that these could be a bit of a ‘sledgehammer to crack a nut’ solution for a single paper, but it is an option available to you if you are familiar with them and you have a more granular set marking criteria for each question. We recommend hiding the assignment before adding the grading form or rubric so that students cannot see it.

If you want to know if this method is for you, you can watch a short video demo or  try marking up an example paper provided by Turnitin. A video tailored to UCL’s 24 hour exam process is given below. This video has captions.

Things to think about with this approach:

• Rubrics and grading forms take a little bit of setting up, and are probably best used where you have previous experience in doing them.
• In some exams it is common to put a mark (e.g. a tick) on each page to indicate that the page has been read. To replicate this you might define a QuickMark called ‘page read’ and put it on each page, or annotate with the same words
• The marked paper often becomes a resource to go back to if there are any errors or omissions in the grading process. You might wish to both write the marks on the paper using the annotation tools or in the general feedback area, and also lodge them in a spreadsheet for uploading to Portico.
• Turnitin does not support double blind marking effectively. It is rarely used for paper-based exams (since the second marker could always see the markings of the first marker on the paper) but if it was needed one marker could mark online and the second could download the papers for offline marking (e.g the ‘marker form’ method below).

You can view additional guidance on using Turnitin Feedback Studio.

---

Annotation using PDF documents

Where you annotation needs are more sophisticated, or you want to ‘write’ on the paper using a graphics tablet or a tablet and pencil/stylus, then this option may suit you better.

Upon notification (notification form) Digital Education will supply your department with PDF copies of the students’ work, uploaded to a OneDrive account set up by your department.

For this to happen, Exam Liaison Officers / Teaching Administrators will need to set up a OneDrive folder and notify Digital Education that they wish to have PDF copies of the files. We have a video tutorial (with captions) on this process below.

You can then use tools you already have or prefer to use to do your marking. There is more flexibility here, and we will not be able to advise and support every PDF tool available or give precise instructions for every workflow used by departments, but we give some examples here.

Marking on an iPad using OneDrive

Many staff have reported using an iPad with Apple Pencil or an Android tablet with a stylus to be a very effective marking tool. The Microsoft OneDrive app supports both platforms and provides rapid access to scripts and some annotation tools as shown in the video below (which also has captions). The OneDrive app is free, and connects to your UCL OneDrive account via Single Sign On.

There’s further guidance from Microsoft on each individual annotation tool.

The Apple Files app can also connect to OneDrive and has a similar (and perhaps more powerful) annotation tool. Thanks to David Bowler for mentioning this in the first comment on this blog post.

Marking on a Mac using Preview

Preview on a Mac is often taken for granted but is actually quite a sophisticated tool and includes some basic annotation functions. Here is some guidance from Apple on using it.

Marking on a PC or Surface Pro using Microsoft Drawboard PDF

Microsoft Drawboard PDF is a very comprehensive annotation tool, but is only available for Windows 10 and is really designed to be used with a Surface Pro or a desktop with a graphics tablet. Dewi Lewis from UCL Chemistry has produced a video illustrating the annotation tools available and how to mark a set of files easily. UCL does not have a site-wide licence for Drawboard PDF, but it is available at a very modest price if departments choose to buy it.

Marking on a PC, Mac or Linux machine using a PDF annotation program.

Of course there are plenty of third party tools that support annotating PDF documents. Some requirement payment to access the annotation facilities (or to save files that have been annotated) but two that do not are Xodo and Foxit PDF.

Things to think about with this approach:

• Your marking process: if you use double blind marking you might need to make two copies of the files, one for each marker. If you use check marking then a single copy will suffice.
• You will need to ensure the files are stored securely and can be accessed by the relevant departmental staff in case of any query. You might share the exam submission files with key contacts such as teaching administrators or directors of teaching.
• Some of the products listed above have a small charge, as would any stylus or pencil that staff would need. These cannot be supplied centrally, so you may need a process for staff claiming back the costs from departments.

---

Using a ‘marker file’

Accessing the students’ scripts is done using Turnitin in Moodle, which allows all the papers to be viewed online individually or downloaded in one go. Then a separate document is kept (either one per script, or one overall) containing the marks and marker feedback for each comment. If double-blind marking is being used, then it is easy to see that two such documents or sets of documents could be kept in this way.

---

Printing scripts and marking on paper

Although we have moved to online submission this year, colleagues are still welcome to print documents and mark on paper. However there is no central printing service available for completed scripts to be printed, and this would have to be managed individually or locally by departments.

---

The evidence about marking online

In this video Dr Mary Richardson, Associate Professor in Educational Assessment at the IOE, gives a guide to how online marking can differ from paper-based marking and offers some tips for those new to online marking. The video has captions.

---

Training sessions and support

Digital Education will be running daily training sessions for teachers covering the ground in this blog post. These will run at 12-1pm every weekday from Tuesday 12 May.

• Marking 24 hour exams: options and guidance – daily from Tuesday 12 May 2020, 12pm – 1pm
• To join a session, navigate to the Teaching and Assessment Continuity – Staff Development Sessions Moodle course, self-enrol, and select the appropriate session.

No booking necessary.

We are also providing additional support for students during the exam period. Our support hours will be (UK time):

• Monday 11.30am-8.30pm
• Tuesday-Thursday 8am-8pm
• Friday: 7.30am-3.30pm

Details of support mechanisms are given in the exam section on each Moodle module where an exam is taking place.

Updated 21 October 2020 to include graphics tablets and webcam options
Updated 5 January 2022 with minor updates to links

We start this blog post with a short exercise for the reader: search Google Images or an image library for the word teacher.

I hope that you see a range of diverse people in the pictures. But I bet that most of them will have a pen or chalk in their hand, and a board or surface at their side. The act of writing and talking as a form of instruction is perhaps the most commonly perceived characteristic of who a teacher is and what they do.

The move to remote teaching has made writing as a form of teaching more difficult. A mouse may be great for controlling a computer, but it’s hard to use it to write smoothly on screen.

So in this post, we will look at how we might continue to use writing as a form of teaching, even from home.

Before we start, a note on accessibility. Whilst we teach remotely, so of the facilities that our students might have relied upon (like captions, transcripts, note-takers or signers) might not be available, so we may need to make alternative accommodations. Please do think about this before you plan your teaching and contact Digital Education if you would like advice.

Using a visualiser at home

In the classroom or lecture hall, many rooms will have a visualiser (also called a document camera) for showing writing and objects. These are typically hooked up to a projection screen through the AV system and are also captured by lecture capture. UCL uses a mix of table-top visualiser and ceiling mounted visualisers from a company called Wolfvision.

You can also buy more consumer-level products in this category. At home I use an IPEZO VZ-R camera, which costs about £200.

The camera connects via USB and can be used in two ways:

• Directly as a webcam, either replacing your main video feed or in Zoom you can share a second camera in the same way you might share your screen
• Using the IPEVO Visualizer software which is available as a free download for multiple platforms, and then sharing this application

Either method works well, but I’d recommend the latter as webcam video is often heavily compressed. This method works well with all our supported platforms, including Lecturecast (Echo 360) Universal Capture Personal, Zoom, Microsoft Teams and Blackboard Collaborate.

The camera is compact but sturdy. It has an in-built light, and facilities such as zoom and autofocus, and can capture a surface slightly less than A3. It also has an inbuilt microphone which is likely to be closer to you when you are speaking than one in a laptop.

The software can control the features of the camera, and can make recordings that you could upload to Lecturecast, for example. The controls neatly get out of the way when they aren’t being used, so that you can focus on the content. The image below shows the auto-focus in operation, clearly capturing a book cover.

The Visualizer application can easily be shared in a Blackboard Collaborate room, and you can then move the window out of the way and focus on the participants. The image below shows me writing some simple maths on a notepad, captured by the camera, using the screen sharing feature of Collaborate to show this to other participants.

IPEVO offer another model, the V4K, for £100, which has a lower resolution sensor (but still good) and no light. Of course other companies are available, and I’ve heard recommendations of Genee and Aver models. These types of visualisers are often used in secondary schools, so any teachers you know may be able to give suitable advice.

One thing I particularly like about the IPEVO models is the design. The one I have folds fairly flat, so is easy to store away, so my living room doesn’t have to look like a lecture theatre. Here’s a comparison of its size when folded compared to a bottle of wine. And yes, it does fit in a wine rack.

---

Writing on an iPad or other tablet

Using an Apple Pencil on an iPad or a stylus for other devices, you can use a notes or sketch app to write on an iPad. This works fine as a standalone process, but you can also use software such as Reflector to mirror your device to your PC or Mac and screen share it as part of a call using Blackboard Collaborate, Zoom etc., or to record it from your screen using Lecturecast Personal Capture. If you have a Mac, you can also plug in your USB to Lightning cable and use Quicktime to mirror your screen by selecting your iPad as the camera source (thanks to Dr Steven Schofield from UCL Physics and Astronomy for that tip).

The short clip below shows Steven leading a revision lecture with his students. He reports that “I gave a two hour live revision lecture today; I was able to use the screen mirroring software to share my iPad Mini screen with the class via Blackboard Collaborate. It went really well and I had good feedback from the students. I think the overall experience is so much better than just talking to prepared slides. ”

---

Using a webcam

If you have a webcam that isn’t built in to your computer, you can use this too. We’ve tried a couple of webcams and it can work well, but may not have the full controls or handy features like the light that the VZ-R has.

If you have a laptop webcam above the screen, there’s  further option using a mirror to give you a whiteboard surface. Again IPEVO has an option for this, or you can look at making your own using a 3D printer or just cardboard cutouts. John Umekubo has blogged about this, and John Mitchell from UCL has put it into practice. The much lower cost means it could be a great option for students to show their workings.

---

Using a graphics tablet

Graphics tablets are often associated with drawing and more artistic work, but can also be used for writing. It can be less natural as your writing is preserved only on screen rather than on the tablet itself, a difference between writing on paper. Wacom are the brand leader with many higher end products for artists, but for simple writing tasks more modest brands are also suitable. We’ve had good reports of the XP-Pen and Viekk brands. Feedback suggests that the size of the graphics tablet is more important than features like number of levels of pressure sensitivity if you are using it predominantly for writing rather than art.

---

Using a mobile phone or tablet

You may have seen tweets where inventive teachers have rigged up their phones to use as cameras. But how do you do this?

Well, IPEVO provide an app for that too. To make this work, you need three things:

• An Apple or Android phone on the same network as your computer (PC, Mac, Chromebook and Linux supported)
• An installation of IPEVO iDocCam on your phone
• An installation of IPEVO Visualizer on your computer

Launch the app on your phone, and then the Visualizer software on your computer, and your phone will appear in the list of available cameras.

And here is an example of it in use, with my old IPhone SE acting as the camera:

The final bit is to rig the phone up in a secure way, about 40cm above your desk. You might use a desk mount, or a pile of books, or some blu tack and string. How  you do this is up to you, but this tweet from UCL’s Matt Whyndham might give you some inspiration. Happy writing!

Earlier this year, I celebrated a decade at UCL.

Just as I joined, there was another new recruit to UCL – Moodle. This open source virtual learning environment had recently replaced WebCT which we used before, and my initial task was to support the migration of 300 courses from one to the other. Since then we’ve regularly upgraded Moodle and added new facilities such as Turnitin, Lecturecast and Blackboard Collaborate into it. It now has 7000 live courses and is used by nearly every teacher and student at UCL.

We also have other services – from voting handsets to portfolios. And we also know that there’s lots of other web-based services that people use.

Time flies by, and after ten years we think it’s right to ask if this environment is right for us? Does it need to change? Are we making the most of what we’ve got? Is there something better we should be doing instead?

To help us answer these, Digital Education has been listening and learning. We’ve started the process by conducting detailed interviews with 10 staff and 13 students about how they teach and learn. These have raised issues from our spaces and technologies, to our culture and organisation. It’s a rich source of viewpoints, and reflects the diversity and breadth of UCL’s education and people.

We’d like to share some of these findings with you, and give you an opportunity to contribute and prioritise our future developments. To do this, we have arranged four Town Hall meetings:

• Wednesday 27 November 3-4pm, Cruciform LT2;
• Tuesday 3 December, 10-11am, Cruciform LT1;
• Monday 16 December, 12-1pm, Medical Sciences AV Hill Lecture Theatre;
• Wednesday 8 January 2020, 3-4pm, Cruciform LT2.

There’s no need to book – just turn up to any that you wish to attend. The events are aimed at teaching staff but students and other staff are welcome too.

Any questions, please contact Steve Rowett in Digital Education.

The Open University was the venue for the 8th Jisc Learning Analytics Network. I’d not been there before. It was slightly eerie to see what was a clearly reconigsable university campus but without the exciting if slightly claustrophic atmosphere that thousands of students provide. I won’t report on everything, but will give some highlights most relevant to me. There’s more from Niall Sclater on the Jisc Learning Analytics blog.

The day kicked off with Paul Bailey and Michael Webb giving an update on Jisc’s progress. Referring back to their earlier aims they commented that things were going pretty much to plan, but the term ‘learner monitoring’ has thankfully been discarded. Their early work on legal and ethical issues set the tone carefully and has been a solid base.

Perhaps more clearly than I’ve seen before, Jisc have set their goal as nothing less than sector transformation. By collecting and analysing data across the sector they believe they can gain insights that no one institution could alone. Jisc will provide the central infrastructure including a powerful learning records warehouse, along with some standardised data transformation tools, to provide basic predictive and alerts functionality. They will also manage a procurement framework for insitutions who want more sophistication.

The learning records warehouse is a biggie here – currently with 12 institutions on board and around 200 million lines of activity. Both Moodle and Blackboard have plug-ins to feed live data in, and code for mainpulating historic data into the right formats for it.

Paul and Michael launched a new on-boarding guide for institutions at https://analytics.jiscinvolve.org/wp/on-boarding – A 20 step checklist to getting ready for learning analytics. Step 1 is pretty easy though, so anyone can get started!

Bart Rientes from the Open University showed again how important learning analytics is to them and how powerfully they can use it. Mapping all of the activities students undertake into seven different categories (assimilative, finding and handling information, communication, productive, experiential, interactive/adaptive, assessment) gives dashboards allowing course designers to visualise their courses. Add in opportunities for workshops and discussion and you have a great way of encouraging thinking about course design.

Interestingly, Bart reported that there was no correlation between retentition and satisfaction. Happy students fail and unhappy students pass, and vice versa. Which begs the question – do we design courses for maximum retention, or for maximum satisfaction, because we can’t have both!

Andrew Cormack, Chief Regulatory Advisor at Jisc, gave an update on legal horizons. The new General Data Protection Regulations is already on the statute books in the UK but comes into force on 1 May 2018. For a complex issue, his presentation was wonderfully straightforward. I shall try to explain more, but you can read Andrew’s own version at http://www.learning-analytics.info/journals/index.php/JLA/article/view/4554   [I am not a lawyer, so please do your own due diligence].

Much of the change in this new legislation involves the role of consent, which is downplayed somewhat in favour of accountability. This gives logic thus:

• We have a VLE that collects lots of data for its primary purpose – providing staff and students with teaching and learning activities.
• We have a secondary purpose for this data which is improving our education design, helping and supporting learners and we make these explicit upfront. We might also say any things that we won’t do, such as selling the data to third parties.
• We must balance any legitimate interest they have in using the data collected, against any risks of using the data that the data subject might face. But note that this risk does not need to be zero in order for us to go ahead.
• Andrew distinguished between Improvements (that which is general and impersonal, e.g. the way a course is designed or when we schedule classes) and Interventions (which go to an individual student to suggest a change in behaviour). The latter needs informal consent, the former can be based on legitimate interest. He also suggested that consent is better asked later in the day, when you know the precise purpose for the consent.
• So for example in a learning analytics project, we might only obtain consent at the first point where we intervene with a given student. This might be an email which invites them to discuss their progress with the institution, and the act of the doing so gives consent at the same time.

You can follow Andrew as @Janet_LegReg if you want to keep up with the latest info.

Thanks to Jisc for another really good event, and apologies to those I haven’t written about – there was a lot to take in!

At UCL we’ve been pondering what a future learning environment might look like now for about two years. And we are starting to reach some conclusions.

Our analysis of our VLE – and pretty much all of them out there – is that it suffers from two fundamental limitations.

Silos – staff and students see the courses they are enrolled for, and generally can’t look over the fence to see something else. In real life, if a student asked to attend lectures for a course they weren’t registered for, we’d welcome their interest, their breadth, their love of learning. In the VLE we tell them that this is impossible. The VLE limits a student’s education to just what they have paid for, just what they deserve, and just what they need to know. All curiosity is lost.

Control – the teacher sets things up and students do them. No questions asked or even allowed. Forums lay devoid of posts for fear of asking ‘dumb’ questions, or fear of making mistakes. Assignments are submitted with perfunctory duty with the best that a student can hope for getting a green pass on Turnitin and some feedback some weeks later which is ignored anyway as the triumph or the disappointment of the grade awarded is processed. All love of learning is lost.

So we’re looking for something different.

And our inspiration came from an interest place – Brockenhurst College in Hampshire. Now they have a very rural catchment area – some students travel over from the Isle of Wight to attend classes. So of course, they don’t travel if they don’t have classes and therefore feel disconnected from the university.

We realised that part of the challenge at UCL is the same. The distances may be much smaller, but when you are travelling from home or commuting on the Central Line means that the disconnection is just as real.

So we need an environment that promotes connections. It just so happens we also have the Connected Curriculum initiative which will encourage interdisciplinary research-based education, where students do real, authentic work, not just essays for a teacher to mark. Where group work is the norm, not the exception. Where students are not passive recipients, but actively engaged in enquiry.

So it’s all coming together. What we want for UCL is an Academic Social Network.

What do I mean by that? Let’s take each word at a time.

First, it’s Academic. That means it is designed for education. There are plenty of social networks around – Facebook, LinkedIn and Yammer spring to mind – but they are designed for different things, typically business. Whether it means allowing people to ask questions anonymously, embedding LaTex in messages so mathematicians can speak in their own language, or structuring data to be able to find final-year projects, the platform needs to speak to teachers and students as being something for them. It’s about work, but also all of the other things that happen at university; social clubs, sports, societies, volunteering. It’s a safe and trusted place to be because the user trusts the university and knows they are not the product to be sold and re-sold to the highest bidder.

It’s Social. Because learning is social. I don’t just mean group work, but the full gamut of human social interaction. If you talk to students in our learning spaces, they are often working ‘alone, together’; that is they are doing individual tasks but just looking after each other. A student who is tired will be offered a coffee; someone will look after your laptop while you go to the toilet. Students are friends with each other on Facebook, but having staff friends is just ‘weird’. We want a space without complex meanings or difficult relationships but where everyone can connect with each other as part of the university community.

Finally it’s a Network. Universities are big places, and UCL is bigger than most. Networks are a place where you can meet like-minded folk, but also get exposure and understanding of those who study different things, think in different ways, have different approaches to the same challenge. That network extends beyond current staff and students to pre-entry students, alumni, industry and charity partners – all of those that have a stake in the vibrancy and excitement of what a university can be.

So what are we going to do?

We’re going to get one.

That’s quite a lot of work, as we have to do a lot of procurement activities to get what we want.

But for now, we have students and teachers on the ground talking to peers, understanding needs, working our what it means to be part of the UCL community.

We’ve done a lot of thinking, some talking and even more listening. It’s an experiment. We don’t know if it will work. Even if it does, it will probably take many years.

We characterise what we want as follows:

 

It’s our shot at what a Next Generation Digital Learning Environment will look like.

Many thanks to Eileen Kennedy for her work in developing and evaluating these ideas within UCL Digital Education.

The thing we love most in Digital Education is working with our students, hearing their ideas and seeing what they can achieve.

The UCLU Technology Society recently approached us about an API for accessing information at UCL – what is available now and what more could be made available in the future. It’s a difficult question, as we don’t own most of the data we process and don’t have the right to just make it available. But with the UK topping the Open Data Barometer, it’s such to be a question we will have to face.

The leaders in this field in the UK are Southampton and Oxford. Four members of TechSoc and myself recently visited Ash Smith and Chris Gutteridge at Southampton, who have done tons of work in opening up university data – everything from buildings to catering.

The TechSoc have written up the visit as a blog post, so I’ll do no more other than link to their report of the visit.