Rationale

One may argue that UCL students are over-assessed. But, regardless of your take on this, it stands to reason that where examinations are used to assess learning, it is worth investing in delivery systems that give students the best possible experience and outcomes. These systems could conceivably have the following aims, to:

1. Improve the functionality, usability and accessibility of the questions on the exam
2. Offer greater flexibility in terms of the time, place and resources needed to host an exam
3. Improve the support given to both students sitting exams and the staff facilitating and/or grading them
4. Potentially reduce the time between sitting some exams and obtaining results/feedback
5. Potentially increase transparency in marking and confidence in the grades awarded
6. Reduce potential for errors in marking and from transcribing/transferring marks
7. Reduce the workloads of both academic and administrative staff
8. Reduce costs around printing, transporting, securing and storing paper

Acknowledging that assessment and feedback are ongoing areas of student concern, and that the Education Strategy commits us to addressing and resolving persistent challenges in this area (https://www.ucl.ac.uk/teaching-learning/education-strategy), Brunel University’s approach to digitising some examinations is interesting.

E-Assessment Management at Brunel

Whereas UCL uses the quiz tool in Moodle for online exams, Brunel University has subscribed to a cloud-based digital exam platform called WISEflow for theirs. Like Turnitin this subscribed service can be linked to various VLEs including Moodle. In the case of Brunel, this happens to be Blackboard. Tutors create their quizzes/exams, which can be accessed indirectly via a link on a course page in their VLE or directly by logging into the secure WISEflow website.

WISEflow provides tutors with 50 varied question types with quite sophisticated features allowing for different media to be embedded and for example, virtual tools, such as a ruler and a protractor to be used to measure the lines and angles of diagrams in questions. It also supports essay questions that allow a student to respond with a 1,000 to 3,000 word essay. This advantageous for exams in subjects such as law, history and literature, etc. WISEflow also has a number of features to ensure that student’s responses aren’t lost whilst being written.

According to Niels Berg Conradsen of UNIwise,

“WISEflow features a very robust lockdown browser module. It carries a text editor in the lockdown browser that also serves as a word processor. The students can structure their essay with headlines and even embed pictures from their webcam. Moreover, the student has a revision tool at their disposal, so they can revert the essay to former editions if they so desire”.

This platform is sufficiently secure to support bring-your-own-device (BYOD) examinations. At a 1-day event at Brunel on March 17, 2017 attendees, (myself included), had an opportunity to hear positive feedback from Brunel’s staff and students about their experience using WISEflow, and to use it ourselves on our own devices.

WISEflow enables tutors to create and manage digital exams, which they can assign to specific students. They can then mark the materials that the students submit in WISEflow. The marking tools are similar to those in the Moodle and Turnitin assignments. WISEflow can also be used for the submission of coursework.

Benefits of digital exams generally for various stakeholders are listed below and may be also achieved with Moodle quizzes, (although in Moodle’s case additional software would be needed to lock the browser).

Advantages for students:

• It’s feasible for students to take exams remotely
• No need to write long hand for hours (increasingly students may have little sustained handwriting practice)
• It’s much easier to correct mistakes and make changes to responses on online exam scripts than on paper
• It’s feasible for students to get feedback beyond just a grade on exams taken
• Students may be able to get their results faster
• It’s feasible to take the exam on a familiar device, their own laptop
• Accessibility features easier to incorporate

Advantages for tutors, administrators, invigilators:

• Exam papers don’t have to be printed, packaged, secured and distributed to exam centres
• No challenging handwriting to decipher
• No paper scripts to collect, secure or transport
• No scripts to store for the short to long term
• Scripts can be printed as required
• Tools exist for invigilators to monitor students’ progress through a paper they’re sitting
• Tools exist for tutors to analyse students’ responses to each question
• Easy to facilitate access to scripts by moderators, second markers and external examiners
• The cost of a streamlined infrastructure for online exams may not exceed that of the current paper based examination system in place.

Importantly, Brunel also benefited from the involvement and support of their Registry and Examinations Office in the launch of this project.

What WISEflow can do for you…

There are other Providers of Digital Examination Systems that may also be considered.

Some Cons of Acquiring Another App

This piece has highlighted many of the pros of acquiring specialist, purpose-built kit. However, some of the problems that would need to be addressed include:

• Providing students with a “one-stop shop” for accessing their results and feedback. UCL has part sponsored the development of the MyFeedback tool in Moodle for this purpose but could it retrieve this data from a third party product?
• Additional training and support needed
• The integration of a new app with Moodle
• Identifying and equipping enough rooms with adequate WiFi and power points to support BYOD
• Adequate provision of suitable computer suites
• Cost

Cost Differentials

It would be informative to compare the estimated cost of running paper-based exams with that of online exams. Instinctively, one may focus on the costs associated with the actually staging of an exam. By so doing, the paper option may seem much cheaper, if only because computing paraphernalia isn’t required. However, costs may start to even out when preparatory printing, storing and distribution expenses, (not to mention the man-hours needed), are taken into account. Plus, cost associated with moving, securing and storing hard-copy exam scripts.

Where the computing infrastructure is exists for other purposes and would be present even if no online exams to use it, then one could argue in favour of discounting the cost it may otherwise present.

Observations by Attendees

See what others had to say:

#bruneldigitalexams Tweets

In the first plenary session for the ALT-C conference this year, Eric Mazur from Harvard University spoke about how student’s brain activity slows during lectures. The highlighted area to the immediate left of the circled lecture periods in the graph below shows that student’s brains are more active during sleep than during traditional lectures. Eric argues that analysing classroom data is essential to improving teaching.

Eric Mazur presenting a graph showing the brain activity of students during lectures (circled)

So how do students actually learn?

Information transfer is the easy part. The hard part where students need to understand the concepts is often being left to students to do on their own. Eric Mazur realised that most of his own “ah-hah” moments of understanding came outside of the classroom. He now uses voting handsets to involve students in his lectures. After voting he asks students to find someone who disagrees with their answer and then try to convince their neighbour  why their own answer is correct. His collaborative approach to teaching ensures students stay engaged during lectures.

Women in particular thrive in a collaborative environment as opposed to a competitive one, so they perform better when he involves them in his lectures.  He also encourages students to work together to complete their homework.

Lecture demonstrations are not as effective as students doing the activity themselves because students may make incorrect assumptions about what the demonstrator has done to achieve the results. Asking students to predict the outcome of the demonstration, record their observation of the demo and then discuss whether they correctly predicted the outcome of the demonstration with their peers leads to a better understanding of the core concepts.

The reason for this is that “the brain stores models not facts.” You need to give students time to re-adjust their models in the lecture. Otherwise students are more likely to continue to believe in their incorrect models. This effect is known in psychology as cognitive dissonance. Predicting, explaining and discussing the concepts makes a significant difference in the ability for students to absorb the correct models. The graph below shows a significant improvement in understanding by those students who had predicted the demonstration results and an even higher improvement by those who also discussed their predications after the demonstration.

Eric Mazur showing the improvement in results as students are asked to predict and discuss the results of a demonstration

It’s difficult to teach students who have the wrong model, because teachers who understand the correct model find it difficult to understand where these students are coming from. Asking students to show their working out helps teachers to understand their misconceptions. Instead of just marking incorrect answers as wrong and leaving it at that, Eric Mazur argues that teachers should concentrate on understanding the thinking behind the incorrect answers. That way they can help students to re-adjust their thinking to incorporate the correct models.

Read more: Classroom Demonstrations: Learning Tools or Entertainment?

Eric Mazur also asks students to tell him what they find difficult or confusing from their readings before the lecture. He asked students to provide him with at least 2 concepts they found confusing and also some feedback on why they found the items confusing. If they found nothing difficult they had to provide him with two examples of what they found interesting and why. He then adapts his lecture to address the areas students found most difficult to comprehend.  This method is known as just in time teaching. You can find out more about this method in the book Just in time teaching: blending active learning with web technology (Novak et al., Prentice Hall, 1999).

Eric Mazur’s research shows that confused students are around twice as likely to understand a concept than those who claim they understand it

In Eric’s study, those students who mentioned they were confused by a concept were roughly twice as likely to demonstrate understanding than those who said they understood it, so “confusion doesn’t correlate with misunderstanding.”  He concluded that those students who claim to understand are likely to have passively read the material instead of properly comprehending it. It’s important to ask students to reflect on what they have read. One way to do this is to ask students to write their own analogy for difficult concepts. Eric Mazur says that “confusion is an essential part of the learning process…and should be elicited.”

Read more: Understanding Confusion

More information about Eric Mazur’s research is available from his website: http://mazur.harvard.edu