Over breakfast at a recent conference on the use of the Echo360 (Lecturecast) system, I found myself talking to an LTA (Learning Technology Advisor) from a small US community college. He had recently been working with teachers from the Math School at the college, helping them transform their existing paper-based courses for online delivery.
One of the last, and most reluctant, members of staff to go online was a senior member of the school’s teaching staff, who met with the LTA regularly to discuss ideas for the new course. As the course’s live date approached, the LTA suggested that an online discussion forum be included; a place where students could share ideas, or give feedback about the course – the LTA also advised that it was good practice to prime a forum with one or more initial posts to ‘get the ball rolling’. The Maths teacher doubted the value of ‘this kind of thing’ but said that he’d think about it.
The new term began, the course was made live but it was a couple of weeks before the LTA and the Maths teacher had a chance to meet and review how things were going. When they did finally meet the LTA was pleased to hear that the course had been well received and asked his colleague what he had found most useful.
The Maths teacher said that he had taken up the suggestion of including a discussion forum and to get the ball rolling had posted the question for all students – ‘What does Maths mean to you in your life?’. This was a question. that over his years of teaching, he always asked every group of students at their first lecture – observing sadly that he rarely got much of a response.
The teacher said that asking the same question in an online forum had made a big difference, the LTA told me that there were tears in his colleagues eyes as he talked about the many messages in the forum and how a number of students had talked about the beauty and elegance of mathematics, describing a passion for the subject that matched his own – he said the replies had inspired him and that his teaching with this group had an energy and enthusiasm he hadn’t felt for years.

The Lecturecast conference covered many interesting uses of this very impressive technology, but a few months later, trying to think of subject for this blog, it’s the story of the Maths teacher and his students that sticks in my mind and how the use of a much simpler technology gave them a different way to connect.

CALT organised a stimulating presentation by Prof Geoffrey Crisp of the University of Adelaide about assessment in the Web 2.0 world. Much information at http://www.transformingassessment.com and a similar presentation is on slideshare.

Crisp calls for much more ‘authentic’ learning and assessment – the need to set big questions; for instance in aeronautical engineering we should set students a task to build a rocket in 3 years. This allows them to see reasons for the smaller things. The tendency with conventional assessment is for everything to become very granular – little learning outcomes are assessed with discrete assessment tasks which don’t encourage students to make connections, and which encourage surface and strategic rather than deep approaches to learning.

Of course moving away from more traditional forms of assessment entails proving that the alternative works – traditional approaches are very deeply engrained in the culture of institutions and are not easily challenged. Crisp acknowledged that even in his own institution there is some way to go.

Three points to start with:

1.    Assessment tasks should be worth doing – if students can get answers by copying from web, or asking google, or guessing, then the task is not worth doing. We need to stop setting tasks which are about information since information is everywhere.

2.    We should separate out diagnostic assessment from formative assessment. Diagnostic assessment is essential before teaching and can be an excellent way of starting relationship with students at the outset. The teacher can then build their teaching on students’ current level of understanding.

3.    Think about assessment tasks which result in divergent rather than convergent responses.  In the traditional approach we tend to seek convergent responses in which all students are expected to come up with same answer but divergent responses are more authentic.  Peer- and self-review approaches can support this approach.

Bearing this in mind, and drawing on the work by Bobby Elliot (see http://www.scribd.com/doc/461041/Assessment-20), we heard that:

• Assessment 1.0 is traditional assessment – paper-based, classroom-based, synchronous in time and space, formalised and controlled.
• Assessment 1.5 is basic computer assisted assessment – using quizzes which tend to replicate the paper-based experience, and portfolios used mainly as storage for students’ work. Tasks tend to be done alone -competition is encouraged and collaboration is cheating.  They tend to encourage focus on passing the test rather than on gaining knowledge, skills and understanding and don’t lead to deeper levels of learning (indeed Elliot argues that factual knowledge is valueless in the era of Wikipedia and Google.)
• Assessment 2.0 is tool-assisted assessment in which students do things using a variety of tools and resources and then simply use the VLE (typically) to submit the results. This kind of assessment is typically authentic, personalised, negotiated, engaging, recognises existing skills, researched, assesses deeper levels of learning, problem oriented, collaborative, done anywhere peer- and self-assessed, and supported by IT tools especially the open web.

Some nice examples of interactive e-assessment 2.0 design included:

• Examine QuickTime VR image of a geological formation then answer questions based on that – drawing on things wouldn’t be able to see from static image.
• Examine panograph (scrolling and zoomable image) of Bayeux Tapestry and answer questions drawing together different parts – students selecting evidence from different segments of the tapestry.
• Interactive spreadsheets – Excel with macros.  Students can change certain bits and answer questions on resulting trends in graphs. Can have nested response questions so that the answer to the second is based on first. (But there is a need for care with dependences so that a wrong move early on doesn’t lead to total failure).
• Chemical structures using the Molinspiration tool. Students can draw molecular structures using the tool and copy and paste the resulting text string into answer which is held in the VLE quiz tool.
• Problem solving using a tool called IMMEX (‘It Makes You Think’) which tracks how students approach problems.  The tutor adds in real, redundant and false information that the students can draw on to solve the problem.  They can use it all but the more failed attempts they make the fewer marks they get. We saw an archaeology example in which students had to date an artefact.
• Role plays which can be done using regular VLE features such as announcements, discussion forums, wikis.  Students adopt different personas and enter into discussion and debate through those personas.
• Scenario based learning – this is more prescriptive than role play. The recommended tool is Pblinteractive.com
• Simulations – the Bized.co.uk site offers a virtual bank and factory. Students can work within bized then answer questions in the VLE.
• Second Life (virtual world) assessment in which the avatar answers questions which go back into Moodle.

Examples of these and more are available through the http://www.transformingassessment.com/ site – it’s Moodle-based and anyone with a .ac.uk email address can self-register and try out the various tasks. (They also run a series of webinars.)

Crisp argues convincingly for much more authentic and immersive assessment, and for assessments in which  process as well as outcome is evaluated – for example approaches to problem solving;  efficiency; ethical considerations; involvement of others.

A good closing question was whether teachers will be able to construct future assessments or will this be a specialist activity. Is it all going to get too hard for people? There may be a need for more team based approaches in future.

Useful resources

Boud, D., 2009, Assessment 2020 – Seven propositions for assessment reform in higher education, Available at: http://www.iml.uts.edu.au/assessment-futures/Assessment-2020_propositions_final.pdf

Crisp, G., 2007, The e-Assessment Handbook. Continuum International Publishing Group Ltd

Crisp, G., 2009, Designing and using e-Assessments. HERDSA Guide, Higher Education Research Society of Australasia

Elliott, B., 2008. Assessment 2.0 – Modernising assessment in the age of Web 2.0. Available at: http://www.scribd.com/doc/461041/Assessment-20.

This year’s ALT-C (Association for Learning Technology) conference opened with a bang – echoes of which continued to resound for the rest of the week. The source was an entertaining and provocative talk by Donald Clark about why the lecture is not fit for purpose (prefaced by an apology for delivering a lecture about why we should not deliver lectures). Clark noted that the word lecture originally meant ‘sermon’ and argued that although some teachers claim interactivity in their lectures often this is an illusion with questions tending to be rhetorical. There is a pretence that critical thinking is being taught; the lecture was never intended to promote critical thinking – it evolved from preaching where the audience was not expected to question material. Clark argues that if you’re going to lecture then make sure you do it well and think about going ‘stadium style’ to huge numbers – using technology to broadcast it if you want.

Many of Clark’s examples come from physics – a subject he is fascinated by but was unable to engage with as a student. It is difficult to understand and to teach, and so was a good basis for his talk. He recounted how he had embarked on a degree in physics but gave up after a year as he had failed to learn from the lectures he was given – typically a lecturer would walk into the auditorium and write long series’ of equations on panels of blackboards without any real attempt at explanation or engagement. Often all would be done without even facing the students. Clark told how even Isaac Newton was a dreadful teacher – no one would turn up; in contrast Richard Feynman was a superb teacher who cared deeply about teaching; Feynman even published his lectures but was critical of lecture as method. Eric Mazur noticed that lecturers revert to anecdote when talking about teaching (great quote – ‘data is not the plural of anecdote’). Academics can be disparaging about undergraduates; they are typically very far ahead of the undergraduate level – especially in physics – that they can’t easily get back to that level. When thinking about and discussing teaching they typically do not use scientific method and persist in using ineffective approaches. Mazur developed a revised approach involving pre-reading, getting students to take notes, seating weaker students at front and smarter ones at the four corners – this was demonstrated to bring up overall standards.

The Institute of Theoretical Physics in Trieste recognized the issue of introverted lecturers and adopted a policy of recording all lectures to ensure that students had a second chance to watch. The recording was initially taking photos every few minutes and recording audio throughout. This approach was judged to be a big improvement.

Clark quoted MIT’s Walter Lewin who has produced some fantastic videos for teaching ‘it’s better to see a first class lecture on video than a mediocre one in the flesh’ – for any teacher a drop off in attendance over the term should ring alarm bells. He also quoted from Donald Bligh’s book ‘ ‘What’s the use of lectures?’ which cites ten pedagogical problems with the lecture:

1. They tend to be 1 hour.
2. There is the tyranny of time (ie specific time and place) – why not use YouTube, OCW etc.
3. And the tyranny of location.
4. It can be difficult to maintain psychological attention – boredom sets in…
5. Cognitive overload – too much info too quickly – teachers should simplify materials and pare back radically.
6. Episodic and semantic memory – lectures shove semantic stuff at you which is unlikely to stick unless something in there is really striking or memorable; teachers need to learn how to use media.
7. Learning by doing – doesn’t happen in a lecture.
8. Spaced practice – Ebbingham demonstrated that we need repeated practice to learn – how does this relate to lecture-based learning?
9. Not collaborative – people don’t want to sit next to each other – as evidenced by the typical practice of leaving an empty space between you and your neighbour – let alone talk to each other in this kind of environment.
10. Personality problems – we should not don’t assume every teacher has go be a researcher and vice versa; some academics are simply not cut out for teaching.

Clark’s final words of advice is to aim for transformation of your approach – redesign the whole course, don’t just bolt on technology. Oh and abolish ‘lecturer’ as a job title.

http://donaldclarkplanb.blogspot.com
Twitter – @donaldclark

Postscript

The talk was entertaining, lively, thought provoking, and indeed provoked a number of the audience quite effectively – the Twitter backchannel was buzzing with protests that Clark was being condescending to academics, and that whilst some of his criticisms were valid he failed to offer any clear solutions. Some of the tweeting was pretty abrasive and a couple of people managed to put together fairly substantial blog posts outlining their criticisms within hours of the talk.  It did seem – as someone put it – that the wisdom of the crowd had turned into the baying of the twittering mob… Clark himself has posted a blog posting about being tweckled in this way – https://www.blogger.com/comment.g?blogID=21077063&postID=2355649275760409611 (and the comments, and his responses…).