Aloha ELESIG London
By Mira Vogel, on 31 March 2015
A summary of the first meeting of the London regional group of the Evaluation of Learners’ Experiences of E-Learning national special interest group a.k.a. ELESIG (and breathe). It took place on Tuesday 24th March, 11.00am-1.00pm, at Birkbeck University of London. The talks weren’t recorded but you can find slides on the ELESIG London Group discussion forum.
Eileen Kennedy presented a case study on the UCL Institute of Education’s ‘What future for education’ Mooc. The Mooc had a repeating weekly structure of reflection task, a recorded interview, open access readings, posting to a Padlet wall on a theme (‘Where do you learn?’ for example), a Google Hangout, and a review & reflection (the latter was a main way for the Mooc team to gather feedback). Eileen’s study of the learner experience aimed to find out whether the design of the Mooc could enable a dialogic educational experience, at scale, and whether the learning led students to interrogate their prior assumptions. The end-of-Mooc survey yielded some appreciation for most of the elements of the Mooc, but the real-time hangouts were hard to join. Respondents wanted external validation of their learning in the form of a statement of accomplishment and a peer grading system they were confident was rigorous. To supplement this survey data, the evaluation team mapped their findings to Laurillard’s conversational framework, matrix of elements including what the learners did, justification for including this type of element in this situation, the specific role of the element in the Mooc, and the evidence collected or needed. We discussed ways to make the rationale of the course design more explicit to students to help them identify hinge points in their learning. The yearning for attention and recognition raised the matter of the relationship between Mooc providers and learners, and the role of caring. We noted that the Mooc is destined to be packaged up as an on-demand Mooc, which seems to be part of a global trend in response to lack of resource to run it.
Ghazaleh Cousin presented on an evaluation of the Panopto lecture capture service at Imperial. Beyond the basic Panopto reports about who accessed which recording and for how long, questions include whether viewing is associated with differences in students’ results, which sessions are most popular, and which days are most popular. Since Panopto’s data is currently quite limited, Imperial are contributing feature requests. We discussed whether students who perform better are watching the videos more. To address this, video could be made which discouraged students from fixating on memorising explanations. We touched only briefly on methods – the team did not have immediate opportunities to arrange questionnaires and interviews, and opted to make sense of the Panopto data as a way to generate deeper questions. At the more challenging methodological end, there was interest in comparing learning from lecture recordings to learning from lecture graphics or lecture pedagogies.
Damien Darcy presented on uses of video at Birkbeck. Before Birkbeck’s Panopto roll-out, use of video at Birkbeck was sporadic, professional or slightly Blair Witchy, and it wasn’t clear how to record a lecture. Video was treated in a technocentric way isolated from educational concerns of assessment or student engagement. Damien carried out an exploratory study with the Law department, as large scale Panopto users, with a methodology he referred to as ‘guerilla ethnography’. His questions were: was it working, was it used (properly) by staff, how were students using it? He confirmed that decontextualised training doesn’t carry across to the rigours of the lecture hall, and superstitions about how technologies work persist. He related a sense of control, pride and ownership to increasing proficiency. Panopto data showed that peak viewing was often immediately after the lecture, and there were signs that if the lecture wasn’t up quickly it wouldn’t get watched. Watching was often social, often while doing other things, and was predictably uneven with spikes at particular points and particular times related to assessment. As video was normalised student expectations became more exacting, with requests for consistent tagging and titles and the inclusion of an overview. To contain their video initiative, Organisational Psychology had initiated a dialogue with students about what to record – i.e. not everything – and what to leave as ephemeral. Damien’s next steps would be to find out more about student reactions and perceptions, lecturer motivations, and how the identity of the lecture is changing. Methods would include surveys, focus groups, and a range of ethnographic studies looking at changes to the identity of lecture and lecturer. Questions would be informed by Panopto data.
We then discussed next steps for ELESIG London – in no particular order:
- Case-making for resourcing evaluation activities.
- Understanding and negotiating institutional barriers to evaluation.
- How to take the findings from an evaluation and create narratives of impact.
- Micro-evaluation possibilities: what kinds of evaluation can you do if you have only been given ten minutes? One day? Ten days? As you go along?
- Methods masterclasses including ethnography and data wrangling
- Can learning experiences be designed so it becomes possible to relate a change the evaluation identifies in students to a specific aspect of course design or learning?
- Incorporating evaluation into developing new programmes.
- Should the group have outputs?
- Can we improve the generalisability of findings by coordinating our evaluation activities across institutions?
- Not encroaching on other London e-learning groups such as the M25LTG – keeping focus on evaluation (e.g. methods, data, analysis, interpretation, politics and strategic importance).
- Twitter rota for the national ELESIG account by region rather than by individual.
The coordinators (Leo Havemann and Mira Vogel) will be incorporating these ideas into plans for the next meeting in summer.
If you are interested in attending or keeping up with ELESIG London goings-on or you’d like to contact a coordinator, then join the London Group on Ning.