“In a digital era, how can we design assessments differently and better?” 

This is the question that a group of more than 35 key partners in UCL’s teaching and assessment community gathered to consider last week. Hosted jointly by ARENA and Digital Education, the group comprised academics, students, the Digital Assessment team, Faculty Learning Technologists, professional services staff and representatives from UNIwise (suppliers of our digital assessment platform, AssessmentUCL).  

Attendees were split into teams of mixed disciplines to share their experience of assessment at UCL and bring forward ideas and recommendations on how digital assessments could look in the future. 

The breadth of representation made for a rich and varied discussion and enabled each partner to express their principle areas of focus: 

• From our student representatives we heard a genuine desire both for continued improvement enabled through assessment feedback not just marks and for assessments to test modern-day marketplace skills (e.g. distilling information, writing reports).   
• Our academic representatives expressed their overarching concern to do the right thing by our students: seeking to understand and share methods and tools for designing assessments to help students in future life.
• Our Digital Assessment team focussed on ways to build and share capabilities within faculties enabling us to connect academics, students and technologists and ways to understand and share how technology can support innovative assessment design both now and in the future.  
• Like our students, our Faculty Learning Technologists focussed on the importance of feedback (or “feedforward”) which is easily accessible, timely and meaningful: enabling students to act upon it.  
• Our Professional Services representatives focused on how to connect people to ensure that fantastic work around assessment design is shared widely and to ensure that academics and faculty teams are all aware of the tools and supporting resources available to them. 
• Our UNIwise representatives were interested in all points raised: keen to consider how future enhancements to the AssessmentUCL platform might facilitate the continued evolvement of digital assessment. 

Group discussions were wide ranging and often raised more questions than answers but surfaced a clear desire to continue the conversations about the issues raised and to focus on how we share knowledge to maximise the depth of expertise in assessment design across departments. 

Next steps 

As a result of the discussion, Arena and the Digital Assessment team will focus their attention on the following key themes over the coming months:  

1. Enabling assessment design knowledge sharing: Helping academics and others involved in assessment design to understand what is possible and how to achieve it. Ensuring clear information channels and networks are established to enable academics and others involved in assessment design to share experience and learn from the experience of others as well as raising awareness of existing resources and support.  
2. Continuing to improve the markers journey: Exploring how to enable flexibility in marking where one may allocate all work to all markers and each takes one off the top of the pile and can go to the next unmarked (or not yet started to mark script), to ensure that there is no loss of marks when marking allocations change pre/post marking. 
3. Continuing the conversation: Building on the foundations and links established during this event, we plan to set up a learning lab through which staff and students can continue the discussion around how we can design digital assessments differently and better using existing mechanisms like the Chart tool, as well as rethinking how assessments are delivered currently to make them more applicable to real-world situtations and careers.
4. Working collaboratively with suppliers and academic colleagues to shape enhancements / design solutions to particular issues: We hope to connect key members in Departments with our suppliers Uniwise to workshop and work through desirability and current system functionalities.
   

If you would like to join events like this in the future, please let us know by contacting assessments@ucl.ac.uk.

If so, why not join the 2022 Bloomsbury Learning Exchange (BLE) Cohort and work towards CMALT accreditation? You may be a tutor, a PGTA, an ELO, a TA, a Librarian, a Learning Technologist, etc. Anyone in fact, who is a staff member at one of the institutions affiliated to the BLE (namely, Birkbeck, LSHTM, RVC, SOAS, UCL, City, University of London and UoL) and has been using technologies effectively to teach or support teaching and learning. Indeed, in the last two years many more UCL staff members have been doing just that.

What is CMALT?

CMALT stands for Certified Membership of the Association for Learning Technology and the CMALT Accreditation Framework provides pathways to peer-assessed accreditation for a cross-section of learning technology focused professionals, educators and administrators in the UK and internationally.

Join the BLE CMALT Cohort

Registration for the BLE CMALT Cohort is now open! (Until March 1st 2022*). If you missed our two Introduction to CMALT sessions, our slides are available to view here and the session’s recording is here. Attached is the CMALT Prospectus; further information about CMALT and the Association for Learning Technology can be accessed here: https://www.alt.ac.uk/certified-membership

When you have decided, which of the 3 CMALT pathways you may wish to pursue, (download the CMALT Prospectus for details), then please complete our sign up form. Thereafter more information will be emailed to you. The first cohort meeting was or will be on Thursday 3rd Feb, 1 – 2.30pm and was recorded. Future meetings will take place on the first Thursday of every month for no more than 8 months.

Late starters are very welcome! (Until March 1st 2022*). So, if you are interested please do get in touch using the aforementioned form.

*It is also possible to pursue CMALT accreditation independently at any time. See CMALT Support for details.

In this seminar, hosted by the UCL Knowledge Lab Learning Technologies Unit, Dr Bronwen Swinnerton presented outputs from the UnbundledHE project: “The Unbundled University: Researching emerging models in an unequal landscape”, a cross-institutional study conducted by the University of Cape Town and University of Leeds.

The project considers the intersection of marketisation, unbundling, and digital technology and its effects on educational inequalities in South Africa and the United Kingdom. The following blog post summarises some of the issues surrounding the concept of educational unbundling and the project’s report into the current state of unbundled provision in both countries.

Marketisation

Recent years have seen a growth in demand for HE internationally but this has occurred alongside global economic shocks, such as the 2007 recession. The economic downturn has seen greater pressure placed on universities to report on the impact of central government funding streams and a resurgence of debates surrounding education as a public versus private good. This debate has tended to foreground the benefits of higher education to private individuals and consequently prompted a shift towards funding by the individual, leading to increases in fees and application of exogenous market principles to the HE environment. Such pressures have led HEIs to look to generate third stream revenue and reach new or alternative markets to fill funding gaps.

Digital technology

Digital technology is now ubiquitous in everyday life and HE, with all universities making use of technology in teaching and learning to some extent. Alongside the need to reach new audiences many have also begun to engage with online education, which has seen dramatic growth since the introduction of MOOCs. These courses have been in great demand from non-traditional audiences, such as adult professionals, with platform providers and university partners now pivoting toward the use of short online courses to deliver CPD and widening access initiatives, enhance teaching quality, and promote ‘massification’ or scalable learning opportunities. It is agreed that while universities can undoubtedly reach more prospective learners with online education it is no longer a ‘second best’ or indeed cheaper delivery method.

Unbundling

Unbundling is explained by McCowan (2017) as being either ‘disaggregation’, where what was sold together is now sold separately, e.g. tracks from an album, or ‘no frills’, a basic version of a product, e.g. budget airlines. There now exists a definitive application to HE, with the project defining unbundling as a disaggregation of educational provision, e.g. degree programmes, into component parts, e.g. modules, for delivery by and to multiple stakeholders. This is often achieved using digital approaches, and is manifesting itself through alternative digital credentials, or microcredentials, often delivered via what were MOOC platforms.

Rationale for offering unbundled educational opportunities can be multifarious and their production and delivery at odds with existing or longstanding institutional processes. The approach is typically: 

• Internal, whereby HEIs choose how and when to break apart existing provision and offer content to learners directly;
• In collaboration with service or platform providers, in which an HEI procures services to support specific stages of the course lifecycle;
• By working with online programme management companies (OPMs), frequently with full service or white-labelled delivery and revenue share arrangements.

The unbundling landscape in SA and the UK

Dr Swinnerton highlighted key findings from the project, including the outcomes of interviews with policy makers, HE leaders, edtech developers, and private company CEOs. Mapping information (delivered via kumu.io software) was drawn from publicly available data, such as university, private company, and online and distance education websites, press releases, and other media. 

Interactive mapping of the HE landscape in SA and the UK demonstrated HEIs, suppliers, and OPMs and the relationships between them across provision of unbundled online education. The maps could be filtered according to league table strata, and what was immediately apparent across both contexts was disparity in coverage, with partnership opportunities broadly unavailable to lower-ranked institutions. Private companies in this space, notably OPMs, chiefly target elite institutions with established brands to drive profit-making business models. This was particularly clear in SA, with historically disadvantaged universities having little to no opportunity to engage in the online education space with support from private sector organisations, and with the suggestion that this had the potential to propagate stark inequalities already inherent within the SA HE system. Similar issues were present in the UK context, but with acknowledgement that educational inequalities and the digital divide are not quite so strong outside the context of online delivery.

See also

Dr Bronwen J Swinnerton, Senior Research Fellow in Digital Education at the University of Leeds

McCowan, T. 2017. Higher education, unbundling, and the end of the university as we know it. Oxford Review of Education 43(6); pp.733-748.

Swinnerton, B , Ivancheva, M, Coop, T et al. (6 more authors). 2018. The Unbundled University: Researching emerging models in an unequal landscape. Preliminary findings from fieldwork in South Africa. In: Bajić, M, Dohn, NB, de Laat, M, Jandrić, P and Ryberg, T, (eds.) Proceedings of the 11th International Conference on Networked Learning 2018. Networked Learning 2018, 14-16 May 2018, Zagreb, Croatia; pp. 218-226.

Alternative presentation delivered by UCT’s Sukaina Walji at WCOL19 with an increased focus on OPM relationships. “Degrees of (un)ease: Emerging relationships between OPMs and University Stakeholders in an unbundling landscape”

Thurs 28th – Fri 29th June

Hosted by Birkbeck, University of London

Following the University of London’s successful conference Demystifying User Experience Design & Testing last year, the Bloomsbury Learning Environment (BLE) in partnership with the University of London (UoL) is holding a free, two-day event for staff based at UoL member institutions on Thursday 28th and Friday 29th June.

On these two days, we are offering three distinct workshops, which are each focused on different applications of UX. Come along to all three or select those that interest you. Places are limited, so don’t delay registering your place!

Day 1: Thursday 28th June

am: User Research: focus groups, user testing and user feedback
pm: User Centred Content

Thursday 28th June: Morning workshop

User Research: focus groups, user testing, surveys and user feedback

A practical session with guest speakers sharing their insights into user research and associated applications.
Led by Naomi Bain, Web Officer (Training and User Experience) – Birkbeck, University of London

0930 Coffee & Registration
• Introduction (Naomi Bain)
• Keynote: conducting f2f user testing (Jane Lessiter, Goldsmiths)
• Case studies: sharing experiences of user research
• Practical session: how to conduct a web user testing session. This session will include tips, discussion, sharing experiences, questions and trying out the roles of tester and testee (Naomi)
End by 1300

Thursday 28th June: Afternoon workshop

User Centred Content

An overview of the online tools available to help you to plan and review your own content. Mapping users against online content – bring along a piece of your own content to review! Finishing with a panel Q&A discussion around content strategy and governance.
Led by Melanie Read, Head of Digital – University of London

1345 Registration, with a prompt start at 1400
• Welcome, Introductions and icebreaker
• Content planning – what tools do you use for planning content.
• Content mapping – against the difference users types and then creating content suitable to that user.
• Content strategy and governance
• Panel discussion: how to manage governance
End by 1630

Day 2: Friday 29th June

am: Moodle and Accessibility

Friday 29th June: Morning workshop

Moodle and Accessibility

This workshop will focus specifically on Moodle and the considerations and requirements to ensure courses are accessible to all users.
Led by Sarah Sherman, Service Manager – Bloomsbury Learning Environment

0930 Coffee and registration
• Welcome & Introductions (Sarah Sherman, BLE)
• Presentation 1: Birkbeck For All (Leo Havemann, Birkbeck)
• Presentation 2: Policy for Accessibility (Nic Charlton, University of London)
• Presentation 3: Working with Moodle (Nic Christodoulou, SOAS)
• Presentation 4: Accessibility initiatives at UCL (Jess Gramp & Paul Thompson, UCL)
• Presentation 5: Checking for accessibility in Moodle (Herve Didiot-Cook, Blackboard)
• Panel discussion
• Workshop activity: developing Moodle accessibility guidelines for practitioners
End by 1300

Book your place now.

For further details about the event, please contact Sarah Sherman or Melanie Read

Arne Hofmann and Helen Matthews from the UCL Joint Faculty (Arts and Humanities and Social and Historical Science) have hit on a successful format for a practice sharing session. Speakers make brief presentations and then disperse to ‘stations’ around the room so that participants can circulate and discuss. At the end of the event is a plenary discussion.

The third event in this series had a digital education focus. Sanjay Karia who heads up IT for SLASH kindly contributed display screens for the stations. I matchmade colleagues in Digital Education with the presenters based on interest; they made notes of the conversations (using MS Teams as recommended by IT for SLASH) and I largely owe this blogpost to them.

The splendid presenters and their presentations, some of which include links to examples of student work:

• Mark Lake (Senior Lecturer, Archaeology) described undergraduate students blogging for ARCL3097 Archeology in the World  – a particularly gutsy initiative given it was a compulsory module for final year undergraduates in the NSS zone [Mark’s slides as PDF].
• Riitta Valijarvi (Senior Teaching Fellow, Finnish) talked about her Wikipedia Translatathon, a cultural and linguistic event marking the centenary of Finland which brought together students, Finnish or Finnish speaking staff from UCL, members of the public, and Wikimedia UK staff.
• Jakob Stougaard-Nielsen (Senior Lecturer, SELCS) discussed students producing digital objects for the Qualitative Thinking module of the BASc [Jakob’s slides as PDF];
• Jacky Derrick (Deputy Module Convenor, History) described how first year undergraduate student groups produce web sites together on a subject which gets them engaging with London;
• Nick Grindle and Jesper Hansen (Senior Teaching Fellows, Arena Centre) reviewed their experiences organising peer feedback via the fearsome-looking but actually wonderful Moodle Workshop activity [Nick’s slides as PDF];
• Maria Sibiryakova (Senior Teaching Fellow, Russian) talked about how the multimedia discuss app VoiceThread can advance the four skills of language learning [Maria’s slides as PDF];
• Jonathan Holmes (Professor of Physical Geography) and Nick Mann (Learning Resources Coordinator, Geography) on designing digital multiple choice exams [Jonathan’s and Nick’s slides as PDF];
• Clive Young (Digital Education Advisory Team Leader) on meeting the UCL minimum quality standard known as ‘the UCL E-Learning Baseline‘.

Here are some of the themes from the event.

How should students be inducted to new technical platforms? For some cohorts this was hardly an issues and staff soon felt comfortable abandoning the training session at the beginning of the module in favour of a drop-in as the deadline approached. However, there are disciplinary differences and not all groups can be guaranteed to have somebody particularly comfortable with using technologies, so the drop-ins are important.  Based on my own experience inducting some large cohorts to Mahara, if it’s done at all then it’s best done when the students have some vision about what they want to do there – i.e. not at the very start of the module, and close enough to the deadline that there is no hiatus between the induction and putting the knowledge to use. In addition, students seemed to know less about copyright and intellectual property than the technologies, so some modules had incorporated sessions on those.

How do we assess digital multimodal work? Formative assessment was considered very worthwhile, especially where students were new to the activity. Currently there is sometimes a criterion related to appropriate use of the mode or format, such as “use of text formatting and good quality images and/or multimedia which clearly enhance the text”. Often there is an element of writing in the work which would be run through Turnitin according to departmental policy. I think it is probably fair to say that (like most of the sector) we are in transition to explicitly recognising the distinctive qualities of digital multimodal composition. I have seen how, in many cases, new and potentially challenging practices need to be eased through teaching committees by anchoring them to the accepted standards and criteria – at least for the first few iterations. With time and experience comes new awareness and recognition of distinctive practices which work well in a given context. Jakob’s slides are particularly detailed on this – the BASc have been giving this kind of thing consideration from day one.

Where should digital multimodal work be positioned in the curriculum? There was a general sense that modularisation tends to isolate digital activities within programmes. This could lead either to them not being built upon (where they happened early) or having their academic validity questioned (where they happened later). Support includes showing students exemplars of blogs and creating opportunities for them to carry out guided marking to help them grasp standards and apply the assessment criteria to their own work.

What if students question the academic validity of a digital activity? Where new forms of digital assessment are introduced later in a programme, expect students to query whether it is really necessary for their degree. The challenge, summarised by Mark, is to pre-emptively “tackle student perception” by advocating for the activity in terms of student learning and success. Archaeology in the World saw their evaluation questionnaire results slowly improve as the tutors learned to advocate for the activity, and students came to recognise it as useful.

When can students’ work be public? In cases like the Wikipedia event, the work is born public. In other cases this is something to be negotiated with students – but there is often groundwork to do beforehand. Students need guidance to use media that is itself licensed to be made public. Where the work happens in groups, licensing their work needs to be a joint and unanimous decision with a take-down policy.

How can different skills levels be accommodated? The intermediate Russian language students were at different levels, which meant that multimedia production such as recordings of poetry read aloud helped them practice speaking (one of the four skills of modern language learning), and the individualised recorded feedback they were given helped them with listening (another of the skills). VoiceThread brought a privacy and timeliness to the feedback which had not previously existed – exposing students to the risk of embedding their mistakes. Another approach to different skills levels is to create groups of students on the assumption that they will either sustain each other in acquiring the skills or divide the labour according to skills, and a third is to give extra guidance to students who need it (as with the Finnish-English Wikipedia Translatathon).

How can the new practice be made to work first time? When the Arena Centre pioneered large scale use of the Moodle Workshop activity for peer feedback, they worked closely with Digital Education – we made those early deadlines our own, and together we prepared for different contingencies. As well as working closely with Digital Education, Geography subjected their digital examination to a number of rigorous checks involving academic and professional services colleagues, students, and internal and external examiners. Digital Education has produced the Baseline to support the quality aspects – these are not intuitive. One participant remarked to me later that he had been skeptical, bordering on resentful, of the Baseline until he started working through it, at which point he realised how useful it is.

What do students get out of the digital side of things? Some indicative comments from Jakob’s students: “learnt to consider digital content in a very different way”; “through creating a Digital Object rather than a traditional essay, I was able to engage with our topic at a much deeper level”, and “I have also developed transferable skills”. Mark received correspondence that the activity “really made me think and synthesise in a new way”. Nick’s and Jesper’s Arena participants have been very positive about giving and receiving peer feedback.

~~~

There are a few things I’d change about how I organised the event. One is that either it should be extended by half an hour (to two hours) or else the number of speakers should be reduced. As it was, we overran and I was very sad to have to cut off a very interesting plenary discussion just as colleagues were beginning to really want to talk with each other. Another is that teaching languages has a distinct set of needs which justify a dedicated event. I might also consider asking the presenters to circulate rather than the participants (though I can see pros and cons there).

That aside, it was a lively, spontaneous, humorous, sophisticated event which balanced different sets of needs – educational, disciplinary, colleagues and students. It is so often the case that when colleagues have the opportunity to seek each other out based on mutual interest, the fruits soon make themselves evident. One participant told me he went from the event straight to his department’s Staff Student Consultative Committee where he proposed an idea which was accepted. “That’s impact”, he said.

The next UCL Digital Literacy Special Interest Group (UCL DL SIG) will be taking place on Friday February 16th from 2pm – 5pm (ticket link at the end of this post).

Digital content is increasingly being used in learning, teaching and research across the Higher Education sector. This has led to a significant change in research practices across disciplines, which include knowledge creation and dissemination through social media and repositories. Complex software tools are being used for data analysis in Arts and Humanities as well as Sciences, and large data sets are being made available to the research community, leading to a blurring of the organisational and support responsibilities of academic stakeholders. This timely event takes a look at digital scholarship at large, and considers new initiatives and opportunities within UCL to address the challenges associated with this disruptive shift.

Event Programme

Developing Digital Scholarship: Emerging Practices in Academic Libraries – Alison MacKenzie, Dean of Learning Services at Edge Hill University and Lindsay Martin Assistant Head of Learning Services at Edge Hill University.

The impact of digital on libraries has extended far beyond its transformation of content, to the development of services, the extension and enhancement of access to research and to teaching and learning systems.As a result,the fluidity of the digital environment can often be at odds with the more systematic approaches to development traditionally taken by academic libraries, which has also led to a new generation of roles and shifting responsibilities with staff training and development often playing ‘catch-up’. One of the key challenges to emerge is how best to demonstrate expertise in digital scholarship which draws on the specialist technical knowledge of the profession and maintains and grows its relevance for staff, students and researchers.

From digital scholarship to digital scholar  – Alison Hicks, Lecturer UCL Department of Information Studies.

Drawing on her experience working as an academic librarian in the United States, Alison’s presentation centres on the capacities that are needed to participate in practices of digital scholarship, as well as the inherent risks and challenges of engaging in open and networked spaces.

Introduction to Digital Scholarship and Open Research – Daniel van Strien, Research Data Support Officer UCL Library Services.

Daniel will be presenting on a session which aims to help participants make a practical start in practicing open science and digital scholarship he is a Research Data Support Officer within UCL Library Services with an interest in digital scholarship and new approaches to research.

Where’s your digital at? – Moira Wright, Digital Literacy Officer, UCL Digital Education.

With an interest in student digital and information literacy skills for employability. Moira will be talking about the Jisc Digital Capability Discovery Tool and how to get involved in the UCL beta pilot.

Research IT Services – Tom Couch, UCL Research IT Services (RITS).

Whilst many of the existing users of Research IT Services are pushing for more of the same but better, the broadening base of digitally engaged researchers from different disciplines requires more experimentation with new technologies and services. Tom Couch reports on some recent projects that have helped RITS to engage and support new groups of researchers.

Please use this link to book your ticket via Eventbrite

We’re using the Jisc definition of digital literacy: ‘the capabilities which fit someone for living, learning and working in a digital society’.
The UCL DL SIG was created for UCL staff to promote the use of technology in learning, provide a platform to ask questions, exchange ideas and also to get support from colleagues beyond UCL Digital Education.

Photo by Tim Mossholder on Unsplash