Written by Dr Kristina Curtis (UCL Centre for Behaviour Change)

This blog article is dedicated to a panel discussion from the CBC Conference Enabling Behaviour Change to Build Back Better for health & sustainability: 1 – 3 Nov 2021. An interdisciplinary panel of experts took us on a journey from how we define and measure engagement to how we learn from big data and produce a generalisable science of factors that promote engagement.

We kicked off the engagement panel with Dr Kevin Doherty from Technical University Denmark, who began his quest to conceptualise engagement from a review of the computer science literature. Kevin was struck by the wide range of perspectives and conceptualisations of engagement from ‘a quality of user experiences with technology’ (O’Brien and Toms, 2008) to ‘..the phenomena of being captivated and motivated’ (James et al. 2011). To help understand where these different perspectives were coming from, Kevin and his team turned to the original theories, revealing a huge range from most cited Flow Theory (50 mentions) to least cited Richness Theory (2 mentions). Still struggling to pin down what exactly was meant by engagement, Kevin and his team looked next at measurement for further insights. They uncovered a long list of measures such as questionnaires, behaviour logging, facial analysis, and gesture tracking. Next, they looked at the design strategies used to design for engagement and reported a variety of strategies including recommendations such as ‘usability’, ‘immersing users’ and ‘inspiring fun’. As these strategies were employed across a range of contexts and technologies, they were still no closer to pinning down exactly what was meant by ‘engagement’. A combination of Kevin’s applied work and academic review led to thinking about engagement not only in terms of what it is, but how we might design for engagement as a process and how users engage with technology as a process in itself. Kevin ended his talk around remaining challenges, one being how we build interdisciplinary teams and the necessary knowledge to facilitate this kind of approach to engagement.

Our next panellist was Professor Rik Crutzen from Maastricht University brought us on to the topic of measuring engagement. Rik stated as with any psychological construct, it’s important to ‘disentangle’ this conceptualisation from how it is measured. These are not two separate entities but related to one another. As with measuring psychological constructs, we use a plethora of methods to measure engagement such as surveys and reaction times. Interestingly and in contrary to some of the audience’s views, Rik argued against a standardised way of measuring engagement. Instead we ought to keep a ‘decentralised’ meaning, as ultimately it will depend on the context and how we are conceptualising engagement in the first place. Rik’s second point related to the distinction between measuring intended and actual use of an intervention and methods employed to understand usage behaviours. For example, ‘Think aloud’ procedures can help to uncover if certain user groups go to different places on a website and whether any content is misinterpreted, allowing an organisation to continually refine their intervention. Rik’s final point addressed the novel ways to reduce participant burden and increase engagement. For example, avatars can be used as a proxy for psychological constructs and skip logic in questionnaires can help to reduce participant burden via shorter questionnaires. Indeed, reducing participant burden is key to sustaining users’ engagement with an intervention and must be prioritised.

Next on the stage was the panel’s industry expert: Charlotte Summers COO and Director of Research at DDM Health. Charlotte Introduced GRO – a digital platform providing disease-specific education, behaviour change support and feedback to optimize health and wellbeing. Gro has been developed with leading clinicians to address all modifiable risks. Charlotte shared encouraging research findings from peer-reviewed published health outcomes including: Type 2 diabetes remission, weight loss, reduce medication, reduced anxiety, depression and perceived stress. So, what’s in this successful platform? Charlotte described the following features: personalised resources; one-to-one behaviour change coaching; goal setting; community support and integrated tracking. It also uses AI software to optimise engagement and health outcomes. This platform is a patient-healthcare platform to allow real-time health monitoring and engagement information. Charlotte demonstrated through a case study how ‘big data’ can be used to provide answers on intended vs actual use. They noticed that it was actually the younger family members using the app on behalf of their older relatives that did not speak the language or were not digitally literate. These insights allowed them to further refine the app and offer even more personalisation, with a recognition that is important to engage not just the individual, but the family as a whole in helping to manage type 2 diabetes.

The final panellist was Dr Olga Perski from UCL who took us through a stimulating presentation on whether current approaches are getting us closer to identifying ‘timeless’ design elements. Olga drew our attention to the current state-of-the-art around identifying design elements that reliably influence engagement. This of course is important for designing websites/apps that people want to use and saves money through reduced testing of design iterations. Olga’s talk highlighted the importance of an interdisciplinary team with expertise in how we use a combination of user-centred design (e.g., co-design workshops) and experimental methods (e.g., factorial trials) to better understand design elements that influence engagement with health and wellbeing websites/apps. Findings from these kind of research methods have provided important insights on broader design elements such as ‘ease of use’, ‘personalisation’, and ‘interactivity’ which are important drivers of engagement. However, the challenge exists in the myriad of ways these design elements can be operationalised. One could argue, this is also where the creativity and innovation comes into play and again the importance of testing design features with the relevant target audience. In addition, Olga points out that design preferences will change over time and across contexts, which is in part due to the changing social and cultural norms/expectations. Research shows that users tend to rely heavily on “social proof” to help navigate which technologies to adopt. With deeply contextual and ever changing social, cultural norms and expectations, it’s looking increasingly challenging to identify “timeless” design elements…For example, although chatbots were seen as moderately acceptable by internet users in 2019, they will likely be more commonplace and widely acceptable in 2030. So, what’s the solution? Olga left us with the idea of embracing complexity. In line with the common theme running throughout this discussion panel, design principles, as with the conceptualisation and measurement of engagement, will largely depend upon context.

Papers panellists drew on are listed below:

Doherty, K et al (2018). A mobile app for the self-report of psychological well-being during pregnancy (BrightSelf): qualitative design study. JMIR mental health, 5(4)

Doherty, K., & Doherty, G. (2018). Engagement in HCI: conception, theory and measurement. ACM Computing Surveys (CSUR), 51(5), 1-39

Doherty, K., & Doherty, G. (2018). Engagement in HCI: conception, theory and measurement. ACM Computing Surveys (CSUR), 51(5), 1-39

Doherty, K. (2019). Designing the Self-Report of Wellbeing in Pregnancy (Doctoral dissertation, Trinity College Dublin).

Doherty, K et al (2018). A mobile app for the self-report of psychological well-being during pregnancy (BrightSelf): qualitative design study. JMIR mental health, 5(4)

Metz, G., Roosjen, H., Zweers, W., & Crutzen, R. (in press). Evaluating use of web-based interventions: an example of a Dutch sexual health intervention. Health Promotion International.

Perski, O., Blandford, A., West, R., & Michie, S. (2017). Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Translational behavioral medicine, 7(2), 254-267.https://doi.org/10.31234/osf.io/zhj86

Perski, O., & Short, C. E. (2021). Acceptability of digital health interventions: embracing the complexity. Translational Behavioral Medicine.

Saslow, L. R., Summers, C., Aikens, J. E., & Unwin, D. J. (2018). Outcomes of a digitally delivered low-carbohydrate type 2 diabetes self-management program: 1-year results of a single-arm longitudinal study. JMIR diabetes, 3(3), e12

Short, C.E., DeSmet, A., … & Crutzen, R. (2018). Measuring engagement in e- & mHealth behaviour change interventions: viewpoint of methodologies. Journal of Medical Internet Research, 20, e292. https://doi.org/10.2196/jmir.9397

Short, C.E., Smit, E.S., & Crutzen, R. (in press). Measuring psychological constructs in computer-tailored interventions: novel possibilities to reduce participant burden and increase engagement. The European Health Psychologist. https://doi.org/10.31234/osf.io/hz593

By Liz Sillence; Psychology and Communication Technology  (PaCT) Lab, Northumbria University

Whilst online health groups have a long history of offering a place for social and emotional support we are starting to recognise that these communities are providing a key role in supporting decision-making and in some cases affecting behaviour change.

In a recent study, a colleague and I examined the different ways in which engagement with an online support group influenced decision-making around health (Sillence & Bussey, in press). (more…)

By: Paolo Satta, project manager at Day One

Designing an effective incentive system is not an easy task. Besides delicate choices that can decide the fate of the entire system (Which actions should be incentivised? How? How often?), a hidden threat can jeopardise even the most sophisticated concept: cheating by the very same people that should be incentivised.

Irrespective of the system, we know that someone is going to try cheating on it – for getting higher rewards, surpassing the others or just for self-gratification in fooling the system.

How can you build up an incentive system with antibodies that protect it against cheating attacks?

The first rule of thumb is to create incentives which are perceived to be fair by the beneficiaries.

(more…)