X Close

CBC Digi-Hub Blog



Expanding our Toolkit to Evaluate Commercial Mobile Health Apps

By Artur Direito, on 29 March 2018

By Dr. Danielle Jake-Schoffman, University of Massachusetts Medical School

While there has been a proliferation of commercial mobile apps to support health behavior change in recent years, the evidence to encourage using these apps is limited in a variety of domains. This is not a groundbreaking conclusion given a flurry of review papers that have examined a selection of commercial apps and have consistently found that they come up short in terms of their use of evidence-based guidelines and behavior change techniques. However, evaluations of commercial apps and the types of solutions to bridging potential deficiencies in their features have been limited in scope, also limiting our ability to determine the value of using apps in research and clinical settings.

We recently published a detailed guide to methods for evaluating apps across the domains of content, usability, and efficacy. Within each domain, we describe a variety of research questions and methods that can be used to explore commercial products to determine their suitability for use in research and practice. The goal of the paper is to shed light on the diversity of methods that could and likely should be used to better evaluate commercial apps and other technology products. This pushes research questions of behavioral scientists past the domains most closely related to our training—e.g., content analysis—and stretches us to consider for example factors that could influence usability of apps and what can potentially be done to enhance commercial apps or work around their shortcomings in order to avoid having to start development from scratch. While user-centered design principles and usability testing are becoming more common in the development of researcher-driven apps, their use is still rare in the evaluation of commercial apps. There is a need for more field and laboratory testing of commercial apps, with large user bases and sophisticated design elements, to better understand what is working well and why. Further, there could be opportunities to partner with industry to use research insights to offer suggestions for the improvement of future iterations of apps.

Ultimately, the speed and dynamics of technology development from the industry pipeline will likely continue to outpace that of researcher-developed products. Instead of criticizing the lack of behavioral science included in these commercial apps, perhaps it is time to focus on other methods of working with what exists and bridging gaps with supplemental materials or training provided to research participants where needed. If researchers still elect to develop their own apps, conducting content, usability, and efficacy testing of what exists commercially can still shed important light on features that might be critical to successful development and user adoption.

Although many commercial health apps are lacking from a behavioral science perspective, consumer use of these tools continues to increase (recent survey showed app use has tripled since 2014).

It is critical for the research community to understand what these tools offer, what they are missing, and how we can leverage their strengths and popularity to move the needle on behavior change at scale.

Discussion questions:

  • What role should commercial health apps play in future behavior change research?
  • Should health behavior researchers focus on development of new apps or finding ways to make use of existing commercial products?


Dr. Danielle Jake-Schoffman

Danielle Jake-Schoffman, PhD is currently a postdoctoral research fellow in the PRACCTIS (Prevention and Control of Cancer Training in Implementation Science) program at the University of Massachusetts Medical School. Her research focuses on leveraging mobile technology to operationalize evidence-based strategies for chronic disease prevention, specifically, for the promotion of physical activity, healthy eating, and weight loss. Follow Dr. Jake-Schoffman on Twitter (@DrJakeSchoffman) for all things #digitalhealth and behavioral science.


2 Responses to “Expanding our Toolkit to Evaluate Commercial Mobile Health Apps”

  • 1
    Christopher Bristow wrote on 29 March 2018:

    As a commercial developer of a behaviour change platform we would welcome input from research partners and we would be willing to collaborate, but to date we have never been approached. Does that mean that researchers prefer a build-it approach or could it be that they are just not aware of the number of products that could be adapted for their research projects?
    In answer to your questions, I think you would find a willing market amongst commercial health app developers, and there are many obvious benefits for the researchers including a working base product with features for the designated market and an easier, faster way to commercialisation (your last point about BC at scale). I have never been one for reinventing the wheel, so I would always first look for a product I could adapt or utilise in some way.
    But that just may be my pragmatic business approach!
    It would be interesting to know how much time researchers spend on reviewing the existing tech market before embarking on the build-it approach.

  • 2
    Danielle Jake-Schofmfan wrote on 30 March 2018:

    Christopher– thank you for your insights! It’s great to know learn of a new potential industry collaborator– I’ll be in touch! But, in response to your question re: evaluating the landscape before opting to build from scratch, I think many researchers do try to ensure they aren’t reinventing the wheel. However, I think we could be doing more to leverage existing commercial products even if they are lacking in some of the behavior change theory-informed content we desire. That’s why we advocate for using a range of methods (beyond content analysis) to determine what the commercial products offer that can be helpful (you can read more here: https://mhealth.jmir.org/2017/12/e190). And certainly partnerships with industry partners are a critical piece of this puzzle as well.

Leave a Reply