Everything you ever needed to know about Registered Reports (*even if you weren’t afraid to ask)
By Kirsty, on 3 July 2020
The concept of Registered Reports was developed in response to a vast range of meta-analysis over the past few years (1) that showed that a lot of research being published exhibited bias. Different papers analysed publication bias, hindsight bias, and selective reporting which demonstrated that published works overwhelmingly showed predominantly positive results. There were also significant issues with reproducibility and transparency as people were not sharing sufficient results or enough detail in methodologies to allow for the results stated to be replicated.
The nature of good research is to investigate, to take a hypothesis and test it dispassionately, discovering the results and presenting them as new or confirmed knowledge – whether the hypothesis is proven or not, it’s all knowledge! Unfortunately, this isn’t always the reality. The issue at the core of all of this is the research environment itself – good, objective practice is not always what gets rewarded.
Researchers are often given the message (intentionally or otherwise) by publishers, funders, institutions and colleagues that positive, world-changing, elegant and simple results are prized above all others. It’s these results that researchers and publishers want to publish. Since publishing is key to career advancement, this inevitably influences how researchers carry out their work. Leaving aside for now deliberate falsification of results, and the arguments about alternative ways of disseminating them, we can see how this leads to behaviours that produce at best selective and incomplete results, and at worst downright misleading ones. Behaviours like HARKing (Hypothesising After Results are Known) and P-hacking, testing variables until you find something significant, are particularly problematic. Since researches also tend to cite positive results more than negative ones, this positive publication bias continues to be amplified after publication, too.
So, what can be done?
Registered reports are a new way of getting your work accepted for publication. The idea is that you submit only part of a study, like the first half of an article, and the peer review is conducted on your idea and study design, before any data is collected. If you are successful, you receive an Acceptance in Principle, do your research and write it up. If the research hypothesis and methods section that has already been accepted hasn’t changed significantly, the final acceptance is based on a peer review of the remaining parts of the article. This second round of peer review focuses on how complete and robust your work is, completely uninfluenced by the results of the study.
If I want to use Registered Reports for my next project, where do I go?
There is a list of journals that are supporting Registered Reports on the Center for Open Science website.
Naturally, if you aren’t working in a hypothesis-driven subject area, Registered Reports might not be for you, but there are other options to consider. They are a little less well known, but still have their own benefits:
Results Blind Peer Review is very similar to registered reports. The article is submitted and reviewed as normal, but the results are withheld until after the first peer review stage.
Exploratory reports are the newest method and lean more towards supporting exploratory research in less hypothesis-driven subject areas. This allows meta-analysis and confirmatory research, and more flexibility in the flow of the research from design to results.
Do registered reports really work?
The logic behind Registered Reports has already proven its worth: compared with papers published in the traditional way, a much higher percentage of Registered Reports have ‘null’ results. This suggests they really do reduce publication bias. And they’re still cited at similar rates to conventional articles.
These forays into addressing bias are just a beginning. The issue isn’t going to go away overnight but these journals, with the support of researchers, can start to redress the balance and make sure that results that disprove hypotheses get as much air-time as those that prove them. These journals show that there is a way for researchers and publishers to work together to address the problems of publication bias. Meanwhile, institutions are changing their promotion and assessment criteria to ensure that the research quality, rather than where the work is published, is taken more into account. A similar path is being taken by funders and even the REF. We all need to support these journals and together commit to an open research culture.
(1) Chambers, C. D. and Tzavella, L. (2020) Registered Reports: Past, Present and Future. MetaArXiv. doi: 10.31222/osf.io/43298.