Week 4: All About Pre-Registration

Leave a comment

Presently there is a large degree of variability regarding the understanding and application of pre-registration in psychological science. From what I have seen on social media, read in papers and other scholarly work, and from reading actual pre-registrations from different labs, there is no agreed upon definition of pre-registration or guiding principles for when and how to implement a pre-registration. This is perhaps to be expected at such an early stage of adoption by academics not used to publicly sharing their ideas prior to testing them, with a non-trivial number of academics remaining skeptical of the practice. The goal of this week’s class was to introduce the students to a definition of pre-registration, to discuss some common “yes, but…” reasons for not pre-registering hypotheses, methods and data analytic plans, and to share some resources for how to implement a pre-registration.

Here is my working definition of pre-registration: Stating as clearly and specifically as possible what you plan to do, and how, before doing it, in a manner that is verifiable by others.

If you have an idea that you would like to test with new data or existing data, you can share your idea and plans for testing it before doing so. The alternative is not sharing this information before testing your idea, meaning it either (a) gets shared to some degree in a manuscript that is written after testing your idea, or (b) not shared because you chose not to write a manuscript that is written after testing your idea. In my opinion sharing before versus (maybe) after testing your idea is the better option. I therefore suggest that academics pre-register all of their planned research pursuits. In this post I will attempt to explain why.

There is no one correct way to implement a pre-registration, and a pre-registration itself is no guarantee that your hypotheses, methods, and/or data analytic approach were sound. Stating in a manuscript that your idea was pre-registered also does not imply the degree of specificity of your hypothesis, or that you followed your pre-registered protocol as specified. Importantly, however, these things are now verifiable by reading the pre-registration materials. It is worth taking the time to learn how to best communicate your intentions in advance of a given research pursuit, perhaps seeking feedback from other experts during this process, with the assumption that consumers of your research will take the time to read your pre-registration materials.

Common “Yes, but…” Arguments Against Pre-Registration

Here are four common “yes, but…” arguments I hear regarding why a given researcher cannot implement pre-registrations for his or her research:

  • It only applies when you have specific, confirmatory hypotheses. “My work is often theoretically guided but I do not always test specific, confirmatory hypotheses from the outset.”
  • It is simply not feasible or practical for complex study designs (e.g., longitudinal designs, large scale observational studies).
  • The data are already collected so (a) “I have nothing to pre-register”, and/or (b) “I have already analyzed some of the data so I can’t pre-register.”
  • It puts limits on what can be done with the data. “I may have some hypotheses and plans to test them, but as a scientist I need to go to where the data takes me and therefore do not want to be limited to only the analyses I could think of in advance. Pre-registration can stifle creativity and even scientific discovery.”

The short answer to each of these arguments is: nope. According to the definition of pre-registration I put forward, it is always to possible to state what you plan to do before you do it, as long as you are open and transparent about the state of the research pursuit in question. Here are some longer answers to these four arguments:

  • Pre-registration is not only for purely confirmatory research. It applies equally well for research that is largely exploratory, or somewhere in between exploratory and confirmatory. If, for example, you plan to collect self-report personality data from a large group of individuals and follow them over time to observe variability in different personality traits but you are not sure what that variability should look like, you can say that in a pre-registration. If you are not sure if the association between some theoretical constructs you assessed in your study should resemble patterns of mediation or moderation and want to test both, you can say that in a pre-registration. If you want to collect responses on many scales that may/may not be correlated with each other from a sample of students in an effort to select some of these scales for use in another study, you can say that in a pre-registration. Here is the pattern that is unfolding: after saying what you would like to do with your data collection and/or data analysis simply add “I can say that in a pre-registration”. Pre-registering vague ideas or exploratory research also helps prevent the researcher from using the words “As predicted…” in future publications using results from this research.
  • It is not necessary for a pre-registration to include every single possible hypothesis and accompanying data analytic plan for a given data set. If you first plan to analyze a subset of the data from, for example, a large sample of married couples assessed over two years (with data collected at 8 time points), you can pre-register those plans. If you decide to analyze a different set of ideas with different data collected from this sample, you can pre-register that at another time.
  • Data may already be collected, but it is still possible to state in advance your idea and how you plan to use existing data to test this idea. Be upfront with your prior experiences with this data set and how your new ideas were generated.
  • Pre-registration does not put limits on what you can do, but rather helps distinguish between analyses that were planned in advance from those that were conducted post-hoc (or between more confirmatory and exploratory analyses). Ideas and data analytic decisions that are made because of experiences working with the data (ideas and decisions you did not have prior to working with the data) are exploratory and should be labeled as such. Of course follow-up analyses are often needed and can lead to new, perhaps unexpected, patterns of results (that will need to be replicated at some point with independent data to properly test these hypotheses).

At this point in my conversations with those skeptical of pre-registration, they often say “Ok so I can pre-register my ideas, but it will not fix ALL the problems!” Agreed. The goal of pre-registration, though, is not to fix all the problems with the academic research process. It can help solve the problem of Hypothesizing After Results are Known (HARKing), or stating in a manuscript that is written after all of the data analyses have been completed that the hypotheses put forward in that manuscript were crafted exactly as specified prior to collecting data and/or data analysis. Solving that problem would be a huge achievement.

The skeptic at this point often suggests that researchers could simply game the system by pre-registering their ideas and data analytic plans after looking at their data! If so, they benefit at the expense of honest, hard working scientists. Yes, researchers could do that. But if they did, they would be committing fraud, not very different from the likes of former successful scientists in our field that faked their data and subsequently lost their jobs. If we assume that outright fraud is rare in our field now, I think we can further assume that it will remain rare with respect to pre-registration fraud.

For the converted skeptic, this is where we discuss what tools are available to implement pre-registrations, and what information should be included in a pre-registration. I will save that discussion for another day, but there are some very useful resources available to assist with putting together useful and informative pre-registrations for all your research needs, such as:

 

Leave a Reply

Your email address will not be published. Required fields are marked *