The president of APS is nervous about pre-registration, or the idea of writing down study goals and hypotheses prior to collecting and/or analyzing data. One concern is that we do not have any data on whether or not pre-registration puts limits on exploration within research programs. If researchers are required to pre-register study goals and/or hypotheses, and given that in many instances good ideas are developed after seeing the data (not always before), then many good ideas may never be tested. This is of course a fair question worthy of discussion.*

But what we perhaps need to know first is approximately how much of our collective research is exploratory at present? We know that over 90% of all journal articles report statistically significant effects (no citation required), presumably for hypotheses developed prior to data collection. If so, then these data analyses have been presumably conducted in a confirmatory manner (i.e., to test hypotheses developed prior to data collection and/or analyses). Pre-registering these confirmatory hypotheses should therefore not be problematic or stifle discovery, particular given current options that make pre-registering hypotheses very easy (e.g., the Open Science Framework, aspredicted.org). If these confirmatory hypotheses took time to develop via exploratory research, then this suggests a massive amount of exploratory research is currently not being reported in any publication outlet; this research represents the large part of the iceberg hidden beneath public perception, with the small confirmatory bit of research peeking into public awareness. If so, we should collectively figure out a way to make this large body of exploratory research, and the details of how these explorations helped researchers develop their confirmatory hypotheses, publicly available. This is important stuff!**

To the extent the current literature, however, is not primarily presenting a priori hypotheses and confirmatory data analyses, then it will contain a blend of confirmatory hypotheses and hypotheses developed during and/or after data analyses (i.e., exploration within the research program). Given that over 90% of all journal articles report statistically significant effects, and that not all articles contain sections that clearly delineate confirmatory hypotheses and those developed from exploration with the data being presented, it is therefore an open question of how much research is confirmatory versus exploratory. Pre-registration of study goals and/or hypotheses, both confirmatory and exploratory (and everything in between), may be one way to answer this question. And perhaps before setting up large scale randomized control trials to determine if pre-registration can limit exploration, we should know just how much exploration is actually going on, as well as the links between this exploration and confirmatory hypotheses that are subsequently developed. Many of us seem to agree that exploration is very important, so let’s make an effort to document our explorations more clearly and openly.

* Russ Poldrack is on record as not being nervous about pre-registration

** “stuff” is a technical term of course