In one word: evaluation.

The motto of the Royal Society is Nullius in verba, or “Take nobody’s word for it”. When evaluating scientific claims, this means focusing on the appropriateness of the research process (e.g., study design, analytic design, interpretation of outcomes, and so on) rather than accepting the claims based on the authority of the individual(s) proposing them. To evaluate the reported outcomes of a study, it is therefore critical to have access to details of the research process that produced those outcomes.

For many years now, these details have typically been shared in the print pages of academic journals. Academic journals typically have page budgets, or a contractually agreed upon number of print pages that can appear in a one year period. Longer individual papers therefore translate into fewer papers published in a given journal each year, meaning that Editors have often asked authors to shorten their papers during the review process. Given the complexity of many experimental designs, and the volume of information collected in many large scale observational studies, it is not feasible to include all methodological details or results within each paper. Authors have typically provided as much detail about how the study was conducted they felt necessary to interpret the results presented, with the promise that omitted details were available upon request if so desired.

But are these details really available upon request? There are many reasons to believe that at least some details would not be available upon request. For example, individuals may change careers and no longer have access to these materials; over time we all die and are not able to provide these details; issues with storage (of both physical and digital copies) can arise that render the details no longer accessible; people are busy and may not have the time to search for these details when asked. And the list goes on. As suggested by the readings for this week, it turns out that the majority of requested research details are not in fact available upon request! This includes access to data and analytic code, things that are important to be available given the non-trivial amount of statistical reporting errors in the literature. Nullius in verba implies that we should not take the word of researchers that say “available upon request”.

The paper by Simmons, Nelson, and Simonsohn (2011) in Psychological Science, as well as the interesting information presented on www.flexiblemeasures.com, also suggest that evaluation of scientific claims requires knowing what research decisions (e.g., hypotheses, use of measures, data analytic plans) were made prior to collecting data and/or data analyses versus after. For example, regarding the use of the Competitive Reaction Time Task (CRTT) in research assessing aggressive behavior, it has been shown that responses to this task have been quantified for data analysis in 156 different ways within 130 academic publications (http://www.flexiblemeasures.com/crtt/)!  It seems likely that some of the decisions regarding how to quantify the CRTT within these publications were made during the process of data analysis (i.e., using a data-driven approach to determine a quantification strategy that yielded “significant” results), but it is impossible to determine which ones in the absence of pre-registration of data analytic strategies. Simmons et al. (2011) also demonstrate how the application of different Questionable Research Practices (QRPs) (e.g., checking for significance, collect more data, check again; trying out a few different DVs; adding covariates and/or moderators) can inflate the Type I error rate well beyond the 5% level typically adopted by researchers using a frequentist data analytic approach. We need to know when decisions were made during the research process, and why, in order to properly evaluate the reported results of the research.

Current technology allows for sharing important details of the research process as the research is being considered, conducted, analyzed, and written up into a manuscript. We are no longer limited by how many print pages a traditional journal is budgeted to print each year and then mail to subscribers. We are also not limited with respect to when we share details of the research process—we no longer have to wait to publish everything we can squeeze into a manuscript.

This is an exciting time to be a scientist, but of course it is scary for some scientists to consider all of these changes occurring in a fairly short period of time. In my opinion, greater openness and transparency of the research process is the future of science. There is so much that is being done, and that can be done, to ensure our research is open and reproducible. Next week we start at the beginning of the research process.