On September 12, 2016, experimental psychologist Christopher Ferguson created a “go-fund-me” page to raise funds for access to an existing data set that was used to advance scientific arguments in a scientific publication (link here). In Ferguson’s own words: “So I spoke with the Flourishing Families project staff who manage the dataset from which the study was published and which was authored by one of their scholars.  They agreed to send the data file, but require I cover the expenses for the data file preparation ($300/hour, $450 in total; you can see the invoice here).” Ferguson’s request has generated a lot of discussion on social media (this link as well), with many individuals disappointed that data used to support ideas put forward in a scientific publication are only available after a big fee is paid. Others feel a fee is warranted given the amount of effort required to put together the data requested into one file, as well as instructions regarding how to use the data file. And in the words of one commenter, “But I also know people who work with giant longitudinal datasets, and preparing just the codebook for one of those, in a way that will make sense to people outside the research team, can take weeks.” (highlighting added by me).

As someone that has collected data over time from large numbers of romantically involved couples, I agree that it would it take some time to prepare these data sets and codebooks for others to understand. But I think this is a shame really, and is a problem in need of a solution. If it takes me weeks to prepare documentation to explain my dataset organization to outsiders, I am guessing it would take the same amount of time to explain the same dataset organization to my future self (e.g., when running new analyses with an existing data set), or a new graduate student that wants to use the data to test new ideas, not to mention people outside of the lab. This seems highly inefficient for in-lab research activities, and represents the potential loss of valuable data to the field given that others may never have access to my data in the event that (a) I am too busy to spend weeks (or even hours for other data sets) putting everything together for others to make sense of my data, and (b) I die before I put these documents together (I am 43 with a love of red meat, so I could drop dead tomorrow. I think twice before buying green bananas).

So what is my proposed solution? Organize your data and code from the start with the assumption that you will need to share this information (see also “Why scientists must share their research code”). Create a data management plan at the beginning of all your research projects. Consider how the data will be organized, where it will be stored, and where the code for data cleaning/variable generation, analyses, and plots will be stored. Create meta-data (information about your dataset) along the way, updating as needed; consider where to store this meta-data from the beginning. If you follow these steps, your data, meta-data, and code can be available for sharing in a manner understandable to other competent researchers in a matter of minutes, not weeks. Even for complex data sets. Your future self will thank you. Your future graduate students will thank you. Your future colleagues will praise your foresight long after you are dead, as your [organized] data will live on.

Update: see Candice Morey’s post on the same topic.

The president of APS is nervous about pre-registration, or the idea of writing down study goals and hypotheses prior to collecting and/or analyzing data. One concern is that we do not have any data on whether or not pre-registration puts limits on exploration within research programs. If researchers are required to pre-register study goals and/or hypotheses, and given that in many instances good ideas are developed after seeing the data (not always before), then many good ideas may never be tested. This is of course a fair question worthy of discussion.*

But what we perhaps need to know first is approximately how much of our collective research is exploratory at present? We know that over 90% of all journal articles report statistically significant effects (no citation required), presumably for hypotheses developed prior to data collection. If so, then these data analyses have been presumably conducted in a confirmatory manner (i.e., to test hypotheses developed prior to data collection and/or analyses). Pre-registering these confirmatory hypotheses should therefore not be problematic or stifle discovery, particular given current options that make pre-registering hypotheses very easy (e.g., the Open Science Framework, aspredicted.org). If these confirmatory hypotheses took time to develop via exploratory research, then this suggests a massive amount of exploratory research is currently not being reported in any publication outlet; this research represents the large part of the iceberg hidden beneath public perception, with the small confirmatory bit of research peeking into public awareness. If so, we should collectively figure out a way to make this large body of exploratory research, and the details of how these explorations helped researchers develop their confirmatory hypotheses, publicly available. This is important stuff!**

To the extent the current literature, however, is not primarily presenting a priori hypotheses and confirmatory data analyses, then it will contain a blend of confirmatory hypotheses and hypotheses developed during and/or after data analyses (i.e., exploration within the research program). Given that over 90% of all journal articles report statistically significant effects, and that not all articles contain sections that clearly delineate confirmatory hypotheses and those developed from exploration with the data being presented, it is therefore an open question of how much research is confirmatory versus exploratory. Pre-registration of study goals and/or hypotheses, both confirmatory and exploratory (and everything in between), may be one way to answer this question. And perhaps before setting up large scale randomized control trials to determine if pre-registration can limit exploration, we should know just how much exploration is actually going on, as well as the links between this exploration and confirmatory hypotheses that are subsequently developed. Many of us seem to agree that exploration is very important, so let’s make an effort to document our explorations more clearly and openly.

* Russ Poldrack is on record as not being nervous about pre-registration

** “stuff” is a technical term of course

Here are 12 things that I** think are true regarding the future of how we do science based on meta-scientific discussions over the past few years and observing the varied developments in many different places (many that have sprung up as part of these discussions):

Ten years*** from now,

1. How we share the results of our scientific efforts will not look the same. The “article” format will very likely continue to be the primary vehicle for sharing research results, but the publishing and distribution of articles will take on many different forms (as is already beginning to occur).
2. Science will be more open. It seems highly unlikely that researchers will return to only sharing limited output of the research process, at the end of the research process, in print publications. Instead, it seems that mechanisms for sharing even more of the research process will continue to grow. And it will grow in ways not anticipated today.
3. Trust in the integrity and reproducibility of data will become more important than trust in the research, because data and code will be openly available for the vast majority of articles (perhaps reaching between 75-90%).
4. Researchers will be officially credited with what are today not yet considered contributions (e.g., shiny-apps, blogs, published data sets, etc.). Publications will always be important, but other types of contributions will be recognized by the scientific community and by University administrators.
5. Findings that are replicable (whether context specific results, or results obtained across many contexts) will be valued more than those that are “surprising”.
6. Statistics training in graduate programs will still be insufficient. But, statistics training will be viewed less as a necessary evil and simply necessary.
7. Computational science will become more popular and prevalent. Some computational approaches include the use of (a) machine learning, (b) simulation of complex models, (c) advanced statistical approaches, and (d) understanding large and complex data sets (e.g. big data).
8. When researchers call upon theory it will be more frequently to generate hypotheses rather than help explain an obtained pattern of results post-hoc. Hopefully this means that more theories start to be subjected to risky tests, and thus developed in a cumulative manner (but this second part is not a prediction, but rather a wish).
9. More articles will contain efforts to directly replicate initial results in a line of research. Direct replications will complement, not replace, conceptual replications.
10. More effort will be made to distinguish between confirmatory and exploratory analyses. Pre-registration of hypotheses and data analytic plans will be a big part of this effort.
11. There will be a growing use of within-person, compared to between-person, research designs. One reason for the growing popularity of these designs will be the realization of the greater statistical power and sensitivity of these designs.
12. Because of all of the above, a much larger proportion of published findings will be (demonstrably) replicable across independent labs. The “Many Labs 100”**** project will support this prediction.

It is my sincere wish to be around ten years from now to see the degree to which these things I think are true come to be. It will also be interesting to observe developments that were not on the horizon (or least on my horizon) today!

* sorry Wally Lamb

** not just me actually. Etienne LeBel (@eplebel; curatescience.org) contributed some novel ideas to this post. I also want to see him around ten years from now.

*** seemed like a good timeline

**** hard to predict, really, how many “Many Labs” projects will have been ushered into being in ten years

A few years ago I made changes to the syllabus for my graduate course on research methods in social psychology in response to what is widely refereed to as the “replicability crisis”. The biggest change was to remove the typical “research proposal” requirement that students completed individually and replace it with a hands-on group replication project (you can view the syllabus of the course here: https://osf.io/nxytf/, updated version from 2019 here https://osf.io/mt2va/). In 13 weeks (the length of the course), the group replication project proceeds as follows:

• Week 1: Groups of up to 4 students are established. Their first task is to identify social psychological research published within the past few years that (a) collected data online (e.g., via a University subject pool, or Amazon’s Mechanical Turk) to minimize time on data collection, and (b) is of interest to group members. Students typically share their research interests when introducing themselves to the class, and form groups with students that share similar interests (e.g., close relationships, stereotypes and prejudice, judgement and decision making).

• Week 2: as a class, we review the research identified by each group, paying close attention to study details (e.g., is it possible to reproduce the study methods and procedures from the methods section alone?; do descriptions of statistical models and results make sense?). Following this discussion, we select one published study for each group to closely replicate.

• Week 3: Each group writes a “replication report” that attempts to reproduce as best as possible the exact methods, procedures, and statistical models used in the study to be replicated. Groups make note of important study information that is not available in the published manuscript (e.g., instructions for participant recruitment, items for scales created by the authors, how covariates were coded and entered into statistical models). Next, groups draft an email to be sent to the corresponding author discussing the project. After I review and edit the email, the replication report is sent along with this email as well as questions asking for additional information as needed (example of email sent to corresponding authors: https://osf.io/q7ps8/; example of replication report: https://osf.io/nrkej/).

• Week 4: Groups edit the replication report as needed based on feedback from the corresponding author. For the 8 groups, across 3 classes, that have worked on these projects the corresponding authors have responded with very helpful feedback within days in every instance except 1 so far! Groups then submit a research ethics application, and wait (example ethics application: https://osf.io/bncq4/). It typically takes 4-6 weeks to obtain ethics approval.

• While awaiting ethics approval, groups prepare all study materials and procedures on Qualtrics. Groups are also encouraged to prepare data analytic code for the primary analyses. Shortly after receiving final ethics approval, groups pre-register study hypotheses on the Open Science Framework (OSF; e.g., https://osf.io/exnqu/). This typically occurs around week 10.

• Following pre-registration, data collection begins via Amazon’s Mechanical Turk (other options are available, but we have used Mturk to this point). Data is typically collected with 48 hours. To date I have funded data collection for each project.

• Each group analyzes their data, conducts follow-up analyses as needed, and prepares a written report to be submitted at the end of the course (week 13).

• After all reports are graded, and the course is officially over, I meet with each group to discuss the next steps for their projects. If needed we contact the corresponding author of the original study to ask for additional study information. In all instances we collect at least one additional large sample of participants to provide a more thorough test of hypotheses (in some instances we have collected three or four additional samples, adding additional conditions or sampling from a different population). When appropriate the results of the replication samples and original study are meta-analyzed (e.g., https://osf.io/y86zp/).

• I work with each group to prepare a manuscript for publication in a peer reviewed journal.
  • UPDATE (March, 2022): eight papers have now been published
    • Connors et al. (2016)
    • Balzarini et al. (2017)
    • Balakrishnan et al. (2017)
    • Babcock et al. (2017)
    • Clerke et al. (2018)
    • Seigel et al. (2020)
    • Koessler et al. (2021) (link to published version)
    • Jensen et al. (2021)

Lessons Learned by the Students:

1. It is not possible to completely reproduce the methods and procedures of the published studies selected to date without requiring additional information from corresponding authors. This surprises the students, and allows for in class discussions of how to make their own research more reproducible by others (e.g., posting study materials and procedures on the OSF).
2. The data analytic approach used in the original research is also often not described in enough detail to reproduce without asking questions of the corresponding author.
3. The students take great care to work with original authors in a respectful manner, and to reproduce the study design as closely as possible. This approach has resulted in the original researchers being very helpful and responsive.
4. When the results from their replication attempts (with large samples) are not similar to the published results the students are genuinely surprised.* This allows for a class discussion on the possible reasons for the inability to replicate published results. It also highlights to them the importance of replicating their own results, when feasible, prior to submitting manuscripts for publication.
5. The students learn that the methods and results sections of published papers are very, very valuable. For example, they now focus more attention on statistical power, if the paper includes a direct/close replication or not, and the effect size of key findings. They also pay more attention to the reproducibility of the methods and procedures as discussed in the paper. If a given study was low powered, not replicated, and yielded a large effect size, and furthermore if the methods and procedures as discussed are not reproducible, they are now more likely to question the conclusions of the authors as put forward in the discussion section.

Overall, the students develop what I feel is a healthy skepticism of results from one study (someone else’s study or their own). They learn the importance of statistical power first hand, and that published effect sizes may overestimate the true effect size. They also learn to value the importance of sharing study materials and procedures to increase the reproducibility of their own research. In my opinion, lectures and readings alone are not as effective at teaching these lessons compared to running an actual close replication of published research.

* Of the eight replication projects completed in my class to date, evidence consistent with the original published research has been obtained in 2 projects