Tag Archives: Pre-registration

A Goal of Preregistration: Evaluation

Leave a comment

This is a third instalment of my focus on preregistration of research projects in psychology. The first was an overview of my lecture for a graduate course I teach on open and reproducible science. The second focused on the who, what, when, where, and why’s of preregistration. This third post is a brief follow up and provides a visualization of the primary goal of preregistration–evaluation. We have endured the whole “preregistration is not a prison” debate, and now some are debating that preregistration is only applicable for a very specific and narrow type of research design.

In the visualization below I provide a definition of preregistration that is fairly common and widely accepted (see my second post linked above). I also state a goal of preregistration–evaluation. The sticking point for some is the question, “evaluation of what?” Your own answer to that question likely predicts with high accuracy what types of research projects you think “can” be preregistered, or “should” be preregistered. But as I briefly demonstrate below, there is more than 1 thing that is greatly aided in terms of evaluating research output/conclusions via the existence of a preregistration for that research. And yes I am “sub-blogging” (is that a thing?), because I use as an example secondary data analysis projects (that can and should be preregistered).

That is all for now peeps.

Week 6: Sharing Materials, Procedures, and Data Analytic Plans

Leave a comment

In this class we discussed the importance of sharing study materials, procedures, and as many decisions as possible regarding planned (and exploratory) analyses. Sharing materials and detailed procedures allows other researchers to reproduce the study in their own labs without needing to contact the original researcher(s). This is important because the original researcher(s) may no longer have access to old study materials (e.g., lost, changed jobs, old files thrown out, has left academics and did not bring materials/files with him/her), and eventually all researchers will die. Publicly sharing research materials and procedures helps ensure that your science does not die with you.

A few years ago my lab decided to conduct a close replication of an important study in the field of relationship science—study 3 of Murray et al. (2002). In this study, both partners of 67 heterosexual relationships participated in a laboratory experiment to see if individuals relatively high or low in self-esteem responded differently in the face of a relationship threat. Partners were seated in the same room, but with their backs to each other, to answer a number of paper and pencil questionnaires. The researchers manipulated relationship threat in half of the couples by leading one partner to believe that his or her partner perceived there to be many problems in the relationship (a very clever manipulation to be honest). The predicted two-way interaction between self-reported self-esteem and experimental condition emerged for some of the self-reported dependent variables, showing that in the “relationship threat” condition individuals with low self-esteem minimized the importance of the relationship whereas those with high self-esteem affirmed their love for their partners. After reading the paper closely to assemble the materials and scales needed to reproduce the study procedures, we realized that many of these scales were created by the original researchers and that we would need to ask the corresponding author (Dr. Sandra Murray) for copies or we could not run the planned replication study. Dr. Murray responded positively to our request, and luckily had copies of the study materials that she forwarded to us. We were able to run our replication study, and a pre-print of the article containing all of the study details and links to OSF (for materials, data, code) can be found here.

The moral of this story is that if we (or others of course) did not attempt to replicate this study, and if Dr. Murray no longer had access to these study materials, it simply would not have been possible for anyone else to closely reproduce the full set of procedures for this study going forward. It seems that this is the case for the majority of all published research in the field of psychology—the reported results remain in print, but the study materials and lab notes discussing how to, for example, properly employ manipulations are lost to future generations of researchers. But it does not have to be this way anymore.

The New Normal

Ideally, for every research project the researcher(s) should publicly share the following on a site such as the Open Science Framework (OSF) (go here to learn more about how to use the OSF to start sharing with the research community):

  • A list of all measures, with citations and web links where available, used in the study. List in order of presentation to participants, or discuss procedures used to randomize the order of presentation
  • A copy of all measures used in the study, except for copyrighted material. This is particularly important for measures created in-house given that the items are not readily available to others
  • Provide instructions for how to score all scales used in the study (e.g., indicate items to be reverse coded, to create scores for each participants calculate an average of all items on the scale, and so on)
  • Detailed description of any manipulations. Did you use single or double blinding?
  • Detailed description of the interactions between researchers and participants in lab settings
  • Description of the study design
  • Consider creating a flow-chart of the study design, discussing what happened at each stage of the study from beginning to end
  • Post pictures of the study setting (e.g., lab rooms) where appropriate
  • Consider creating a methods video (i.e., record a typical run of the experimental protocol with mock participants)

Sharing data analytic plans created prior to conducting analyses is also important to help clarify the difference between what was expected up front and what was assessed after getting started with the planned analyses. This plan should be included in the study pre-registration (see here for more information on pre-registration).

Here are some things to consider including in a data analytic plan:

  • If not included elsewhere, discuss the stopping rule to be used to terminate data collection (e.g., after recruiting a minimum of x number of participants data collection will cease)
  • Indicate rules for removing participants from the data set (e.g., failed attention checks, scores that are considered outliers based on some pre-determined criteria, participants that did not meet pre-determined inclusion and/or exclusion criteria)
  • Consider what types of descriptive data are important to present for your study (e.g., correlations, means, and standard deviations of study variables, frequency of responses on key variables)

Confirmatory Analyses

  • Define your alpha level (e.g., .05, .01, .005)
  • Do you plan to use any type of correction for your alpha level given the number of planned tests?
  • Given your hypotheses and the methods used to test your hypotheses, what types of statistical analyses are appropriate to use? Discuss what tests you plan to use, what variables will be used as predictor and outcome variables, and the critical effect(s) for each planned model.
  • What options will you consider if your planned models violate assumptions of that particular test?
  • How do you intend to conduct simple effects analyses (if applicable)?
  • Where appropriate, consider providing a figure of expected outcomes for each planned model
  • What type of effect size will you be reporting?
  • Will you present 95% confidence intervals of effects? Or some other measure of sensitivity?

 Exploratory Analyses

  • Can include many of the same elements relevant for confirmatory analyses (e.g., define your alpha, consideration of assumptions of tests to be used)
  • Provide a guiding framework for how exploratory analyses will be approached (an example of this can be found here)

Remember, sharing is caring.

Reference:

Murray, S. L., Rose, P., Bellavia, G., Holmes, J., & Kusche, A. (2002). When rejection stings: How self-esteem constrains relationship- enhancement processes. Journal of Personality and Social Psychology, 83, 556–573.

Week 4: All About Pre-Registration

Leave a comment

Presently there is a large degree of variability regarding the understanding and application of pre-registration in psychological science. From what I have seen on social media, read in papers and other scholarly work, and from reading actual pre-registrations from different labs, there is no agreed upon definition of pre-registration or guiding principles for when and how to implement a pre-registration. This is perhaps to be expected at such an early stage of adoption by academics not used to publicly sharing their ideas prior to testing them, with a non-trivial number of academics remaining skeptical of the practice. The goal of this week’s class was to introduce the students to a definition of pre-registration, to discuss some common “yes, but…” reasons for not pre-registering hypotheses, methods and data analytic plans, and to share some resources for how to implement a pre-registration.

Here is my working definition of pre-registration: Stating as clearly and specifically as possible what you plan to do, and how, before doing it, in a manner that is verifiable by others.

If you have an idea that you would like to test with new data or existing data, you can share your idea and plans for testing it before doing so. The alternative is not sharing this information before testing your idea, meaning it either (a) gets shared to some degree in a manuscript that is written after testing your idea, or (b) not shared because you chose not to write a manuscript that is written after testing your idea. In my opinion sharing before versus (maybe) after testing your idea is the better option. I therefore suggest that academics pre-register all of their planned research pursuits. In this post I will attempt to explain why.

There is no one correct way to implement a pre-registration, and a pre-registration itself is no guarantee that your hypotheses, methods, and/or data analytic approach were sound. Stating in a manuscript that your idea was pre-registered also does not imply the degree of specificity of your hypothesis, or that you followed your pre-registered protocol as specified. Importantly, however, these things are now verifiable by reading the pre-registration materials. It is worth taking the time to learn how to best communicate your intentions in advance of a given research pursuit, perhaps seeking feedback from other experts during this process, with the assumption that consumers of your research will take the time to read your pre-registration materials.

Common “Yes, but…” Arguments Against Pre-Registration

Here are four common “yes, but…” arguments I hear regarding why a given researcher cannot implement pre-registrations for his or her research:

  • It only applies when you have specific, confirmatory hypotheses. “My work is often theoretically guided but I do not always test specific, confirmatory hypotheses from the outset.”
  • It is simply not feasible or practical for complex study designs (e.g., longitudinal designs, large scale observational studies).
  • The data are already collected so (a) “I have nothing to pre-register”, and/or (b) “I have already analyzed some of the data so I can’t pre-register.”
  • It puts limits on what can be done with the data. “I may have some hypotheses and plans to test them, but as a scientist I need to go to where the data takes me and therefore do not want to be limited to only the analyses I could think of in advance. Pre-registration can stifle creativity and even scientific discovery.”

The short answer to each of these arguments is: nope. According to the definition of pre-registration I put forward, it is always to possible to state what you plan to do before you do it, as long as you are open and transparent about the state of the research pursuit in question. Here are some longer answers to these four arguments:

  • Pre-registration is not only for purely confirmatory research. It applies equally well for research that is largely exploratory, or somewhere in between exploratory and confirmatory. If, for example, you plan to collect self-report personality data from a large group of individuals and follow them over time to observe variability in different personality traits but you are not sure what that variability should look like, you can say that in a pre-registration. If you are not sure if the association between some theoretical constructs you assessed in your study should resemble patterns of mediation or moderation and want to test both, you can say that in a pre-registration. If you want to collect responses on many scales that may/may not be correlated with each other from a sample of students in an effort to select some of these scales for use in another study, you can say that in a pre-registration. Here is the pattern that is unfolding: after saying what you would like to do with your data collection and/or data analysis simply add “I can say that in a pre-registration”. Pre-registering vague ideas or exploratory research also helps prevent the researcher from using the words “As predicted…” in future publications using results from this research.
  • It is not necessary for a pre-registration to include every single possible hypothesis and accompanying data analytic plan for a given data set. If you first plan to analyze a subset of the data from, for example, a large sample of married couples assessed over two years (with data collected at 8 time points), you can pre-register those plans. If you decide to analyze a different set of ideas with different data collected from this sample, you can pre-register that at another time.
  • Data may already be collected, but it is still possible to state in advance your idea and how you plan to use existing data to test this idea. Be upfront with your prior experiences with this data set and how your new ideas were generated.
  • Pre-registration does not put limits on what you can do, but rather helps distinguish between analyses that were planned in advance from those that were conducted post-hoc (or between more confirmatory and exploratory analyses). Ideas and data analytic decisions that are made because of experiences working with the data (ideas and decisions you did not have prior to working with the data) are exploratory and should be labeled as such. Of course follow-up analyses are often needed and can lead to new, perhaps unexpected, patterns of results (that will need to be replicated at some point with independent data to properly test these hypotheses).

At this point in my conversations with those skeptical of pre-registration, they often say “Ok so I can pre-register my ideas, but it will not fix ALL the problems!” Agreed. The goal of pre-registration, though, is not to fix all the problems with the academic research process. It can help solve the problem of Hypothesizing After Results are Known (HARKing), or stating in a manuscript that is written after all of the data analyses have been completed that the hypotheses put forward in that manuscript were crafted exactly as specified prior to collecting data and/or data analysis. Solving that problem would be a huge achievement.

The skeptic at this point often suggests that researchers could simply game the system by pre-registering their ideas and data analytic plans after looking at their data! If so, they benefit at the expense of honest, hard working scientists. Yes, researchers could do that. But if they did, they would be committing fraud, not very different from the likes of former successful scientists in our field that faked their data and subsequently lost their jobs. If we assume that outright fraud is rare in our field now, I think we can further assume that it will remain rare with respect to pre-registration fraud.

For the converted skeptic, this is where we discuss what tools are available to implement pre-registrations, and what information should be included in a pre-registration. I will save that discussion for another day, but there are some very useful resources available to assist with putting together useful and informative pre-registrations for all your research needs, such as:

 

A Commitment to Better Research Practices (BRPs) in Psychological Science

Leave a comment

Scientific research is an attempt to identify a working truth about the world that is as independent of ideology as possible.  As we appear to be entering a time of heightened skepticism about the value of scientific information, we feel it is important to emphasize and foster research practices that enhance the integrity of scientific data and thus scientific information. We have therefore created a list of better research practices that we believe, if followed, would enhance the reproducibility and reliability of psychological science. The proposed methodological practices are applicable for exploratory or confirmatory research, and for observational or experimental methods.

  1. If testing a specific hypothesis, pre-register your research[1], so others can know that the forthcoming tests are informative. Report the planned analyses as confirmatory, and report any other analyses or any deviations from the planned analyses as exploratory.
  2. If conducting exploratory research, present it as exploratory. Then, document the research by posting materials, such as measures, procedures, and analytical code so future researchers can benefit from them. Also, make research expectations and plans in advance of analyses—little, if any, research is truly exploratory. State the goals and parameters of your study as clearly as possible before beginning data analysis.
  3. Consider data sharing options prior to data collection (e.g., complete a data management plan; include necessary language in the consent form), and make data and associated meta-data needed to reproduce results available to others, preferably in a trusted and stable repository. Note that this does not imply full public disclosure of all data. If there are reasons why data can’t be made available (e.g., containing clinically sensitive information), clarify that up-front and delineate the path available for others to acquire your data in order to reproduce your analyses.
  4. If some form of hypothesis testing is being used or an attempt is being made to accurately estimate an effect size, use power analysis to plan research before conducting it so that it is maximally informative.
  5. To the best of your ability maximize the power of your research to reach the power necessary to test the smallest effect size you are interested in testing (e.g., increase sample size, use within-subjects designs, use better, more precise measures, use stronger manipulations, etc.). Also, in order to increase the power of your research, consider collaborating with other labs, for example via StudySwap (https://osf.io/view/studyswap/). Be open to sharing existing data with other labs in order to pool data for a more robust study.
  6. If you find a result that you believe to be informative, make sure the result is robust. For smaller lab studies this means directly replicating your own work or, even better, having another lab replicate your finding, again via something like StudySwap.  For larger studies, this may mean finding highly similar data, archival or otherwise, to replicate results. When other large studies are known in advance, seek to pool data before analysis. If the samples are large enough, consider employing cross-validation techniques, such as splitting samples into random halves, to confirm results. For unique studies, checking robustness may mean testing multiple alternative models and/or statistical controls to see if the effect is robust to multiple alternative hypotheses, confounds, and analytical approaches.
  7. Avoid performing conceptual replications of your own research in the absence of evidence that the original result is robust and/or without pre-registering the study. A pre-registered direct replication is the best evidence that an original result is robust.
  8. Once some level of evidence has been achieved that the effect is robust (e.g., a successful direct replication), by all means do conceptual replications, as conceptual replications can provide important evidence for the generalizability of a finding and the robustness of a theory.
  9. To the extent possible, report null findings. In science, null news from reasonably powered studies is informative news.
  10. To the extent possible, report small effects. Given the uncertainty about the robustness of results across psychological science, we do not have a clear understanding of when effect sizes are “too small” to matter. As many effects previously thought to be large are small, be open to finding evidence of effects of many sizes, particularly under conditions of large N and sound measurement.
  11. When others are interested in replicating your work be cooperative if they ask for input. Of course, one of the benefits of pre-registration is that there may be less of a need to interact with those interested in replicating your work.
  12. If researchers fail to replicate your work continue to be cooperative. Even in an ideal world where all studies are appropriately powered, there will still be failures to replicate because of sampling variance alone. If the failed replication was done well and had high power to detect the effect, at least consider the possibility that your original result could be a false positive. Given this inevitability, and the possibility of true moderators of an effect, aspire to work with researchers who fail to find your effect so as to provide more data and information to the larger scientific community that is heavily invested in knowing what is true or not about your findings.

We should note that these proposed practices are complementary to other statements of commitment, such as the commitment to research transparency (http://www.researchtransparency.org/). We would also note that the proposed practices are aspirational.  Ideally, our field will adopt many, of not all of these practices.  But, we also understand that change is difficult and takes time.  In the interim, it would be ideal to reward any movement toward better research practices.

Brent W. Roberts, Rolf A. Zwaan, Lorne Campbell

[1] van ’t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12. doi:10.1016/j.jesp.2016.03.004