Tag Archives: open science

The 5Ws of Preregistration

Leave a comment

This post is a long awaited follow up to my February 2018 post titled “All About Pre-Registration”. I began that post with the following: “Presently there is a large degree of variability regarding the understanding and application of pre-registration in psychological science.” Six years later, there STILL is not consensus about many aspects of preregistration (yes, I am dropping the “-“). During this time I have personally found it a little odd how there has been so many opinions expressed in published papers and particularly social media about preregistration yet the people making these comments likely share divergent views *of* preregistration. Some of the arguments against preregistration that make me shake my head the most are that “it is not a panacea” (so the bar for introducing practices to potentially improve our science is that it has to be a solution for ALL problems in our field?), “it stifles creativity and exploration” (well, maybe for you…), “I cannot be expected to know all my hypotheses with my big study that has longitudinal components” (totally get it, but when I read your papers you often say “as expected” or “as predicted”, so…), “preregistration is only for purely confirmatory research” (ok, but what is your definition of preregistration because it is not one I am familiar with), and “people deviate from them so preregistration does not work” (work for what exactly? And let’s forget for a moment that the only way to obtain information regarding deviations between research intentions as stated in a preregistration and actions as written about in a published article are because the preregistration exists). But I also hear arguments from proponents of preregistration say things like “preregistration is not needed for that type of research” (for now of course preregistration is not something that seems to be “needed” to publish research papers in general in that it is not universally mandatory, but this statement implies there is only value for preregistering certain kinds of research activities as well as certain types of information regarding these research activities).

Given the lack of general agreement about the nature and goals of preregistration, I have put together in this post answers to the 5Ws of preregistration: Who? What? Where? When? and Why? The answers are obviously not agreed upon by everyone (see the first paragraph of this post), and they are based on my own experiences with preregistration and other open science practices the past ten years. That includes working with my own lab to generate preregistrations for our research that ranges from primarily confirmatory to primarily exploratory (and everything in between), discussing the practice with colleagues when invited to present on the topic at conferences as well as invited departmental talks, reading social media posts way to often, and publishing papers on the topic (e.g., Campbell, Loving, LeBel, 2014; LeBel, E.P., Campbell, L., & Loving, T.J., 2017; Moshontz, H., Campbell, L., … Chartier, C.R., 2018; Nosek, B.A., Beck, E., Campbell, L., Flake, J.K., Harwicke, T.E., Mellor, D.T., van’t Veer, A.E., & Vazire, S., 2019).

To begin, I put together a brief cheat sheet of the five Ws of preregistration and some brief answers below:

The 5Ws of Preregistration:

Who? – Any individual or group of researchers that evaluate ideas via the collection and analysis of data.

What? – Preregistration is both a concept as well a concrete action, or actions. A working definition of preregistration that is generally agreed upon by those tasked with engaging in the practice is therefore needed to encourage progress on standardizing the practice of preregistration. An agreed upon definition and subsequent translation of this definition to a standard set of practices regarding preregistration currently does not exist.  

When? – Typically preregistration should occur prior to the proposed actions that are specified in the preregistration are carried out.

Where? – Preregistration information should be timestamped and publicly available in perpetuity without the option to delete shared information. Any updates or additions to a preregistration should be transparent to anyone viewing the preregistration information.

Why? – The goal, or goals, motivating the practice of preregistration. There is currently a great deal of disagreement regarding the purpose of preregistration, meaning it is possible for one person to declare that preregistration “works” and another to declare that it “does not work”, with each statement being correct given the different inferred goals of the practice. This is not a desirable situation.

In the next sections, I provide expanded answers to the five Ws of preregistration (this is the first draft of this post on January 30, 2024, and I will likely add to it and lightly edit).

Who?

In my field of psychology, people typically develop and advance their careers by generating research questions and/or hypotheses that can be assessed by designing research studies and comparing study outcomes with expectations. Results that contrast with expectations can be very useful to generate novel insights. A researcher may also evaluate data in the absence of expectations or with vague expectations, developing new research ideas based on the pattern of observations across the set of available variables in a given data set. Overall, any researcher that obtains data for the purpose of discovery has the option to share up front in an open and transparent manner their research intentions and planned actions.

What?

In January of 2024 I conducted an informal internet search for definitions of preregistration. The top hits of my search appear below.

APA.org: “Preregistration allows researchers to specify and share details of their research in a public registry before conducting the study.”

COS.io: “When you preregister your research, you’re simply specifying your research plan in advance of your study and submitting it to a registry. Preregistration separates hypothesis-generating (exploratory) from hypothesis-testing (confirmatory) research.”

“What is pre-registration? Pre-registration involves making hypotheses, analytic plans, and any other relevant methodological information for a study publicly available before collecting data.”

COS from May 13, 2023: Preregistration is the practice of documenting your research plan at the beginning of your study and storing that plan in a read-only public repository such as OSF Registries or the National Library of Medicine’s Clinical Trials Registry.

Surrey.ac.uk: “The goal is to create a transparent plan, ahead of beginning your study/accessing existing data. So, your preregistration will include details about the study, hypotheses (if you have them), your design, sampling plan, search strategy (for reviews) variables, exclusion and inclusion criteria, and the analysis plan.”

The Administration for Children and Families (USA): “Pre-registration is the practice of deciding your research and analysis plan prior to starting your study and sharing it publicly, like submitting it to a registry.”

PLOS.org: Preregistration is the practice of formally depositing a study design in a repository—and, optionally, submitting it for peer review at a journal—before conducting a scientific investigation.

Common Themes: Publicly sharing details of planned research projects in an appropriate registry prior to carrying out the planned research actions.

My working definition of preregistration:

Stating as clearly and specifically as possible what you plan to do, and how, before doing it, in a manner that is verifiable by others (From: https://www.lornecampbell.org/?p=181)

All of the above definitions of preregistration are consistent in that they refer to publicly sharing intentions prior to actions. Where many differences of opinion occur is not with these somewhat vague definitions, but instead with the translation from the conceptual (“create and publicly share research plan prior to carrying out research plan”) to the concrete (from “you can share plans for all of your research” to “sharing only needed and/or useful for specific data analytic decisions for confirmatory hypotheses and nothing else”). It seems to me, therefore, that there is actually some general agreement regarding the higher order definition of preregistration, but important differences between researchers on the translation of this definition for different types of research.

In the figure below, I refer to research intentions and actions. With respect to intentions, I roughly dissect research projects into explorations and hypothesis testing research. Explorations loosely include purely descriptive research (e.g., assessing base rates of behavior in a given population) as well as “fishing” (e.g., calculating correlations between study variables, running a lot of models that include different variables or same variables with different combinations of items, trimming the sample, and so on). Hypothesis testing includes evaluating vague hypotheses (e.g., two variables should be positively correlated, the mean for group A should be higher than group B) as well as very concrete hypotheses (e.g., using data obtained via specific measures the correlation between two variables should be in a given range, or the mean for Groups A and B should be in a given range and differ by more than 1 unit). These categories are simply meant to cover research that at one end has very limited expectations regarding outcomes to research that has very specific expectations regarding outcomes. With respect to actions, I list four categories of the types of concrete information regarding a researcher’s intentions that can be publicly shared. What is obvious from this conceptualization of the research process as it relates to preregistration is that it is primarily the degree of specificity of the information shared (actions) that changes as a function of the research intentions, NOT whether information should be shared or not shared. In other words, you can preregister all of your research projects, with some preregistrations including a lot of very specific information compared to others. If at this point you disagree with what I just said regarding preregistration, it is very likely we have different views on the goal(s) of preregistration. So keep reading until the end.

When?

Prior to doing what you plan to do (will be updated with more information)

Where?

OSF, aspredicted.org, git/github (will be updated with more information)

Why?

            Overall, what is the desired outcome, or outcomes, associated with the practice of preregistration? Ask people that believe in the value or preregistration, and those that do not, this question and I predict that you will get a variety of responses. Whereas researchers may have general agreement about a higher order definition of preregistration, there are important differences of opinion with respect what types of research should be preregistered and I believe this is largely because of disparate beliefs about the goal(s) of preregistration. What does it help with?

Before moving on, when researchers ask this type of question it sounds like this in my head: “If you want me to share my research intentions with you before I have done said research then you need to prove to me that there is some value in doing so or else I declare it is all a waste of time.” Here is something we wrote in a paper published in 2014: “Ideally we should not need to persuade researchers of the benefits of disclosing details of the research process; instead, researchers should need to provide solid rationale for not openly sharing these details.” (Campbell, Loving, & LeBel, 2014, p. 542). I suppose what I am about to say now is somewhat controversial, but consistent with the 2014 version of myself I believe researchers should preregister by default unless there is solid rationale, that needs to be shared, for not doing so.

Primary Goal of Preregistration: The open and transparently shared decision making process, from intentions to proposed actions, for a research endeavour to allow for enhanced evaluation.

The availability of this information should allow for achieving many sub-goals (when relevant to a particular research endeavour), such as:

  • Defining research questions and/or pre-planned hypotheses as formed by the researcher(s) prior to collecting data and/or examining existing data
    • Often described as delineating exploratory from confirmatory tests of research questions and/or hypotheses
      • Whereas some explorations can be planned in advance (e.g., obtaining base rate information for some outcomes in a given group or groups; wanting to see differences in a variety of measures between particular groups; estimating correlations among responses to particular measures), other explorations occur during the process of examining the data obtained to assess the research questions and/or hypotheses
  • Detailing with appropriate specificity the research methods the researcher(s) plan to use to evaluate the research questions and/or hypotheses, including:
    • Target population and planned sample
    • Rationale for sample size
    • The planned information to be obtained and how the researcher(s) will obtain this information (e.g., want to obtain the age of participants in the sample, so will ask study participants to indicate their age in years and months or ask study participants to indicate their date of birth; want to assess self-esteem so will ask participants to complete a particular self-esteem scale or scales)
    • Any planned manipulation in the study design, described in appropriate detail to allow others to implement the manipulation in a like manner
    • The planned procedures for conducting all aspects of the study
    • Any data exclusion rules
    • Plans for how data will be analyzed in specific detail as appropriate given the nature of the research questions and/or hypotheses as well as the methods used to obtain the data

Secondary Goals of Preregistration: Preregistration is believed by some advocates to help increase the quality of published research reports in the following ways:

  • Eliminate HARKing, or hypothesizing after the results are known. In other words, viewing the results from a given set of analyses and then tailoring hypotheses to match the results obtained
  • Eliminate p-hacking, or conducting many statistical tests in many different ways in order to obtain a p value that is lower than the traditionally accepted cut off of .05 and then presenting this result (or results) as the only test that was conducted to test the hypothesis
  • Eliminate outcome switching, or stating that a given result was the primary test planned all along when in fact it was not
  • Enhance the severity of a statistical test of a concrete hypothesis by following the pre-specified data analytic plan
  • Enhance the reproducibility of the study methods and procedures, as well as the data analyses. Each type of reproducibility can assist with (a) the evaluation of study claims, as well as (b) re-running the study to determine the replicability of the findings in another sample of participants

Any preregistration, even one that is poorly constructed, helps achieve the primary goal of being open and transparent regarding one’s intentions and proposed actions. Some preregistrations may assist with achieving the secondary goals, whereas some may not. But being open and transparent with your research intentions and proposed actions in a preregistration achieves the primary goal of preregistration. So do it.

References

Campbell, L., Loving, T.J., & LeBel, E.P. (2014). Enhancing transparency of the research process to increase accuracy of findings: A guide for relationship researchers. Personal Relationships, 21, 531-545. DOI: 10.1111/pere.12053

LeBel, E.P., Campbell, L., & Loving, T.J. (2017). Benefits of open and high-powered research outweigh costs. Journal of Personality and Social Psychology, 113, 230-243. DOI: 10.1037/pspi0000049

Moshontz, H., Campbell, L., … Chartier, C.R. (2018). The psychological science accelerator: Advancing psychology through a distributed collaborative network. Advances in Methods and Practices in Psychology Science, 1, 510-515. DOI: 10.1177/2515245918797607

Nosek, B.A., Beck, E., Campbell, L., Flake, J.K., Harwicke, T.E., Mellor, D.T., van’t Veer, A.E., & Vazire, S. (2019). Preregistration is hard, and worthwhile. Trends in Cognitive Science. DOI: 10.1016/j.tics.2019.07.009

Psychology Grad Course: Open and Reproducible Science

Leave a comment

I have written several posts about the open and reproducible graduate course for psychology that I created and have taught a few times. This post is simply to pull together all relevant links to one location. Below I link to two versions of the syllabus as well as to my lecture summaries for each class. The lecture summaries do not reflect the only material discussed in each class, but they do highlight the main theme for each class.

Syllabus:

  • Syllabus from 2018 (first time teaching this course)
  • Syllabus from 2020

Lecture Summaries (based on 2018 syllabus):

  • Week 1: Introduction to the course
  • Week 2: Why Should Science be Open and Reproducible?
  • Week 3: Open Notebook
  • Week 4: All About Pre-Registration
  • Week 5: Power and Study Design
  • Week 6: Sharing Procedures, Materials, and Data Analytic Plans
  • Week 7: Data Management Plans
  • Weeks 8 and 9: Sharing Data as well as Code/Syntax
  • Week 10: Openly Sharing Research Reports/Manuscripts
  • Week 11: Extended Research Networks
  • Week 12: Transforming Discovery

Week 12: Transforming Discovery

Leave a comment

In this final week of the class I discussed the idea of transforming the process of discovery in our own labs. When I started my undergraduate program in 1992 I learned about the discoveries of others in my courses and textbooks. When I started my graduate program in 1996 I also started to implement what I had learned about the process of discovery with my own research. During this time I often asked myself, and others, “what is typically done by other researchers in this area of research?” I wanted to make sure I was doing what was considered acceptable at the time. For example, using a particular measure of attachment orientations because it seems to be used a lot and thus would not be questioned by reviewers. Or using a particular analytic method because it was frequently used and thus seemingly defensible during the review process. And when it came time to analyze data I learned to run a lot of different models with a number of different combinations of responses across multiple measures to find results that were consistent with our original expectations. Then when it came to write manuscripts, learning what results to not only include but also exclude. It simply felt to me that “this is how science is done” in our field, so I did it that way.

But really it did not have to done in the manner briefly described above. I was searching for what I thought was the best way to be a scientist and in the process also discovered the existing norms for how to be a scientist. Sometimes existing norms overlap with what is best for scientific discovery and dissemination, but sometimes they don’t. I don’t claim to know exactly how scientists should go about making their discoveries, but I do feel strongly that when we feel confident enough to share our results publicly we need to also share to the best of our abilities at the time how exactly we obtained those results. For me that is like a latent construct of open science practices–be open and transparent–that influences what I do throughout the research process. In the early days of the open science movement, I felt that actions could speak just as loud as all the words that were flying around. When others were wondering out loud if sharing details of the research process was worth it, might be costly to the researcher, etc., I felt that I could simply point to our own experiences as examples of how it could be done. Yes, you can preregister hypotheses and *still* conduct analyses not planned in advance. Yes, you can even preregister exploratory research and their is value in doing so. Yes, you can share the measures you used in your study. Yes, you can be open and transparent with longitudinal research designs. Yes, you can share data all the time (sometimes publicly, sometimes via other means). Yes, you can share the syntax you used to produce the results that your presented in your manuscript. Yes, you can preprint your work. And on and on. Debates are fun and all, but when many researchers pondered whether they could/should do these things, we simply did them. One of my hopes was that when new graduate students were learning about the process of discovery, they might stumble across some of our open and transparent research practices and think it was something they could do, something that was becoming normative in the field. Whereas some colleagues saw open science practices as warning flags for the process of discovery, I encouraged my students to see them as challenges for which we have the opportunity to develop solutions to our own process of discovery. With the existence of today’s technologies there are numerous ways to share our research process and make it available for scrutiny, and no solid arguments for keeping this process “available upon request”. That is also the reason why I wanted to teach a course on open and reproducible science. I wanted to do what I could to share these tools with these early career researchers in hopes that they would see value in adopting them in their own research.

My final take home message here: when sharing the results of your research also share how you obtained those results as openly and transparently as possible.

I feel relieved to finally finish this series of posts to accompany the weekly lectures for my course on open and reproducible science. It seemed obvious to me that being open and transparent could also apply to the courses we teach, meaning we could share syllabi and course notes. This series of posts serves as my own personal lecture notes for each class. If you have read them I hope they have been of some value.

Week 6: Sharing Materials, Procedures, and Data Analytic Plans

Leave a comment

In this class we discussed the importance of sharing study materials, procedures, and as many decisions as possible regarding planned (and exploratory) analyses. Sharing materials and detailed procedures allows other researchers to reproduce the study in their own labs without needing to contact the original researcher(s). This is important because the original researcher(s) may no longer have access to old study materials (e.g., lost, changed jobs, old files thrown out, has left academics and did not bring materials/files with him/her), and eventually all researchers will die. Publicly sharing research materials and procedures helps ensure that your science does not die with you.

A few years ago my lab decided to conduct a close replication of an important study in the field of relationship science—study 3 of Murray et al. (2002). In this study, both partners of 67 heterosexual relationships participated in a laboratory experiment to see if individuals relatively high or low in self-esteem responded differently in the face of a relationship threat. Partners were seated in the same room, but with their backs to each other, to answer a number of paper and pencil questionnaires. The researchers manipulated relationship threat in half of the couples by leading one partner to believe that his or her partner perceived there to be many problems in the relationship (a very clever manipulation to be honest). The predicted two-way interaction between self-reported self-esteem and experimental condition emerged for some of the self-reported dependent variables, showing that in the “relationship threat” condition individuals with low self-esteem minimized the importance of the relationship whereas those with high self-esteem affirmed their love for their partners. After reading the paper closely to assemble the materials and scales needed to reproduce the study procedures, we realized that many of these scales were created by the original researchers and that we would need to ask the corresponding author (Dr. Sandra Murray) for copies or we could not run the planned replication study. Dr. Murray responded positively to our request, and luckily had copies of the study materials that she forwarded to us. We were able to run our replication study, and a pre-print of the article containing all of the study details and links to OSF (for materials, data, code) can be found here.

The moral of this story is that if we (or others of course) did not attempt to replicate this study, and if Dr. Murray no longer had access to these study materials, it simply would not have been possible for anyone else to closely reproduce the full set of procedures for this study going forward. It seems that this is the case for the majority of all published research in the field of psychology—the reported results remain in print, but the study materials and lab notes discussing how to, for example, properly employ manipulations are lost to future generations of researchers. But it does not have to be this way anymore.

The New Normal

Ideally, for every research project the researcher(s) should publicly share the following on a site such as the Open Science Framework (OSF) (go here to learn more about how to use the OSF to start sharing with the research community):

  • A list of all measures, with citations and web links where available, used in the study. List in order of presentation to participants, or discuss procedures used to randomize the order of presentation
  • A copy of all measures used in the study, except for copyrighted material. This is particularly important for measures created in-house given that the items are not readily available to others
  • Provide instructions for how to score all scales used in the study (e.g., indicate items to be reverse coded, to create scores for each participants calculate an average of all items on the scale, and so on)
  • Detailed description of any manipulations. Did you use single or double blinding?
  • Detailed description of the interactions between researchers and participants in lab settings
  • Description of the study design
  • Consider creating a flow-chart of the study design, discussing what happened at each stage of the study from beginning to end
  • Post pictures of the study setting (e.g., lab rooms) where appropriate
  • Consider creating a methods video (i.e., record a typical run of the experimental protocol with mock participants)

Sharing data analytic plans created prior to conducting analyses is also important to help clarify the difference between what was expected up front and what was assessed after getting started with the planned analyses. This plan should be included in the study pre-registration (see here for more information on pre-registration).

Here are some things to consider including in a data analytic plan:

  • If not included elsewhere, discuss the stopping rule to be used to terminate data collection (e.g., after recruiting a minimum of x number of participants data collection will cease)
  • Indicate rules for removing participants from the data set (e.g., failed attention checks, scores that are considered outliers based on some pre-determined criteria, participants that did not meet pre-determined inclusion and/or exclusion criteria)
  • Consider what types of descriptive data are important to present for your study (e.g., correlations, means, and standard deviations of study variables, frequency of responses on key variables)

Confirmatory Analyses

  • Define your alpha level (e.g., .05, .01, .005)
  • Do you plan to use any type of correction for your alpha level given the number of planned tests?
  • Given your hypotheses and the methods used to test your hypotheses, what types of statistical analyses are appropriate to use? Discuss what tests you plan to use, what variables will be used as predictor and outcome variables, and the critical effect(s) for each planned model.
  • What options will you consider if your planned models violate assumptions of that particular test?
  • How do you intend to conduct simple effects analyses (if applicable)?
  • Where appropriate, consider providing a figure of expected outcomes for each planned model
  • What type of effect size will you be reporting?
  • Will you present 95% confidence intervals of effects? Or some other measure of sensitivity?

 Exploratory Analyses

  • Can include many of the same elements relevant for confirmatory analyses (e.g., define your alpha, consideration of assumptions of tests to be used)
  • Provide a guiding framework for how exploratory analyses will be approached (an example of this can be found here)

Remember, sharing is caring.

Reference:

Murray, S. L., Rose, P., Bellavia, G., Holmes, J., & Kusche, A. (2002). When rejection stings: How self-esteem constrains relationship- enhancement processes. Journal of Personality and Social Psychology, 83, 556–573.

A Commitment to Better Research Practices (BRPs) in Psychological Science

Leave a comment

Scientific research is an attempt to identify a working truth about the world that is as independent of ideology as possible.  As we appear to be entering a time of heightened skepticism about the value of scientific information, we feel it is important to emphasize and foster research practices that enhance the integrity of scientific data and thus scientific information. We have therefore created a list of better research practices that we believe, if followed, would enhance the reproducibility and reliability of psychological science. The proposed methodological practices are applicable for exploratory or confirmatory research, and for observational or experimental methods.

  1. If testing a specific hypothesis, pre-register your research[1], so others can know that the forthcoming tests are informative. Report the planned analyses as confirmatory, and report any other analyses or any deviations from the planned analyses as exploratory.
  2. If conducting exploratory research, present it as exploratory. Then, document the research by posting materials, such as measures, procedures, and analytical code so future researchers can benefit from them. Also, make research expectations and plans in advance of analyses—little, if any, research is truly exploratory. State the goals and parameters of your study as clearly as possible before beginning data analysis.
  3. Consider data sharing options prior to data collection (e.g., complete a data management plan; include necessary language in the consent form), and make data and associated meta-data needed to reproduce results available to others, preferably in a trusted and stable repository. Note that this does not imply full public disclosure of all data. If there are reasons why data can’t be made available (e.g., containing clinically sensitive information), clarify that up-front and delineate the path available for others to acquire your data in order to reproduce your analyses.
  4. If some form of hypothesis testing is being used or an attempt is being made to accurately estimate an effect size, use power analysis to plan research before conducting it so that it is maximally informative.
  5. To the best of your ability maximize the power of your research to reach the power necessary to test the smallest effect size you are interested in testing (e.g., increase sample size, use within-subjects designs, use better, more precise measures, use stronger manipulations, etc.). Also, in order to increase the power of your research, consider collaborating with other labs, for example via StudySwap (https://osf.io/view/studyswap/). Be open to sharing existing data with other labs in order to pool data for a more robust study.
  6. If you find a result that you believe to be informative, make sure the result is robust. For smaller lab studies this means directly replicating your own work or, even better, having another lab replicate your finding, again via something like StudySwap.  For larger studies, this may mean finding highly similar data, archival or otherwise, to replicate results. When other large studies are known in advance, seek to pool data before analysis. If the samples are large enough, consider employing cross-validation techniques, such as splitting samples into random halves, to confirm results. For unique studies, checking robustness may mean testing multiple alternative models and/or statistical controls to see if the effect is robust to multiple alternative hypotheses, confounds, and analytical approaches.
  7. Avoid performing conceptual replications of your own research in the absence of evidence that the original result is robust and/or without pre-registering the study. A pre-registered direct replication is the best evidence that an original result is robust.
  8. Once some level of evidence has been achieved that the effect is robust (e.g., a successful direct replication), by all means do conceptual replications, as conceptual replications can provide important evidence for the generalizability of a finding and the robustness of a theory.
  9. To the extent possible, report null findings. In science, null news from reasonably powered studies is informative news.
  10. To the extent possible, report small effects. Given the uncertainty about the robustness of results across psychological science, we do not have a clear understanding of when effect sizes are “too small” to matter. As many effects previously thought to be large are small, be open to finding evidence of effects of many sizes, particularly under conditions of large N and sound measurement.
  11. When others are interested in replicating your work be cooperative if they ask for input. Of course, one of the benefits of pre-registration is that there may be less of a need to interact with those interested in replicating your work.
  12. If researchers fail to replicate your work continue to be cooperative. Even in an ideal world where all studies are appropriately powered, there will still be failures to replicate because of sampling variance alone. If the failed replication was done well and had high power to detect the effect, at least consider the possibility that your original result could be a false positive. Given this inevitability, and the possibility of true moderators of an effect, aspire to work with researchers who fail to find your effect so as to provide more data and information to the larger scientific community that is heavily invested in knowing what is true or not about your findings.

We should note that these proposed practices are complementary to other statements of commitment, such as the commitment to research transparency (http://www.researchtransparency.org/). We would also note that the proposed practices are aspirational.  Ideally, our field will adopt many, of not all of these practices.  But, we also understand that change is difficult and takes time.  In the interim, it would be ideal to reward any movement toward better research practices.

Brent W. Roberts, Rolf A. Zwaan, Lorne Campbell

[1] van ’t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12. doi:10.1016/j.jesp.2016.03.004

My 2016 Open Science Tour

Leave a comment

I have been asked to discuss my views on open science and replication, particularly in my field of social psychology, nine times in 2016 (see my “Open Science Tour” dates below). During these talks, and in discussions that followed, people wanted to know what exactly is open science, and how might a researcher go about employing open science practices?

Overall, many similar questions were asked of me from faculty and students so I thought I would create a list of these frequently asked questions. I do not provide a summary of my responses to these questions, instead wanting readers to consider how they would respond. So, how would you answer these questions? (public google doc for posting answers)

  1. Given that many findings are not, and in many cases cannot, be predicted in advance, how can I pre-register my hypotheses?
  2. If my research is not confirmatory, do I need to use open science practices? Isn’t open science only “needed” when very clear hypotheses are being tested?
  3. How can I share data?
    • What data do I “need” to share? (All of it? Raw data? Aggregated data?)
    • What platforms are available for data sharing? (and what is the “best” one?)
    • What format/software should be used?
    • Is this really necessary?
    • How should I present this to my research ethics board?
  4. Can I publicly share materials that are copyrighted?
  5. What is a data analytic plan?
  6. Is it really important to share code/syntax from my analyses?
  7. Can’t researchers simply “game the system”? That is, conduct research first, then pre-register after results are known (PRARKing), and submit for publication?
  8. Can shared data, or even methods/procedures, be treated as unique “citable units”?
  9. If I pilot test a procedure in order to obtain the desired effects, should the “failed” pilot studies be reported?
    • If so won’t this bias the literature by diluting the evidence in favor of the desired/predicted effect obtained in later studies?
  10. How much importance should I place on statistical power?
    • Given that effect sizes are not necessarily knowable in advance, and straightforward procedures are not available for more complex designs, is it reasonable to expect a power analysis for every study/every analysis?
  11. If I use open science practices but others do not, can they benefit more in terms of publishing more papers because of fewer “restrictions” on them?
    • If yes, how is this fair?

Unique question from students:

  1. Could adopting open science practices result in fewer publications?
  2. Might hiring committees be biased against applicants that are pro open science?
  3. If a student wants to engage in open science practices, but his/her advisor is against this, what should this student do?
  4. If a student wants to publish studies with null findings, but his/her advisor is against this, what should this student do?
  5. Will I “need” to start engaging in open science practices soon?
  6. Will it look good, or bad, to have a replication study (studies) on my CV?
  7. What is the web address for the open science framework? How do I get started?

My Open Science tour dates in 2016 (links to slides provided):

  • January 28, Pre-Conference of the Society of Personality and Social Psychology (SPSP), San Diego, USA
  • June 10, Conference of the Canadian Psychological Association, Victoria, Canada
  • October 3, York University (Psychology), Canada (audio recording)
  • October 11, University of Toronto (Psychology), Canada
  • October 19, University of Guelph (Family Relations and Applied Nutrition), Canada
  • October 21, Illinois State University, (Psychology), USA
  • November 11, Victoria University Wellington (Psychology), New Zealand
  • November 24, University of Western Ontario (Clinical Area), Canada
  • December 2, University of Western Ontario (Developmental Area), Canada

How to Publish an Open Access Edited Volume on the Open Science Framework (OSF)

Leave a comment

Edited volumes are collections of chapters on a particular topic by various experts. In my own experience as a co-editor of three (3) edited volumes, the editors select the topic, select and invite the experts (or authors), and identify a publisher. Once secured, a publisher typically offers a cash advance to the editor(s) along with a small percentage of sales going forward in the form of royalties. The publisher may also provide reviewing services for the collection of chapters, and will advertise the edited volume when it is released. The two primary ways for consumers to access the chapters is to (a) purchase the book, or (b) obtain a copy of the book from a library.

With technological advances it is now possible to publish edited volumes without a professional publishing company. Why would someone choose to not use a publishing company? Indeed, they are literally publication experts. Perhaps the biggest reason is that the resulting volume will be open access, or available to anyone with a connection to the internet, free of charge. There are also some career advantages to sharing knowledge open access. Also, a publishing company is simply not needed for all publication projects.

There are very likely many different ways to publish an edited volume without using a professional publishing company. Below, I outline one possibility that involves using the Open Science Framework (OSF). Suggestions for improving these suggested steps are welcome.

Steps to Using the OSF to publish an Open Access Edited Volume

  1. Identify a topic for the edited volume, and then identify a list of experts that you would like to invite to contribute chapters.
  2. If you do not have an OSF account, create one (it is free). Create a new project page for your edited volume, and give it the title of the proposed edited volume. Select one of the licensing options for your project to grant copyright permission for this work.
  3. Draft a proposal for your edited volume (e.g., the need for this particular collection of chapters, goals of the volume, target audience, and so on). Add this file to the project page.
  4. Send an email inviting potential authors, providing a link to your OSF project page so they can read your proposal.
    • You can make the project page public from the start and simply share the link, or,
    • You can keep the project page private during the development of the edited volume and “share” a read-only link to the project page with prospective authors only.
  5. Ask all authors that accepted the invitation to create on OSF account. Then create a component for each individual chapter; components are part of the parent project, but are treated as independent entities in the OSF. Use the proposed title for each chapter as the title of the component. Add the author(s) as administrators for the relevant component (e.g., A. Smith has agreed to author chapter #4; add A. Smith as an administrator of component #4).
  6. Ask authors to upload a copy of their first draft by the selected deadline. Provide feedback on every chapter.
    • One option is to download a copy of the chapter, make edits using the track changes option, and then upload a copy of the edited chapter using the same title as the original in order to take advantage of the “version control” function of the OSF (i.e., all versions of the chapter will be available on the project page in chronological order, with the most recent version at the top of the list).
  7. Ask authors to upload their revised chapter using the same title (again to take advantage of the “version control” function of the OSF).
  8. When the chapters are completed, “register” the project and all components. This will “freeze” all of the files, meaning changes can no longer be made. The registered components, or chapters, represent the final version of edited volume. Then…
    • Make all of the components, as well as the main project registration, public;
    • Enable the “comments” option so that anyone can post comments within each component (e.g., to discuss the material presented in the chapter);
    • Click the link to obtain a Digital Object Identifier (DOI) for each component (i.e., chapter).
  9. Advertise the edited volume
    • Use social media, including Facebook discussion groups and Twitter (among others). Encourage readers to leave comments for each chapter on the OSF pages;
    • Ask your University to issue a press release;
    • Ask your librarian for tips on how to advertise your new Open Access edited volume (librarians are an excellent resource!!).

Prior to following these steps to create your own Open Access edited volume on the OSF (or by using a different approach), there are some pros and cons to consider:

Pros

  • You have created an edited volume that is completely Open Access
  • The volume cost no money to create, no money to advertise, and no money to purchase
  • Given that the chapters are available to a wider audience than a traditional edited volume released by a for profit publishing company, it is likely that they will actually reach a wider audience as well and have a greater scientific impact

Cons

  • You do not receive a cash advance or royalties
  • You do not receive any assistance from a publisher for reviewing or advertising
  • This approach is new compared to traditional publishing, and therefore you may be concerned that you will not receive proper credit from others (e.g., people evaluating your contributions to science when deciding to hand out grant funds, jobs, promotions, and so on)

Final Thoughts

There is usually more than one way to achieve the same aim. Professional publishing companies work with academics to create many edited volumes every year, but creating an edited volume does not inherently require the assistance of a professional publishing company. The purpose of this post was to present one alternative using the functionality of the Open Science Framework to publish an edited volume that is Open Access. I am sure there are even more ways to achieve this aim.

Teaching Open Science

Leave a comment

In November 2015 I gave a workshop at the University of Toronto Mississauga on “Doing Open Science” (slides: https://osf.io/kz2u5/). During, and following, the workshop I spoke with attendees and heard two particular responses from this audience of graduate students and post-docs. First, they all believed that open science is becoming more important in our field. Second, most of them were unsure how to get started with open science in their own research. In fact, these are the two responses I hear most from others when discussing open science—it seems important, but how do I do it in my own lab?

More resources are now becoming available including a manual of best practices offered by BITSS and a list of course syllabi on the topic hosted on the Open Science Framework (OSF). My recent blog on organizing my own open science offered some suggestions for how to adopt open science practices (see also this paper). A Facebook post to the Psychology Methods Discussion Group asking how to pre-register study details also generated some useful feedback. Perusing public registrations of research projects on the OSF can also provide many examples of how to share details of the research process. And the newly introduced AsPredicted.org is a site devoted to making pre-registration very straightforward and fairly simple. Information is therefore becoming more available if one is motivated to look for it.

Psychology graduate programs typically have students take courses on statistical approaches to data analysis as well as on research methods. In these courses students read texts and papers, and learn where to find additional information. They also learn the values of their academic elders regarding the scientific process (e.g., predicting outcomes using statistical analyses with particular methodological designs). It seems to me, however, that going forward it is critical that we start routinely teaching open science practices to our students so (a) they know where to find information on open science, and (b) they learn that the research community that is training them values open science. It also seems practical to introduce material (or courses) on open science given that many journals are beginning to incentivize open science practices. Graduate students that adopt open science practices (as part of science 2.0) may therefore have an advantage in the job market compared to students that maintain the traditional closed science practices. As one final incentive to embrace the teaching of open science to your students, there are now awards available for doing it!