Tag Archives: replication

Week 11: Extended Research Networks

Leave a comment

In this class I introduced to students the idea of scaling up open science practices for use in extended research networks. When I first taught this course many of these initiatives were relatively new and some were untested, and the students were excited about the possibility of these large scale collaborations. I will only discuss a few of the current extended research networks in this post.

One of the earlier extended research networks that incorporated open science practices was the Registered Replication Reports (RRR) initiative originally offered by Perspectives on Psychological Science and originally headed up by Daniel Simons. The basic idea was that individuals or groups of researchers would propose a study that they would like to re-run on a large scale with a number of independent labs after input from the original author(s). When a proposal was approved the submitting researchers worked closely with Dan and the original author(s) to reproduce the methods of the original study as closely as possible (or in some cases settle on a particular method/approach they felt was optimal to assess the effect of interest). A call would then go out to the research community, asking others to use the agreed upon methods/measures and collect a given amount of data to contribute to the project. All study details were shared with this extended group on the Open Science Framework. When the data was collected members of the extended research team submitted their data to the person in charge of overseeing the statistical analyses. This person was not part of any of the research groups, and the statistical analyses as well as the syntax used to run these analyses were agreed upon in advance. The team that submitted the original proposal for the replication project worked with Dan and the original author to draft a methods/results section in advance of knowing the results; the goal was to be able to drop the results into the already prepared manuscript. When all was ready the results would be revealed. I participated in one of these projects (I wrote about it here). Here is a link to the final product. Overall these projects were focused on large scale replication research, and presently RRRs are now offered via the journal Advances in Methods and Practices in Psychological Science.

Another successful extended research network is the Many Babies initiative. From their website, Many Babies “is a collaborative project for replication and best practices in developmental psychology research. Our goal is to bring researchers together to address difficult outstanding theoretical and methodological questions about the nature of early development and how it is studied.” The basic idea here is to enhance collaboration between labs all over the world that collect data from babies in an open and transparent manner. This also helps with increasing sample sizes, given that any individual lab faces challenges collecting data from large samples of babies. Check it out.

Lastly I will mention the Psychological Science Accelerator. From their website: “The Psychological Science Accelerator is a globally distributed network of psychological science laboratories with 1328 members representing 84 countries on all six populated continents, that coordinates data collection for democratically selected studies.” They have many committees to assist with every aspect of the research process for everyone involved (e.g., translation, ethics review, statistical analyses, and so on), and the entire process is guided by open and transparent research interests. I was part of some of the early discussions of this initiative and am very impressed with the leadership team during the handful of years it has existed. They are truly inspirational. This type of large scale extended research network seems to be an ideal manner to test ideas with lots of data, but more importantly data from all over the world. This allows for testing group/cultural differences in the effects of interest. Check out the results from the first project of this initiative here.

I have not gone into any detail on the Many Labs projects that sparked a lot of discussion, or other initiatives that sought to bring together researchers from different Universities and countries to collectively test hypotheses in an open and transparent manner. Overall, there are many exciting options available to researchers at all stages of their career to get involved in these extended research networks.

Week 10: Openly Sharing Research Reports/Manuscripts

Leave a comment

When I first taught this course, pre-print servers, or other online resources for sharing research reports and manuscripts, were not as popular or well known as they are today (Spring 2023). My goal with this class was to introduce the idea of sharing manuscripts prior to/after publication as well in lieu of publication in a peer reviewed journal. I showed them a few different options available at the time, including the one hosted by the library system at Western University (where we are located).

Overall the students seemed concerned about how it would be perceived to share a manuscript publicly before it was accepted for publication at a peer reviewed journal (e.g., “will the journal want to publish my paper if I have already “published” it?”). As part of this discussion I showed them sherpa romeo, a site that allows one to view the open access policies of a lot of journals and thus help one determine if they can/should share a preprint of a manuscript. The students were also concerned, however, with sharing a copy of the paper that was accepted for publication in case the journal would forbid this practice (and maybe even revoke acceptance of a manuscript); sherpa romeo is helpful here as well. A lot of fear associated with sharing outside the mainstream publication system! Fair enough, that is why I teach this material in the class and have an open discussion where I make sure to listen to the concerns of the students.

In this class I also discuss thinking beyond the typical research report as material worthy of sharing publicly. For example, stimuli used in the research that will not be part of the manuscript but others may want to use for their own research. I discussed how they could share this material in such a way that it could be both used but also cited. It was appealing to the students to think that they could have aspects of their research beyond the manuscript itself appear in, for example, google scholar and also be cited. The same goes for unique methods as well as data sets. Lastly, we discussed the idea of open peer review and associated pros and cons.

I have been sharing preprints for many years now, mostly (but not exclusively) on psyarxiv. Most of the manuscripts shared their are now published in peer reviewed journals, but some are not. For example, here is a brief paper now published at the Journal of Research in Personality that is also on psyarxiv. Google scholar tells me the published paper has been cited a whopping 4 times. But as you can see on psyarxiv it has been downloaded over 2000 times to date. This may mean absolutely nothing, but perhaps it means that the paper is having an impact not measured by citations alone. Also, you can see on psyarxiv that after the paper is published in a peer reviewed journal the author(s) can update the preprint with the published DOI. One example of a manuscript that exists only as a preprint focuses on a qualitative analysis of “ghosting” (in this case relationship dissolution by ending all contact with a partner) that was lead by former awesome graduate student Rebecca Koessler. This paper has been downloaded over 3000 times, suggesting it has been helpful in some way to others; if it had remained tucked away in our hard drives only it would obviously not have had this level of attention. Interestingly enough this preprint has also been cited 11 times according to Google scholar. From this perspective it was therefore of value to share this research as a preprint even though it was not published in a peer reviewed journal. My approach to open science practices has been to lead by example, so I appreciate that my own experiences with sharing preprints has resulted in noticeable attention to the research when the paper is published in a peer reviewed journal or not. I will likely use these papers, and others, as examples of sharing preprints if I teach this course again.

My 2016 Open Science Tour

Leave a comment

I have been asked to discuss my views on open science and replication, particularly in my field of social psychology, nine times in 2016 (see my “Open Science Tour” dates below). During these talks, and in discussions that followed, people wanted to know what exactly is open science, and how might a researcher go about employing open science practices?

Overall, many similar questions were asked of me from faculty and students so I thought I would create a list of these frequently asked questions. I do not provide a summary of my responses to these questions, instead wanting readers to consider how they would respond. So, how would you answer these questions? (public google doc for posting answers)

  1. Given that many findings are not, and in many cases cannot, be predicted in advance, how can I pre-register my hypotheses?
  2. If my research is not confirmatory, do I need to use open science practices? Isn’t open science only “needed” when very clear hypotheses are being tested?
  3. How can I share data?
    • What data do I “need” to share? (All of it? Raw data? Aggregated data?)
    • What platforms are available for data sharing? (and what is the “best” one?)
    • What format/software should be used?
    • Is this really necessary?
    • How should I present this to my research ethics board?
  4. Can I publicly share materials that are copyrighted?
  5. What is a data analytic plan?
  6. Is it really important to share code/syntax from my analyses?
  7. Can’t researchers simply “game the system”? That is, conduct research first, then pre-register after results are known (PRARKing), and submit for publication?
  8. Can shared data, or even methods/procedures, be treated as unique “citable units”?
  9. If I pilot test a procedure in order to obtain the desired effects, should the “failed” pilot studies be reported?
    • If so won’t this bias the literature by diluting the evidence in favor of the desired/predicted effect obtained in later studies?
  10. How much importance should I place on statistical power?
    • Given that effect sizes are not necessarily knowable in advance, and straightforward procedures are not available for more complex designs, is it reasonable to expect a power analysis for every study/every analysis?
  11. If I use open science practices but others do not, can they benefit more in terms of publishing more papers because of fewer “restrictions” on them?
    • If yes, how is this fair?

Unique question from students:

  1. Could adopting open science practices result in fewer publications?
  2. Might hiring committees be biased against applicants that are pro open science?
  3. If a student wants to engage in open science practices, but his/her advisor is against this, what should this student do?
  4. If a student wants to publish studies with null findings, but his/her advisor is against this, what should this student do?
  5. Will I “need” to start engaging in open science practices soon?
  6. Will it look good, or bad, to have a replication study (studies) on my CV?
  7. What is the web address for the open science framework? How do I get started?

My Open Science tour dates in 2016 (links to slides provided):

  • January 28, Pre-Conference of the Society of Personality and Social Psychology (SPSP), San Diego, USA
  • June 10, Conference of the Canadian Psychological Association, Victoria, Canada
  • October 3, York University (Psychology), Canada (audio recording)
  • October 11, University of Toronto (Psychology), Canada
  • October 19, University of Guelph (Family Relations and Applied Nutrition), Canada
  • October 21, Illinois State University, (Psychology), USA
  • November 11, Victoria University Wellington (Psychology), New Zealand
  • November 24, University of Western Ontario (Clinical Area), Canada
  • December 2, University of Western Ontario (Developmental Area), Canada

An Inside Perspective of a Registered Replication Report (RRR)

2 Comments

Update: Dan Simons and Bobbie Spellman discuss this Registered Replication Report, and others, on NPR “Science Friday”

In the spring of 2014 we (i.e., Irene Cheung, Lorne Campbell and Etienne LeBel) decided to submit a proposal to Perspectives on Psychological Science for a Registered Replication Report (RRR) focusing on Study 1 of Finkel, Rusbult, Kumashiro and Hannon’s (2002) paper testing the causal association between commitment and forgiveness. The product of over 2 years of work by many people including us, the tireless Dan Simons (Editor of the RRR series), a cooperative and always responsive Eli Finkel (the lead author of the research to be replicated), and researchers from 15 other labs all over the world, is now finally published online (http://www.psychologicalscience.org/pdf/Finkel_RRR_FINAL.pdf). Here is our inside perspective of how the process unfolded for this RRR.

The initial vetting stage for the RRR was fairly straightforward. We answered some simple questions on the Replication Pre-Proposal Form, and provided the rationale for why we believed Study 1 of Finkel et al.’s (2002) manuscript was a good candidate for an RRR (e.g., the paper is highly cited, is theoretically important, no prior direct replications have been published). After receiving positive feedback, we were asked to provide a more thorough breakdown of the original study and the feasibility of having multiple labs all over the world conduct the same project independently. In a Replication Proposal and Review Form totaling 47 pages, we provided information regarding (a) the original study and effect(s) of interest, (b) sample characteristics of original and proposed replication studies (including power analysis), (c) researcher characteristics (including relevant training of the researcher collecting data from participants), (d) experimental design of original and proposed studies, (e) data collection (including any proposed differences from the original study, and (f) target data analysis (of both the original and planned replication studies). After receiving excellent feedback and making many edits, a draft of this document was sent to the original corresponding author (Eli Finkel). Eli very quickly provided thorough feedback, and forwarded copies of the original study materials. He also provided thoughtful feedback throughout the process as we made many decisions on how to conduct the replication study and ultimately vetted the final protocol. The RRR editors eventually gave us the green light to go forward with the project.

We were then required to organize the project. The study was programmed on Qualtrics, the protocol requirements were created, the project page on the Open Science Framework (OSF) was established, and eventually a call went out for interested researchers to submit a proposal to independently run the study and contribute data. It is near impossible to estimate the number of emails sent around between Dan, our team, and Eli during this time, as well as the number of small changes made to all of the materials along the way. Leading up to the fall of 2015, all participating labs were ready to start collecting data. Participating labs simply needed to download the necessary materials from the OSF project page, and Dan provided support to many of the labs throughout the process. Prior to data collection, the study was pre-registered on the OSF. Data collection was complete by January 2016, and it was time to prepare the R code needed to analyze the data from each lab as well as conduct the planned meta-analyses. Our team helped test the code, and then Edison Choe (working for APS) wrote the full set of code (verified by Courtney Sodenberg from the OSF). The code needed to be tweaked many times as well as to make small adjustments. All labs then ran the code with their data, and submitted the data and results to their own OSF pages, while our team wrote the manuscript before seeing the full set of results from all labs. Dan and Eli provided feedback on numerous occasions, and the full set of results was not released to us until the manuscript was considered acceptable by all parties. After the results were released we incorporated them into the manuscript and wrote a discussion section. Eli then wrote a response. After making many small edits, and sending copious amounts of email around to Dan and Eli, the manuscript was complete. All participating labs were then provided a copy of the manuscript to review for any required edits, and asked not to discuss the results with anyone not associated with the RRR until the paper was published online. Not surprisingly, a few more edits were indeed required. When completed, the manuscript was sent to the publisher and appeared online first within a week.

Overall, this was a monumental task. The manuscript can be read in minutes, the results digested in a few quick glances at the forest plots. Getting to this point, however, required the time, attention and effort of many individuals over 2 years. Seeing an RRR through to completion requires a lot of dedication, hard work, and painstaking attention to detail; it is not to be entered into lightly. But the process itself, in our opinion, represents the best of what Science can be—researchers working together in an open and transparent manner and sharing the outcome of the research process regardless of the outcome. And the outcome of this process is a wonderful set of publicly available data that helps provide more accurate estimates of the originally reported effect size(s). It is a model of what the scientific process should be, and is slowly becoming.

From the proposing authors of this RRR:

Irene Cheung

Lorne Campbell

Etienne LeBel