I have been asked to discuss my views on open science and replication, particularly in my field of social psychology, nine times in 2016 (see my “Open Science Tour” dates below). During these talks, and in discussions that followed, people wanted to know what exactly is open science, and how might a researcher go about employing open science practices?
Overall, many similar questions were asked of me from faculty and students so I thought I would create a list of these frequently asked questions. I do not provide a summary of my responses to these questions, instead wanting readers to consider how they would respond. So, how would you answer these questions? (public google doc for posting answers)
- Given that many findings are not, and in many cases cannot, be predicted in advance, how can I pre-register my hypotheses?
- If my research is not confirmatory, do I need to use open science practices? Isn’t open science only “needed” when very clear hypotheses are being tested?
- How can I share data?
- What data do I “need” to share? (All of it? Raw data? Aggregated data?)
- What platforms are available for data sharing? (and what is the “best” one?)
- What format/software should be used?
- Is this really necessary?
- How should I present this to my research ethics board?
- Can I publicly share materials that are copyrighted?
- What is a data analytic plan?
- Is it really important to share code/syntax from my analyses?
- Can’t researchers simply “game the system”? That is, conduct research first, then pre-register after results are known (PRARKing), and submit for publication?
- Can shared data, or even methods/procedures, be treated as unique “citable units”?
- If I pilot test a procedure in order to obtain the desired effects, should the “failed” pilot studies be reported?
- If so won’t this bias the literature by diluting the evidence in favor of the desired/predicted effect obtained in later studies?
- How much importance should I place on statistical power?
- Given that effect sizes are not necessarily knowable in advance, and straightforward procedures are not available for more complex designs, is it reasonable to expect a power analysis for every study/every analysis?
- If I use open science practices but others do not, can they benefit more in terms of publishing more papers because of fewer “restrictions” on them?
- If yes, how is this fair?
Unique question from students:
- Could adopting open science practices result in fewer publications?
- Might hiring committees be biased against applicants that are pro open science?
- If a student wants to engage in open science practices, but his/her advisor is against this, what should this student do?
- If a student wants to publish studies with null findings, but his/her advisor is against this, what should this student do?
- Will I “need” to start engaging in open science practices soon?
- Will it look good, or bad, to have a replication study (studies) on my CV?
- What is the web address for the open science framework? How do I get started?
My Open Science tour dates in 2016 (links to slides provided):
- January 28, Pre-Conference of the Society of Personality and Social Psychology (SPSP), San Diego, USA
- June 10, Conference of the Canadian Psychological Association, Victoria, Canada
- October 3, York University (Psychology), Canada (audio recording)
- October 11, University of Toronto (Psychology), Canada
- October 19, University of Guelph (Family Relations and Applied Nutrition), Canada
- October 21, Illinois State University, (Psychology), USA
- November 11, Victoria University Wellington (Psychology), New Zealand
- November 24, University of Western Ontario (Clinical Area), Canada
- December 2, University of Western Ontario (Developmental Area), Canada
Edited volumes are collections of chapters on a particular topic by various experts. In my own experience as a co-editor of three (3) edited volumes, the editors select the topic, select and invite the experts (or authors), and identify a publisher. Once secured, a publisher typically offers a cash advance to the editor(s) along with a small percentage of sales going forward in the form of royalties. The publisher may also provide reviewing services for the collection of chapters, and will advertise the edited volume when it is released. The two primary ways for consumers to access the chapters is to (a) purchase the book, or (b) obtain a copy of the book from a library.
With technological advances it is now possible to publish edited volumes without a professional publishing company. Why would someone choose to not use a publishing company? Indeed, they are literally publication experts. Perhaps the biggest reason is that the resulting volume will be open access, or available to anyone with a connection to the internet, free of charge. There are also some career advantages to sharing knowledge open access. Also, a publishing company is simply not needed for all publication projects.
There are very likely many different ways to publish an edited volume without using a professional publishing company. Below, I outline one possibility that involves using the Open Science Framework (OSF). Suggestions for improving these suggested steps are welcome.
Steps to Using the OSF to publish an Open Access Edited Volume
- Identify a topic for the edited volume, and then identify a list of experts that you would like to invite to contribute chapters.
- If you do not have an OSF account, create one (it is free). Create a new project page for your edited volume, and give it the title of the proposed edited volume. Select one of the licensing options for your project to grant copyright permission for this work.
- Draft a proposal for your edited volume (e.g., the need for this particular collection of chapters, goals of the volume, target audience, and so on). Add this file to the project page.
- Send an email inviting potential authors, providing a link to your OSF project page so they can read your proposal.
- You can make the project page public from the start and simply share the link, or,
- You can keep the project page private during the development of the edited volume and “share” a read-only link to the project page with prospective authors only.
- Ask all authors that accepted the invitation to create on OSF account. Then create a component for each individual chapter; components are part of the parent project, but are treated as independent entities in the OSF. Use the proposed title for each chapter as the title of the component. Add the author(s) as administrators for the relevant component (e.g., A. Smith has agreed to author chapter #4; add A. Smith as an administrator of component #4).
- Ask authors to upload a copy of their first draft by the selected deadline. Provide feedback on every chapter.
- One option is to download a copy of the chapter, make edits using the track changes option, and then upload a copy of the edited chapter using the same title as the original in order to take advantage of the “version control” function of the OSF (i.e., all versions of the chapter will be available on the project page in chronological order, with the most recent version at the top of the list).
- Ask authors to upload their revised chapter using the same title (again to take advantage of the “version control” function of the OSF).
- When the chapters are completed, “register” the project and all components. This will “freeze” all of the files, meaning changes can no longer be made. The registered components, or chapters, represent the final version of edited volume. Then…
- Make all of the components, as well as the main project registration, public;
- Enable the “comments” option so that anyone can post comments within each component (e.g., to discuss the material presented in the chapter);
- Click the link to obtain a Digital Object Identifier (DOI) for each component (i.e., chapter).
- Advertise the edited volume
- Use social media, including Facebook discussion groups and Twitter (among others). Encourage readers to leave comments for each chapter on the OSF pages;
- Ask your University to issue a press release;
- Ask your librarian for tips on how to advertise your new Open Access edited volume (librarians are an excellent resource!!).
Prior to following these steps to create your own Open Access edited volume on the OSF (or by using a different approach), there are some pros and cons to consider:
- You have created an edited volume that is completely Open Access
- The volume cost no money to create, no money to advertise, and no money to purchase
- Given that the chapters are available to a wider audience than a traditional edited volume released by a for profit publishing company, it is likely that they will actually reach a wider audience as well and have a greater scientific impact
- You do not receive a cash advance or royalties
- You do not receive any assistance from a publisher for reviewing or advertising
- This approach is new compared to traditional publishing, and therefore you may be concerned that you will not receive proper credit from others (e.g., people evaluating your contributions to science when deciding to hand out grant funds, jobs, promotions, and so on)
There is usually more than one way to achieve the same aim. Professional publishing companies work with academics to create many edited volumes every year, but creating an edited volume does not inherently require the assistance of a professional publishing company. The purpose of this post was to present one alternative using the functionality of the Open Science Framework to publish an edited volume that is Open Access. I am sure there are even more ways to achieve this aim.
This post is open to read and review on The Winnower.
In November 2015 I gave a workshop at the University of Toronto Mississauga on “Doing Open Science” (slides: https://osf.io/kz2u5/). During, and following, the workshop I spoke with attendees and heard two particular responses from this audience of graduate students and post-docs. First, they all believed that open science is becoming more important in our field. Second, most of them were unsure how to get started with open science in their own research. In fact, these are the two responses I hear most from others when discussing open science—it seems important, but how do I do it in my own lab?
More resources are now becoming available including a manual of best practices offered by BITSS and a list of course syllabi on the topic hosted on the Open Science Framework (OSF). My recent blog on organizing my own open science offered some suggestions for how to adopt open science practices (see also this paper). A Facebook post to the Psychology Methods Discussion Group asking how to pre-register study details also generated some useful feedback. Perusing public registrations of research projects on the OSF can also provide many examples of how to share details of the research process. And the newly introduced AsPredicted.org is a site devoted to making pre-registration very straightforward and fairly simple. Information is therefore becoming more available if one is motivated to look for it.
Psychology graduate programs typically have students take courses on statistical approaches to data analysis as well as on research methods. In these courses students read texts and papers, and learn where to find additional information. They also learn the values of their academic elders regarding the scientific process (e.g., predicting outcomes using statistical analyses with particular methodological designs). It seems to me, however, that going forward it is critical that we start routinely teaching open science practices to our students so (a) they know where to find information on open science, and (b) they learn that the research community that is training them values open science. It also seems practical to introduce material (or courses) on open science given that many journals are beginning to incentivize open science practices. Graduate students that adopt open science practices (as part of science 2.0) may therefore have an advantage in the job market compared to students that maintain the traditional closed science practices. As one final incentive to embrace the teaching of open science to your students, there are now awards available for doing it!
DOI: 10.15200/winn.144842.20295 provided by The Winnower, a DIY scholarly publishing platform