Below I make two observations, after making an assumption on pre-publication peer-review:

Assumption:

Pre-publication peer review is viewed as an essential step in the publication process. It represents a “stamp of approval”—colleagues with expertise in this area of research agree that the research presented in the published manuscript properly tests an original research question/hypothesis (or is at least super cool and clever). Without pre-publication peer review a published article is not taken seriously by the field at large (e.g., an article published on someone’s website site but not yet published by a journal after one or more rounds of pre-publication peer review is not really considered a “publication” by most of my colleagues at least; or at least as less of a publication).

Observations:

(1) statistical power, generally obtained by recruiting high numbers of participants in our studies (yes, it is more complex than high N), is becoming more valued in the psychological sciences (no citations needed—general knowledge). Having 30% power was so pre one year ago.

(2) Pre-publication peer review typically involves the opinions of two “expert” reviewers (sometimes 3, maybe even 4), and one editor, on only one manuscript.

Now, if we consider reviewers as participants in a study designed to answer questions about the value of a given article, what kind of inter-rater agreement would we expect from 2-4 participants for 1 article? What kind of power would this study have to identify true scientific merit? This is essentially a poorly designed mixed quantitative/qualitative case study. If this truly were a study it would be rejected by reviewers, if the editor did not already issue a desk reject—the methods are atrocious. I know, it is like comparing apples and oranges. That said it seems to me that a standard we are beginning to apply with much more rigour when conducting our research (i.e., increasing statistical power) is one that is woefully absent from the process of how we as a field evaluate the merits of this research after it is conducted.

And as it turns out, there really is no evidence that pre-publication peer review is good for science: https://www.timeshighereducation.co.uk/content/the-peer-review-drugs-dont-work. In this piece written by Richard Smith, he offers up an alternative to pre-publication peer review: “With the World Wide Web everything can be published, and the world can decide what’s important and what isn’t.” In other words, publish what you like and rely on post-publication peer review to separate the wheat from the chaff. With post-publication peer review there is no limit to the number of people that can provide expert commentary, and such commentary can be made at any time after the manuscript is published (not only during the “review” process).

In fact all of us could follow this advice right now if we truly wanted to. How? Here is one example (and of course there are other options available): publicly post all study materials, data sets, and data analytic code on the Open Science Framework (OSF). Write a manuscript to put the results in context. Publicly post that manuscript on the OSF, and obtain a DOI by clicking that option when posting—your manuscript is now published with no pre-publication peer review, no Open Access publication fees, and no waiting months and months for reviews from a traditional journal with high rejection rates largely because they are limited by how many paper pages the publisher has contractually agreed to print per year. Click the option to allow comments to be posted by anyone, and advertise your paper on social media and other outlets and encourage colleagues to provide critical commentary. Revise your manuscript based on the comments. Repeat with other projects.

This suggestion may sound radical, but consider that in the above scenario ANY consumer of the research can look at the study scales, re-analyze the data if desired, and make suggestions to the author on additional data analyses or for follow-up studies. A dialogue between people posting comments about the manuscript could develop that is publicly available for other consumers of the research to read through. Just, wow. Now think of the last paper you read in the “top” journal in your field—odds are you will not have access to study materials, data, data analytic code, reviewer’s comments, or the editor’s comments. Both papers are published. If these two hypothetical papers were on the same topic, a topic that you wanted to follow up with your own research, which paper would you rely on more heavily when developing your study?

Herbert Stein once said, “If something cannot go on forever, it will stop.” Given the existence of the World Wide Web, the inefficiencies of pre-publication peer review, and let’s also throw in a tired system of privately controlled academic publication that is still largely ruled by paper and huge subscription costs, it seems that how we have gone about publishing our research for many years now cannot go on forever, and it will stop. Other ways of publishing our research will take its place. Or I am completely wrong.