Researchers need to publish manuscripts to both advance science and their careers. This latter fact was made clear recently as I was a member of a departmental committee tasked with evaluating the performance of each faculty member. When it came time to discuss publications there were two themes: (1) how many publications did this person have in the given time frame (we “expect” approximately 3 per year), and (2) were they published in top journals? No mention of the actual research conducted, or the strength of the methods employed, because the committee was not asked to read any of these publications. The number of publications and the prestige of publication outlet therefore served as proxies for research performance.
I want to focus this discussion on the prestige of academic journals. What makes a journal a Top Journal? We all seem to know one when we see one, but specifically defining why one journal is better than another is tricky. When I ask this question of others a variety of factors are discussed, such as (a) rejection rate (higher rejection rate seems to equal better journal), (b) a place where the successful academics tend to publish their work, (c) the impact of the research published in a journal on the rest of the field, (d) visibility of the journal (i.e., are most people in the field familiar with the journal?), (e) perception that the research needs to be particularly novel/ground-breaking to be published in the journal, and so on. These all seem like reasonable points, but it does suggest a fairly static hierarchy of journal prestige, and the quality of the research discussed in particular manuscripts within each journal is implied as a function of the prestige of the journal.
I then conducted some Internet searches to see how journal prestige is assessed. Most ranking systems that I could find rely largely on citation counts of articles published within a journal (e.g., the much loved, and loathed, impact factor). For example, the SCImago journal ranking provides information on thousands of journals, including a long list of psychology journals that can be ranked on a few different factors, including: SJR (a measure of “prestige”), H-index (number of articles in a journal cited a given number of times), and Impact Factor. A great feature of this site is that you can download the rankings in an excel file. I downloaded the file, and decided to add some ranking information recently put together by Uli Schimmack: the R-index of the journal.* I direct readers to Uli’s blog to learn more about the R-index and how it is calculated, but briefly it uses the estimates of post-hoc power calculated for each published article, and R-index scores increase when power is higher and decreases when publication bias is present (according to Uli, and awaiting verification). So, it is calculated based on information presented in each paper published in a given journal (i.e., based on p-values calculated from reported statistical tests that are then used to estimate post-hoc power; see also the N-pact factor), rather than on how many times each paper published in a given journal is cited by others.
So, do indices based on citation counts correlate with an index based on post-hoc power of published articles? No. The scatterplot below indicates a slightly negative relation between the R-index and IF of a journal. There is R code available to recreate this scatterplot and to to calculate correlations with the SJR and H-index as well here (spoiler alert—the R-index does not significantly correlate with them either).
So, Top Journals do not seem to publish studies with relatively more post-hoc power
and thus results more likely to replicate compared to lower tier (dare I say “specialty”) journals (at least according to the R-index). Is journal prestige therefore merely popularity?
* the R-index currently ranks 54 psychology journals (more to come I believe). The R-index was calculated for each section of JPSP, whereas the SCImago rankings gives scores for the entire journal; I therefore selected the highest R-index score for JPSP. A few journals listed in the R-index ranking were not included in the SCImago rankings. Overall, a total of 50 R-indices were entered into the data file. Also, I used the R-index calculated for articles published in journals between 2010-2014 to be more consistent with the time frame of the SCImago rankings (based on 2014 numbers)