An investigation unmasks the inadequate quality standards applied by some journals requiring article processing charges.
(This article is based on Dell’Anno, Roberto, Caferra, Rocco, Morone, Andrea, 2020. “A “Trojan Horse” in the peer-review process of fee-charging economic journals,” Journal of Informetrics, Elsevier, vol. 14(3).)
According to the slogan that “numbers don’t lie”, the quantitative analysis of research production (i.e. bibliometrics) is becoming an unexceptional element to determine career paths of academics. Several scholars1-2 pointed out the negative consequences of the over-competitiveness in Academia and an uncritical use of bibliometrics. Got to hop on that hamster wheel it is not mandatory but, in the era of bibliometric evaluation, if you don’t take part in the race, someone will check your ranking for you! One of the most relevant side effects of this structural change in the academic profession is the proliferation of misconducts in scientific publishing both from the author’s side (e.g. plagiarism; salami-slicing; impropriety of authorship, falsification of data, etc.) and from the publisher’s side (e.g. editorial favoritism3 and predatory publishing4). Among these dishonorable academic practices, one of the most recent and increasing scientific misconducts is Predatory publishing. It is an exploitative business model, where the publisher requires an article processing charge (APC or publication fee) to cover the costs of peer review administration and management, professional production and dissemination of published papers”. For publishers of “traditional” (No APC) journals, a virtuous circle that incentives a proper peer-review process exists. No APC publishers have an incentive selecting rigorous and competent editors, because they invest journals’ resources on the best manuscripts. The best articles attract more citations than poor works, these quotations increase journal Impact Factor (IF) and, in turn, publishers’ revenue will increase through more library subscriptions and sponsors. Editor’s incentive for a proper screening process for scientific quality is also evident. Journals with higher IF increase the prestige of the editorial board. For APC journals this virtuous circle is broken. Publishers have an incentive to accept anything submitted because more publications mean higher revenue without the third wheel of citations (alias scientific community).
Several scholars raise doubts on the APC model for their too bland peer review process. The peer review process – that it is the only widely accepted method for scientific research validation – indeed may be easily biased by Publisher’s influence on unscrupulous editors permitting an improper or fictitious peer review process.
As Bohannon5 shown by a sting operation, this suspect has found empirical support. He submitted a bogus manuscript to biological, chemical and medical journals requiring APC and his fake article received 157 acceptances on 255 submissions. However, no one has examined the issue systematically and applying an experimental design. We did.
The Experiment – Ulysses at work
We define an experimental research strategy to establish the existence of a cause-and-effect relationship between the APC publishing business model and the inappropriateness of the peer review process. To accomplish this goal, we submit a bait-manuscript to two groups of journals that are equal one to each other in terms of bibliometric characteristics, but different in their business model. Accordingly, we compare journal’s acceptance rate for two groups of journals: the treatment group – including journal requiring publication fee (APC journals) – and the control group – including the journals not requiring publication fees (No APC) -.
In the first step, we select the treatment group by screening the list of the Italian Research Quality Assessment Exercise (in Italian: VQR) in order to pick – among the 2073 international scientific journals indexed in the areas of “Business” and “Economics” – all the journals which require payment of APC. We found that 52 journals match this condition.
In the second step, we select the control group by choosing 52 journals (like artificial twins), from the same list. This selection consisted to pick, for each journal included in the treatment group, a No APC journal that belongs to the same area (i.e. “Business” or “Economics”); it is indexed in the same bibliometric database (ISI Web of Science and/or Scopus or neither); and had a comparable quality in terms of bibliometric statistics (e.g., IF, h-index, etc.). Once this sample was defined, we submitted a bait manuscript to the editorial offices of these 104 journals.
The analysis of results was based on a sample of 73 journals. It consisted of excluding the 31 journals that sent back the manuscript for “technical” reasons. Specifically, editorial offices do not consider the submission for six types of editorial impediments: some journals share the same editorial office, therefore, it was not possible a multiple submission of the same article; some editorial offices required to modify the bait-manuscript in accordance with the format style guidelines; for other journals, the submissions are no longer allowed or they required to attach personal documents of the submitter or accepting only paper with editor invitation or also to pay a submission fee. Accordingly, this sample counted 34 journals requiring APC and 39 journals not requiring APC.
The Treatment – The Trojan Horse
Following Bohannon’s strategy, we wrote a bait manuscript entitled: “Crime and Economic Growth. An empirical analysis for Germany”. The topic of the study is chosen to fit the greatest possible number of economic journals’ aims.
In order to make multiple communications between (fake) authors and editors easier, we used two false identities for the two areas of economic research (i.e. “Business” and “Economics”). The name chosen was the translation of “Misunderstood Genius” in Welsh (Camddeall Athrylith) and Haitian Creole (Jeni Konpreyansyon) languages. For the affiliations, we have combined the generic “National University of” with the capital cities of the Welsh and Haitian institutions.
The bait had a credible layout for an economic journal article. It included 5 sections (Introduction, Literature review, Methodology, Results and Discussion, Conclusions) and a data appendix.
The hoax was written in Italian and automatically translated to English using Google Translator. The manuscript counted, at least, 70 errors that a skilled peer reviewer should identify and suggest the editor to reject the manuscript. For instance, in addition to the sloppy quality of English, there were: wrong derivatives in the theoretical model; erroneous interpretations of statistical significance of coefficients and goodness of fit statistics; the main conclusions and policy implications were completely reversed compared to the empirical outcomes reported in the manuscript; data sources were fictitious; some invented references; etc.
Data Collection – The Trojan War(s)
The sting operation of Prof. Genius took place in the first semester of 2019.
After excluding for “technical” reasons 31 journals on 104, of the remaining 73 journals, 43 were desk-rejected (i.e. without peer review) because they did not fit with the journal’s aim. Only 20 editors on 34 (58.8%) of the APC journals rejected the paper for the poor quality of the manuscript (see figure 1).
|One referee of APC journal stated: “The paper is well written and I believe this is the best manuscript I have read for [journal name redacted]”.|
In 10 cases the Genius received an invite to revise and resubmit the manuscript. In this circumstance, he re-submitted the original version without any changes but, in the cover letter for the editor and reviewers, he stated: “Dear Editor, please find attached the revised manuscript. Sincerely [author’s name]”. As this second round of peer review process concerns, in 4 cases the re-submitted paper was rejected while for 6 journals, the (false) revised version was accepted for publication. Last but not least, for 8 journals the manuscript was accepted as it was submitted without requiring any revision!
The results of Experiment -The Fall of Troy
Once Trojan horses were in the city, the end of the story depends on the internal rules of the city, out of the metaphor, if the journal follows an APC rather than No APC business model. For journals requiring publication fee, in 14 on 34 cases (41.2%) the paper was accepted. For the traditional journals (No APC) editors always rejected the paper (acceptance rate 0%).
The Moral of the Story
A multifaced picture emerges on academic publishing using the APC business model. On the one hand, we cannot argue that all the APC journals are predatory – indeed a number of these journals applied an appropriate peer review process and rejected the manuscript. On the other hand, there is a (too) large presence of predatory journals among them. We should not throw the baby out with the bathwater; however, an acceptance rate for a bogus manuscript of 42% should be considered as an important warning for the scientific community and policymakers.
The APC business model has shown its inappropriateness to preserve the quality of scientific publishing. The conflict of interests between journals and authors to publish poor researches predictably disappears. Finally, we find that, at least, in the economic area, with the exclusion of ISI web of science and DOAJ (Directory of Open Access Journals), the other famous bibliometric databases and some white lists include, or fail to discover, journals with predatory practices. On 14 journals falling in the trap: 7 (50%) belong to Committee on Publication Ethics list; 4 (29%) are not included in the Beall’s list; 4 (29%) are indexed in IDEAS; 2 (14%) are indexed in Scopus database.
It is important to point out that, although the APC is also the fee requested to publish gold open access (OA), our experiment does not test whether OA journals are predatory. For this reason, an extensive interpretation of our result as supporting the hypothesis that OA is correlated to the inappropriateness of the peer review process is incorrect. Indeed, we find that all the APC journals included in DOAJ rejected the bogus manuscript.
This study has two main policy implications.
The first one deals with the issue of the efficient use of public resources in academic research. The payment of APCs is usually sponsored by scholar’s institution, and for useless articles, it constitutes a clear waste of research funds (e.g. Assistant Prof. Misunderstood Genius had to pay more than 4,900 US dollars as APCs for the 14 accepted manuscripts).
The second implication deals with the effect that predatory journals have on academic recruitment and research institutions’ assessment. Bibliometric statistics are often used in a comparative way. For that reason, if the scientific community, (honest) publishers and public institutions will not seriously join forces to ban predatory journals, in few years, also the use of the bibliometrics, as a complementary assessment criterion of the researchers’ activity, may be seriously questioned.
- Carson, L., et al., Disruptive Science and Technology, 1(4), 183-190 (2013).
- Baccini A, et al. PLoS ONE 14(9) (2019)
- Reingewertz, Y., Lutmar, C., 12(1), 74-86, (2018).
- Beall, J. Nature 489, 179 (2012).
- Bohannon, J. Science, 342, 6154 (2013).
- Fanelli, D. PLoS ONE, 5, 4 (2010)
- Sorokowski, P., et al., Nature, 543, 481 (2017).