Reese Richardson

Happy Peer Review Week!

[ TLDR: To make my latest tumble down a deep rabbit hole marginally less bumpy, I downloaded >250K peer review documents for >35K articles in six open access journals published by BMC (an imprint of Springer Nature). As of September 23, 2025, I have now downloaded >600K peer review documents for >85K articles at 14 BMC journals. You can find links to download these archives in full here. ]

Since my earliest exposure to bioinformatics, I have had an averse fascination with articles that claim to develop gene expression-based “prognostic signatures”. This flavor of article has recently become very popular, probably because it is usually based entirely on publicly available data and can be easily generated following a simple formula:

1. Pick a type of cancer (e.g., breast cancer)
2. Pick a publicly available gene expression dataset for that cancer (most likely from the Cancer Genome Atlas, TCGA)
3. Pick a class of genes (e.g., genes involved with ferroptosis)
4. Feed the expression values of these genes into a regularized regression (e.g., a LASSO-penalized Cox regression) that predicts the duration of a patient’s overall or disease-free survival using only the expression values of a much smaller set of genes.
5. (Optional) Validate the obtained model with another publicly available dataset.
6. Perform a post-hoc literature search for specious reasons why the genes selected by your regression might be related to survival time in your cancer of interest.

Weakly-motivated studies utilizing easily-exploitable public datasets are in vogue right now (see also COSIG’s entry on formulaic research). “Prognostic signature” studies are especially irksome to me because they pollute the human gene research literature and the results they report are usually utterly unimpressive. For instance, this article reports that their three-gene model for predicting survival with bladder cancer achieves an area under the receiver operating characteristic curve of around 0.65—barely better at predicting survival than just guessing. 

Usually, the question “how on earth did this make it past peer review?” is not immediately answerable. However, this particular article was published in BMC Cancer, an open-access Springer Nature journal that practices “transparent peer review”, whereby peer review reports and authors’ responses to them are made available to read alongside the published article. This revelation brought me to the first-round review report of an anonymous Reviewer #2, which gave me something entirely different to be mad about.

This review spends 300 words saying absolutely nothing substantive. As an illustration of its banality, I’ve reproduced the entirety of the review below:

Dear Editor, thank you so much for inviting me to revise this manuscript about bladder cancer.

This study addresses a current topic.

The manuscript is quite well written and organized. English could be improved.

Figures and tables are comprehensive and clear.

The introduction explains in a clear and coherent manner the background of this study.

We suggest the following modifications:

Introduction section: although the authors correctly included important papers in this setting, we believe the evolving systemic treatment scenario for bladder cancer should be further discussed and some recent papers added within the introduction (PMID: 36695827; PMID: 33225800; PMID: 36533070; PMID: 33516645; PMID: 38791914), only for a matter of consistency. We think it might be useful to introduce the topic of this interesting study.

Methods and Statistical Analysis: nothing to add.

Discussion section: Very interesting and timely discussion. Of note, the authors should expand the Discussion section, including a more personal perspective to reflect on. For example, they could answer the following questions — in order to facilitate the understanding of this complex topic to readers: what potential does this study hold? What are the knowledge gaps and how do researchers tackle them? How do you see this area unfolding in the next 5 years? We think it would be extremely interesting for the readers.

However, we think the authors should be acknowledged for their work. In fact, they correctly addressed an important topic in bladder tumors, the methods sound good and their discussion is well balanced.

One additional little flaw: the authors could better explain the limitations of their work, in the last part of the Discussion.

We believe this article is suitable for publication in the journal although some revisions are needed. The main strengths of this paper are that it addresses an interesting and very timely question and provides a clear answer, with some limitations.

We suggest a linguistic revision and the addition of some references for a matter of consistency.

Moreover, the authors should better clarify some points.

Every part of this review reads as if it has been copy-pasted from a template. In fact, that’s exactly what happened: if you search Google for phrases like “only for a matter of consistency”, “area unfolding in the next 5 years” or “correctly included important”, you will find dozens of peer reviews that are functionally identical to this one. Nearly all of them feature the same request for citations to barely-relevant “recent papers” and almost all of the requested citations feature one author in common: an Italian oncologist named Alessandro Rizzo.

Bear with me, now—I think that Rizzo might have written these reviews. Some of them (example) are actually signed by Rizzo. I even found one article where the authors reach through the veil of anonymity to scold Rizzo for shamelessly fishing for citations. At the time of writing, Rizzo’s Web of Science profile shows 487 verified peer reviews since 2020 (since only a fraction of peer reviews get counted through Web of Science, this is probably a massive undercount).

Coercive citation requests like this are alarmingly common, but the scale of this operation was something else. I initially saw it two ways: either Rizzo was the laziest peer reviewer alive or oncology’s most industrious self-promoter. 

However, as it turns out, the situation is even stranger than it appears: some of the non-anonymous peer reviews sharing these exact recycled phrases and requests for citations to Rizzo are not signed by Rizzo, but by one of his colleagues. This review, for instance, is one of several attributed to Angela Dalia Ricci, Rizzo’s frequent coauthor. In other instances, like for this PLOS One article, Rizzo is listed as the article’s handling editor, but an anonymous Reviewer #1 uses all the same language as Rizzo’s reviews elsewhere while also thanking the editor for selecting them as a peer reviewer (“Dear Editor, thank you so much for inviting me to revise this manuscript”)!

I shared this case with María de los Ángeles Oviedo-García, who recently discovered a set of apparent “review mills”, large-scale operations whereby signed and anonymous reviewers use incredibly similar peer review templates to facilitate fast peer review and direct citations to themselves and colleagues. María agreed that we are probably seeing something very similar here and immediately identified dozens more reviews matching this template, including several more signed by Ricci and one signed by Giovanni Brandi, another frequent coauthor.

To expand the search further without relying on copious Google searches, I identified several BMC/Springer Nature journals with transparent peer review that had multiple articles citing one or more of Rizzo’s articles. I then downloaded every peer review report and author response published in the journals BMC Cancer (18,311 articles with 119,703 peer review documents), BMC Gastroenterology (5,085 articles, 35,143 documents), BMC Medical Genomics (2,178 articles, 14,938 documents), BMC Medicine (4,605 articles, 30,432 documents), BMC Surgery (3,465 articles, 24,314 documents) and BMC Women’s Health (4,131 articles, 30,764 documents). Searching these reports for the previously-mentioned phrases as well as all PubMed identifiers associated with Rizzo’s articles, I found 45 articles with review reports apparently from this review mill.

At the time of writing, with manual searches through Google and automated searches of journals’ complete peer review archives (manually-checked), María and I have catalogued 104 unique articles published from 2021 onward that have at least one review report from this apparent mill (publishers include BMC/Springer Nature, AME Publishing Company, MDPI, PLOS, BMJ Group, PeerJ and Qeios). We will continue to add cases to this spreadsheet as we (or you!) find them (note that manually-identified and automatically-identified cases currently reside in two different sheets in this workbook).

This apparent review mill is probably much larger than we can discover without insider access to publishers’ peer review records. Obviously, it represents an immense deviation from the expected ethos of peer review. If I were an author of an article that received one of these reviews, I would be mad enough about the platitudinous review and citation grubbing even without knowing that hundreds of other authors had gone through the same experience. I would want whoever willfully participated in this scheme to be held to account. However, I’ll try to direct my indignation where blame is certain: authors and readers trust publishers to facilitate meaningful and effective peer review through their own internal quality control and through the editors they appoint. Here, these publishers failed spectacularly and they should have to answer for how they will prevent this sort of thing from continuing.

---

Anyway, in observance of Peer Review Week, I’ve made all peer review files, extracted text and associated metadata from these six BMC journals available as datasets on Zenodo for anyone to download in bulk:

• BMC Cancer (downloaded 31 August 2025)
• BMC Gastroenterology (downloaded 31 August 2025)
• BMC Medical Genomics (downloaded 31 August 2025)
• BMC Medicine (downloaded 31 August 2025)
• BMC Surgery (downloaded 14 September 2025)
• BMC Women’s Health (downloaded 14 September 2025)

You might find these datasets useful if you are a metascientist studying the process of peer review more generally or if you are interested in tracking other questionable peer review patterns (I may have already identified a couple just by looking for unexpectedly frequent n-grams).

A note: publishers practicing transparent peer review can and should make these archives publicly available without restriction as bulk datasets. PLOS does it already for all of their article content and metadata, not just peer review reports. This data is already available article-by-article on an open-access basis. Plus, the process of webscraping sucks for everyone involved.