Think preprints are unreliable? Analysis of 70,000 studies might change your mind

A recent analysis of nearly 70,000 biomedical studies challenges the perception that preprints—scientific manuscripts shared publicly before undergoing formal peer review—are inherently unreliable. According to a study published on the preprint server bioRxiv, the central conclusions of most biomedical preprints remain largely unchanged after they are subjected to peer review and published in journals. This finding suggests that preprints may serve as a dependable source of scientific information despite ongoing skepticism within certain academic circles. The research, led by Ruslan Rust, a neuroscientist at the University of Southern California, aimed to investigate whether peer review significantly alters the content of preprinted studies. To achieve this, Rust and his team employed a large language model (LLM) to extract the primary scientific conclusions from the abstracts of 72,644 biomedical manuscripts initially uploaded to bioRxiv between 2018 and 2025. The LLM then evaluated the degree of change between the preprint abstracts and their final peer-reviewed versions. Of these, 39.9 percent retained their main conclusions without modification, while another 50 percent experienced only minor revisions. Only slightly more than 10 percent underwent substantial changes. According to the study, when significant alterations did occur, they tended to make the conclusions more cautious rather than more confident. Approximately 8.4 percent of revised findings adopted more conservative language, while just 4.2 percent became more assertive. These variations highlight the nuanced impact of peer review on the framing of scientific claims. The extent of revision also differed across disciplines. Bioinformatics studies showed the lowest rate of major changes, with only 7.2 percent experiencing substantial modifications. In contrast, microbiology studies had the highest rate, with 17.5 percent undergoing major revisions. Additionally, the frequency of major revisions decreased over time, dropping from 17 percent in 2019 to 5.7 percent in 2024. Julian Sienkiewicz, a researcher specializing in artificial intelligence tools and data exploration at the Warsaw University of Technology, suggested that the decline in major revisions might reflect an increasing workload on peer reviewers, potentially leading to less thorough evaluations. However, Rust proposed an alternative explanation: the way preprints are utilized has evolved since the launch of platforms like bioRxiv. During the early years, especially during the height of the COVID-19 pandemic, there was immense pressure on scientists to share findings quickly, resulting in numerous papers requiring extensive revisions prior to publication. More recently, some researchers may have incorporated initial reviewer feedback directly into their preprint drafts, thereby reducing the need for later revisions. In addition to examining revisions, the study also looked at the rates of retraction. Papers that first appeared as preprints were retracted at a rate of 8.1 per 10,000, compared to 18.7 per 10,000 for those that had never been posted as preprints. While the researchers acknowledge that this comparison is observational and does not establish causation, it raises intriguing questions about the potential role of preprints in preventing erroneous findings from reaching the public domain. Reactions to the findings have been mixed. On the professional networking platform LinkedIn, some researchers pointed out that preprints are influenced by selection bias, as the choice of which studies to post can vary widely depending on the individuals involved and the nature of the research itself. Despite these concerns, the study provides compelling evidence that preprints are generally stable and resistant to major changes after peer review, offering a valuable resource for the scientific community.