PREreview of Public Understanding of Preprints: How Audiences Make Sense of Unreviewed Research in the News

Overview

This study examines the understanding of students and the US general population regarding preprints. The authors carried out deductive and inductive content analysis of 1,702 preprint definitions obtained as survey free-text responses from study participants who read COVID-related news articles with or without preprint disclosure. The authors found that only a minority of participants could define preprints in ways that capture the key characteristics of preprints. Additionally, this study suggests that providing brief definitions of preprints in news articles may not be as helpful in increasing reader understanding as some guidelines purport. This study provides insight into the public understanding of preprints and points to potential ways to improve media reporting of research posted as preprints.

We want to thank the authors for posting an interesting, original, and thought-provoking piece of research.

This review was compiled by the ASAPbio Meta-Research Crowd.

Major comments

What percentage of preprint definitions included more than one of (A) not peer reviewed, (B) released publicly, and/or (C) not published in a scientific journal? How would the findings change, if the bar of “accurate definition” is raised to include at least 2 among A, B, and C? 

• I would argue this is an important consideration because having only one of A, B, C codes does not make for an accurate definition of preprint. For example, a manuscript that is not peer-reviewed could be a manuscript shared in confidentiality with colleagues or reporters (not a preprint, but satisfies code A). Similarly, a manuscript that is released publicly could be an open-access journal article (not a preprint, but satisfies code B)...

We are not expert statisticians, however we would like to query whether the use of bivariate correlation tests is  appropriate for examining factors associated with “technically accurate preprint understanding”?

• Bivariate correlation tests measure whether and how two variables covary linearly, often used for two continuous variables. Most of the independent variables in this study (e.g., Education, PIUS, Factual Scientific Literacy) are categorical variables. In particular, the authors provided a numerical mean for Education in Supplemental Material 6. How was this computed?

• In Table 3, the authors conducted seven tests. Is there a concern with multiple hypothesis testing with this analysis?

• Lastly, is logistic regression a more appropriate statistical test here? Logistic regression models the log odds of a dichotomous outcome as a linear combination of the predictor variables.

We agree that adding a brief definition of preprints to news stories is an important conclusion.

The discussion about the interpretations of the term ‘preprint’ suggest that it may be worth the community thinking about using a different term that more accurately reflects the status of a piece of research that hasn’t been peer-reviewed. One of us has previously had conversations with those in the industry where the term used was thought to be irrelevant, despite being misleading (neither pre- anything nor in print), but this research suggests otherwise. That could be a useful additional conclusion for this work.

Minor comments

The abstract didn't mention the main findings regarding student and general population understanding differences, namely, that students performed significantly better than the general population.

In previous studies, did participants perceive both preprints and peer-reviewed content as credible or with scepticism? I.e. could it be a result of a lack of trust in peer review? More generally, what expectations are there about how a general audience should react to peer-reviewed work versus preprints?

The study design is well thought out, with a clear division between deductive and inductive content analysis.

I would urge the authors to make deidentified responses and corresponding codes available as a supplemental dataset, if possible (i.e., complies with IRB)

The term ‘unfinalized science’ could be misleading. Is science ever really finalized, even with journal publication?

How was PIUS for each study participant calculated? Averaging across the 7 items?

Table 1: median may be a better summary statistic to use, esp if underlying variable does not follow a normal distribution

Printing and publication: I enjoyed reading this section, it gives a great window into how some groups define preprints.

Participant characteristics: considering putting this before the other sections in the discussion.

US citizens implies citizenship/nationality. The methods section only described the groups as "US general population recruited using Qualtrics Panel Services." Did the authors ask the study participants to clarify their citizenship or nationality status? or where study participants resides?

Suggestions for further work

This study focuses on the US, it would be interesting to see studies looking at preprint perceptions globally.

I wonder if the authors made any distinctions among the students who were surveyed about their majors. It would be interesting to see the responses of science students vs humanities, business, etc. It would also be interesting if the other participants were asked about their occupation.

The authors allude to some misconceptions that came up in study participants’ definitions of preprint. It would be interesting to conduct a formal analysis to see what are the common misconceptions that students and the general public have about preprint.

Reflections on inductive analysis: It might be interesting to compare the average age of students vs the general public as different generations have different ways of thinking, so maybe the general population relates preprints to the literal meaning of preprint as they are more used to reading printed things. In contrast, younger generations are more used to electronic versions of publications.

An interesting extension would be to ask researchers and journalists how they would like the general public to interpret new reports based on preprints. How authoritative would they expect a general audience to consider such a report? This could be compared with the results here to see how the gap between the expectation of perception and the actual perception could be closed.

Competing interests

The authors declare that they have no competing interests.