Peer review process best method to determine scientific funding, study finds

LAWRENCE — While everybody these days has an opinion about science that affects us all, the wisest approach still involves consulting with key experts in the field. That also applies to funding research, according to a new article affirming that peer review is the most effective method of helping government agencies predict what ideas can best further scientific advancement.

“Peer review is even more necessary to biomedicine because some of those findings could be a matter of life and death,” said Donna Ginther, the Dean’s Professor of Economics at the University of Kansas.

Ginther is the co-author of “Administrative Discretion in Scientific Funding: Evidence from a Prestigious Postdoctoral Training Program.” The article appears in the current issue of Research Policy.

“You always want to have a reality check for your work,” she said.

Her study uses data from the National Institutes of Health (NIH) to determine that peer review – compared to agency program officer discretion – “more closely predicts high-quality science and future research independence.”

“The peer process review is very controversial. Many people are calling for a change,” said Ginther, who co-wrote the article with Misty Heggeness of the U.S. Census Bureau. “But we found that the peer review process for scientific grants, which is what we’re studying, is rigorous, structured and well-defined. We’re finding that process is working pretty well.”

They also discovered that “regression discontinuity design,” which is the method typically used to examine the effect of funding, doesn’t always apply to scientific-funding models.

She explained how every grant that is reviewed receives a score. At NIH, low scores are better than high scores: If you earn a one, you’re funded, and if you get a 20, you’re not funded. If peer review works as advertised, every proposal is funded based on the order of this score.

“But then we know there’s some break where you don’t have any money left,” she said. “If you compare the last proposal funded to the first proposal that wasn’t, that’s a regression discontinuity design. You’re looking at ones just above and below that line and asking, ‘What is the effect of those proposals of similar quality?’” 

“As we dug into the data, we realized, ‘Oh, they’re not doing it the way they say they do it.’ And since they’re not using a regression discontinuity design, and they’re reaching for proposals that didn’t score as well and skipping proposals that should have been funded if everything was funded in order, you can look at the effectiveness of peer review itself.”

Although this particular research involves the realm of science, it raises the question of whether it’s applicable to other fields. For example, is peer review crucial to the humanities?

“Academia is based a lot on peer review,” Ginther said. “I can’t necessarily speak to the humanities because they are more book-focused. But it’s always beneficial to have a fresh set of eyes on work to ensure it has the potential to make a contribution. And I think when you’re talking about large sums of money like NIH gives out, it’s really important.”

Now that the world is searching for ways of dealing with COVID-19 as quickly as possible, Ginther said scientific peer review could be fundamentally changed. It may begin to reflect how other fields approach the process.

“Economics is very open,” she said. “The profession shares working papers. The results of this study have been circulating for a couple of years. However, life science, up until the coronavirus, kept everything under wraps before it was published. And what we’re seeing right now is research findings are being put out before they’ve been peer-reviewed. Given the size of this crisis, making preprints available for studies of the coronavirus is critically important.”

A 17-year veteran of KU, Ginther specializes in labor economics. She is also the director of the Institute for Policy & Social Research, an interdisciplinary campus center for faculty and students doing funded work in the social and behavioral sciences.

“It’s the ongoing research of mine to see how following the money influences the sciences,” she said.

Ginther hopes “Administrative Discretion in Scientific Funding” can help settle any controversy concerning the necessity and relevance of peer review.

“There’s a lot of debate about this question in the science policy community,” she said. “But now the science policy people aren’t paying attention because they’re in a lab trying to save the world.”

Top photo: iStock