Is Peer Review Working?

Michael Eisen argues that the whole bedrock of the scientific enterprise is broken, and that the recent expose of how open access journals were happy to publish a bogus paper doesn’t tell the whole story. “Peer review is a joke” he asserts.

Any scientist can quickly point to dozens of papers – including, and perhaps especially, in high impact journals – that are deeply, deeply flawed – the arsenic DNA story is one of many recent examples. As you probably know there has been a lot of smoke lately about the “reproducibility” problem in biomedical science, in which people have found that a majority of published papers report facts that turn out not to be true. This all adds up to showing that peer review simply doesn’t work.

In a delicious bit of argumentative inspiration Eisen comes up with a wonderful analogy:

But unlike the fly by night OA publishers who steal a little bit of money, the subscription publishers’ long con has far more serious consequences. Not only do they traffic in billions rather than thousands of dollars and denying the vast majority of people on Earth access to the findings of publicly funded research, the impact and glamour they sell us to make us willing participants in their grift has serious consequences. Every time they publish because it is sexy, and not because it is right, science is distorted. It distorts research. It distorts funding. And it often distorts public policy.

To suggest – as Science (though not Bohannon) are trying to do – that the problem with scientific publishing is that open access enables internet scamming is like saying that the problem with the international finance system is that it enables Nigerian wire transfer scams.

Emphasis added, for purposes of emphasis.

Comments:

  1. > Any scientist can quickly point to dozens of papers – including, and perhaps especially, in high impact journals – that are deeply, deeply flawed

    I don't immeadiately recognise this, for climate journals. I could point to papers that are overhyped, or a bit boring. But deeply flawed? No, not really. Only a tiny tiny proportion. But, we can test this - we can attempt to name some. But can we quickly name dozens?

  2. “Peer review is a joke”

    I'm a bit horrified to see this in Beyond P3. Really? How is peer review meant to function? The clue's in the name. Thats its limitation and its power. What other approach is being suggested? Is it being argued that peer review is irrevocably corrupted by money? You want to support that argument? If so, some actual evidence?

    I think maybe I must be missing the point and should stop typing now.

  3. OK, so more typing. The thing about peer review: it has to function as a dynamic system, and the point about that is, it also must constantly face opportunities to adapt. That's what it's *meant* to do. That has to include situations where people publish things that are wrong. I'm about to publish a couple of papers from my PhD that haven't been exposed to my "peers" yet. One already got corrections I'm currently fixing before final publication (hopefully) and then a proper public slagging off (hopefully a constructive and developing slagging off!)

    What do we think peer review should be? Have we been warped by denial? Error is integral to the process. Visible, constant - gradual, controlled but occasionally completely bloody wrong and upside-down - error. What's this argument about peer review being a joke???

  4. the whole bedrock of the scientific enterprise is broken

    What nonsense! Dangerous clueless nonsense to propagate cluelessness.

    Peer review is just the first filter, so that the journals don't burst and minimize rubbish. Peer review needs some holes. It should be just a raw filter. Ask Lynn Margulis' ghost (on the one hand). Ask any overworked reviewer (on the other hand). Think about how long it often takes to really digest a paper, for a single reader or the whole relevant group of scientists.

    After publication comes the real digestion and evaluation.

    -------------------

    Here's a nice example of how review continues to work after post-publish review after peer review:
    Lynn Margulis, Richard Dawkins, and more, in the Homage to Darwin Debate at Balliol College, Oxford University UK (May 8, 2009)

  5. I can't really participate in that test, because I'm out of productive science business. And my science, mathematics is a special beast. Anyhow,

    I don't recall any rubbish paper. ZERO.
    Perhaps because I usually did not read journals, but papers.

    The journals are just containers. There were only 2-3 journals I regularly opened when they arrived at the library. And that were moments of leisure, prying about the jungle. Usually I sought out papers from references in other papers/books. Or from recommendations. Or the latest preprints of my Gurus. Most often I was tracing the root system of a particular outstanding paper/theory I studied.

  6. I think that the more serious problem is that the modern journals detach the 'story' part of a scientific article from the data. Ironically, this has grown in the internet era, not shrunk. The meat of most papers is now in the supplemental material, and the casual reader has to take the journal story more and more on faith. This makes peer review much more difficult.

    Furthermore, the tendency to hype the science comes in part from the failed idea at National Science Foundation that they only have money to fund 'transformational' science. So, every proposal has to make itself 'transformational, and a bias has been inserted to find something exciting. This like trying to build the roof of a house without putting in the foundation and walls. In my own field, the mantra is 'hypothesis-driven' even though we are for the most part data limited. So now we are hypothesis infested.

  7. Dan, I think your point is fine if inelegantly expressed. Perhaps the article was posted to get a reaction, but the action and reaction template is breaking the bank. We have real information, and it is being overwhelmed by lies. Some very clever people with tons of money are using the finest PR and strategy money can buy to create a parallel universe that looks plausible to less hypereducated people (a vast majority).

    It is not difficult to understand that copying is easier than creating, and the authenticity of the original can be gamed to create an appearance of validity. In the years I have been trying to find ways to get past this, I have found my best points turned inside out, upside down, or otherwise distorted to make them support opposite views in one way or another. It is frustrating that one's best work is stolen and perverted, but I know of no way to prevent this happening except to remain silent, which is also unacceptable.

    In a world where warnings about real danger are urgently needed not just at the higher levels but on the streets so we can make significant changes in how we function to conserve, protect, and preserve, it is tragic that this is so.

  8. There are two points here.

    1) Peer review does not function very well as a filter. Good stuff gets damaged and delayed by the existing process, and bad stuff gets through.

    2) The Big Two are increasingly choosing papers for controversy and grandiosity rather than quality or scientific importance, and thus falling into the same traps that commercial journalism does.

    I think both are defensible.


Leave a Reply

Your email address will not be published.