Peer Review II
Been thinkin’ about this problem of government peer review (though the monsterpost hasn’t been pushed beyond Tuesday’s edition, alas) and my intuition just flipflopped. Maybe industrial peer review is actually a good idea, sort of.
The fundamental problem with peer review is that reviewers are
lazy busy. It’s easy to read a paper, do a few back-of-the-envelope calculations, and nod your head when the methods used seem reasonable. It’s much harder to actually pick through all the calculations, tinker with critical parameters, etc etc. If there are devils in the details, many reviewers will let them lie.
I’m not sure that existing “reputational” incentives are strong enough to prevent this. A reputation for good peer reviews just gets you more peer reviews, which are neither glorious nor superly-duperly lucrative. And there’s variously imposed deadlines and cash-conscious funding agencies to consider, too, so Ye Olde Dysinteressed Reviewer can’t always be trusted to pick apart all the things that need picking.
Someone with a vested interest in picking apart a study, on the other hand, can always be trusted to pick it apart to the fullest extent possible. Which is great if you’re a scientist trying to improve your study, and you have the perspective necessary to see which criticisms are important and which are absurd. Not so great if you’re Senator Joe Blow who can’t tell relative error from earwax. Vested interests are big on earwax.
Which leaves us with fundamentally the same problem of finding able, disinterested parties to sort the errors from the earwax… or possibly producing a population of Super Expert Citizens who are able to use scientific common sense, which is also a popular fantasy.