How Much NSF Grant Money Can One Bad Apple Control?

First: Scientiae #3 and Philosophia Naturalis #8 are both out. Go read!

Second: a friend of mine was rejected for an NSF training fellowship. Two of the three reviewers liked her proposal, but the third review was very negative, which, well… sometimes it happens. But in the section evaluating the “broader impacts”, instead of addressing what she actually wrote, he (reviewers are anonymous, I use the male pronoun here for convenience) noted that she was “truly intelligent” and also a woman, and opined that this alone should fulfill the requirement of broad participation. He hoped that she would resubmit a proposal that was not complete garbage, because she works in an important subfield and has potential. (In case you are curious, the full question to which the reviewer was responding was What are the broader impacts of the proposed activity? In addition, please consider the prospective benefits to the applicant, the scientific discipline and the United States.)

Now, this review made us both want to puke on the reviewer’s shoes, but we found it difficult to articulate why. There are any number of horrible implications one can read into that brief paragraph: that women’s proposals should be judged by lower standards, that truly intelligent women are very rare, that the original broader impacts statement did not contain anything interesting to talk about, that the reviewer was thinking about her gender the entire time he was writing the rest of the review and was too distracted by wondering about boobiez to pay attention to the science, etc. Presumably the reviewer would vehemently deny all of these things, but they aren’t all things that operate at a conscious level.

Personally, what makes me roll my eyes the most is the thought that after he had finished trashing her application, this reviewer gave himself a pat on the back for saying something nice and supporting the cause of women in science. Which is not to imply that he should have held back, if he really thought her proposal was garbage, but he should’ve realized that “She’s smart, and a woman!” tacked on to the end of an extremely negative review would smell like a road that’s been traveled by a caravan of sanctimonious donkeys.

One of the benefits of a multi-person review panel is that if you happen to draw a bad card, someone who’s not going to give your proposal a fair shake because of your ethnicity or gender or institutional affiliation or you’re not using the lab technique he invented or whatever, the other two reviewers can balance that out. However, in a highly competitive grant program like the NSF’s, you will eventually find yourself competing against proposals that didn’t have a weirdo on the panel, or didn’t trigger the weirdo’s biases, and got three good reviews as a result. Which means that adding more reviewers is not a good defense against the political nature of the process.

This particular fellowship was not awarded through a blind review process, wherein the scientific merits of a project are evaluated before the panel knows who submitted it (another criterion for awarding NSF fellowships, the competence/promise of the scientist, obviously can’t be evaluated blindly). I really think it ought to be. I’m not sure whether or not blind review produces significant changes in the racial or gender distribution of fellowship recipients (and am too lazy to research the matter right now, but perhaps the lazyweb can do it for me?), but it’d be really nice if we could get negative feedback on the scientific merits of our proposals without wondering if racism or sexism played a role.