Nature Medicine | Spoonful of Medicine

The jury’s (way) out

When it’s not a trip, my day job gets in the way of my posting something to Spoonful. This week we closed the June issue of Nature Medicine, and right now I’m at the airport, about to start another ‘tour’. So, while I wait for the PA system to herd us to the plane, I thought I would blog about my day as a juror.

For the second year running, a very dear friend of mine invited me to be part of the jury for an award that her organization gives to young scientists. And for the second year running, it turned out to be a fun day out.

The award recognizes young scientists in all disciplines. As a result, the jury (composed of nearly 40 people) was a very eclectic mix of basic scientists, engineers, physicists, matemathicians, you name it. There were even editors like me, whose only expertise lies on the inexact science of rejecting papers.

The discussion, which took the best part of the day to go over something like 40 finalists, was free-form. One of us would go over the candidate, and the rest would ask questions or bring up caveats about each of them, trying to understand the importance of their contributions.

I must confess that, halfway through the session, I started feeling sorry for those scholars who have to decide on, say, people’s grants.

In the case of our jury, we found it was pretty tricky to decide how much weight to give to the candidates’ letters of recommendation, to their number of papers and the journals in which they were published (someone referred to the high-profile journals as ‘vanity journals’, which struck a chord with me), to the number of citations, and to a plethora of factors that, one way or another, represent the blood, sweat and tears of a scientist.

Does a scientist who has three patents in the past five years, but only three papers, each of which had been cited just three times deserve more recognition than the scientist with five Nature papers and 1000 citations? Does a scientist who works in a hot field and has made nice contributions deserve more credit than another one who works in a less glamorous, lonelier field and has made equally profound contributions?

These questions aren’t always easy to answer, but my friend got it right because she quickly realized that there’s strength in numbers. So, if you have 40 judges from different fields and with different ways of evaluating science, chances are that the truly outstanding pieces of work will triumph over the rest. I’m therefore confident that our eclectic group made the right choices.

But returning to those grant evaluators, I was saying that I felt sorry for them because, if we managed to screw up and made a mistake yesterday, it won’t make much of a difference for the scientists who should have won and didn’t despite our best efforts. They’ll carry on and will still be great scientists. In the case of grant decisions, though, getting them wrong can lead to a lab’s shutdown, to fired postdocs and to truncated careers, which, alas, are becoming more and more common.

jury.jpg

Photo by JasonUnbound

Comments

  1. Report this comment

    Achaetes McNeil said:

    What are you talking about, dude? You editors have as much power over people careers as a grant officer. Have you ever pondered how many people didn’t get tenure because you rejected their papers? I find it very disingenious and hypocritical of yours to “feel sorry” for people who evaluate grants. I’d say “Physician, heal thyself”.

  2. Report this comment

    JCL said:

    I wouldn’t exactly compare the effect of a rejection letter from us to the effect of having your grant turned down.

    If I make a bad editorial decision and the paper is good, then another high-profile journal will benefit from my mistake and you’ll get your career-advancing paper. So, if anything, my journal is screwed over as a result of poor judgement from my team and me.

    But if your NIH grant is turned down, where are you gonna go? Sure, there’s plenty of smaller grants, the same way that there are plenty of specialized journals. But who would want to adopt such a survival strategy, as opposed to having enough resources to do good research?

  3. Report this comment

    Sergio Stagnaro MD said:

    An Italian famous Mina’s song sounds: “Parole, Parole, Parole”. You surely understand both Italian PAROLE and what I mean. I am an aged researcher in the field of physic semeiotics, who may not hope to receive awards. However I reflect on possible jury’s consideration of my Lory’s experiment, based on NON Local Realm in biological systems, besides the well known Local Realm. As a consequence of such as discovery, now a days physicians can make a CLINICAL diagnosis, even if a patient is hundreds of KM away, due to entanglement phenomena.

  4. Report this comment

    Sesli Chat Sohbet said:

    I agree with Achaetes ….JCL your editors do have as much power as a grant officer over people….I think you should re-evaluate some decisions..!!

  5. Report this comment

    Chris Muller said:

    I have to agree with Achaetes here in that with people struggling to publish in certain journals, there is so much snobbery that publication in these so-called “other high-profile journals” is not considered as prestigious. The following excerpts sound rather flippant, insensitive and suggestive of misplaced sympathies, considering the impact that such decisions really have on scientists’ lives. Dr. Stagnaro—I am not at all clear about your meaning! You’re like my Organic Chemistry prof—he’s a wonderful man but none of us really knew what he was talking about! But we were all fond of him because among all those indifferent, careerist faculty members, he was a gem (albeit an aged one) who tried his best to help the few students who approached him for help. Cheers!

    “But returning to those grant evaluators, I was saying that I felt sorry for them because, if we managed to screw up and made a mistake yesterday, it won’t make much of a difference for the scientists who should have won and didn’t despite our best efforts. They’ll carry on and will still be great scientists. In the case of grant decisions, though, getting them wrong can lead to a lab’s shutdown, to fired postdocs and to truncated careers, which, alas, are becoming more and more common.”

  6. Report this comment

    JCL said:

    Thanks for those of you who got in touch to say that some of my comments were insensitive. I got the same reaction even from people in my team!

    I should clarify, though, that I didn’t mean to sound flippant. I don’t like to make bad decisions, the same way that no-one likes to make a mistake at work. The fact remains, though, that papers have a way to find their right niche: I sometimes see papers that we turned down published in Cell, Science, the NEJM and even Nature, the same way that I’m sure that the editors of all of these journals see some of their rejections in our pages. As I said, if I made a mistake and your paper is good, it will find a great home. But if you shop the paper around and none of these journals bites, then one ought to conclude that the contribution wasn’t so great after all.

    There aren’t as many sources of R01 grants as there are high-profile journals. So, the stakes are higher for those who have to make decisions on those grants than on a paper.

  7. Report this comment

    sesli sohbet said:

    What are you talking about, dude? You editors have as much power over people careers as a grant officer. Have you ever pondered how many people didn’t get tenure because you rejected their papers? I find it very disingenious and hypocritical of yours to “feel sorry” for people who evaluate grants. I’d say “Physician, heal thyself”.

    COMMENT FROM JCL:

    As I said in the original post and elsewhere: if I reject someone’s paper and the paper is good, it will be published in a high-profile journal because not everyone would be as stupid as I was to not see the importance of the work. I therefore disagree that I’ve wrecked careers through my influence.

  8. Report this comment

    sesli chat said:

    What are you talking about, dude? You editors have as much power over people careers as a grant officer. Have you ever pondered how many people didn’t get tenure because you rejected their papers? I find it very disingenious and

  9. Report this comment

    e-okul said:

    An Italian famous Mina’s song sounds: “Parole, Parole, Parole”. You surely understand both Italian PAROLE and what I mean. I am an aged researcher in the field of physic semeiotics, who may not hope to receive awards.

  10. Report this comment

    sesli sohbet said:

    There aren’t as many sources of R01 grants as there are high-profile journals. So, the stakes are higher for those who have to make decisions on those grants than on a paper.

  11. Report this comment

    Özgür Sesli Chat said:

    If I make a bad editorial decision and the paper is good, then another high-profile journal will benefit from my mistake and you’ll get your career-advancing paper. So, if anything, my journal is screwed over as a result of poor judgement from my team and me.

  12. Report this comment

    Sesli Chat Seslisohbet said:

    Why? You editors have as much power over people careers as a grant officer. Have you ever pondered how many people didn’t get tenure because you rejected their papers?

  13. Report this comment

    Kameralı Sohbet said:

    Thanks for those of you who got in touch to say that some of my comments were insensitive. I got the same reaction even from people in my team!

    I should clarify, though, that I didn’t mean to sound flippant. I don’t like to make bad decisions, the same way that no-one likes to make a mistake at work. The fact remains, though, that papers have a way to find their right niche: I sometimes see papers that we turned down published in Cell, Science, the NEJM and even Nature, the same way that I’m sure that the editors of all of these journals see some of their rejections in our pages. As I said, if I made a mistake and your paper is good, it will find a great home. But if you shop the paper around and none of these journals bites, then one ought to conclude that the contribution wasn’t so great after all.

  14. Report this comment

    Çet said:

    As I said in the original post and elsewhere: if I reject someone’s paper and the paper is good, it will be published in a high-profile journal because not everyone would be as stupid as I was to not see the importance of the work. I therefore disagree that I’ve wrecked careers through my influence.

  15. Report this comment

    Medyum said:

    What are you talking about, dude? You editors have as much power over people careers as a grant officer. Have you ever pondered how many people didn’t get tenure because you rejected their papers? I find it very disingenious and hypocritical of yours to “feel sorry” for people who evaluate grants. I’d say “Physician, heal thyself”.