Peer-to-peer

Two letters about Nature's peer-review mentoring scheme, published in the journal's Correspondence pages earlier this month (Nature 448, 129 and 130; 2007):

Your Editorial 'Mentors of tomorrow' (Nature 447, 754; 2007) highlights a need to encourage ethical, honest and fair peer review by young scientists. Although I applaud the ethos of the argument presented, graduate students such as myself often suffer from anonymity in their field of research, even though our work is often at the cutting edge. A lack of publications can render a student invisible to editors and may result in missed opportunities to offer their services as referees.
I suggest that journals consider introducing a 'PhD student peer-review pool' to which students and their supervisors can sign up. Such a database, including a student's name, area of research expertise and current supervisors, would provide editors with a ready supply of willing referees. Editors could try new referees in the knowledge that they will be supported during the review process by their supervisor, and could provide feedback to the student about the quality of the report.
Refereeing has often been described as a thankless task, but although it does require considerable effort, it also provides invaluable experience in critical interpretation of science. Having recently completed my first review, I believe that it has made me far more objective about my own writing and can only benefit the production of my thesis. I therefore look forward to receiving my next invitation to review.
Angelo P. Pernetta, Centre for Ecology and Hydrology, Wareham, UK

Thank you for your excellent Feature, 'Nature's guide for mentors' (Nature 447, 791–797; 2007). It's definitely one of the best things I've read in Nature in the 25 years I've been reading the journal. To better help students make informed choices about choosing a mentor, it would be enormously useful if public granting agencies such as the US National Institutes of Health would publicly post the 'trainees' lists that are included in training grants for every faculty member in a given PhD training programme. If this were done, students considering applying to those labs would know their actual chance of being mentored successfully
Ben Barres, Department of Neurobiology, Stanford University School of Medicine, USA

See here for earlier Peer to Peer posting containing the Editorial and comments from readers.

Posted by Maxine Clarke on July 27, 2007
Categories: Mentoring and training | Permalink | Comments (0)

Pedro Beltrao of Public Rambling writes a stimulating post exploring whether the editor could be a "value unit" for authors and their publications, independent of the journal. He asks:"could there be freelance editors? Could the editors be separated from the publisher? Imagine [if] I read a paper from a pre-print server, ask some people to peer-review (why would they?) and sell our evaluation to a journal.
Also, can a publisher sell the editorial decision to another publisher? Lets imagine a journal that has a very high rejection rate, the editor asks referees for comments but ultimately the manuscript is rejected. The editor could then ask the authors where they want to send it next and offer to provide the referee report and editorial comments directly to the next journal to expedite the process. Could this journal get paid for this?"
Pedro is not referring here to the transfer system that some publishers, including Nature Publishing Group, offer between their journals, but to the editor taking a role rather like that of an agent in book publishing, as supervisor of the peer-review process and then advocate of the "package" of manuscript and reviews in finding the best possible journal in which to publish the work.

Posted by Maxine Clarke on July 24, 2007
Categories: Quality and value | Permalink | Comments (0)

Mario of Mario's Entangled Bank blog discusses an article in Wired magazine about Nature Precedings. Mario higlights Wired's criticisms of Precedings for its "controversial" title, and for offering to upload scientists' "effluvia and detritus" that would never appear in a journal. To the contrary, he writes, the site (on which he has been a beta tester) states that it "provides a rapid way to disseminate emerging results and new theories, solicit opinions, and record the provenance of ideas." From Mario's post:

"In other words, contrary to detritus and effluvia, Nature Precedings will host research at the cutting edge, some of which may have errors in it that will be "corrected" with the collaborative efforts of the research bazaar. Most of the posted research will, however, be correct (after all, it is your reputation that is at stake) and it's early pre-print disemination will only be of benefit to the scientific community at large. This basically also addresses the last paragraph, "There's also a danger of errors being missed that would have been picked up in the peer review process". On the contrary, the recent PLoS retraction (see Show me the code and Exemplary retraction of high profile paper) confirms what we all academic researchers already know, that the peer review process is far from being fool proof. A forum like Nature Precedings, or arXiv for that matter, aid in the peer review process by disseminating pre-print cutting edge research findings thus allowing errors to be caught early on."

Posted by Maxine Clarke on July 20, 2007
Categories: Systems | Permalink | Comments (0)

The pseudonymous FemaleScienceProfessor is also an editor for a journal. She has reviewed its reviewers:

"I did a quick, statistically invalid analysis of the reviewer data for the past year to see whether the time it took a reviewer to complete the review was random or correlated with seniority. My working hypothesis was that younger scientists do quicker reviews. The dataset is sufficiently large to make an analysis like this reasonable, but I wasn't rigorous about tracking down reviewer time-from-Ph.D. data. I put reviewers in one of several bins: postdoc, assistant professor, mid-career, late-career, retired, and I put research scientists into these same bins based on where they would be in terms of time since Ph.D. if they were tenure-track. It's not a perfect system, but I just wanted to get a sense for any trends. The quickest reviewing groups are the early-career and retired scientists. "

More analysis is provided in FemaleScienceProfessor's post, and there are some reactions in the comments to this post. At the Nature journals we do not publish reviewer statistics of this type, nor do we capture information about the reviwers' gender, seniority level and so on. Is there interest from among our peer-reviewers to know these statistics, and to have published the type of information provided by the preliminary results of FSP?

Posted by Maxine Clarke on July 15, 2007
Categories: Quality and value | Permalink | Comments (0)

From a Correspondence in this week's Nature (448, 129; 12 July 2007)
Much emphasis has been put on the importance of policies that require authors to be transparent about financial conflicts of interest. Nature, for example, requires most authors to submit a declaration of any competing financial interests in relation to the work described in a submitted article. The reason why journals have these policies is, presumably, to safeguard the objectivity of the research. Transparency is thought to promote objectivity because if readers are aware of potential financial conflicts, they can critically evaluate the ways in which such interests may have affected the research — for example, in the selection of evidence, interpretation of results, or research methodology.
Yet transparency is insufficient as a safeguard of objectivity. Scientific expertise is necessary to correctly evaluate whether conflicts have biased the research, yet financial conflicts are revealed only when an article is published. This prevents peer-reviewers — who are in the best position to evaluate the possible influence of the conflicts on interest — from having access to the information. Thus, it is not clear to us how revealing financial interests in a statement accompanying publication of an article can allow readers to make accurate assessments of bias.
In addition, these policies foster an abrogation of scientific responsibility by the research community, because they put the burden of critical evaluation on the public, who in the main are not scientifically knowledgeable at a detailed level. This aspect is of particular concern for papers in journals such as Nature, which are likely to be widely disseminated to the public by the media. Even if biases are identified after publication and a correction made, such criticisms tend not to be publicized to the same extent as the original article.
If the aim of conflict-of-interest policies is to promote objectivity and inform readers and the public, we believe a more effective approach would be for authors to be required to reveal possible financial competing interests, not only to the public after publication, but also to reviewers during the peer-review process.
Inmaculada de Melo-Martín
Weill Cornell Medical College, New York
Kristen Intemann
Montana State University, Montana

Note from the editors: Nature’s policy is to ask peer-reviewers to exclude themselves in cases where there is a significant conflict of interest, financial or otherwise. We do not reveal authors' competing interest declarations to peer-reviewers while the manuscript is under consideration.

Posted by Maxine Clarke on July 11, 2007
Categories: Ethics | Permalink | Comments (2)

The ethics of journalism don't work for science | comment | EducationGuardian.co.uk

Professor (of philosophy) Jonathan Wolff describes (in the article at the above link) hearing a lecture and reading a subsequent published article in a scientific journal (Science) by Professor Naomi Oreskes. Prof Oreskes surveyed hundreds of peer-reviewed articles on climate change, none of which "denied that the Earth was warming or that human action was at least partially responsible." The sceptics, she argued, were largely members of independent think-tanks, publishing their own reports without external review.
Yet when Prof Oreskes published her article, writes Prof Wolff, she "was immediately shot down by bloggers, journalists and think-tankers, who mixed insults about her honesty with more plausible-sounding complaints about her methodology. Oreskes replied, with great restraint, that she would wait for the peer-reviewed criticisms."
His observation led Prof Wolff to contrast journalistic and scientific ethics. In reporting political arguments, each claim must be countered so that a lively debate can take place and readers come to their own views, he writes. Journalists are mistaken in applying the same ethical code of 'balance' to scientific reporting. "Whenever a story on climate change is produced, a maverick nay-sayer is rolled out for the sake of balance. But this misleads the public into thinking that a few lone voices have equal weight to the scientific orthodoxy." Prof Wolf also provides the example of the few people who deny a role for HIV in AIDS, yet make a disproportionate amount of noise.
Can non-scientists understand scientific discussions, asks Wolff? "We all study science for a few years, but learn - or at least remember - very little about methodology. Science is presented as a body of known truths. As adults, though, we need to know not the atomic number of chlorine, but how to assess evidence for or against a theory."

Posted by Maxine Clarke on July 03, 2007
Categories: Ethics, Policy and Public | Permalink | Comments (2)