In his Nature Network summary of a recent publishing meeting, Yiorgos Apidianakis describes his opinion that a peer-review “score” is a more desirable (and efficient) indicator of scientific excellence than the currently used impact factor of the journal that publishes the work. Charles G. Jennings’s wrote, as part of Nature’s peer-review debate, “It is common to bemoan the over-reliance on quantitative markers such as impact factors for assessing scientists’ abilities (and indeed there is much to bemoan), but until committee members have time to read every paper on every applicant’s CV, they will have to rely at least in part on proxy indicators.” Dr Apidianakis believes that an ideal indicator would be a score from a unified peer-reviewing system, or a central agency “that will thoroughly, rigorously and objectively evaluate any given work to be published, using again specialized scientists as reviewers. Having an evaluation score from such an agency, scientists can include this score to their publication record.”
Dr Apidianakis thinks that such a system could work in practice, using the example of the NIH evaluation scoring system — while admitting it would be very expensive. Irrespective of this practical obstacle (as well as others) I wonder how the quality of the agency’s “score” can be standardized? The establishment of, in effect, one giant journal of research, with all the world’s scientists signed up to it as peer-reviewers, is a stimulating concept. But, given the many divergences of views within fields, how could a centralized scoring system work? The IPCC has attempted a similar kind of approach for one discipline, climate change — even though most scientists in the field broadly agree with the IPCC’s assessment of research output, this consensus requires massive bureaucratic baggage, including many international meetings and vast reports justifying decisions. Yet there is a substantial minority of scientific dissentors, and many members of the public are sceptical of a unified approach resulting in “science by endorsement”.
The current peer-review system works very well “in miniature”, whether as operated by those journals able to call on the most thoughtful scientists in a field who do the most cutting edge research, or by literature reviews or research ranking services, usually written or operated by one or a few individuals. Scientific research itself has benefited from large author collaborations across the whole spectrum of disciplines, from astronomy and nuclear physics to genomics and cell signalling, facing the challenge of making sense of vast quantities of data. Are there the same intellectual and innovative advantages to be gained by a single-managed peer-review system for all of science?