Correlation of metrics with expert judgement
This Correspondence by Stevan Harnad of the universities of Montreal, Canada and Southampton, UK, was published in Nature last week (Nature 457, 785; 2009):
Your Editorial 'Experts still needed' (Nature 457, 7–8; 2009, free to access online) is correct in that no metric alone can substitute for expert evaluation, because no single metric (including citation counts) is correlated strongly enough with expert judgements for it to take their place. But some individual metrics, such as citation counts, are nevertheless significantly correlated with expert judgements. It is likely that a battery of multiple metrics, when considered jointly, will be even more strongly correlated.
The UK Research Assessment Exercise (RAE) provides such an opportunity, alongside the wealth of potential performance indicators that are increasingly available online. Both enable a candidate battery of metrics — such as citations, co-citations, downloads, tags and growth/decay metrics — to be systematically validated against expert judgements, field by field. The 2008 RAE has also provided data that make it possible to do this validation exercise now, across all disciplines, on an important nationwide scale.
