Peter A. Todd of the National University of Singapore, and Richard J. Ladle of Oxford University, write in Correspondence this week (Nature 451, 244; 2008):

On 22 November, the Higher Education Funding Council for England announced that the assessment and funding of science-based disciplines will in future be “based on citation rates per paper, aggregated for each subject group at each institution”.

Changes in performance indicators always strongly influence individual and institutional behaviour and ‘citation game-playing’ will no doubt become a staple of coffee-room conversation. What is less clear is how the citation practices of authors may influence bibliometric indicators.

Citation practices are known to be imperfect. The documented problems include excessive citation of an author’s own work. Papers cited can be inappropriate or ambiguous in their support and, in some cases, the authors may not have read the papers they cite. Authors may form ‘citation coalitions’ within research networks. They may fail to provide citations to intellectual precursors or to work reporting conflicting conclusions. There are geographical and language biases. The increasing number of many-authored papers makes it impossible to have a clean-cut general metric in which one author is associated with one paper.

Taken together, these factors represent a problematic degree of error for the proposed bibliometric system of assessment. They place added responsibility on journal editors and reviewers as arbiters of appropriate author conduct.

Unfortunately, there are no simple solutions. Currently, identifying poor citation practices is not emphasized in the peer-review process, so perhaps journals could adopt a system of random citation audits, or periodically request evidence of citation appropriateness from authors. In reality, time constraints and the sheer volume of submissions to many journals mean that such measures are unlikely to be implemented soon.

Until referencing practices improve, we would argue that using citation rates to assess performance is fundamentally flawed.

From this week’s issue of Nature, a news story by Declan Butler “Free journal-ranking tool enters citation market” (Nature , 451, 6; 2008)

“A new Internet database lets users generate on-the-fly citation statistics of published research papers for free. The tool also calculates papers’ impact factors using a new algorithm similar to ”https://en.wikipedia.org/wiki/PageRank">PageRank, the algorithm Google uses to rank web pages. The open-access database is collaborating with Elsevier, the giant Amsterdam-based science publisher, and its underlying data come from Scopus, a subscription abstracts database created by Elsevier in 2004.

The SCImago Journal & Country Rank database was launched in December by SCImago, a data-mining and visualization group at the universities of Granada, Extremadura, Carlos III and Alcalá de Henares, all in Spain. It ranks journals and countries using such citation metrics as the popular, if controversial, Hirsch Index. It also includes a new metric: the SCImago Journal Rank (SJR)."

The article goes on to discuss the new metric, and to compare it with others, notably the dominant Impact Factor of Thomson Scientific. Thomson has set up a web forum to respond formally to an editorial in the Journal of Cell Biology, in which Mike Rossner and colleagues point to discrepancies between their “independent audit” calculations of Impact Factors based on Thomson data, and Thomson’s own calculated Impact Factors from the same data.

Citation-based quality metrics were discussed on Nautilus earlier this year. One of those, the h (for highly cited) index, was covered recently in a News story, and is the subject of two Correspondence letters in the current issue of Nature.

Michael C. Wendel of Washington University Medical School writes (Nature 449, 403; 2007):

The h-index (the number n of a researcher’s papers that have received at least n citations) may paint a more objective picture of productivity than some metrics, as your News story ‘Achievement index climbs the ranks’ (Nature 448, 737; 2007) points out. But for all such metrics, context is critical.

Many citations are used simply to flesh out a paper’s introduction, having no real significance to the work. Citations are also sometimes made in a negative context, or to fraudulent or retracted publications. Other confounding factors include the practice of ‘gratuitous authorship’ and the so-called ‘Matthew effect’, whereby well-established researchers and projects are cited disproportionately more often than those that are less widely known. Finally, bibliometrics do not compensate for the well-known citation bias that favours review articles.

Clint D. Kelly and Michael D. Jennions of Australian National University, write (Nature 449, 403; 2007):

The h-index seems to be breaking away from the bibliometric pack, in the race to become a favoured measure of scientific performance (‘Achievement index climbs the ranks’ Nature 448, 737; 2007). However, if the h-index is to become an assessment tool commonly used by university administrators and government bureaucrats, those using it should be aware of its pitfalls.

As noted in your News story, tallying how many papers a researcher publishes (their productivity) gives undue merit to those who publish many inconsequential papers. But at least for ecologists and evolutionary biologists, the h-index is highly correlated with productivity.

This is worrisome, because the h-index is easily misconstrued as an equitable measure of research quality. We offer two examples.

First, female ecologists and evolutionary biologists publish fewer papers than their male counterparts, and they have significantly lower h-indices. Should administrators therefore conclude that men are better researchers? No. The gender difference vanishes if we control for productivity. It seems unlikely that this phenomenon is restricted to ecology and evolution.

Second, the h-index increases with age and using the ratio of the two can be problematic. Therefore, reliably comparing the performance of younger researchers with older ones is difficult.

Your views are welcome.

What’s so wonderful about citations? asks Cambridge professor Peter Murray-Rust. Prof Murray-Rust has looked on Google Scholar for a paper which according to the publisher has more than 100,000 accesses, and found that it has 92 citations over the same period, which translates into one citation for every 1,000 (or so) downloads.

Prof Murray-Rust applied the same logic to himself. He was told by a publisher that his paper had been downloaded 6,000 times, so expected to find about 6 citations on Google Scholar — but in the event found only one. “I’m not saying there are better ways – there probably aren’t”, he writes. “If we make downloads a metric, then people will try to distort them. But let’s not take this [citation analysis] as seriously as we do.”

Antonio G. Valdecasas and Uta Grothkopf write:

Leaving aside the adequacy of using citation counts to evaluate scientists’ merit (as discussed for instance by Peter Lawrence in Nature 422, 259-261; 2003), there is the associated problem of exhaustive gathering of citations for certain kinds of publications, in particular monographs and book series that are not routinely covered by the Science Citation Index (SCI).

In countries like Spain where, as Lawrence points out, bureaucrats carry out scientists’ assessment with “formulaic precision”, there is, therefore, a risk of underrating this type of publication. Examples can be as extreme as to receive a dictum of almost ‘no impact’ for a monograph whose articles have been cited more than 150 times in journals included in the SCI, 19 times in non-SCI journals, 10 times in proceedings and which contains several articles in ‘rare’ languages such as Russian and Chinese. These citation results can be easily found through a simple search in Google Scholar. Not a bad citation number for a discipline (morphometrics) that has a mean number of citations for the period 1996-2005 of 8.5 (n= 1,408) citations per article, but most of them would be lost if one relies exclusively on the SCI.

Maybe the days of the SCI are numbered, as is already the case in disciplines such as astronomy, where alternative services are used. If impact is to be used as a metric that affects people directly, then databases like Google Scholar — free, accessible to everybody, and non-discriminatory against languages other than English — could provide a tool of universal coverage for bureaucrats and evaluation committees to discover the real impact of publications and hence to be less biased in the distribution of benefits.

Antonio G. Valdecasas

Museo Nacional de Ciencias Naturales, Madrid, Spain

Uta Grothkopf

European Southern Observatory, Garching, Germany.

Scopus, the abstract and citation database of peer-reviewed literature and quality web sources, will incorporate the h-index soon. The h-index considers the publication records of an individual, the number of papers published over n years and the number of citations for each paper. The result is a single number, the h-index. To provide the user with additional clarity Scopus sys it will include visual aids that present a transparent overview of citation and publication patterns over time, revealing whether the h-index is dependent on a few highly cited papers or that the author’s papers have a relatively consistent volume of citations.

See here for a recent Nature Commentary on the h-index; and see here for a 2005 Nature news story “Index aims for fair ranking of scientists”.

In the Editorial of the April issue of Nature Clinical Practice Rheumatology (3, 189; 2007), Editor in Chief Peter E. Lipsky writes: "The IF [Impact Factor] was envisioned over 50 years ago with the purpose of eliminating “the uncritical citation of fraudulent, incomplete or obsolete data by making it possible for the conscientious scholar to be aware of criticisms of earlier work” (Garfield E ¹⁹⁵⁵ Science 122: 108–111). The IF has subsequently morphed into an institutionalized means of ranking the quality of scientific journals and, by implication, the individual articles published within them; for researchers, the IF influences employability, promotion, grant acceptability and bonus payments, and has been likened to a popularity contest."

Dr Lipsky discusses various flaws in the Impact Factor evaluation system, such as the lack of transparency of the formula by which the IF is calculated; that an erroneous but frequently cited article will bolster the ranking; disproportionate representation of review articles; and the differences in publication and citation frequencies in different disciplines. He asks whether his journal even actually wants an IF when it becomes eligible, concluding: “We do not feel that the current IF will reflect either our quality or our potential influence on clinical practice. Only if a new validated metric is developed that can evaluate the true quality and value of journal articles can we make real progress in improving the communication of new information in clinical medicine.”

The full editorial is available at the journal’s website or as a PDF here: Download file.

Comments are very welcome.