Nature Geoscience’s September editorial (1, 563; 2008) broaches the subject of impact measures. From the editorial:

The ripples of the revolution in science evaluation have long reached the relatively uncompetitive backwaters of the geosciences. Indeed, Nature Geoscience received questions regarding its likely future impact factor before it was even accepted into Thomson Scientific’s Web of Science in April this year. So here are a few thoughts on the topic from us, long before our own impact factor (due in 2010) may skew our perspective.

Citation patterns vary hugely between disciplines. The impact factors of Nature and Science have ranged between 26 and 32 in the past few years. But a quick estimate, based on a sample of papers, suggests that geoscience papers in these journals score an impact factor of around 15 when evaluated on their own. This is high considering that the impact factors of journals publishing exclusively geoscience research have not exceeded 5 in the past several years. But far higher citation counts in the biological sciences drive up the statistics of journals that publish across disciplines.

The timescales of the publication cycle in a field determine a journal’s impact factor. These are defined as all citations in one year to citable content published in the two preceding years, thereby excluding all references more than two years from publication. This can be problematic for the slower-moving sciences. For example, the ten most cited papers in Geology in 2004 were collectively referred to about 1.5 times more often in 2007 than in 2006 — citations that have never entered the index.

For geoscientists, taking guidance from impact factors alone would mean favouring interdisciplinary journals (whereas many biologists would, for the same reasons, favour their own disciplines). It would also lead to reading preferentially short-lived, quickly cited papers over those that develop more slowly — not necessarily a good idea. Other more time-consuming ways of assessing quality are therefore needed to supplement the quick and easy number check.

The 2007 Impact Factors are now out (published on 17 June 2008). The ten Nature Publishing Group journals with the highest Impact Factors are as follows:

1 NAT REV MOL CELL BIO 31.921

2 NAT REV CANCER 29.190

3 NATURE 28.751

4 NAT REV IMMUNOL 28.300

5 NAT MEDICINE 26.382

6 NAT IMMUNOLOGY 26.218

7 NAT GENETICS 25.556

8 NAT REV NEUROSCI 24.520

9 NAT REV DRUG DISCOV 23.308

10 NAT BIOTECHNOLOGY 22.848

The Impact Factors of the Nature journals that publish original research are:

1 NATURE 28.751

2 NAT MEDICINE 26.382

3 NAT IMMUNOLOGY 26.218

4 NAT GENETICS 25.556

5 NAT BIOTECHNOLOGY 22.848

6 NAT MATERIALS 19.782

7 NAT CELL BIOLOGY 17.623

8 NAT NEUROSCIENCE 15.664

9 NAT METHODS 15.478

10 NAT NANOTECHNOLOGY 14.917

11 NAT PHYSICS 14.677

12 NAT CHEM BIOLOGY 13.683

13 NAT STRUCT MOL BIOLOGY 11.085

(Nature Photonics and Nature Geoscience are not old enough to have been awarded an Impact Factor this year.)

Readers can create their own lists of journals by subject area, title, Impact Factor or publisher, at ISI Web of Knowledge.

There is a free-access account at the ThomsonISI website which explains how the Impact Factor for journals is calculated.

Discussion of the 2007 Impact Factors, and of citation in science in general, is taking place at the Nature Network Citation in Science group, which you are warmly invited to join.

The June editorial in Nature Neuroscience (11, 619; 2008) discusses the relationship between web traffic and citations. The journal’s preliminary analysis indicates that the number of downloads a paper receives immediately following its appearance online correlates very well with its citation frequency years after publication. Noah Gray, one of the Nature Neuroscience editors, has written a post at Action Potential, the journal’s blog, to provide more of the details behind the data and analysis, and to initiate discussion. He writes (edited for length):

Everyone has their own pet problem with impact factors, but despite these concerns, these numbers are typically used to rate the importance or prominence of a particular journal, and thus by proxy, the importance of the individual papers published within. This is a seriously flawed use of association, leading scientists to often equate the total number of citations with scientific impact, which can be fraught with problems. Searching for an alternative measure of impact that is perhaps free of the “bias of authority” (citing a paper because it is from a famous lab) or the “lemming bias” (citing a paper just because everyone else seems to do so whenever broaching a particular subject) led us to explore readership….

The “number of downloads” measure potentially provides a piece of an alternative solution for deciphering the impact of an individual paper. In this current scientific climate where tenure and grant funding decisions are influenced by flawed metrics like impact factor, it is important to make good use of all available technology in an attempt to realize a better system of measuring the scientific impact of any particular paper. This analysis is obviously preliminary and flawed in its own ways, but perhaps metrics such as paper downloads can find a place in a compilation of aggregated stats, painting a more accurate and informative picture of manuscript influence.

The Nature Neuroscience editorial.

The Action Potential post and discussion.

Nature Network Citation in science forum discussion.

Nature Network Citation in science group homepage.

Futher reading: Connotea bookmarks “citation”

Further reading: Connotea bookmarks “impact factor” : thanks to Bob O’Hara for this library.

Peter A. Todd of the National University of Singapore, and Richard J. Ladle of Oxford University, write in Correspondence this week (Nature 451, 244; 2008):

On 22 November, the Higher Education Funding Council for England announced that the assessment and funding of science-based disciplines will in future be “based on citation rates per paper, aggregated for each subject group at each institution”.

Changes in performance indicators always strongly influence individual and institutional behaviour and ‘citation game-playing’ will no doubt become a staple of coffee-room conversation. What is less clear is how the citation practices of authors may influence bibliometric indicators.

Citation practices are known to be imperfect. The documented problems include excessive citation of an author’s own work. Papers cited can be inappropriate or ambiguous in their support and, in some cases, the authors may not have read the papers they cite. Authors may form ‘citation coalitions’ within research networks. They may fail to provide citations to intellectual precursors or to work reporting conflicting conclusions. There are geographical and language biases. The increasing number of many-authored papers makes it impossible to have a clean-cut general metric in which one author is associated with one paper.

Taken together, these factors represent a problematic degree of error for the proposed bibliometric system of assessment. They place added responsibility on journal editors and reviewers as arbiters of appropriate author conduct.

Unfortunately, there are no simple solutions. Currently, identifying poor citation practices is not emphasized in the peer-review process, so perhaps journals could adopt a system of random citation audits, or periodically request evidence of citation appropriateness from authors. In reality, time constraints and the sheer volume of submissions to many journals mean that such measures are unlikely to be implemented soon.

Until referencing practices improve, we would argue that using citation rates to assess performance is fundamentally flawed.