IF all over again

If you can meet with Triumph and Disaster
And treat those two impostors just the same;
If you can bear to hear the truth you’ve spoken
Twisted by knaves to make a trap for fools,
Or watch the things you gave your life to, broken,
And stoop and build ’em up with worn-out tools:
Rudyard Kipling

Yes it is that time of the year again when Thomson Reuters publishes its Journal Citation Report (JCR) and everyone involved in science publishing gets obsessed with Impact Factors (IF). I’m not going to go through the arguments about how little Impact Factors really mean, and I’m certainly not going to try and forecast the health or otherwise of a publishing venture based on a change of 0.3 in its IF. But I thought you might want to know what Nature Protocols’s 2012 IF is. So cue drum roll …

It’s 7.96 down a couple of points from last year’s 9.92

Or

It’s 11.74 up from last year’s 10.20

Nothing is simple when it comes to Impact Factors. They are sort of an estimate of the average number of citations that a paper in a particular journal gets, but they are actually the number of citations a journal gets in a year to articles published in the previous two (or five) years divided by the number of articles published in those years that it seem appropriate to cite (‘simples!’). Herein lies the apparent contradiction in the numbers I gave above. Nature Protocol’s Impact Factor based on citations in 2012 to protocols published in 2010 and 2011–the two year impact factor (IF2)– is 7.96. The Impact factor based on citations in 2012 to protocols published between 2007 and 2011–the five year Impact Factor (IF5)–is 11.74.

For most journals there isn’t a whole lot of difference between the IF2 and the IF5, certainly less than 10% so when someone says Impact Factor they normally mean IF2. There are a few journals with big differences between the two values. The journal with the highest IF2 of all, CA: A Cancer Journal for Clinicians published by the American Cancer Society, has an IF2 of 153.46 and a IF5 of 88.55 which I interpret as meaning that what it publishes is extremely relevant for a couple of years (and so is highly cited) but after that it quickly loses its importance.

Conversely that Nature Protocols has a higher IF5 than IF2 could be taken as an indication that the protocols we publish remain relevant well beyond the first years after presentation. A measure that might bear that out would be the cited half-life of the journal. This is defined as the median age of the articles published in Nature Protocols that were cited in a given year (i.e. 2012). For Nature Protocols it is 4.9 (in 2011 it was 4.2), but that really doesn’t say a lot as Nature Protocols is a relatively young journal which has only been in existence since 2006. The maximum value for cited half-life we could have got would have been 6 and if there was no change in the rate of citation of our protocols over time a value of 3 would have been expected. We will need to be at least in our teens before I will put much store by cited half-life.

There is another confounding factor in all this for Nature Protocols and its name is DAVID.

In December 2008 we published online a protocol by Richard Lempicki and colleagues at National Cancer Institute at Frederick, Maryland called “Systematic and integrative analysis of large gene lists using DAVID bioinformatics resources” (Nature Protocols 4, 44-57 doi:10.1038/nprot.2008.211 (2009)). It is our most cited paper having been cited more than 3,000 times. It is in fact the most highly cited paper from any Nature  journal, including Nature itself, published in 2009 (yes I know it was published in 2008 but it was in the January 2009 issue of the journal and so that makes it officially a 2009 paper). In 2012 alone it was cited upwards of 1,000 times. However since it was published in 2009 those citation do not contribute to our  IF2 although they do to our  IF5. It is difficult to get the figures to calculate the exact effect of a single paper on IF but a fair approximation would be to say that had those 1,000 citations been included in the calculation of our IF2 then it would have been a bit less than 3 points higher, while excluding them from our  IF5 would reduce that by about 0.8 making both values in the region of 10.9.

Yep, you’ve got it! Citations to a single paper seem to account for all the difference in our Impact Factors. Which simply shows again that IF may be a great poem, but it is a poor measure of the scientific literature.

 

Protocols Discussion Forum is Moving

As you may or may not know Nature Protocols has a Discussion Forum where anyone having problems with their experiments can ask questions and hopefully receive helpful advice from other researchers with some experience in whatever is causing the problems. For the last several years this has been hosted on Nature Network but as of, well right about now, we are shifting its platform to a Google Group.

We have always thought of Nature Protocols as more than a journal and we try to engage with our readers to help them in their research. We have done this through commenting on our manuscripts; this blog; our YouTube channel and the discussion forum. We are always on the look out for technologies that might be helpful.

That’s why we are now moving the Discussion Forum onto the Google Groups platform.  We hope that this will greatly improve the effectiveness of the discussions by making it even easier for anyone to become involved in the forum. It should be easier to access on mobile devices, and easier to post on as comments can be made and discussion topics started simply by sending an email to nature-protocols-discussion@googlegroups.com.There are other technical advantages for us as well, not least a better spam filtering system so hopefully far fewer topics or posts will be offering fake passports for sale!

There are two ways to access the new look discussion forum, either through the Discussion page on this blog or directly at Google Groups . We very much hope that this move will increase the usefulness of the Discussion Forum by making it easier for researchers to take part, easier for the topics to be found by people searching for answers using search engines, and easier for topics to be connected to from throughout the Web.

The Current Discussion Forum isn’t going away but will remain as an archive. However it will not be possible to post new topics or reply to existing threads. I have however copied across all discussions started this year into the new platform as well as some of the most active from 2012. If there are any threads that you would like to see continued please let me know and I will copy those across as well.

I hope that everyone will find the new forum at least as helpful as it’s predecessor.

So, what shall we talk about? As an eponymous fictional talk show psychiatrist didn’t quite say:

“We’re listening.”

 

The Season of the Impact Factor

It is that time of year again! A bit like Christmas, now is the time when journal editors wait expectantly to open their presents; or in this case present singular. I’m talking about the annual release of Impact Factors from ISI. I’m not going to go through all the reasons why Impact Factors may not be the best way to judge scientific research. Katharine made some comments about that last year and I haven’t changed my opinion much since I railed against their tyranny at another journal. But Impact Factors and the other citation metrics probably say something about a journal so I thought I’d give you the numbers for Nature Protocols and say what I’m taking from them.

A picture from the cover

My favourite cover image from 2011.

So the 2011 Impact Factor for Nature Protocols is 9.924 which I’m really happy about. It would be a presentable number for a journal publishing primary research, and we don’t publish primary research. I would expect protocols to be formally cited less often. Also it is up 1.562 points from the 2010 value so Nature Protocols is probably being cited more than 12 months ago and hopefully that is an indication that our articles are being used more than 12 months ago. If our articles are being used more than they were then that’s good news.

Impact Factors are quite good for making comparisons between similar journals, but sadly none of the other main publishers of protocols is indexed by ISI and so don’t get given Impact Factors. I analysed the citation of those journals a few months ago and I’ll let you draw your own conclusions about what thier IF might be if they were given one. What ISI does do is put Nature Protocols in a group of journals publishing “Biochemical Research Methods”. Our colleagues at Nature Methods top that list with an IF of 19.276. However it is a bit like comparing chalk and cheese as other journals in this group are things like Acta Crystallographica Section D: Biological Crystallography (IF 12.619), Current Opinion in Biotechnology (IF 7.711) and even PLoS Computational Biology (IF 5.215). Still in the group we rank third, up from fourth last year, so again pleasing.

The numbers that please me are those that suggest that Nature Protocols articles have some longevity. If you calculate IF over 5 years instead of 2 it rises to 10.201 suggesting that Protocols are relevant well beyond their initial publication. Many journals, possibly even most (I’m eyeballing rather than having done a proper analysis)  have a lower 5-year IF than their standard 2-year IF.

For a slightly more sophisticated measure there is the Eigen Factor score. This is calculated in a similar way to Google’s page rank algorithm in that not all links are of equal value and self citations aren’t counted at all.  For 2011 ours is 0.10716. I’m not completely clear how to interpret that but it is more than for 2010 so that must be good!

Last up the Article Influence Score. This is derived from the Eigen Factor and then normalised for the size of the journal so that the ‘average’ journal will have an Influence of 1.000. Ours for 2011 is 4.422 which is again more than it was last year and shows, to misquote the immortal Yogi, that we are “smarter than the average journal (Boo-Boo)”.

So it’s all good (for as much as Impact Factors and similar measures can tell you anything about a journal).  But at Nature Protocols we don’t really care about the value of our IF. In the next 12 months we will try to commission and publish even better protocols so that more scientists will be helped in their research. If someone can quantify that then I’d like to hear about it.

March’s SoNYC: On setting the research record straight – Sound familiar?

Dorothy Clyde (Dot), is a Senior Editor at Nature Protocols and has been with the journal since its inception in January 2006. In her previous life as a research scientist, she spent close to a decade studying various aspects of fruit fly development and genetics. In her guest post she explains the role an editor plays in avoiding plagiarism, giving advice to all parties. 

Imitation may be the highest form of flattery, but I doubt if anyone feels particularly flattered when they come across their own work – copied verbatim and unattributed – in someone else’s publication.  Unfortunately, instances of plagiarism are becoming more commonplace. As a result, tackling plagiarism has now become part of an editor’s job, and at Nature Protocols we work closely with our authors to avoid it. Once upon a time plagiarism was difficult to detect, but now that so much of the scientific literature is online there are a number of services that help us spot when parts of a submitted manuscript have already appeared elsewhere.

Thankfully, plagiarism of the classic kind is rare; when we do come across plagiarism, it usually takes the form of self-plagiarism (where an author reproduces uncited sections of text from one of their previous publications), or duplicate submissions (where the same manuscript is submitted to more than one journal, often simultaneously). Self-plagiarism, if detected early enough, is usually relatively straightforward to resolve. Duplicate submissions, on the other hand, are one of the more frustrating aspects of an editor’s job, as the problem often does not become apparent until the later stages of the editorial process – or worse, until both versions have been accepted and published in different journals!

Why is self-plagiarism a particular problem for Nature Protocols?

In order to ensure that our protocols are reproducible, Nature Protocols has a policy that all our protocols are based on a previous primary research paper from the author’s lab, in which the technique was used to generate data. If this supporting paper is very methodological, there may be some degree of overlap with the Nature Protocols manuscript. Alternatively, the authors may have published a protocol-type paper on a similar topic in another journal. In these circumstances, it is easy for sections of duplicated text to creep in, especially if the authors are unaware of the problems associated with self-plagiarism.

With respect to duplicate submission, we hope that all authors realise that it is unacceptable to submit identical manuscripts to more than one journal at the same time. However, duplicate submission (and self-plagiarism) problems can arise when manuscripts that were seemingly unrelated at initial submission, progressively become more similar as they pass through the editorial process, for example, when additional material is added in response to referee or editorial comments. In addition, some authors view protocols differently to other publication types and do not see a potential conflict in publishing the same protocol in more than one journal.

The role of editors in avoiding plagiarism

It is our job as editors to ensure our authors are fully informed about our plagiarism policies and to detect potential problems early in the editorial process. So how do we set out to achieve this at Nature Protocols?

  1. When an author agrees to submit a full manuscript to Nature Protocols, they will be sent an e-mail that reminds them we take plagiarism seriously and it must be avoided, directing them to “NPG’s policy on plagiarism” for further information. This e-mail also requests that authors make the editor aware of any methods/protocol papers they have previously written or have agreed to write on a similar topic – and, if possible, to provide us with a copy of these papers. We also ask authors to provide us with a copy of their supporting primary research paper. Editors will have already carried out extensive literature checks by this stage and may ask authors to upload specific publications that these searches have identified as being potentially overlapping or a source of potential problems. In such cases, the editor will make it clear that we expect the Nature Protocol to add substantial value to the existing literature and request that the author outlines to the editor how the manuscript expands upon previous publications.
  2. Authors are reminded of our policy again when their manuscript is returned to them for revision. It is also made clear in the editorial comments that duplication of text must be avoided and that all our manuscripts are cross-checked against the published literature before being accepted for publication. Thus, all authors are informed at least twice of our position on plagiarism.
  3. Nature Protocols (along with all other NPG journals) participates in the “CrossCheck initiative.” Prior to accepting any manuscript for publication, it is compared to the CrossCheck database using “iThenticate”.  iThenticate will generate an overall similarity score and provide a summary report that highlights instances of duplicated text in the submitted manuscript and links back to the original source(s). The editor then carefully checks each ‘hit’ in the report to determine its significance. Large sections, or multiple smaller sections, of highly similar text are an immediate red flag; if such sections are a clear indication of deliberate plagiarism or if the duplicated text cannot be removed by careful revision by the author (for example, by extensive rewriting or appropriate referencing) the manuscript will be rejected. We understand that paraphrasing sections of the Procedure can be difficult and not always helpful so in these cases it may make most sense to ensure the original source is prominently cited. However, the software will also turn up hits that are not plagiarism, such as isolated occurrences of partially similar sentences. Some standard phrases will be present in most of our manuscripts and can also be excluded, for example text reminding authors to follow institutional and national guidelines when performing experiments on animals. Another example where our editorial requirements make some duplication inevitable is the Materials section and so hits limited to this section can usually be overlooked.

Taking action

If self-plagiarism is detected before a manuscript is accepted for publication, the author is made aware of the offending sections and asked to either cite the original source or to rewrite the duplicated text. Duplicate submissions will be rejected outright as soon as they are detected. No manuscript will be accepted for publication until we are satisfied that all text is original and appropriately referenced.

Once a manuscript has been published, the course of action will depend on a number of factors, including: proof of intent; severity of plagiarism; policies of other journals involved. In cases where it is judged that the plagiarism is relatively minor and unintentional, authors will most likely be given the opportunity to correct the publication record by including additional citations or rewriting sections of text in the form of an official correction. In more serious cases, it is likely that we will retract the Protocol. That is about the limit of what we as editors can do ourselves although an author’s institution may well decide to investigate. In theUS, there is also the US Office of Research Integrity (ORI) which will investigate complaints of plagiarism. We are of course happy to cooperate with Institutions, the ORI or equivalent authorities.

Limitations of our approach

Unfortunately, there will be cases of plagiarism that slip past the Nature Protocols editors unnoticed. Plagiarism detection software, such as iThenticate, is not foolproof; determined individuals will find a way to ensure their manuscripts evade detection. Services such as PubMed, Scopus or CrossCheck do not index content from every journal; neither do they index all content types from participating journals. And by definition, only published papers are included in these databases. Thus, concurrent submission of identical manuscripts to more than one journal is a serious problem with no immediate solution.

A closing plea to authors

Authors – remember, your editor is your friend when it comes to avoiding plagiarism. It is in everyone’s best interests to identify and resolve potential issues early in the publication process. To facilitate this, I would urge all authors to be honest and transparent with your editor. And if you are unsure about how the policies apply to your manuscript –just ask!

Science Online NYC (SoNYC) is a monthly discussion series held in New York City where invited panellists and the in-person and online audiences talk about a particular topic related to how science is carried out and communicated online. For this month’s SoNYC the topic for discussion is:  Setting the research record straight. We’re looking at issues such as retractions and plagiarism and how they relate to real or perceived increases in research misconduct.  More details about this month’s SoNYC can be found here.

To complement the event, we’re running a series of guest posts discussing what steps publications are taking to deal with fraudulent research practices and what is being done to investigate and deter such practices. We’ve already heard from Richard Van Noorden, Assistant News Editor at Nature. He gave us an overview of  what retractions can tell us about setting the research record straight, highlighting some recent high profile cases of retraction, explaining why retraction rates appear to be increasing. We also compiled a Storify from a session at February’s AAAS meeting in Vancouver on Global Challenges to Peer Review which touched on some of the challenges faced by journal editors. More guest posts coming soon.

Nature Protocols and Impact Factors

When we first launched "Nature Protocols ":https://www.nature.com/nprot I was keen to get our protocols indexed by all the big players. Whilst there was going to be a marketing campaign with ads in Nature and banner ads all over nature.com I knew we’d only get real users when people did their literature search and one of our protocols came top of the list. We wanted to be something researchers needed and found useful. However, alongside getting listed in the ISI Web of Knowledge came the news we’d also be eligible for an Impact Factor. It was clearly something scientists cared about. A lot of our early correspondence was about what our Impact Factor was. We had to be very apologetic and explain that we didn’t have one – first we’d need to publish some protocols, and then wait to see if they were cited. Personally I was relieved we didn’t have one because I wasn’t sure what an impact factor would mean for us, even if we did have one.

The Impact Factor comes into its own as a comparative factor. That’s why my colleagues upstairs at Nature are delighted with theirs, as at 36.101 it puts them ahead of their competitors. And also why journals with impact factors of 1 or 2 can also be delighted, as such an Impact Factor can be the highest in their specialist field. But when I looked into Impact Factors I realised that no protocols publications had them, so as a comparative factor, our impact factor would be rather meaningless. When and why would people cite protocols? We certainly wouldn’t be cited because we presented exciting new data and reported a major advance in scientific knowledge. I hoped that we’d be cited if we were used – and so lots of use would lead to a high impact factor. We’d obviously only get cited if the experiment worked, and so a higher impact factor would indicate good, reliable protocols. But, maybe we wouldn’t get cited, as citing the source of your protocol didn’t seem to be the ‘done thing’ at the time.

Five years down the line – with a 2 year ISI Impact Factor of 8.362 – I am pleased to report that we are being cited. We have an Immediacy Index of 1.547 and "Article Influence Score ":https://en.wikipedia.org/wiki/Article_influence_score of 3.799. We’re all rather pleased, as we know our Impact Factor matters to some of our authors. We’re also curious – which articles have people been citing? Are they the methods people have been using? Have they been getting good results? How do these articles correlate with the articles that are most downloaded from our site?

Our most highly cited article according to ISI is Locating proteins in the cell using TargetP, SignalP and related tools by Emanuelsson et al. This is different from the most highly cited article according to Scopus – "Systematic and integrative analysis of large gene lists using DAVID bioinformatics resources ":https://dx.doi.org/10.1038/nprot.2008.211 – by Huang et al., although this comes in at second place. At third we have "Analyzing real-time PCR data by the comparative C-T method ":https://dx.doi.org/10.1038/nprot.2008.73 by Schmittgen et al. And in fourth, "In-gel digestion for mass spectrometric characterization of proteins and proteomes ":https://dx.doi.org/10.1038/nprot.2006.468 by Shevchenko _et al. _and in fifth, Protein structure prediction on the Web: a case study using the Phyre server. Interestingly there are lots of bioinformatics protocols in this list.

Of course, our 2010 2 year Impact Factor is only based on articles published in 2008 and 2009, so only two of those protocols, those in second and third place, will have been included. And also, only the cites in 2010. I was – and remain – curious about our five year impact factor. Whilst methods are developing all the time, protocols do tend to have a long life. A tweak here and there (hopefully highlighted as a comment to the protocol), but we hope our more useful protocols will still be in use, and cited, 5 years on. I guess we’ll have to wait and see

Happy Birthday Nature Protocols – 5 today!

Five years ago today we launched Nature Protocols. It was the culmination of 6 months of hard work for the editorial team. Dot Clyde and myself had both started work at NPG at the start of January with a long list of things to sort out. First we had to decide on our scope, answering questions such as do physicists use protocols? Next we had to decide on a format. Although we had a rough guide from our colleagues on Nature Methods who had published some protocols, we had much more freedom. No page budgets and an online format. Dot had just left the lab, as had the rest of the editorial team we recruited (Bronwen, Alex, Andy, Baldo and Hannah), so there was much discussion of what scientists REALLY wanted to know about. Next there was peer reviewing – what would we ask the peer reviewers to do? Ideally we’d have liked them to repeat the procedure, but clearly that was not practical (although on occasion we have had peer reviewers do just that!) And then there was presentation – sorting out exactly how we would present all that information we gathered. Plus who was going to copy edit it all. Then of course, last, but most certainly not least, who would we invite to write? And would they agree to write for this new type of publication?

Thankfully many of the authors we approached were as enthusiastic about the concept of Nature Protocols as we were. We quickly received plenty of manuscripts to send out to peer review (helped by the fact that the protocols were already written and much tried-and-tested in our authors labs). Many of the peer reviewers we approached were pleased to help, and on 12th May we put our first example protocols on our web page. It was great to finally have some actual content, alongside the information about what we were doing.

Then on June 30th 2006 we published our first batch of content – a whopping 87 Nature Protocols protocols. Over the following months we published many more so by 29th December we’d published 322. By January 2008 we had over 1000 protocols on our site, including almost 800 Nature Protocols protocols.

In addition, we’d also worked on our system for researchers to post protocols-in-progress onto the site. Then called the Protocols Network, this launched with 14 protocols, all contributed by authors of recent papers published in other Nature publications. Today that system is bigger and better than ever before, and was recently relaunched as the Protocol Exchange, including lots of added featured we didn’t quite get round to including in 2006.

So, 5 years on, where is Nature Protocols now? The editors are all a little older (here we all are back in 2006):

Copy (2) of S4200028.JPG

On the roof terrace in the summer of 2006. Back row, from left: Andy, Alex, Dot, Baldo. Front row, from left: Katharine, Hannah, Naomi (our first editorial assistant), Bronwen.

The feedback we’ve had from authors and users has been fantastic and largely positive, which is the best reward an editor can have. Our most cited article on Scopus – "Systematic and integrative analysis of large gene lists using DAVID bioinformatics resources by Huang, Sherman and Lempicki ":https://dx.doi.org/10.1038/nprot.2008.211 – published in December 2008 has been cited 847 times. We’ve published old favourites, such as the single step method of RNA isolation by acid guanidinium thiocyanate-phenol-choroform by Piotr Chomczynski & Nicoletta Sacchi and the latest protocols published in Nature, such as "Kazutoshi Takahashi, Keisuke Okita, Masato Nakagawa & Shinya Yamanakas’ protocol for the induction of pluripotent stem cells from fibroblast cultures ":https://dx.doi.org/10.1038/nprot.2007.418. We’ve followed dramatic progress in certain fields – for example, the new protocols available using next generation sequencing have revolutionised research progress. It’s now possible to rapidly sequence single cells, as demonstrated by Fuchou Tang, Catalin Barbacioru, Ellen Nordman, Bin Li, Nanlan Xu, Vladimir I Bashkirov, Kaiqin Lao & M Azim Surani who used RNA-Seq analysis to capture the transcriptome landscape of a single cell. We continue to enjoy following the progress of stem cell research, which, as well as making dramatic progress actually generating stem cells, has produced improved protocols for researchers in other areas, such as the ability to generate gene knockout rats, described by Chang Tong, Guanyi Huang, Charles Ashton, Ping Li & Qi-Long Ying.

And the future? Well, I’ve taken a step back now and so that’s a question for our current Chief Editor, Chris Surridge, but I am sure we will continue to concentrate on our core aim back from 2006 of “publishing the protocols being used to answer outstanding biological and biomedical science research questions, including methods grounded in physics and chemistry that have a practical application to the study of biological problems”. All that’s left for me to do now is give a final thanks to all who have contributed to the success of Nature Protocols so far and say, heres to the next 5 years!

Focusing on stem cells

Since Nature Protocols launched we’ve been having a shall-we-shan’t-we discussion about the relative merits of article series. On one hand they seem such a great idea – bringing together lots of protocols in a specific area to make life easier for researchers in that area. On the other hand we worry they’ll alienate researchers working on other things – why do the mass spectrometrists get special treatment and not us? And how do you decide what to include on such a page – anything vaguely connected or just the hard core of a particular topic? Then finally, as an Editor I just find myself wondering why I am making more work for myself…

But back in 2009 we took the plunge and launched our Stem Cell Series page. At the time stem cells were all the rage – if you watched the TV news at night you’d think all diseases were on the verge of being cured by these fantastic new cells. Unless you lived in the US, where they were not viewed so kindly by some.

Recently the page has slipped a bit in priorities – launching the Protocol Exchange was quite a job. But we feel it’s now time to revitalise it. Stem cells remain a hot research topic, and the optimism about their potential remains. Each week more and more possible applications are being devised. I’m pleased to see that our recently published stem cell protocols are reflecting this increased variety. Currently featured, in addition to our typical protocols showing how to get your stem cells initially, or how to go from stem cells to your cells of choice, there are some intriguing ways to look at your stem cells. Some of these methods can be applied to cells other than stem cells so do check it out even if your favourite cell is not a stem cell.

The juries still out on whether series pages are a useful addition to our site. So, if you’ve not checked for a while, or if you’ve not looked at all, we do suggest you take a peek. And do let us know what you think – should we run series or not? And if so, what topics should we cover?