Nature open peer-review trial: Conclusions
For those following Nature's open peer review trial (previously blogged here, here and here), we have now written up the results. For Nature subscribers, there's also an editorial. This excerpt from the editorial provides a good summary:
In the trial, the papers selected for traditional peer review were, in a parallel option offered to authors, hosted for public comment. In the event, 5% of authors took up this option. Although most authors found at least some value in the comments they received, they were small in number, and editors did not think they contributed significantly to their decisions.This was not a controlled experiment, so in no sense does it disprove the hypothesis that open peer review could one day become accepted practice. But this experience, along with informal discussions with researchers, suggests that most of them are too busy, and lack sufficient career incentive, to venture onto a venue such as Nature's website and post public, critical assessments of their peers' work.
The disappointing aspect was not the author participation (which was in line with our expectations) or general levels of interest and web traffic (both good), but the number and average quality of the comments. So either (i) open peer review doesn't work, (ii) the particular approach we used doesn't work, or (iii) scientists aren't ready for it yet. The trial results alone don't allow us to tease apart these possibilities, but my personal bias is to favour (iii), and perhaps (ii), rather than (i).
The fact that the scientific literature migrated online relatively soon after the emergence of the web often leads people to believe that, among all segments of society, scientists are trailblazers in this medium. In fact, scientists as a whole are very conservative when it comes to making the most of information technology. Academic economists are keener bloggers, business people (as well as teenagers) make much heavier use of social networking applications, and law professors have a more obvious presence in virtual worlds like Second Life. Science will eventually catch up, and as a scientific communication company it's Nature's job to facilitate that. To that end, the Nature editorial concludes on a more positive note:
Another form of peer review emerges after publication, when work is replicated — or not. If this kind of discussion is to make it into the open, rather than be confined to gossip at conferences, it requires a forum where peers are able to comment on individual papers, with minimal editorial intervention. Would commenting on Nature papers be more widely adopted by researchers after they have been formally published than before? We intend to introduce this function next year, and find out.
One final point: Whilst contemplating the relative failure of a bottom-up, community-driven approach, at least in this particular trial, it's also worth reminding ourselves that more traditional editorially driven approaches produce their own fair share of disasters. Take, for example the coverage of this very story in yesterday's Wall Street Journal (in an article titled "Journal Nature Drops Open-Editing Experiment" that I'm afraid I can't currently find online). In it the author:
- Fails to distinguish between open peer review, open access and open editing (a la Wikipedia).
- Refers to this as an "experiment" instead of a trial (see the first quote above for an explanation of why this is important).
- Doesn't acknowledge that this was always supposed to be a limited-time trial, which we have merely decided not to repeat or extend at this stage.
Compared to that clueless, misleading piece, Nature's open peer review trial was a big success — at least we learnt something from it.