Deepak and Cameron both have interesting posts up that analyse the comments, notes and ratings left at PLoS One since inception. Making this data available was a cool thing for PLoS to do as they’re one of the few publishers / journals taking a serious stab at implementing commenting properly (Highwire’s BMJ and BioMedCentral being others. And us, of course, shortly ;)).
Bora sent us the data too. I got caught up in other stuff and didn’t have time to analyze the dataset by the agreed date, but now I’m thinking that was actually a good thing as there’s some interesting discussion surrounding the stats happening in the comment threads on BBGM and Science in the Open.
It’ll be a couple of weeks before I get round to an analysis (though somebody else in Web Publishing might beat me to it!) but my first impressions were that comments on PLoS One are working out quite well. To be fair the starting point is pretty low – it’s not like commenting has really been pushed in many other places – but if comments on papers do more good (by adding some value, say) than harm (by confusing users, fragmenting discussion too much) then IMHO they’re a success to some extent, and that’s certainly the case here.
I wouldn’t go as far as Cameron and suggest that the commenting on PLoS One is in some way like post publication peer review (though I might be misrepresenting his views here?)… the majority of the comments I saw in the PLoS and BMC datasets were nothing to do with reviewing the paper – journal club comments aside – and I’m not convinced that comment threads as they’re traditionally implemented would be the best medium for review in any case. Aggregators showing you all the conversation around a DOI (as Deepak, Richard and Cameron have suggested in the FF discussion) though… now we’re talking.
Something else worth mentioning quickly, to defend BMC’s honour if nothing else: bear in mind they started in 2002. Facebook, MySpace and Firefox didn’t yet exist. ‘Blog’ wasn’t in the OED. Commenting wasn’t nearly as widespread on mainstream media websites as it is today. You think it’s hard to get scientists to leave comments now? PLoS definitely has a much higher comment to paper ratio, but cut BMC’s 2% “since the dawn of commenting time” stat some slack. 😉