Comparing Wikipedia and Britannica
This week's issue of Nature has a news feature that reports on an experiment our journalists conducted to discover how reliable the scientific content of Wikipedia is compared to Britannica's.
They sent 50 pairs of Wikipedia and Britannica articles on scientific topics to recognised experts and, without telling them which article came from which source, asked them to count the numbers of errors (mistakes, misleading statements or omissions). Among the 42 replies, Britannica content had an average of just under 3 errors per article whilst Wikipedia had an average of just under 4 errors — not as much difference, perhaps, as most people would expect. There is also an accompanying Nature editorial.
We also took the opportunity to interview Jimmy Wales about this when he came to visit the other week. You can hear the results in the 15 December 2005 Nature Podcast.
Going through the subject-by-subject results, I make the final score 22-10 in favour or Britannica, with 10 draws. The total numbers of errors are 123 in Britannica and 162 in Wikipedia (which therefore has just over 30% more errors than it's traditional rival).
Of course this only covers scientific content, and it doesn't look at every aspect of an encyclopedia's value (e.g., timeliness, depth, the quality of its references), but I think it's a useful addition to the less rigorous and anecdotal information that seems to have fuelled the debates about Wikipedia so far.
If you believe that an encyclopedia should be judged by it's weakest entries (in general I don't), or if you're the subject of an error or slur (thankfully I'm nowhere near famous enough), then the anecdotal outliers might be more important to you than averaged results. But most readers simply want to know whether a source can generally be relied upon. What these results say to me is that Wikipedia isn't bad in this regard — and that if it's really important to get your facts right then even Britannica isn't completely dependable. (In fact, I'm not sure that anything is.) Put this together with the fact that Wikipedia is generating content at many orders of magnitude lower cost than traditional encyclopedias and you've got yourself a classic Christensenesque disruption in reference publishing.
A key outstanding question is whether or not Wikipedia can ever surpass Britannica in quality. Since it evidently already does in some subjects, I think the answer is yes, but we will have to wait and see. Frankly, I still can't get over the fact that it works at all.