Peer-to-Peer

Ethics: Trust and reputation on the web

William Arms

Online publications have several ways to give themselves a good name.

Trust and reputation are fundamental to scholarly publishing. The web provides tantalizing new ways to publish, but can these win the trust that is crucial to scientific acceptance? Peer review is the traditional way of building trust, but it is slow and expensive; some topics are difficult to review and reviewers miss mistakes. What alternatives do authors have?


One surprisingly effective practice, which has become standard in computer science, is for authors to place preprints of their papers on their own websites when they submit them to journals for review. On publication, the final version replaces the preprint. Readers must judge preprints for themselves, but can be guided by other information on the website, such as the author’s résumé, a full list of publications, professional affiliations, and a list of grants. A knowledgeable reader may find this contextual information more than sufficient to judge the quality and relevance of a paper.

Preprint repositories, organized by discipline or by institution, are another way for authors to distribute papers before or without review. The best known is ArXiv, which dominates physics, and is important in mathematics and related fields. It offers authors immediate, worldwide distribution of their research, an opportunity for others to comment on (and authors to revise) a paper before it is submitted to a journal, and a comprehensive archive. Dissemination through the archive fixes the date when the research is made public, which can be important for establishing priority.

ArXiv was created by physicists for physicists, but although it has gained the trust of its community, other similar initiatives have faltered. DSpace, for example, is an attempt by university librarians to encourage faculty members to place their papers online, but has so far failed to attract many authors. One difficulty is that universities are so diverse that institutional repositories cannot be tailored to a specific community in the way that ArXiv works for physics.

A more revolutionary approach is to let readers edit articles without consulting the original authors. The online encyclopaedia Wikipedia takes this approach. Within simple guidelines, anybody can create a new encyclopaedia entry, or edit an existing one. Many sections of Wikipedia are superb. It is an excellent reference work for mathematics; thousands of individuals have combined to create accurate and well-written entries. An informal comparison by Nature of 42 scientific articles found an error rate not much higher than in the Encyclopaedia Britannica11.

This success does not come easily and can breed its own problems: vandalism, bias, and disputes between editors who reverse each other’s changes. But Wikipedia strives to create a climate of trust and continually makes small but significant revisions to its procedures. Other, less carefully managed attempts at cooperative writing have failed.

In conventional publishing, editors and reviewers are selected for their expertise, and trust is gained through reviewers’ authority. On the web, an alternative is to allow anybody to submit a review, and build trust based on consensus. One example is the book-review system at Amazon. Many of Amazon’s reviews are excellent, but problems arise from the power of online reviews to influence book sales and authors’ reputations. A recent study documents the ways in which Amazon reviews are unreliable, including widespread plagiarism, abuse by rival authors, and reviewers who praise everything2. Amazon’s reviews are often useful, but nobody should trust them to be unbiased or well informed.

To build trust in reviews, some websites use hierarchical reputation-management systems that let readers grade reviewers. One of the pioneers was Epinions.com, in which reviewers collaborate to rate consumer goods. These systems have worked in commercial settings, but the manipulation of their ratings is a serious problem. There is no successful example in any scientific field.

A journal’s reputation is usually linked to a name. In conventional publishing it may be the name of a society (the American Physical Society), a university (Oxford University Press), a publisher (Wiley), or simply the name of the journal itself (Nature). In the past, reputation was hierarchical and built up over a long period of time. A university gave its name to a press, which attached its name to a publication, which in turn provided credentials for authors and reviewers. On the web, several new publications have succeeded in building reputation without being tied to a conventional name. An example is D-Lib Magazine, which has become the leading publication for research in digital libraries. The magazine, which has no peer review, built its reputation through support for authors: by providing expert editors, rapid turnaround of manuscripts, punctuality, open access for readers, and a friendly copyright policy. Many established authors value these attributes more than the imprimatur of peer review.

All the examples in this article are heavily used. Professional advancement in science is still based on peer-reviewed journals, but in some disciplines the benefits of rapid dissemination and open access, the abundance of clues about quality other than peer review, and the excellent quality of many unreviewed publications have gained the trust of the working scientist.

REFERENCES

1. Giles, J. Nature 438, 900–901 (2005).

2. David, S. & Pinch, T. J. Soc. Sci. Res. Network https://papers.ssrn.com/abstract=857505(2005).

William Arms is Professor of Computer Science at Cornell University, Ithaca, New York 14850, USA. He is also at the National Science Digital Library.

See this article in Nature’s web focus here

Comments

Comments are closed.