Systems: Opening up the process

Erik Sandewall

A hybrid system of peer review

Traditional peer review serves two purposes: to give feedback to the authors, helping them to improve their manuscript; and to control the quality of published articles. I believe that more value can be obtained by incorporating an open element in the peer-review system.

The Electronic Transactions on Artificial Intelligence (ETAI) is an open-access journal using open peer review. It was launched in 1997, making it one of the earliest experiments in open reviewing. At ETAI, the peer-review process is divided into two steps – ‘reviewing’ and ‘refereeing’. The purpose of reviewing is to communicate with the peer community and provide feedback for the authors. The purpose of refereeing is to define a quality threshold before acceptance to the journal. This effectively separates the traditional goals of peer review into two distinct functions.

The following is the standard reviewing process in ETAI; other schemes apply for special issues that are based on workshops or conferences. The reviewing process begins as soon as an article is submitted to ETAI and received by one of its editors, at which time it is posted on the ETAI website and advertised to the peer community via e-mail. A discussion period of three months is started, where there is no anonymity and all comments are made openly. Manuscripts are only considered if they fall within the scope of one of the ETAI subject areas. This eliminates the submission of pseudoscience papers and keeps discussions within the boundaries of ETAI editorial expertise.

The discussion period can be extended beyond three months, for example to allow a lively discussion to come to a conclusion. Afterwards, the authors are given a chance to revise their paper, which is then sent out for expert review, in which the identity of referees is confidential. At this stage referees are only supposed to return a verdict of ‘pass’ or ‘fail’, as any more detailed feedback should already have been received during the review period. The refereeing stage is therefore rapid compared to other journals in the same area. The combined time for referee decision plus website publication is often less than a month, and at most three months in 70% of the cases. If accepted, the article is then included in an issue of the journal.

This scheme is made possible by modern information technology, in particular the Internet. It has several advantages. For authors, there is more transparency: they see all comments, so manipulation of the peer-review process is very difficult. For reviewers, there is the reward of being acknowledged for writing an extended and constructive comment on an article. And for the readers, it is often interesting to see the interaction between authors and commentators during the review period. The quality of the dialogue is good because both the commentator and author have time to think before responding, the process continues over three months, and it is open to everyone.

Changing the concept of ‘publication’

ETAI’s open-review model makes it necessary to rethink the concept of ‘publication’. What is the formal status of an article during its three-month discussion period?

This is not merely a philosophical question, for we are often asked what happens if someone steals an author’s idea while it is being discussed. Our answer is that the article is in fact published as soon as the discussion starts. Priority of the result counts from the date of publication in this sense. Paradoxically, the journal only publishes previously published articles – but only after the review and refereeing process. Although this may present a problem for some fields of research – particularly if the media get hold of an incomplete or subsequently corrected story and confuse the message – we have not found it to be a problem for us. The media are not very interested in new varieties of logic-based knowledge representation, or proofs of the properties of an algorithm. If our approach were to be used in higher-profile fields, papers under discussion should be clearly marked as such, so that it is clear the results are not yet validated.

So who publishes the article to begin with? The founders of ETAI created a separate organization – Linköping University Electronic Press – to act as the formal publisher of the article from day 1 of the discussion period. The modern term ‘institutional repository’ serves the same purpose. All articles at discussion stage are given a unique, persistent URL that acts as a unique identifier and links to the full text and discussion. If the paper is later rejected, it remains ‘published’ in the sense that the URL still exists, but it is not published in the journal.

What about the review discussion? It is published on the website. One of the ETAI areas also issues a separate journal – a kind of newsletter – to bring the reviews together with news from within the research area, including reports from conferences and meetings and announcements about new research programmes. This way the difference between research articles and comments is clearer, although the journal’s website does, of course, link all articles to their review discussion.

What the authors say

The ETAI system is quite tough on authors. They must be prepared to answer quickly to criticism during the review period, and they risk having their article rejected in public. Nevertheless, those who have agreed to undergo these hardships report favourably on the experience. One recurrent comment is that the scheme provides good advertising for their work. In fact, even critical comments are welcomed; the worst thing that can happen is that no one cares about an article. If the author gets a tough question and can give a good answer then she or he is better off. All comments are screened by an ETAI editor, who ensures that they are both relevant and above a basic quality threshold. This screening has the added advantage of being able to catch comments that are purely self-promotional, although we have yet to receive any such submissions.

In fact, many of the notions and expectations from traditional journal publishing cease to apply in the ETAI model. For example, authors in traditional journals are eager to get their articles published as quickly as possible to claim priority. But in ETAI, priority counts from the first day of discussion, so the exact date of acceptance is not as important. The fact that there is less pressure to do the reviewing very fast is arguably an advantage of the ETAI scheme as it allows reviews to be done more carefully.

Most traditional journals also consider the anonymity of reviewers to be essential. Are researchers prepared to write open reviews and risk making enemies of rejected authors? The answer is that a critical review in ETAI does not constitute a threat; instead, it helps to improve the article and increase its visibility. Both reviewers and authors have adopted this view without problems as far as we know.

In some cases the discussion begins by itself; but in many cases the editor will invite a few peers to specifically write reviews for a paper under consideration. This often leads to further discussion where additional review contributions are obtained spontaneously. Discussions on any given paper typically involve comments from three or four people, with a maximum of six in a single discussion.

Some articles are rejected, of course, but the rejection rate is relatively low compared with conventional journals. This does not seem to represent lower standards, but a more restrictive submission practice from the authors. In a conventional journal it does not cost much to submit an article prematurely: if it is rejected, at least you get some feedback. In ETAI you are more careful, and one may argue that fewer weak articles are submitted. Perhaps this is an important lesson for journals that are now finding it more and more difficult to get reviewers for the articles submitted to them.

Erik Sandewall is professor of computer science at Linköping University, Sweden, general editor of ETAI, and director of Linköping University Electronic Press. His research areas are in representation of knowledge, cognitive robotics and human–robot dialogue systems. He can be contacted via ETAI.

Read more See this article in Nature’s web focus here


Comments are closed.