The Impact of Electronic Publishing on the Academic Community
Session 3: The content and quality of academic communication
Peer review and electronic publishing
CEA-Saclay, Service de Physique Théorique, F-91191 Gif-sur-Yvette Cedex, France
©Portland Press Ltd., 1997.
The point of view which follows originates from a physicist, head of a laboratory of theoretical physics, head editor of Journal of Physics A (published by Institute of Physics Publishing) and editor of Nuclear Physics B (Elsevier) and Fortschritte der Physik (Akademie Verlag).
Electronic dissemination of scientific information, because it offers a cheap, easy, fast and unscreened way to reach the whole scientific community, forces us to reconsider the role of peer review, one of the cornerstones of paper publication. The traditional refereeing system is already on the verge of collapse, due to the increasing number of submitted articles and their increasing specialization. It is now doubtful that it can survive unchanged in the electronic age.
We shall argue here that some form of evaluation of scientific papers remains necessary, but the way this could be achieved could ultimately be quite different. We shall suggest that dynamic and open refereeing could offer alternatives, the precise implementation being a matter of pragmatic experimentation.
The role of refereeing in the publication process
Let us first discuss the role of refereeing in the way we now perceive it. It is here important to distinguish two, sometimes conflicting, points of view; that of the scientific community and that of the author (a somewhat schizophrenic situation).
The community expects the referee to perform the following tasks: (i) eliminate wrong papers; (ii) fight against plagiarism and redundancy; (iii) fight against dilution of information; (iv) help to improve, when necessary, the quality of articles from the point of view of content and readability; and (v) decide about formal publication, and simultaneously determine the final form of articles.
For the author: (i) before electronic media, publication in journals was the main tool of scientific communication. This had already partially been circumvented by the paper preprint systems, at least in well-established laboratories. This question is now rapidly becoming obsolete in the sense that public and free databases provide a better way to reach the scientific community than journals that are not everywhere available. Note that this leads to a decoupling between the two issues of scientific dissemination and validation of research results. (ii) For many authors, having their articles accepted for publication is the main way of having their work recognized. The general perception of the journal's reputation in the community is an important element of their strategy. This reputation is part related to the quality of editors and referees.
It is essential to keep this problem in mind in any proposal to modify the present system. Many authors no longer care about their papers actually appearing in a journal (they often still refer to papers by the reference in the Los Alamos database even after publication). But they consider it as essential to have their paper accepted for publication. They believe this to be important for their academic career. As an obvious consequence the present publication system is becoming unstable. Indeed a rational economic behaviour would be to submit articles to journals but not to subscribe to them. Another consequence is that it becomes more difficult to start new journals even if they are quite inexpensive. Indeed authors are induced to submit to journals of high reputation even if they are poorly distributed.
The crisis of the referee system in physics
A few empirical remarks: (i) the number of published pages increases exponentially. The reasons I can only try to guess: increase in the number of physicists; increase in the physicist 'productivity' due to external pressure (the feeling that the number of published papers is essential for a scientific career); and improved production tools (computers). In fact, many published papers can really be characterized as progress reports.
(ii) Referees have increasing difficulties performing their duties. They receive too many papers which are poorly written and with a highly specialized content. To assess the relevance and novelty of papers therefore becomes a very difficult task. Moreover the less interesting papers are the more demanding in terms of the referee's time. If a referee becomes too selective the author generally fights back, asks for another referee, then goes to another journal. Finally, if existing journals become too selective, new journals are created.
Moreover, in a world in which the success of a journal is in part measured by the very number of published pages, it cannot be expected that publishers themselves will try to discourage authors.
In fact, the present system favours authors against referees: a paper is considered innocent unless a referee can prove it guilty (except in letter journals where the rather subjective urgency criterion can always been used as a final argument).
(iii) Two intrinsic shortcomings of the peer-review system in paper publication must also be emphasized. In the paper form the only visible effect of the evaluation is rejection or acceptance for publication. This strongly discriminates between papers of comparable marginal interest which have been submitted to more or less selective referees, while two published papers are treated in the same way, independently of their relative importance. The final outcome hardly reflects the quality of the referee's report and may even caricature it. Of course, there is some implicit evaluation related to the reputation of journals. However this reputation varies in time, depends on topics and finally induces dubious submission strategies.
The evaluation remains unique, at least if the paper is accepted. Once published, a paper's content can only be discussed through another publication. Errata published months later, which nobody reads, are the only way to point out errors. All this in turn contributes to the dilution of scientific information.
Finally, to answer an objection: indeed, established journals still reject a fraction of submitted papers (30% for example in high-energy physics), but it should be noted that this fraction has remained approximately constant while the number of papers has increased dramatically. In fact, the referees are able to stop a large fraction of wrong papers, they eliminate some redundancy, but they achieve little in the matter of dilution of information and improvement of manuscripts. Therefore the refereeing, without being irrelevant, simply becomes increasingly ineffective.
(iv) Conclusion: we produce an information of decreasing value, the rate of dilution of scientific information has become excessive, and in the traditional production system nothing can really be done to prevent further deterioration. Drastic changes are required, and in the electronic age, as I shall try to demonstrate, new options become available.
Should some form of peer review be maintained?
It is obvious that some form of evaluation of scientific articles is needed. While experts in a field know which articles are relevant, newcomers to a field may have more problems. This problem will become increasingly serious with time, due to the accumulation of articles and some loss of memory. Of course one can argue that some form of scientific evaluation already occurs spontaneously under the form of citations. Moreover, in electronic form, articles can be linked through references (as is done routinely in the Los Alamos or Stanford Linear Accelerator databases) and it becomes much easier to find out by whom and why an article is cited. The existing system can further be improved by pointing to the location where the citation occurs in the article. Such an evaluation system has several advantages, it is cheap, easy and automatic. It improves with time. Is it thus sufficient?
I do not believe so for several reasons. It is not very efficient because the reasons for a citation are not always obvious and may require extensive reading of other articles. It is very sensitive to fashions and topical subcultures (the number of papers authors cite varies very much). If it remains the unique mode of evaluation it can be easily subverted. Finally, such a system is unlikely to be accepted in the immediate future by researchers who want their work to be reviewed and judged on the basis of article submission.
Dynamical and open refereeing
One of the new features of electronic publication is the possibility of linking electronic documents, as we have already discussed. An electronic article does not exist any more as an isolated object, but in direct relation with others. This environment is dynamical and open in the sense that it evolves under the action of other actors. As a consequence important documents can remain alive. One might argue that such a phenomenon already existed for paper documents, but the speed and ease of navigation between linked electronic documents creates a totally new situation. A new system of evaluation should therefore take full advantage of this possibility. Actually, this evaluation should be only a part of a more complex process to organize and structure the whole information available (the main task of scientific publishers in the future?).
The result of peer review could then take the form of comments linked to articles. This would provide a much more accurate evaluation. The evaluation would not necessarily be unique. Conflicting opinions could in limiting cases be admitted and judgments could evolve with time.
How could such a system be organized? Two important problems are: who will decide which comments are acceptable? and, should comments be spontaneous? Unscreened spontaneous comments is certainly not the solution for obvious reasons. Screening of comments thus requires, as in paper publication, some kind of editorial board (or several?) who will review the comments (useful comments should be concise, precise and as factual as possible). The system would bear some analogy to what happens in the comment sections of present journals (with the author being given some opportunity to reply), except that comments would not be considered as new publications.
Spontaneous refereeing, if it worked, would have many advantages. Signed and dated comments could be appended to articles (the author himself would also be allowed to add his own remarks or corrections). It may be hoped that scientists would be more inclined to comment articles they enjoyed reading, than articles of marginal interest they only read, under pressure, as referees.
Would spontaneous refereeing be sufficient? Probably not, at least in the near future. A further step would be that the editorial board selects referees and ask them to review some number of articles of their own choice per year.
However this again may not solve all problems. As I mentioned at the beginning, a new system to work must be acceptable to authors. Many authors need their work to be evaluated. If they work in topics temporarily out of fashion spontaneous refereeing may not be sufficient. Also newcomers to a field could for some time be ignored. Therefore we may at least for some time be forced to supplement spontaneous refereeing with solicited evaluation, as in the traditional system. However the new system provides referees with new weapons. They will no longer be forced to make a sometimes very difficult decision. Unflattering comments of the form: "correct article of marginal interest, very hard to read", will increase the pressure on authors to improve the quality of their production.
Finally the evaluation, to be useful, has to be easily accessible, and this concerns both the precise form of the evaluation, and the electronic tools to retrieve it (the Michelin guide to articles on high-temperature superconductivity?). This also leaves some interesting opportunities for publishers.
A few comments
Unsolicited comments clearly should be signed. If this form of comments starts playing an essential role, the widespread anonymous refereeing system, a system which had some virtues, would be endangered. Whether this would have any significant impact is unclear.
We already have some problems with articles replaced in the database by revised version. Some general rules must be established to prevent comments becoming irrelevant because the article has been significantly altered.
Since opinions about the status of articles on a database still vary, a question remains: should the author's authorization be required before attaching a comment to an article, or should the option only be given to reply?
I have some concern about the pricing models which seem to be emerging, in which subscription would pay for paper articles (something which has little added value), the additional work involved in producing useful electronic versions would be free. More reasonable pricing models in the future could be: (i) the author pays page charges which cover the cost of evaluation; (ii) 'publishers sell comments, which become part of a more general process of discussing topics, articles, etc.; or (iii) institutions take care of the evaluation process, as they do for instance for research grants.
The new possibility offered to physicists of having all articles in a field gathered in a unique database (not necessarily physically but at least logically unique), freely accessible, represents major progress and should absolutely be maintained. This requires modification of our system of evaluation of scientific activity. Indeed, if we just wait, the old system will end in chaos, and an abrupt and unprepared transition may be quite painful both for scientists who need to publish to be recognized, and for the whole publishing community.
What precise form this evaluation and commenting process will ultimately take is as yet unclear; we have to be imaginative, bold and pragmatic. Some experimenting will be required before we find a new stable mode of operation. One of the difficulties is to find a path which starts from the present system and continuously moves to something radically different. However, if we succeed we can produce a much more efficient and useful model of scientific communication. This would be of tremendous importance for the whole scientific community.
©Portland Press Ltd., 1997.
Charles Darwin House
12 Roger Street
Tel: +44 (0)20 7685 2425
Fax: +44 (0)20 7685 2468
Portland Press Ltd.
Charles Darwin House
12 Roger Street
London WC1N 2JU
Tel: +44(0) 20 7685 2410
Fax: +44(0) 20 7685 2469