The Impact of Electronic Publishing on the Academic Community
Session 3: The content and quality of academic communication
A system for electronic peer review
Andrew G. Cohen
Physics Department, Boston University, 590 Commonwealth Ave, Boston, MA 02215, U.S.A.
©Andrew G. Cohen, 1997.
I describe an electronic system for implementing peer review in theoretical high-energy physics as an 'overlay' to the existing e-print (electronic print) archive at Los Alamos National Laboratory.
The traditional functions of a scholarly journal, including disseminating, validating and archiving the results of academic research, can be divided broadly into two categories: those dealing with the actual manipulation of the representation of these results (commonly in the form of a research article, but also raw scientific data, database entries, etc.); and those dealing with auxiliary information (e.g. refereeing, indexing, commentary and similar items). The widespread use of electronic media currently allows a more-or-less total separation of these two functions. I want to argue that, in addition to being possible, this separation is desirable, allowing objectives such as peer review to be accomplished more easily and more cheaply than the usual entanglement of these two ideas.
In the field of theoretical high-energy physics the foundation for this separation already exists, in the form of the Los Alamos e-print archive [which can be accessed at the URL (universal resource locator) http://xxx.lanl.gov/]. This archive is the continuation of a 25-year tradition in particle physics of making documents available prior to formal review in the form of 'preprints'. Documents are submitted to this archive by the author, where they are (automatically) made freely available in various forms through a Web, e-mail and ftp interface. The use of this system, first implemented five years ago, is nearly universal: all research publications in theoretical high-energy physics are available from this archive. It now provides at no direct cost to the researcher the functions in the first category described earlier.
What this archive currently lacks is the functionality of the second category. All documents (except those that do not produce viewable output, or those that do not qualify as research in theoretical high-energy physics) are accepted without regard to content, format or other conventional publication criteria: the archive functions as a means of dissemination and archiving of all work in high-energy theory, but does not provide the auxiliary information that is associated with traditional publication. This state of affairs has led to changes in the practice of high-energy physics: recent journal issues are rarely consulted, the distinction between refereed and non-refereed publication is evaporating, the quality (at least subjectively) of individual articles is deteriorating. Consequently a significant task facing high-energy theorists is accommodation to this new mode of operation. What is needed is a scheme for incorporating auxiliary data; initially one which would reproduce those functions of journals which are being eroded, but subsequently would allow the incorporation of new types of auxiliary data.
The existence of the archive with its open, although undocumented, API (application programming interface) allows for easy access and manipulation of the documents themselves. Thus the construction of a system to incorporate auxiliary data must focus on two issues: (i) how to generate auxiliary data; and (ii) how to serve auxiliary data.
We chose to implement one particular example: quality review. This example is designed to mimic that of the traditional refereeing process: validation through anonymous peer review. The resulting datum is a single bit: acceptable or unacceptable according to a set of reviewing standards. The generation of this quality 'mark' is through a traditional correspondence between author and anonymous referee. The process is managed as much as possible by an electronic agent, supervised by a physicist. The human agent makes a final decision on the basis of this correspondence.
The generation of this datum is designed to be low cost, and place the least possible burden on the physicists involved. All transactions involving the actual document being considered are done through the archive. The author 'submits' the document for review by contacting the electronic agent through the World Wide Web, and provides the archive identifier for the document (rather than the document itself). This relieves the review process of the necessity of insuring that a properly viewable document is available for review: documents that do not meet this criterion are rejected by the archive rather than by the peer-review process. The author provides some rudimentary information about the document through an automatic form (such as a topical identifier, category of research, etc.) which allows the electronic agent to automatically assign a particular physicist (an 'editor') to oversee the review, as well as choose appropriate referees. These referees are automatically informed by e-mail of the review request, and provided with the document's archive identifier. Reviewers submit their reports to the electronic agent through the Web, and the editor in charge is notified. Similar interactions between the agent and the author allow for flexibility in the review process: the author may choose to make modifications to the document (new versions are placed on the archive with the same identifier as the original, with a modification record automatically kept), request new referees, etc. The final judgment of the editor constitutes the auxiliary bit: acceptable or unacceptable.
The presentation of this datum is designed to look as much as possible like the similar presentation of a traditional refereed journal. That is, the documents that have been deemed acceptable are collected into issues of an electronic 'journal', and may be referred to as 'published' in this journal (although in fact publication has already taken place on the archive). Of course, the documents themselves continue to reside on the archive: the journal 'issues' are in fact merely electronic pointers (links on a Web page) to the documents in the archive. In addition the archive API allows a mark to be added directly to the document on the archive indicating acceptance, allowing the archive tools (such as searching and indexing) to reflect this new datum.
The electronic agent is an object-oriented system written in Perl, although any language supporting object-oriented features would be suitable. The details of the peer-review process, which was designed to be as traditional as possible, are not particularly relevant. The main point of this exercise was to demonstrate that the separation of document handling from auxiliary issues such as peer review allows for the easy production of systems such as this. The object-oriented paradigm is ideally suited for this separation: the usual advantages of abstraction, reuse and inheritance will help foster an easy transition from traditional publishing methods to electronic systems.
©Andrew G. Cohen, 1997.
Charles Darwin House
12 Roger Street
Tel: +44 (0)20 7685 2425
Fax: +44 (0)20 7685 2468
Portland Press Ltd.
Charles Darwin House
12 Roger Street
London WC1N 2JU
Tel: +44(0) 20 7685 2410
Fax: +44(0) 20 7685 2469