Peer review and academic responsibility
Richard Parncutt, 2009, revised 2011

Peer review is the most important and widespread means of controlling and ensuring quality in academic journals, conferences and projects. The proportion of academic journals and conferences with good peer-review procedures has been rising steadily in recent decades. Academics in most disciplines are donating their time and energy freely to peer-review processes, which give them access to confidential drafts of current research and in that way keep them up-to-date. 

Peer review means that an article submitted to a journal (or an abstract to a conference, or a grant application) is read by (usually at least two, usually anonymous) experts as the submission before being accepted for publication (or presentation, or funding). These peers usually also make suggestions for improvement of the submission, and usually an editor (or conference organiser) ensures that the author(s) of the submission respond constructively to the suggestions before the submission is published (or presented). Rejection is not only a possibility but also happens regularly (if not, the journal is unlikely to be taken seriously by experts in the same area). In the case of conferences, it means a talk or poster is not placed on the program until a corresponding abstract submission or proposal has been similarly evaluated. In the case of grant applications, granting agencies rely on the opinions of international experts before deciding who gets money for research projects. 

A central assumption behind peer review is that a piece of research can only be realistically evaluated by internationally recognized experts in the same specific area, because only they can judge the "truth content" or the submission, that is, the extent to which the claims being made are realistic or feasible. Other people, such as researchers in other areas, administrators, and students are not in a good position to do that. If asked to evaluate a piece of research, non-experts ask themselves whether the research sounds interesting or important, or evaluate it against standards in their own discipline. If it feels good, they will recommend publication. Thus, evaluation by people who are not not internationally acknowledged  experts in the same specific area can lead to the publication of what experts know to be poor research. It also encourages academics to use impressive sounding jargon to impress the non-experts and to hide their weak underlying arguments. The only reliable solution to these tendencies, which can be found in all academic disciplines, is peer review. 

Another important feature of peer review is that reviewers have the option of remaining anonymous. They need not fear any consequences of a negative review - even if the person whose work is being reviewed is a valued colleague or personal friend. This frees the reviewers to give an honest opinion. 

That peer review is a good thing needs hardly be justified.

• It promotes creativity. Psychologists (e.g. Csikszentmihalyi, 1988) have investigated  the separate and combined roles of three elements - society, culture, and the individual creative person - in creative work. "Society" includes the feedback that creative people receive from other people, which in an academic context includes the suggestions that researchers receive from anonymous colleagues in peer-review procedures. 
• It reduces the chance that poor research will be published. Any student who has spent a lot of time and energy studying a piece of research that later turned out to be of poor quality, superceded, or just plain nonsense, will agree that peer review is necessary and important. 
• It reduces the incidence of chauvinist, ethnocentric or authoritarian attitudes among academics. An academic who is repeatedly required to respond constructively to the suggestions and criticisms of anonymous expert colleagues has no choice but to develop democratic attitudes and to act in a collegial fashion. This is a particularly important function of peer review, given that university professors have a strong influence on their students, many of whom later become socially powerful. Peer review encourages realism, modesty, and a practical orientation, but it can also be hard work - which explains why many academics resist it.

Peer review also has its down side. It does not work well if there is a conflict of interest between an author and a reviewer, i.e. if a reviewer is or has been in a close professional or personal relationship with the author of an article. Journals often explicitly ask reviewers to refuse to review in this case, and by all accounts the instruction works. Another problem is that reviewers may review on the basis of what they know about the author rather than the content, or their decision may be biased either consciously or unconsciously by knowledge about the sex or cultural origin of the author/s (sexism, racism). Such problems can be avoided by double-blind review, in which not only the identity of the reviewer/s is concealed from the author/s,  but also vice-versa. The probability of people successfully guessing each other's identity can be reduced by avoiding any statement that may reveal identity, but in practice this possibility can never completely eliminated. For that reason, conflict of interest is best avoided if reviewers are asked not to review if they have or had a close personal or professional relationship with a suspected author. In double-blind procedures, reviewers and authors are often given the option of making their identity known to each other, which encourages the development of new collaborations in the future.

Because it is impossible to completely avoid reviewer bias, some academics have concluded that peer review should be abandoned altogether. But that is not consistent with the experience of journal editors, who know from experience that some kind of quality control is always better than none. Ultimately, publication is always selective - someone has to decide on some basis what will be published and what will be rejected. Clearly, this decision is best made by anonymous experts. In this regard, peer review is a bit like democracy. I am not a great fan of Winston Churchill, but he was right when he said that democracy is the worst form of government except all the others that have been tried. Similarly, peer review is the worst form of academic quality control except all the others that have been tried. 

How to identify a peer-reviewed journal

It is not always immediately clear whether a given journal subjects its submissions to a (good) peer review procedure. You may have to do a little bit of detective work. 

• The first thing to do is look for the journal in the internet. If it doesn't have some kind of homepage, it is unlikely to be peer reviewed. 
• Having found the homepage, read the general description of the journal. If it says that the journal is peer reviewed or refereed, you can usually believe it. For example, I found the following sentence on the homepage of the Journal of New Music Research: "All research articles published in this journal have undergone rigorous peer review". 
• If the general description of the journal does not contain such a statement (perhaps because the editors think it is obvious), check out the submission guidelines (sometimes called "information for authors", "how to submit an article" or similar). Under the heading "Submit an article", the homepage of the journal Music Perception says "Manuscripts are accepted for review on the understanding that...". The word "review" suggests a peer-review procedure. Under the heading "Hinweise zur Einreichung von Manuskripte" the homepage of Jahrbuch Musikpsychologie says "Um sich über den Stand der Begutachtung etc. zu informieren, erhalten die AutorInnen der eingereichten Manuskripte Zugang zum Reviewsystem des Jahrbuches."
• Some well-known academic societies are associated only with peer-reviewed journals. An example is the American Psychological Association. All APA journals are peer-reviewed,
  

Given that a journal has a peer-review system, how can good is it? This question is harder to answer. A good peer-review procedure is one in which an independent sub-editor first approaches leading international experts in the specific area of the submission and these experts are free to express any well-founded opinions about the content of the submission. Often it is impossible to know this unless one has been closely associated with the journal for an extended period. I guess as a rule one could say that journals with a high rejection rate or a high impact factor have good review procedures, but this is not necessarily the case. Rejection rates are often not publicy available and impact factors are problematic in different ways and difficult to compare across disciplines and languages. Another plus is when review procedures can be double-blind, that is both authors and reviewers have the option of remaining anonymous. Finally, in older journals it may be unclear when peer review was introduced, and whether there was a transition period during which it was applied inconsistently.


History of peer review

The increasing prevalence of peer review procedures in modern research publications reflects a fundamental change in international academic culture. We are witnessing a long, slow transition from an old to a new model. The old model has its roots in the 19th Century and earlier. In its extreme modern form, it is characterized by white male professors who prefer to work alone (rather than collaborate), resist evaluation of their academic performance in research or teaching, and expect special respect from other people, especially their students. The old model worked provided that (i) professors could control the quality of the research of their colleagues, and (ii) the professors themselves were good (i.e. the professorial selection committees succeeded in selecting the best candidates). In Austria, the era of the Professorenuniversität ended with the introduction of the university organisation law of 1975 (UOG75), which gave students and lecturers new democratic powers. 

At that point, a new form of research evaluation was necessary to compensate for the diminishing powers of the professors. But that did not happen. Instead, the international peer review model slowly infiltrated Austrian universities. This new model of academic quality control is continuing to develop internationally in parallel with international progress in political democracy, human rights, communication technologies, teaching methods and so on. It is characterized by networking, collegiality, and diversity (of gender, culture and so on). Academics in this new tradition welcome and even seek out independent evaluations of their research and teaching, because they give them ideas and help them to improve and compete. They also prefer to communicate eye to eye with others, including students. They avoid unnecessary implicit differences in power or value, and tend to promote openness, tolerance and democracy.

I don't want to imply that everything about the old model is bad, and everything about the new is good. A highly democratic system is very time-consuming and ultimately conservative, because democratic processes often have the effect of preventing necessary change. A dynamic, forward-looking organisation needs a hierarchical structure of bosses who have a reasonable amount of power to enforce their decisions, and it is normal and ok that they sometimes contradict the will of their colleagues. Thus, the best organisational structures take a middle path between total democracy (which is fair, but quite impractical) and total autocracy (which is efficient, but tends to be corrupt and insensitive), so as to simultaneously maximize both worker satisfaction and productivity. Consider for example Austrian university law: most Austrian academics would agree that an optimal compromise would lie somewhere between the democractic old law (UOG1993) and the authoritarian new law (UG2002).

From this point of view, all universities, academic societies, conferences and journals - in fact all aspects of academic infrastructures - contain a mixture of conservative and progressive forces. In fact, every individual professor may be regarded as a mix of conservative and progressive. In the more conservative infrastructures and world-views, the old, conservative model predominates; in newer infrastructures that have appeared in recent years and decades, the progressive model.


Humanities versus sciences, German versus English

The humanities tend to be more conservative than the sciences - presumably, because humanities dominated academe in the 19th Century, the sciences changed more rapidly in the 20th, and today, results of scientific investigations are more likely to be rigorously tested through practical applications. Independently of that distinction, English-speaking research is generally more progressive than German-speaking research - presumably because the American system and culture place higher value on open competition and reward achievement more than conformity. As a result, peer-review is more common in the English language and sciences than the German language and humanities. Said another way: an awareness of academic responsibility to control quality fairly - with open procedures but concealed identities - is more common in the English language and the sciences. On balance, this puts scientific research in the English language at an advantage.

Today, these issues are of central importance for all academics, but especially for German-speaking humanities scholars. If they fail to take action, their disciplines will continue to decline. They will get even less money and even fewer professorships, and the general public will be even less aware that they exist (let alone know what they do). This is bad news for the human race, because humanities are crucial important for human identity, quality of life, and intercultural communication, and the venerable German-language tradition of research and teaching has enormous potential to contribute positively to 21st-Century global society.

And logically, there is no particular reason why German humanities research should be dragging its feet. The oft-heard argument that peer-review is less appropriate for humanities does not apply if one assumes that humanities and sciences are, in a fundamental sense, equally important - and in both cases evaluation can only be carried out by internationally recognized experts in the same specific discipline. When German speakers use the English term "peer review" in a German context, thereby avoiding the German term Kreuzgutachten, they reinforce the myth that peer review is something that international English-speaking scholarship is forcing on German scholarship, which is doing very well without it. But a quick glance at any of those many university rankings will confirm that the latter is not the case - not even if the intrinsic bias of those rankings towards the English language could somehow be compensated for. Instead, German-language scholarship, the humanities, and in particular German-language humanities, urgently need to develop a new spirit of academic democracy and responsibility, based on peer review.

German-speaking academics and humanities scholars should not aim to emulate existing peer-review procedures, but to improve on them. German-speaking academics have a distinct Vorsprung by comparison to their English-speaking colleagues: they are better at languages, which gives them access to more literature. They are also more sensitive to linguistic nuances and to intercultural differences on the level of language. These advantages can and should be used to their own advantage. Similiarly, humanities scholars from all countries and in all languages have advantages over their scientific colleagues: they, too, are better at languages and more sensitive to linguistic nuance, and they are also more sensitive to cultural differences generally. Again, these advantages can and should be used to create new and better approaches to peer review.

Literature 

Csikszentmihalyi, M. (1988). Society, culture, and person: A systems view of creativity. In R. J. Sternberg (Ed), The nature of creativity: Contemporary psychological perspectives (pp. 325-339). New York: Cambridge University Press. 

The opinions expressed on this page are personal to the author.