Peer review in scientific publications - Science and Technology Committee Contents


Written evidence submitted by Lawrence Souder (PR 16)

ABSTRACT

Whether we tolerate peer review as it stands, fix its specific faults, or replace its whole mechanism, one aspect of its process needs attention—the tone of its discourse. The lack of civil discourse in any context, but especially in science, reflects a troubling lack of trust and sincerity and it betrays a vulnerability to bias and a will to power—all of the things anathema to the ideal of science. Various stakeholders in the peer review system have advanced ways to ameliorate its tone of discourse. None of these ways has had much effect. Tone, in fact, may be a symptom of the corrupting effects of the larger ideology that underlies peer review—competition. This report recommends shifting the emphasis in peer review back to science's key norms—communalism, universalism, and disinterestedness. A civil tone, then, may become both a reflection and a reminder of the value that binds all scientists—interdependence.

REVIEW OF PEER REVIEW

1.  The literature from a variety of research disciplines has acknowledged the imperfect nature of the scholarly peer review system.1 Perhaps the most candid expression comes from Richard Horton, editor of The Lancet: "[W]e know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong."2 The literature reports such peer review indignities as worries over redundant publication,3 failures to detect plagiarism in published papers,4 fears of referee theft of authors' ideas,5 suspicions over a journal's rejections based on gender,6 charges of financial conflicts of interest among authors,7 complaints about harshly worded anonymous referee reports,8 concerns over scientists' lack of candor about their research misconduct,9 and laments over the all-too-frequent need to publish retractions of tainted research.10

2.  In the face of these troubles, a number of minor adjustments, major changes, and complete replacements for the status quo have been advanced. The least aggressive suggestions amount to calls for stricter implementation and policing of existing policies that govern peer review. Some editors, for example, recommend formal training of new referees in topics such as peer review responsibilities and etiquette.11 More radical are the calls for greater transparency in the peer review process such as are implied in critiques of anonymous peer review.12 Some commentators of peer review have gone so far as to propose a complete replacement of the existing system. For example, some progressive commentators have called for new open-authorship review processes based on the model of Wikipedia to address the troubles with traditional peer review.13

3.  In the midst of these deliberations for improving peer review, many of its stakeholders seem resigned to working in the existing imperfect system. Many have likened it to the institution of democracy: it's a very bad system, but the alternatives are so much worse. Campanario was much more sanguine when he asked rhetorically at the end of his 1998 review of literature: "Could science survive if the peer review system were suppressed?" Some scholars have even argued for moral obligations to participate in peer review.14 Regardless of the direction that peer review takes, whether minor adjustments or wholesale changes, one component of this system should be integral—attention to civil discourse. Without civility no social process will function effectively or ethically.

PROBLEM OF TONE

4.  The tone of discourse has always troubled the participants of peer review and now since the Climategate episode it troubles the public, who pays for much of science. In November of 2009 a computer hacker infiltrated an email server at the University of East Anglia's Climatic Research Unit (CRU) and made publically available many private messages among key climate scientists. Given the highly polarized political debate around climate science and, especially, climate-related government regulation, some of these email messages were incendiary. It wasn't so much the scientific content of these messages that caused such a stir, but rather their unusually candid tone that provoked strong criticisms of climate scientists and climate science.

5.  No one should be surprised that scientists, when among their closest colleagues, will let down their guard in the interests perhaps of conversational efficiency and say things like "Mike's Nature trick" and "to hide the decline" to refer to an acceptable method for combining different kinds of data sets. Subsequent investigations into this and other unfortunate choices of expression, in fact, have absolved the writers of any scientific wrongdoing. Nevertheless, other remarks that implied efforts on the part of scientists to stifle dissent in the climate science community, censor data, and tamper with the peer review process provoked many to wonder: to what extent can scientists keep in check their own human impulses to be self-serving, doctrinaire, and even vindictive?

6.  Peer review is easily and helpfully described by Merton's norms.15 According to the norm of communalism, for example, scientists should make their work and data available to their scientific communities. This ideal is particularly evident in the outcome of peer review--the publishing and archiving of research reports. However, communalism is evident even in the actual process of peer review: editors and referees do not expect to be gainfully compensated for their time and efforts. Stakeholders of peer-review systems, in short, should be magnanimous. According to the norm of universalism scientists should ignore age, race, nationality, or gender when they evaluate the research of others. Journal editors have admonished readers to weigh the work of, for example, senior, white, Western, or male researchers no more than that of others. Stakeholders of peer-review systems, in other words, should be impartial. According to the norm of disinterestedness scientists should dissociate their research interests from their personal beliefs, attitudes, and values. To this end many journal editors use blind reviews to minimize the effects of a referee's interests whether actual or perceived, and disclosures of conflicts of interest to alert readers of possible sources of bias. Stakeholders of peer-review systems, in other words, should be selfless.

7.  Even though the email messages from the CRU were not themselves part of a formal peer review, they were ancillary to and commented on that process. Their tone was troubling because it reflected an apparent lack of trust and sincerity, and it seemed to betray a vulnerability to bias and a will to power—sentiments that are all contrary to the ideals of science. The remark, "If the RMS is going to require authors to make ALL data available - raw data PLUS results from all intermediate calculations - I will not submit any further papers to RMS,"16 suggests an impulse to horde information and seems to violate the norm of communalism. The remark, "Can any competitor simply request such datasets via the U.S. FOIA, before we have completed full scientific analysis of these datasets?" 17 calls into question the author's sincerity towards science's ideal of transparency and seems to violate the norm of disinterestedness. And perhaps most troubling is the remark, "Kevin and I will keep them out somehow - even if we have to redefine what the peer-review literature is!"18 implies a malicious will to power and seems like a clear violation of universalism.

8.  Whether these remarks reflect vicious intent or merely careless candor, the fact remains that they are thought to have besmirched the public's view of scientists and even of science. All of the pundits, in fact, along the continuum from climate change deniers to warmists are concerned for the appearance of improprieties among climate scientists. The Telegraph's Christopher Booker predictably criticized these emails, saying: "[T]his is the worst scientific scandal of our generation."19 Even The Guardian's George Monbiot remained humble in the face of damning evidence: "[T]here are some messages that require no spin to make them look bad."20

ATTEMPTS TO ADDRESS TONE

9.  Journal editors have available to them a number of resources to address the troubles that careless tone can bring to peer review. All journal editors who ascribe to the ICMJE Uniform Requirements for Manuscripts can invoke the guideline on tone: "In all instances, editors must make an effort to screen discourteous, inaccurate, or libelous statements and should not allow ad hominem arguments intended to discredit opinions or findings."21 Unfortunately surveys suggest journal editors are not always aware of such guidelines.22

10.  Absent the access to outside guidance, some editors have adapted their own principles of civil discourse from other, more general ethical guidelines. One editor, for example, invoked the golden rule:23 Manuscript referees should use the same polite tone in their critiques that they would want to read in the critiques of their own manuscripts. Yet another editor adapts the principle of respect for persons from the Nuremberg Code as a guide to polite behavior in peer review:24 Any researcher should apply the principle of respect for persons as much to authors in the peer review process as to subjects of human research.

11.  Some editors have taken the problem of uncivil reviews into their own hands and exercised discretion in recruiting and assigning referees for peer review. One editor, for example, recommends weeding out those reviewers who have a habit of making impolite remarks.25 However, the marketplace of scholarly peer review is such that referees are dearer than manuscripts.26 It would take a very confident, if not reckless, editor who could afford to reduce an already limited pool of competent reviewers to choose from.

12.  When no ethical principle can compel civil discourse among peer reviewers, one editor intimates that threats of legal action might.27 The UK's Data Protection Act, for example, specifies that any referee report that is linked to a specific author is regarded as personal data and as such is subject to the author's request for access. Of course, the same protection applies to the records that editors might keep on the quality of the work of their referees. Thus, it is prudent for both editors and reviewers to keep a civil tongue lest they be subject to any actionable comments on record.

APPEAL FOR A RETURN TO CORE VALUES

13.  Perhaps all efforts to ameliorate the tone of peer review are doomed to failure because the stakes are so high. Tone, after all, may be simply a synecdoche for the fierce competitiveness in the peer review system. Since monetary gain is not what draws people to careers in scientific research, it's a cruel irony that money has become such a strong incentive for scientists. Science succeeds more on the strength of collaboration than on competition, so stakeholders in the peer review process should appeal to the value of interdependence. A civility of tone, then, could become a barometer for the depth of interdependence among those in peer review. In this way scientists might see the tone of discourse in peer review as a reflection and a reminder of their interdependence.

14.  Among the ad hominem attacks on fellow scientists and the implied disdain for the ideals of science, an occasional reminder of the sanctity of peer review surfaces in the CRU emails. One remark, for example, seems refreshingly principled: "I'd want to protect another academic's freedom to be contrary and critical, even if I personally believe she is probably wrong."28 Unfortunately, such remarks are rare in the CRU email archive. The prevalence of bad tone is disturbing enough for its indications of scientists' questionable morality, but it's also disturbing for its apparent effects on the public's attitude toward science. To rehabilitate the public's view of scientists and science, we must remove the incentives that encourage scientists to act in ways contrary to the ideals of science, and we need to restore the conditions in their research communities that encourage them to share information freely, judge each other's work impartially, and work towards the pursuit of knowledge for its own sake.

REFERENCES

1 For a comprehensive survey of this literature see my review, "The Ethics of Scholarly Peer Review: A Review of the Literature," Learned Publishing, Volume 24, Issue 1, January 2011, 49-66.

2 Horton, R 2000. Genetically modified food: consternation, confusion, and crack-up. Medical Journal of Australia, 172: 148-149.

3 Schein, M and Paladugu, R 2001. Redundant surgical publications: tip of the iceberg? Surgery, 129(6): 655-661.

4 Mojon-Azzi, SM and Mojon, DS 2004. Scientific misconduct: from salami slicing to data fabrication. Ophthalmologica, 218: 1-3.

5 Anderson, MS, Ronning, EA, De Vries, R, and Martinson, BC 2007. The perverse effects of competition on scientists' work and relationships. Science and Engineering Ethics, 13(4): 437-461.

6 Budden, AE, Tregenza, T, Aarssen, LW, Koricheva, J, Leimu, R, and Lortie, CJ 2008. Double-blind review favours increased representation of female authors. Trends in Ecology & Evolution, 23(1): 4-6.

7 Bosetti, F and Toscano, CD 2008. Is it time to standardize ethics guiding the peer review process? Lipids, 43(2): 107-108.

8 Weber, EJ, Katz, PP, Waeckerle, JF, and Callaham, ML 2002. Author perception of peer review: impact of review quality and acceptance on satisfaction. Journal of the American Medical Association, 287(21): 2790-2793.

9 Redman, BK, Yarandi, HN, and Merz, JF 2008. Empirical developments in retraction. Journal of Medical Ethics, 34(11): 807-809.

10 Tobin, MJ 2000. Reporting research, retraction of results, and responsibility. American Journal of Respiratory and Critical Care Medicine, 162(3): 773-774.

11 Benos, D, Kirk, K, and Hall, J. 2003. How to review a paper. Advances in Physiology Education, 27(2): 47-52.

12 Bence, V and Oppenheim, C 2004. The influence of peer review on the research assessment exercise. Journal of Information Science, 30(4): 347-368.

13 Kelty, CM, Burrus, CS, and Baraniuk, RG 2008. Peer review anew: three principles and a case study in postpublication quality assurance. Proceedings of the IEEE, 96(6): 1000-1011.

14 Cain, J 1999. Why be my colleague's keeper? Moral justifications for peer review. Science and Engineering Ethics, 5(4): 531-540.

15 Merton, R The Sociology of Science; Theoretical and Empirical Investigations. Chicago, University of Chicago Press, 1973.

16 East Anglia Confirmed Emails from the Climate Research Unit, http://www.eastangliaemails.com/emails.php?eid=967&filename=1237496573.txt

17 East Anglia Confirmed Emails from the Climate Research Unit, http://www.eastangliaemails.com/emails.php?eid=950&filename=1231257056.txt

18 East Anglia Confirmed Emails from the Climate Research Unit, http://www.eastangliaemails.com/emails.php?eid=419&filename=1089318616.txt

19 Booker, C, 2009. Climate change: this is the worst scientific scandal of our generation, The Telegraph, 28 Nov 2009 http://www.telegraph.co.uk/comment/columnists/christopherbooker/6679082/Climate-change-this-is-the-worst-scientific-scandal-of-our-generation.html

20 Monbiot, G, 2009. Global warming rigged? Here's the email I'd need to see, The Guardian, 23 November 2009. http://www.guardian.co.uk/commentisfree/cif-green/2009/nov/23/global-warming-leaked-email-climate-scientists

21 International Committee of Medical Journal Editors, Uniform Requirements for Manuscripts Submitted to Biomedical Journals: Writing and Editing for Biomedical Publication, http://www.icmje.org/, accessed 30 July 2010.

22 Wager, E, Fiack, S, Graf, C, Robinson, A, and Rowlands, I. 2009. Science journal editors' views on publication ethics: results of an international survey. Journal of Medical Ethics, 35(6): 348-353.

23 Gough, NR 2009. Training for peer review. Science Signaling, 2(85).

24 Hadjistavropoulos, T and Bieling, P.J. 2000. When reviews attack: ethics, free speech, and the peer review process. Canadian Psychology, 41(3): 152-159.

25 Miller, CC 2006. Peer review in the organizational and management sciences: prevalence and effects of reviewer hostility, bias, and dissensus. Academy of Management, 49(3): 425-431.

26 Tsui, AS and Hollenbeck, JR 2009. Successful authors and effective reviewers. Organizational Research Methods, 12(2): 259-275.

27 Singleton, A 2004. Data protection and peer review. Learned Publishing, 17(4): 195-198.

28 East Anglia Confirmed Emails from the Climate Research Unit, http://www.eastangliaemails.com/emails.php?eid=1065&filename=1256765544.txt.

Lawrence Souder

28 February 2011


 
previous page contents next page


© Parliamentary copyright 2011
Prepared 28 July 2011