Written evidence submitted by Lawrence
Souder (PR 16)
ABSTRACT
Whether we tolerate peer review as it stands, fix
its specific faults, or replace its whole mechanism, one aspect
of its process needs attentionthe tone of its discourse.
The lack of civil discourse in any context, but especially in
science, reflects a troubling lack of trust and sincerity and
it betrays a vulnerability to bias and a will to powerall
of the things anathema to the ideal of science. Various stakeholders
in the peer review system have advanced ways to ameliorate its
tone of discourse. None of these ways has had much effect. Tone,
in fact, may be a symptom of the corrupting effects of the larger
ideology that underlies peer reviewcompetition. This report
recommends shifting the emphasis in peer review back to science's
key normscommunalism, universalism, and disinterestedness.
A civil tone, then, may become both a reflection and a reminder
of the value that binds all scientistsinterdependence.
REVIEW OF
PEER REVIEW
1. The literature from a variety of research
disciplines has acknowledged the imperfect nature of the scholarly
peer review system.1 Perhaps the most candid expression
comes from Richard Horton, editor of The Lancet: "[W]e
know that the system of peer review is biased, unjust, unaccountable,
incomplete, easily fixed, often insulting, usually ignorant, occasionally
foolish, and frequently wrong."2 The literature
reports such peer review indignities as worries over redundant
publication,3 failures to detect plagiarism in published
papers,4 fears of referee theft of authors' ideas,5
suspicions over a journal's rejections based on gender,6
charges of financial conflicts of interest among authors,7
complaints about harshly worded anonymous referee reports,8
concerns over scientists' lack of candor about their research
misconduct,9 and laments over the all-too-frequent
need to publish retractions of tainted research.10
2. In the face of these troubles, a number of
minor adjustments, major changes, and complete replacements for
the status quo have been advanced. The least aggressive suggestions
amount to calls for stricter implementation and policing of existing
policies that govern peer review. Some editors, for example, recommend
formal training of new referees in topics such as peer review
responsibilities and etiquette.11 More radical are
the calls for greater transparency in the peer review process
such as are implied in critiques of anonymous peer review.12
Some commentators of peer review have gone so far as to propose
a complete replacement of the existing system. For example, some
progressive commentators have called for new open-authorship review
processes based on the model of Wikipedia to address the troubles
with traditional peer review.13
3. In the midst of these deliberations for improving
peer review, many of its stakeholders seem resigned to working
in the existing imperfect system. Many have likened it to the
institution of democracy: it's a very bad system, but the alternatives
are so much worse. Campanario was much more sanguine when he asked
rhetorically at the end of his 1998 review of literature: "Could
science survive if the peer review system were suppressed?"
Some scholars have even argued for moral obligations to participate
in peer review.14 Regardless of the direction that
peer review takes, whether minor adjustments or wholesale changes,
one component of this system should be integralattention
to civil discourse. Without civility no social process will function
effectively or ethically.
PROBLEM OF
TONE
4. The tone of discourse has always troubled
the participants of peer review and now since the Climategate
episode it troubles the public, who pays for much of science.
In November of 2009 a computer hacker infiltrated an email server
at the University of East Anglia's Climatic Research Unit (CRU)
and made publically available many private messages among key
climate scientists. Given the highly polarized political debate
around climate science and, especially, climate-related government
regulation, some of these email messages were incendiary. It wasn't
so much the scientific content of these messages that caused such
a stir, but rather their unusually candid tone that provoked strong
criticisms of climate scientists and climate science.
5. No one should be surprised that scientists,
when among their closest colleagues, will let down their guard
in the interests perhaps of conversational efficiency and say
things like "Mike's Nature trick" and "to hide
the decline" to refer to an acceptable method for combining
different kinds of data sets. Subsequent investigations into this
and other unfortunate choices of expression, in fact, have absolved
the writers of any scientific wrongdoing. Nevertheless, other
remarks that implied efforts on the part of scientists to stifle
dissent in the climate science community, censor data, and tamper
with the peer review process provoked many to wonder: to what
extent can scientists keep in check their own human impulses to
be self-serving, doctrinaire, and even vindictive?
6. Peer review is easily and helpfully described
by Merton's norms.15 According to the norm of communalism,
for example, scientists should make their work and data available
to their scientific communities. This ideal is particularly evident
in the outcome of peer review--the publishing and archiving of
research reports. However, communalism is evident even in the
actual process of peer review: editors and referees do not expect
to be gainfully compensated for their time and efforts. Stakeholders
of peer-review systems, in short, should be magnanimous. According
to the norm of universalism scientists should ignore age, race,
nationality, or gender when they evaluate the research of others.
Journal editors have admonished readers to weigh the work of,
for example, senior, white, Western, or male researchers no more
than that of others. Stakeholders of peer-review systems, in other
words, should be impartial. According to the norm of disinterestedness
scientists should dissociate their research interests from their
personal beliefs, attitudes, and values. To this end many journal
editors use blind reviews to minimize the effects of a referee's
interests whether actual or perceived, and disclosures of conflicts
of interest to alert readers of possible sources of bias. Stakeholders
of peer-review systems, in other words, should be selfless.
7. Even though the email messages from the CRU
were not themselves part of a formal peer review, they were ancillary
to and commented on that process. Their tone was troubling because
it reflected an apparent lack of trust and sincerity, and it seemed
to betray a vulnerability to bias and a will to powersentiments
that are all contrary to the ideals of science. The remark, "If
the RMS is going to require authors to make ALL data available
- raw data PLUS results from all intermediate calculations - I
will not submit any further papers to RMS,"16
suggests an impulse to horde information and seems to violate
the norm of communalism. The remark, "Can any competitor
simply request such datasets via the U.S. FOIA, before we have
completed full scientific analysis of these datasets?"
17 calls into question the author's sincerity towards science's
ideal of transparency and seems to violate the norm of disinterestedness.
And perhaps most troubling is the remark, "Kevin and I will
keep them out somehow - even if we have to redefine what the peer-review
literature is!"18 implies a malicious will to
power and seems like a clear violation of universalism.
8. Whether these remarks reflect vicious intent
or merely careless candor, the fact remains that they are thought
to have besmirched the public's view of scientists and even of
science. All of the pundits, in fact, along the continuum from
climate change deniers to warmists are concerned for the appearance
of improprieties among climate scientists. The Telegraph's
Christopher Booker predictably criticized these emails, saying:
"[T]his is the worst scientific scandal of our generation."19
Even The Guardian's George Monbiot remained humble in the
face of damning evidence: "[T]here are some messages that
require no spin to make them look bad."20
ATTEMPTS TO
ADDRESS TONE
9. Journal editors have available to them a number
of resources to address the troubles that careless tone can bring
to peer review. All journal editors who ascribe to the ICMJE Uniform
Requirements for Manuscripts can invoke the guideline on tone:
"In all instances, editors must make an effort to screen
discourteous, inaccurate, or libelous statements and should not
allow ad hominem arguments intended to discredit opinions or findings."21
Unfortunately surveys suggest journal editors are not always
aware of such guidelines.22
10. Absent the access to outside guidance, some
editors have adapted their own principles of civil discourse from
other, more general ethical guidelines. One editor, for example,
invoked the golden rule:23 Manuscript referees should
use the same polite tone in their critiques that they would want
to read in the critiques of their own manuscripts. Yet another
editor adapts the principle of respect for persons from the Nuremberg
Code as a guide to polite behavior in peer review:24
Any researcher should apply the principle of respect for persons
as much to authors in the peer review process as to subjects of
human research.
11. Some editors have taken the problem of uncivil
reviews into their own hands and exercised discretion in recruiting
and assigning referees for peer review. One editor, for example,
recommends weeding out those reviewers who have a habit of making
impolite remarks.25 However, the marketplace of scholarly
peer review is such that referees are dearer than manuscripts.26
It would take a very confident, if not reckless, editor who could
afford to reduce an already limited pool of competent reviewers
to choose from.
12. When no ethical principle can compel civil
discourse among peer reviewers, one editor intimates that threats
of legal action might.27 The UK's Data Protection Act,
for example, specifies that any referee report that is linked
to a specific author is regarded as personal data and as such
is subject to the author's request for access. Of course, the
same protection applies to the records that editors might keep
on the quality of the work of their referees. Thus, it is prudent
for both editors and reviewers to keep a civil tongue lest they
be subject to any actionable comments on record.
APPEAL FOR
A RETURN
TO CORE
VALUES
13. Perhaps all efforts to ameliorate the tone
of peer review are doomed to failure because the stakes are so
high. Tone, after all, may be simply a synecdoche for the fierce
competitiveness in the peer review system. Since monetary gain
is not what draws people to careers in scientific research, it's
a cruel irony that money has become such a strong incentive for
scientists. Science succeeds more on the strength of collaboration
than on competition, so stakeholders in the peer review process
should appeal to the value of interdependence. A civility of tone,
then, could become a barometer for the depth of interdependence
among those in peer review. In this way scientists might see the
tone of discourse in peer review as a reflection and a reminder
of their interdependence.
14. Among the ad hominem attacks on fellow
scientists and the implied disdain for the ideals of science,
an occasional reminder of the sanctity of peer review surfaces
in the CRU emails. One remark, for example, seems refreshingly
principled: "I'd want to protect another academic's freedom
to be contrary and critical, even if I personally believe she
is probably wrong."28 Unfortunately, such remarks
are rare in the CRU email archive. The prevalence of bad tone
is disturbing enough for its indications of scientists' questionable
morality, but it's also disturbing for its apparent effects on
the public's attitude toward science. To rehabilitate the public's
view of scientists and science, we must remove the incentives
that encourage scientists to act in ways contrary to the ideals
of science, and we need to restore the conditions in their research
communities that encourage them to share information freely, judge
each other's work impartially, and work towards the pursuit of
knowledge for its own sake.
REFERENCES
1 For a comprehensive
survey of this literature see my review, "The Ethics of Scholarly
Peer Review: A Review of the Literature," Learned Publishing,
Volume 24, Issue 1, January 2011, 49-66.
2 Horton, R 2000.
Genetically modified food: consternation, confusion, and crack-up.
Medical Journal of Australia, 172: 148-149.
3 Schein, M and Paladugu,
R 2001. Redundant surgical publications: tip of the iceberg? Surgery,
129(6): 655-661.
4 Mojon-Azzi, SM and
Mojon, DS 2004. Scientific misconduct: from salami slicing to
data fabrication. Ophthalmologica, 218: 1-3.
5 Anderson, MS, Ronning,
EA, De Vries, R, and Martinson, BC 2007. The perverse effects
of competition on scientists' work and relationships. Science
and Engineering Ethics, 13(4): 437-461.
6 Budden, AE, Tregenza,
T, Aarssen, LW, Koricheva, J, Leimu, R, and Lortie, CJ 2008. Double-blind
review favours increased representation of female authors. Trends
in Ecology & Evolution, 23(1): 4-6.
7 Bosetti, F and Toscano,
CD 2008. Is it time to standardize ethics guiding the peer review
process? Lipids, 43(2): 107-108.
8 Weber, EJ, Katz,
PP, Waeckerle, JF, and Callaham, ML 2002. Author perception of
peer review: impact of review quality and acceptance on satisfaction.
Journal of the American Medical Association, 287(21): 2790-2793.
9 Redman, BK, Yarandi,
HN, and Merz, JF 2008. Empirical developments in retraction. Journal
of Medical Ethics, 34(11): 807-809.
10 Tobin, MJ 2000.
Reporting research, retraction of results, and responsibility.
American Journal of Respiratory and Critical Care Medicine,
162(3): 773-774.
11 Benos, D, Kirk,
K, and Hall, J. 2003. How to review a paper. Advances in Physiology
Education, 27(2): 47-52.
12 Bence, V and Oppenheim,
C 2004. The influence of peer review on the research assessment
exercise. Journal of Information Science, 30(4): 347-368.
13 Kelty, CM, Burrus,
CS, and Baraniuk, RG 2008. Peer review anew: three principles
and a case study in postpublication quality assurance. Proceedings
of the IEEE, 96(6): 1000-1011.
14 Cain, J 1999. Why
be my colleague's keeper? Moral justifications for peer review.
Science and Engineering Ethics, 5(4): 531-540.
15 Merton, R The
Sociology of Science; Theoretical and Empirical Investigations.
Chicago, University of Chicago Press, 1973.
16 East Anglia Confirmed
Emails from the Climate Research Unit, http://www.eastangliaemails.com/emails.php?eid=967&filename=1237496573.txt
17 East Anglia Confirmed
Emails from the Climate Research Unit, http://www.eastangliaemails.com/emails.php?eid=950&filename=1231257056.txt
18 East Anglia Confirmed
Emails from the Climate Research Unit, http://www.eastangliaemails.com/emails.php?eid=419&filename=1089318616.txt
19 Booker, C, 2009.
Climate change: this is the worst scientific scandal of our generation,
The Telegraph, 28 Nov 2009 http://www.telegraph.co.uk/comment/columnists/christopherbooker/6679082/Climate-change-this-is-the-worst-scientific-scandal-of-our-generation.html
20 Monbiot, G, 2009.
Global warming rigged? Here's the email I'd need to see, The
Guardian, 23 November 2009. http://www.guardian.co.uk/commentisfree/cif-green/2009/nov/23/global-warming-leaked-email-climate-scientists
21 International Committee
of Medical Journal Editors, Uniform Requirements for Manuscripts
Submitted to Biomedical Journals: Writing and Editing for Biomedical
Publication, http://www.icmje.org/, accessed 30 July 2010.
22 Wager, E, Fiack,
S, Graf, C, Robinson, A, and Rowlands, I. 2009. Science journal
editors' views on publication ethics: results of an international
survey. Journal of Medical Ethics, 35(6): 348-353.
23 Gough, NR 2009.
Training for peer review. Science Signaling, 2(85).
24 Hadjistavropoulos,
T and Bieling, P.J. 2000. When reviews attack: ethics, free speech,
and the peer review process. Canadian Psychology, 41(3):
152-159.
25 Miller, CC 2006.
Peer review in the organizational and management sciences: prevalence
and effects of reviewer hostility, bias, and dissensus. Academy
of Management, 49(3): 425-431.
26 Tsui, AS and Hollenbeck,
JR 2009. Successful authors and effective reviewers. Organizational
Research Methods, 12(2): 259-275.
27 Singleton, A 2004.
Data protection and peer review. Learned Publishing, 17(4):
195-198.
28 East Anglia Confirmed
Emails from the Climate Research Unit, http://www.eastangliaemails.com/emails.php?eid=1065&filename=1256765544.txt.
Lawrence Souder
28 February 2011
|