Written evidence submitted by Diane Harley
and Sophia Krzys Acord (PR 88)|
(1) Since 2005, and with generous support from
the A.W. Mellon Foundation, The Future of Scholarly Communication
Project at UC Berkeley's Center for Studies in Higher Education
(CSHE) has been exploring how academic valuesincluding
those related to peer review, publishing, sharing, and collaborationinfluence
scholarly communication practices and engagement with new technological
affordances, open access publishing, and the public good. The
current phase of the project focuses on peer review in the Academy;
this deeper look at peer review is a natural extension of our
findings in Assessing the Future Landscape of Scholarly Communication:
An Exploration of Faculty Values and Needs in Seven Disciplines
(Harley et al. 2010), which stressed the need for a more
nuanced academic reward system that is less dependent on citation
metrics, the slavish adherence to marquee journals and university
presses, and the growing tendency of institutions to outsource
assessment of scholarship to such proxies as default promotion
criteria. This investigation is made urgent by a host of new challenges
facing institutional peer review, such as assessing interdisciplinary
scholarship, hybrid disciplines, the development of new online
forms of edition making and collaborative curation for community
resource use, heavily computational subdisciplines, large-scale
collaborations around grand challenge questions, an increase in
multiple authorship, a growing flood of low-quality publications,
and the call by governments, funding bodies, universities, and
individuals for the open access publication of taxpayer-subsidized
research, including original data sets.
(2) The full document
explores, in particular, the values and costs of the current peer-review
system in academic promotion and publishing. We include discussion
of the tightly intertwined phenomena of peer review in publication
and academic promotion patterns domestically and abroad, variation
in and experimental forms of peer review in the digital environment,
the effects of current academic practices on the publishing system
as a whole, and the possibilities and costs of creating alternative
loci for peer review and publishing that link scholarly societies,
libraries, institutional repositories, and university presses.
We also explore the motivations and ingredients of successful
open access resolutions that are directed at peer-reviewed, article-length
material. This report includes (1) an overview of the state of
peer review in the Academy at large, (2) a set of recommendations
for moving forward, (3) a proposed research agenda to examine
in depth some of the effects of academic status-seeking on the
entire academic enterprise, (4) proceedings from a workshop on
these themes, and (5) four substantial and broadly conceived background
papers on the workshop topics, with associated literature reviews.
What Do We Mean by Peer Review?
(3) The importance of peer review, also known
as scholarly refereeing, flows from being the primary avenue of
quality assessment and control in the academic world. Peer review
has many forms and loci. It acts to signal the quality of a piece
of work, but also functions as a form of gatekeeping to regulate
the entry of new ideas into scholarly fields; it "serves
to maintain overall standards as well as to recognize individual
excellence" (Becher and Trowler 2001, 61). Moreover, it regulates
opportunities throughout a scholar's career, in that it attaches
strongly to reputation and signals a scholar's value in a competitive
academic marketplace. The process and substance of peer review
differs by field, and this diversity and flexibility of peer review
to adapt to disciplinary and subdisciplinary needs, while maintaining
generally high standards, is its strength (cf. Kling and Spector
(4) For clarity of discussion, it is essential
to distinguish among the many forms that peer review can take:
peer review Scholars solicit feedback
on work-in-progress from informal networks (eg, laboratory discussions,
sharing drafts with colleagues, blogs).
peer review Scholars present and circulate
more developed work at invited talks, symposia, and various-sized
conferences to invite comment and citation. (The invitation to
present is itself regulated by an additional level of peer review.)
Posting polished work on personal websites and in repositories
is increasing, although sharing unpublished work openly is highly
variable among disciplines.
peer review The multiple dissemination
outlets for scholarly work (eg, books, journal articles, conference
proceedings, edited volumes) undergo different types of formal
peer review in which peer referees and editors make evaluative
decisions. Different editorial and peer-review models include:
single- or double-blind peer review, student-edited journals in
law, prestigious invited contributions in the humanities, "communicated"
papers in the Proceedings of the National Academy of Sciences
or "lightly" reviewed edited volumes.
peer review Indicators of the significance,
impact, and reception of a scholar's work include: book or performance
reviews, letters to the editor, later citations (including various
bibliometric citation counts), author-meets-critics conference
sessions, article or book prizes, inclusion on course syllabi,
journal clubs, and news and blog coverage, among others.
review of data and other scholarly products In
some fields, peer review is a central criterion to judge other
scholarly products, such as databases, documentary films, websites,
The peer review of data is increasing and creates new problems
in the economies of scholarship for both authors and publishers.
peer review in tenure and promotion cases In
tenure and promotion decisions, peer review is conducted by institutional
representatives, as well as by external referees who are solicited
for letters of support. At most research universities, scholars
are judged by their excellence in three areas: publication, service,
and teaching. (Excellence in the latter two holds little weight
without a stellar publication record and evidence that a scholar's
work is widely read, is judged to be of high quality by internal
and external reviewers, and advances the field.)
review for grants/funding Peer review
at this stage evaluates a scholar's preliminary ideas (and, frequently,
past research record) to determine if he or she will be able to
receive funding for a proposed research program (cf. Lamont 2009;
National Institutes of Health 2008; Weale et al. 2007).
peer review Career work is evaluated for
superlative prizes, awards, and election to elite societies, such
as the National Academies.
(5) Some types of peer review inform others.
For instance, the impact of a scholar's peer-reviewed publications
is integral to the review of a scholar's grant application or
tenure package, and the informal assessment that work-in-progress
receives can influence where it is published (eg, journal editors
may approach scholars at conferences and invite them to publish).
And finally, although the forms of peer review can have different
purposes, a scholar's body of work may, in fact, be peer reviewed
by a relatively small number of people over the course of a career.
Publication-based peer review as it relates to
(6) As Abbott (2008) describes, around the turn
of the nineteenth century informal strategies for manuscript control
gave way to the professional publication-based peer-review system.
The consolidation of formal peer review and publication venue
has led to the latter becoming a general "proxy" for
the level of peer review it carries out. In tenure and promotion
reviews at competitive universities, the emphasis on publishing
in these top-tier outlets is well documented (eg, Becher and Trowler
2001; Boyer 1997; Harley et al. 2010; L Waters 2004; Zuckerman
and Merton 1971).
Overreliance on publisher imprimatur has led to the "outsourcing"
of peer review by linking the quality, relevance, and likely impact
of a piece of work to the symbolic brand of its publisher (including
the publication's Impact Factor).
(7) Traditionally, there is some flexibility
built into how a scholar coming up for tenure and promotion is
judged; "quality over quantity" is the stated ideal
in research-intensive institutions (Harley et al. 2010).
Institutional reviewers may give individual portfolios and published
work a great deal of in-house scrutiny, increase the component
of "campus review" (judgments by individuals in the
university) rather than relying as heavily on external letters
and citation indices, look to secondary indicators in the absence
of large numbers of high-impact publications (such as awards or
other signs that a scholar's work has received unique recognition),
and accept alternative publication formats (eg, journal articles
in lieu of books, ground-breaking instruments in some fields,
and so on).
(8) The emphasis that institutional review places
on publication in the top peer-reviewed outlets, however, is growing,
not decreasing. Senior scholars expect young scholars to meet
the same levels of peer review and certification that they faced.
Consequently, most young scholars do not risk publishing in outlets
that lack prestige; they follow the lead of their mentors and
place enormous value on outlets with established reputations.
Along with the committees that make hiring and promotion decisions,
these young scholars are therefore major actors in the Academy's
inability to break the cycle of publication overproduction. This
overproduction translates into an environment where it is increasingly
commonplace to formally publish work that: is embryonic, of low
quality, should be disseminated more casually, and/or is "salami
sliced" to garner the largest possible number of publications
and to conform to the "smallest publishable unit" format
at many of the top science journals.
(9) These problems are exacerbated by the insidious
and destructive "trickle down" of tenure and promotion
requirements from elite research universities to less competitive
institutions. Compounding this problem further is the mountingand
often unrealisticgovernment pressure on scholars in developed
and emerging economies alike to publish their research in the
most select peer-reviewed outlets, ostensibly to determine the
distribution of government funds (in research assessment exercises)
and/or to meet national imperatives to achieve research distinction
internationally. The global effect is a growing glut of low-quality
publications that strains the entire process of peer review, a
glut that is documented by the increasing number of articles published
every year (Ware and Mabe 2010) and is driven significantly by
scholars in Asia and developing countries (Bell et al.
2007; Holmgren and Schnitzer 2004). Library budgets and preservation
services for this expansion of peer-reviewed publication have
run out. Faculty time spent on peer review, in all of its guises,
is being exhausted.
(10) Bibliometricsparticularly citation
indices and the Impact Factorthat provide scholars with
proxies to gauge the impact of their own work and filter formally
published material post-publication have become important players
in the entire landscape. Bibliometric measures can inform institutional
review and/or the allocation of research grants. They also, for
good or bad, influence where many scholars choose to publish.
A wider array of metrics is becoming available in the digital
environment, creating novel ways of assigning quality and impact
to scholarly work. These include various flavors of citation counts,
bibliograms, webographies, ratings, social bookmarks, download
metrics of articles, and quantitative analyses of reader-generated
open commentary and blog coverage. A real problem with all such
metrics, which we explore in some detail in Background Paper 2,
is that they often substitute quantitative measures (some of which
can be easily gamed, and are of dubious or at best limited value)
for informed and thoughtful judgments by competent and responsible
peers. The overreliance on bibliometrics and external publishing
proxies in career advancement decisions, as well as the institutional
jockeying for higher university rankings, fuels publishing practices
that involve the reassignment of author copyright to entities
that are concerned with making profits over making peer-reviewed
scholarship widely available. An equally troubling reality is
that some of the largest bibliometrics services are controlled
by a few of the largest commercial publishers, such as Elsevier
and Thomson Reuters (Olds 2010).
The imperative to make changes in the system
(11) Given the magnitude of these problems, we
must ask: What value does the current publication-based peer-review
system provide? Which of the myriad forms of peer review that
are used for specific academic purposes (eg, tenure and promotion,
publishing, extramural funding, national and international stature)
should we keep, and which should we modify or abandon? How can
we determine with more accuracy the considerable costs to universities
in subsidizing the entire peer-review process through faculty
salaries on top of the costs to maintain access to the scholarly
record? And, as importantly, how can the significant costs of
high-quality scholarly publishing be borne in the face of calls
for alternative, usually university-based and open access, publishing
models for both journals and books?
(12) How might the Academy move forward productively
in this environment where there is an acknowledged "inflationary
currency" in scholarly publishing and an entrenched system
of peer review that is widely considered to provide an effective
quality filter for busy faculty (and is organized primarily by
publishers, but carried out by faculty)? As we have noted before
in our earlier work, if more nuanced and capacious tenure and
promotion criteria were made explicit at research universities,
it could provide a pragmatic "signaling effect" to other
institutions and government ministries. Blessing faculty to use
a wider array of acceptable alternatives to publish their varied
research output than is currently accepted by tenure and review
committees (and which are currently provided by traditional publishers)
could: maintain the quality of academic peer review and publications,
increase the purchasing power and preservation capabilities of
cash-strapped libraries, better support the free flow of ideas,
relieve overtaxed faculty from the burden of conducting too much
peer review, result in a more economically sustainable publishing
environment overall, and ensure that future generations of scholars
will be able to access scholarship published in digital formats.
Such a change might also lead to a neutralization of the unsustainable
"arms race" to over-publish throughout the Academy.
Some Suggestions for Further Research
(13) As noted throughout this report, the current
problems with peer review in academic publishing and promotion
are due, in great part, to many of the most pernicious effects
of academic status-seeking behavior. Specifically, such a research
agendawhich examines peer-review practices in academic
promotion and publishing, the use of bibliometrics in promotion
and university rankings, and the effectiveness of emergent publishing
modelsshould transect epistemologies of sociology (sociology
of knowledge, network analyses, organizational behavior), economics
(cost/benefit studies, rational and behavioral choice theories),
psychology, anthropology (ethnographies), political science (case
studies of power dynamics, international relations), information
science (bibliometrics, user studies), statistics, and media studies
(digital environments, media ecologies). Research should strive
to be empirical, comparative, and span a complete range of disciplinary
practices within the sciences, social sciences, arts, and humanities.
Suggested research topics and questions include:
primary indicators of effective tenure and promotion review practices
across institutions and higher education sectors internationally.
Institutional and governmental pressure on scholars in developed
and emerging economies to publish research in the most prestigious
publications has resulted in an explosion in the volume of publications
worldwide. This not only strains the efficient and effective practice
of peer review, but also puts at risk research productivity, legitimate
academic publishing endeavors, library acquisition budgets, and
resources for preserving digital-born and digitally migrated materials.
The Academy needs empirical studies of the entire global system
of academic reputation and status seeking in the face of these
challenges. For example, how do practices vary across higher education
sectors and countries? How do research assessment exercises affect
the general quality and number of research publications, as well
as the teaching missions of non-research-intensive institutions?
And what are the actual costs (including social and opportunity
costs) to teaching-intensive institutions of diverting academic
labor from teaching to increasing research output as measured
primarily by publications? In order to identify successful models,
which types of institutions engage in "best practices"
and which rely too heavily on secondary indicators, and why? Can
an agenda to encourage adoption of the good practices we note
above in our recommendations be implemented, and if so, how? For
example, will attempts at reform, such as limiting how many papers
are allowed in tenure and promotion portfolios, encouraging senior
faculty to eschew formal outlets for their research dissemination,
or quid pro quo exchanges of institutional peer reviewing
for promotion decisions, be effective?
a way to make publication costs more transparent so that they
can be appropriately allocated.
the effectiveness and economics of various policies that require
open access to refereed publications as well as the sharing of
data at pre-publication stages of research.
the reasons for failed and successful experiments in alternative
publishing models. For instance, why are there no models for successful
What kinds of services could such journals
provide? Who is in the best position to develop new publishing
models? Are gold open access models ultimately cost-effective
for the Academy?
the success of specific alternatives, such as preprint repositories.
Beyond the hypothesis noted by Harley et al. (2010) that
these systems tend to favor high-paradigm fields with low commercial
value, are there other reasons that more fields have not embraced
a preprint model?
whether bibliometrics or other mechanisms can evolve to filter
scholarship effectively, reliably, and in a way that cannot be
easily gamed or abused.
and assess whether transparent, open, and/or commentary-based
peer-review experiments relieve or add to the burden of reviewing
and filtering relevant literature.
ways to finance adequate publication models to underwrite the
important work of scholarly societies.
Diane Harley and Sophia
Center for Studies in Higher Education
University of California, Berkeley
16 March 2011
11 The full report can be downloaded at http://escholarship.org/uc/item/1xv148c8#page-4 Back
For example, there are clear differences among disciplines, and
many professional schools, such as journalism, architecture, law,
and environmental design, create their own specialized criteria
for judging scholarly output. Back
On the different forms of peer review in PNAS, and their consequences,
see Rand and Pfeiffer (2009). Back
See, for instance, the APA/AIA Task Force on Electronic Publications
(2007) and the EVIA Digital Archive Project for ethnographic field
video in ethnomusicology. Back
Although conference presentations, working papers, (some) edited
volumes, blogs, and other non-peer-reviewed work can help scholars
to establish precedence for their work and may influence the evaluations
written by external reviewers, they do not substitute for peer-reviewed
publications in the institutional review process. (Exceptions
to this include fields like computer science, where conference
papers constitute penultimate publications.) This may be because,
as Borgman (2007) observes, it is easier for institutions to measure
a scholar's outputs (in the form of publications), than to measure
inputs (eg, in the form of research time and other activities). Back
A white paper representing an earlier version of these proposals
was submitted to the NSF's SBE 2020 call for papers examining
Future Research in the Social, Behavioral & Economic Sciences.
Cf: Harley and Acord (2011), Understanding the Drivers and
Dangers of Academic Status Seeking: Studying the Impacts of Embedded
Disciplinary Cultures in a Networked Academy, available at:
Overlay journals are minimalist journals that provide peer review
but not a publishing platform (Suber 2001). Still fairly speculative
at present, an overlay journal would mine self-archived "raw"
author manuscripts from repositories and carry out certain publishing
functions like peer-review management, editing, and perhaps branding
(Swan 2010). The actual published content would continue to reside
in the repository, perhaps with an updated postprint incorporating
any revisions and updated metadata reflecting the journal/society
brand that carried out the peer review. The overlay journal would
then link to the repository content via a traditional table of