Peer review in scientific publications - Science and Technology Committee Contents

Written evidence submitted by Diane Harley and Sophia Krzys Acord (PR 88)

(1)  Since 2005, and with generous support from the A.W. Mellon Foundation, The Future of Scholarly Communication Project at UC Berkeley's Center for Studies in Higher Education (CSHE) has been exploring how academic values—including those related to peer review, publishing, sharing, and collaboration—influence scholarly communication practices and engagement with new technological affordances, open access publishing, and the public good. The current phase of the project focuses on peer review in the Academy; this deeper look at peer review is a natural extension of our findings in Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines (Harley et al. 2010), which stressed the need for a more nuanced academic reward system that is less dependent on citation metrics, the slavish adherence to marquee journals and university presses, and the growing tendency of institutions to outsource assessment of scholarship to such proxies as default promotion criteria. This investigation is made urgent by a host of new challenges facing institutional peer review, such as assessing interdisciplinary scholarship, hybrid disciplines, the development of new online forms of edition making and collaborative curation for community resource use, heavily computational subdisciplines, large-scale collaborations around grand challenge questions, an increase in multiple authorship, a growing flood of low-quality publications, and the call by governments, funding bodies, universities, and individuals for the open access publication of taxpayer-subsidized research, including original data sets.

(2)  The full document[11] explores, in particular, the values and costs of the current peer-review system in academic promotion and publishing. We include discussion of the tightly intertwined phenomena of peer review in publication and academic promotion patterns domestically and abroad, variation in and experimental forms of peer review in the digital environment, the effects of current academic practices on the publishing system as a whole, and the possibilities and costs of creating alternative loci for peer review and publishing that link scholarly societies, libraries, institutional repositories, and university presses. We also explore the motivations and ingredients of successful open access resolutions that are directed at peer-reviewed, article-length material. This report includes (1) an overview of the state of peer review in the Academy at large, (2) a set of recommendations for moving forward, (3) a proposed research agenda to examine in depth some of the effects of academic status-seeking on the entire academic enterprise, (4) proceedings from a workshop on these themes, and (5) four substantial and broadly conceived background papers on the workshop topics, with associated literature reviews.

What Do We Mean by Peer Review?

(3)  The importance of peer review, also known as scholarly refereeing, flows from being the primary avenue of quality assessment and control in the academic world. Peer review has many forms and loci. It acts to signal the quality of a piece of work, but also functions as a form of gatekeeping to regulate the entry of new ideas into scholarly fields; it "serves to maintain overall standards as well as to recognize individual excellence" (Becher and Trowler 2001, 61). Moreover, it regulates opportunities throughout a scholar's career, in that it attaches strongly to reputation and signals a scholar's value in a competitive academic marketplace. The process and substance of peer review differs by field, and this diversity and flexibility of peer review to adapt to disciplinary and subdisciplinary needs, while maintaining generally high standards, is its strength (cf. Kling and Spector 2004).[12]

(4)  For clarity of discussion, it is essential to distinguish among the many forms that peer review can take:

—  Developmental peer review Scholars solicit feedback on work-in-progress from informal networks (eg, laboratory discussions, sharing drafts with colleagues, blogs).

—  Pre-publication peer review Scholars present and circulate more developed work at invited talks, symposia, and various-sized conferences to invite comment and citation. (The invitation to present is itself regulated by an additional level of peer review.) Posting polished work on personal websites and in repositories is increasing, although sharing unpublished work openly is highly variable among disciplines.

—  Publication-based peer review The multiple dissemination outlets for scholarly work (eg, books, journal articles, conference proceedings, edited volumes) undergo different types of formal peer review in which peer referees and editors make evaluative decisions. Different editorial and peer-review models include: single- or double-blind peer review, student-edited journals in law, prestigious invited contributions in the humanities, "communicated" papers in the Proceedings of the National Academy of Sciences (PNAS),[13] or "lightly" reviewed edited volumes.

—  Post-publication peer review Indicators of the significance, impact, and reception of a scholar's work include: book or performance reviews, letters to the editor, later citations (including various bibliometric citation counts), author-meets-critics conference sessions, article or book prizes, inclusion on course syllabi, journal clubs, and news and blog coverage, among others.

—  Peer review of data and other scholarly products In some fields, peer review is a central criterion to judge other scholarly products, such as databases, documentary films, websites, and software.[14] The peer review of data is increasing and creates new problems in the economies of scholarship for both authors and publishers.

—  Institutional peer review in tenure and promotion cases In tenure and promotion decisions, peer review is conducted by institutional representatives, as well as by external referees who are solicited for letters of support. At most research universities, scholars are judged by their excellence in three areas: publication, service, and teaching. (Excellence in the latter two holds little weight without a stellar publication record and evidence that a scholar's work is widely read, is judged to be of high quality by internal and external reviewers, and advances the field.)

—  Peer review for grants/funding Peer review at this stage evaluates a scholar's preliminary ideas (and, frequently, past research record) to determine if he or she will be able to receive funding for a proposed research program (cf. Lamont 2009; National Institutes of Health 2008; Weale et al. 2007).

—  Cumulative peer review Career work is evaluated for superlative prizes, awards, and election to elite societies, such as the National Academies.

(5)  Some types of peer review inform others. For instance, the impact of a scholar's peer-reviewed publications is integral to the review of a scholar's grant application or tenure package, and the informal assessment that work-in-progress receives can influence where it is published (eg, journal editors may approach scholars at conferences and invite them to publish). And finally, although the forms of peer review can have different purposes, a scholar's body of work may, in fact, be peer reviewed by a relatively small number of people over the course of a career.

Publication-based peer review as it relates to institutional review

(6)  As Abbott (2008) describes, around the turn of the nineteenth century informal strategies for manuscript control gave way to the professional publication-based peer-review system. The consolidation of formal peer review and publication venue has led to the latter becoming a general "proxy" for the level of peer review it carries out. In tenure and promotion reviews at competitive universities, the emphasis on publishing in these top-tier outlets is well documented (eg, Becher and Trowler 2001; Boyer 1997; Harley et al. 2010; L Waters 2004; Zuckerman and Merton 1971).[15] Overreliance on publisher imprimatur has led to the "outsourcing" of peer review by linking the quality, relevance, and likely impact of a piece of work to the symbolic brand of its publisher (including the publication's Impact Factor).

(7)  Traditionally, there is some flexibility built into how a scholar coming up for tenure and promotion is judged; "quality over quantity" is the stated ideal in research-intensive institutions (Harley et al. 2010). Institutional reviewers may give individual portfolios and published work a great deal of in-house scrutiny, increase the component of "campus review" (judgments by individuals in the university) rather than relying as heavily on external letters and citation indices, look to secondary indicators in the absence of large numbers of high-impact publications (such as awards or other signs that a scholar's work has received unique recognition), and accept alternative publication formats (eg, journal articles in lieu of books, ground-breaking instruments in some fields, and so on).

(8)  The emphasis that institutional review places on publication in the top peer-reviewed outlets, however, is growing, not decreasing. Senior scholars expect young scholars to meet the same levels of peer review and certification that they faced. Consequently, most young scholars do not risk publishing in outlets that lack prestige; they follow the lead of their mentors and place enormous value on outlets with established reputations. Along with the committees that make hiring and promotion decisions, these young scholars are therefore major actors in the Academy's inability to break the cycle of publication overproduction. This overproduction translates into an environment where it is increasingly commonplace to formally publish work that: is embryonic, of low quality, should be disseminated more casually, and/or is "salami sliced" to garner the largest possible number of publications and to conform to the "smallest publishable unit" format at many of the top science journals.

(9)  These problems are exacerbated by the insidious and destructive "trickle down" of tenure and promotion requirements from elite research universities to less competitive institutions. Compounding this problem further is the mounting—and often unrealistic—government pressure on scholars in developed and emerging economies alike to publish their research in the most select peer-reviewed outlets, ostensibly to determine the distribution of government funds (in research assessment exercises) and/or to meet national imperatives to achieve research distinction internationally. The global effect is a growing glut of low-quality publications that strains the entire process of peer review, a glut that is documented by the increasing number of articles published every year (Ware and Mabe 2010) and is driven significantly by scholars in Asia and developing countries (Bell et al. 2007; Holmgren and Schnitzer 2004). Library budgets and preservation services for this expansion of peer-reviewed publication have run out. Faculty time spent on peer review, in all of its guises, is being exhausted.

(10)  Bibliometrics—particularly citation indices and the Impact Factor—that provide scholars with proxies to gauge the impact of their own work and filter formally published material post-publication have become important players in the entire landscape. Bibliometric measures can inform institutional review and/or the allocation of research grants. They also, for good or bad, influence where many scholars choose to publish. A wider array of metrics is becoming available in the digital environment, creating novel ways of assigning quality and impact to scholarly work. These include various flavors of citation counts, bibliograms, webographies, ratings, social bookmarks, download metrics of articles, and quantitative analyses of reader-generated open commentary and blog coverage. A real problem with all such metrics, which we explore in some detail in Background Paper 2, is that they often substitute quantitative measures (some of which can be easily gamed, and are of dubious or at best limited value) for informed and thoughtful judgments by competent and responsible peers. The overreliance on bibliometrics and external publishing proxies in career advancement decisions, as well as the institutional jockeying for higher university rankings, fuels publishing practices that involve the reassignment of author copyright to entities that are concerned with making profits over making peer-reviewed scholarship widely available. An equally troubling reality is that some of the largest bibliometrics services are controlled by a few of the largest commercial publishers, such as Elsevier and Thomson Reuters (Olds 2010).

The imperative to make changes in the system

(11)  Given the magnitude of these problems, we must ask: What value does the current publication-based peer-review system provide? Which of the myriad forms of peer review that are used for specific academic purposes (eg, tenure and promotion, publishing, extramural funding, national and international stature) should we keep, and which should we modify or abandon? How can we determine with more accuracy the considerable costs to universities in subsidizing the entire peer-review process through faculty salaries on top of the costs to maintain access to the scholarly record? And, as importantly, how can the significant costs of high-quality scholarly publishing be borne in the face of calls for alternative, usually university-based and open access, publishing models for both journals and books?

(12)  How might the Academy move forward productively in this environment where there is an acknowledged "inflationary currency" in scholarly publishing and an entrenched system of peer review that is widely considered to provide an effective quality filter for busy faculty (and is organized primarily by publishers, but carried out by faculty)? As we have noted before in our earlier work, if more nuanced and capacious tenure and promotion criteria were made explicit at research universities, it could provide a pragmatic "signaling effect" to other institutions and government ministries. Blessing faculty to use a wider array of acceptable alternatives to publish their varied research output than is currently accepted by tenure and review committees (and which are currently provided by traditional publishers) could: maintain the quality of academic peer review and publications, increase the purchasing power and preservation capabilities of cash-strapped libraries, better support the free flow of ideas, relieve overtaxed faculty from the burden of conducting too much peer review, result in a more economically sustainable publishing environment overall, and ensure that future generations of scholars will be able to access scholarship published in digital formats. Such a change might also lead to a neutralization of the unsustainable "arms race" to over-publish throughout the Academy.

Some Suggestions for Further Research[16]

(13)  As noted throughout this report, the current problems with peer review in academic publishing and promotion are due, in great part, to many of the most pernicious effects of academic status-seeking behavior. Specifically, such a research agenda—which examines peer-review practices in academic promotion and publishing, the use of bibliometrics in promotion and university rankings, and the effectiveness of emergent publishing models—should transect epistemologies of sociology (sociology of knowledge, network analyses, organizational behavior), economics (cost/benefit studies, rational and behavioral choice theories), psychology, anthropology (ethnographies), political science (case studies of power dynamics, international relations), information science (bibliometrics, user studies), statistics, and media studies (digital environments, media ecologies). Research should strive to be empirical, comparative, and span a complete range of disciplinary practices within the sciences, social sciences, arts, and humanities. Suggested research topics and questions include:

—  Determine primary indicators of effective tenure and promotion review practices across institutions and higher education sectors internationally. Institutional and governmental pressure on scholars in developed and emerging economies to publish research in the most prestigious publications has resulted in an explosion in the volume of publications worldwide. This not only strains the efficient and effective practice of peer review, but also puts at risk research productivity, legitimate academic publishing endeavors, library acquisition budgets, and resources for preserving digital-born and digitally migrated materials. The Academy needs empirical studies of the entire global system of academic reputation and status seeking in the face of these challenges. For example, how do practices vary across higher education sectors and countries? How do research assessment exercises affect the general quality and number of research publications, as well as the teaching missions of non-research-intensive institutions? And what are the actual costs (including social and opportunity costs) to teaching-intensive institutions of diverting academic labor from teaching to increasing research output as measured primarily by publications? In order to identify successful models, which types of institutions engage in "best practices" and which rely too heavily on secondary indicators, and why? Can an agenda to encourage adoption of the good practices we note above in our recommendations be implemented, and if so, how? For example, will attempts at reform, such as limiting how many papers are allowed in tenure and promotion portfolios, encouraging senior faculty to eschew formal outlets for their research dissemination, or quid pro quo exchanges of institutional peer reviewing for promotion decisions, be effective?

—  Discover a way to make publication costs more transparent so that they can be appropriately allocated.

—  Examine the effectiveness and economics of various policies that require open access to refereed publications as well as the sharing of data at pre-publication stages of research.

—  Determine the reasons for failed and successful experiments in alternative publishing models. For instance, why are there no models for successful overlay journals?[17] What kinds of services could such journals provide? Who is in the best position to develop new publishing models? Are gold open access models ultimately cost-effective for the Academy?

—  Explore the success of specific alternatives, such as preprint repositories. Beyond the hypothesis noted by Harley et al. (2010) that these systems tend to favor high-paradigm fields with low commercial value, are there other reasons that more fields have not embraced a preprint model?

—  Assess whether bibliometrics or other mechanisms can evolve to filter scholarship effectively, reliably, and in a way that cannot be easily gamed or abused.

—  Track and assess whether transparent, open, and/or commentary-based peer-review experiments relieve or add to the burden of reviewing and filtering relevant literature.

—  Investigate ways to finance adequate publication models to underwrite the important work of scholarly societies.

Diane Harley and Sophia Krzys Acord
Center for Studies in Higher Education
University of California, Berkeley

16 March 2011

11   The full report can be downloaded at Back

12   For example, there are clear differences among disciplines, and many professional schools, such as journalism, architecture, law, and environmental design, create their own specialized criteria for judging scholarly output. Back

13   On the different forms of peer review in PNAS, and their consequences, see Rand and Pfeiffer (2009). Back

14   See, for instance, the APA/AIA Task Force on Electronic Publications (2007) and the EVIA Digital Archive Project for ethnographic field video in ethnomusicology.  Back

15   Although conference presentations, working papers, (some) edited volumes, blogs, and other non-peer-reviewed work can help scholars to establish precedence for their work and may influence the evaluations written by external reviewers, they do not substitute for peer-reviewed publications in the institutional review process. (Exceptions to this include fields like computer science, where conference papers constitute penultimate publications.) This may be because, as Borgman (2007) observes, it is easier for institutions to measure a scholar's outputs (in the form of publications), than to measure inputs (eg, in the form of research time and other activities). Back

16   A white paper representing an earlier version of these proposals was submitted to the NSF's SBE 2020 call for papers examining Future Research in the Social, Behavioral & Economic Sciences. Cf: Harley and Acord (2011), Understanding the Drivers and Dangers of Academic Status Seeking: Studying the Impacts of Embedded Disciplinary Cultures in a Networked Academy, available at: Back

17   Overlay journals are minimalist journals that provide peer review but not a publishing platform (Suber 2001). Still fairly speculative at present, an overlay journal would mine self-archived "raw" author manuscripts from repositories and carry out certain publishing functions like peer-review management, editing, and perhaps branding (Swan 2010). The actual published content would continue to reside in the repository, perhaps with an updated postprint incorporating any revisions and updated metadata reflecting the journal/society brand that carried out the peer review. The overlay journal would then link to the repository content via a traditional table of contents. Back

previous page contents next page

© Parliamentary copyright 2011
Prepared 28 July 2011