Peer review

Written evidence submitted by Academy of Social Sciences (PR 26)

1. Basis of Evidence: This evidence is submitted on behalf of the Academy of Social Sciences. It was prepared by Professor Robert Dingwall AcSS with the assistance of Madeleine Barrows, Communications Officer for the Academy. The Academy represents 40 learned societies in the social sciences who, between them, have over 85,000 social science members; the Academy also has over 700 individual Academicians elected in recognition of their eminence within the fields of social science and its application to public policy. The responses reflect a canvass among both learned societies and Academicians. Nine submissions were received from seven learned societies and two individuals, one a journal editor and one a government social scientist. These refer both to experience as authors and reviewers and to editorial perspectives from 14 major journals. Some member societies indicated that they were likely to make independent submissions. The response also draws on Professor Dingwall’s own previous experience as editor-in-chief (2000-06) of the highest impact UK sociology journal and his current work as editor of a small socio-legal journal and of a major US-owned journal.

2. Background: Although this consultation is directed mainly towards peer review in relation to the content of academic journals, it is widely used in other academic contexts and any recommendations should also take account of these.

a. Most major publishers peer review books and monographs, which are important vehicles for dissemination in the social sciences.

b. Most research funding by research councils, government departments and charities relies on the peer review of grant applications.

c. Most internal promotions within universities require the peer review of candidates, particularly for advancement to personal chairs.

3. What is peer review? In the social sciences, this is almost invariably a double-blind process. The author does not know who the reviewer is and vice versa. Although it is sometimes possible for either party to guess the other’s identity, journals adopt a variety of practical strategies to try to minimize the possibility of this happening and many have formal statements requiring reviewers to disqualify themselves if they recognize an author and feel that this constitutes a conflict of interest. Formally, then, a paper is reviewed solely on the basis of its content rather than on the reputation of its author or the institution where they are working. This contrasts with the approach adopted by a number of medical and scientific fields where reviewing is not blind and where the possibility of, conscious or unconscious, bias is introduced.

4. Strengths of peer review: Respondents were unanimous in their support for peer review as the best available system for assuring the quality of work published in academic journals.

a. Work is evaluated by relevant experts in the field, who are familiar with existing literature and standards of rigour.

b. Work is evaluated impartially and without reference to the identity of its author or the standing of their institution.

c. Provides a guarantee for the public of the validity and scientific warrant of knowledge produced with public funds and placed in the public domain.

d. Provides diversity and independence of judgement because of the diversity of journals, editors and selection networks for referees. Effectively, it invented, and regulated, crowd-sourcing before the term came into use.

5. Weaknesses of peer review: Respondents generally agreed that the process of peer review could inhibit innovation and interdisciplinary work and that it was under some stress as a result of structural changes in higher education.

a. Peer review can be a barrier to the publication of innovative work, or work that departs from a current consensus within a discipline. This varies somewhat according to the ways in which journals are edited. Some are rather mechanical in terms of relying on referee scorecards and others allow editors more discretion to publish papers that reviewers may not fully endorse but which are considered likely to provoke an important debate. The way in which current market arrangements for journals sustain diversity is an important corrective, although, in the UK, it is countered by the tendency of research assessment processes to favour publication in discipline-oriented journals that constitute the established ‘mainstream’ of the field.

b. The intensification of academic work, which is a global phenomenon, has made it increasingly difficult for journals to persuade established scholars to undertake the unrewarded community service of reviewing. Junior scholars find themselves discouraged because reviewing is not recognized by tenure committees or performance review processes. This introduces delays, while willing referees are located, and can mean that papers are not actually reviewed by people who are in absolutely the right place to exercise expert judgement. There is a kind of Gresham’s Law, where a minority of willing scholars find themselves increasingly burdened by requests and gradually withdraw their goodwill in order to protect their time for activities that are rewarded or required by their management. It is increasingly common to have to approach 5-6 people to find 2 who are willing to referee. Their reports can be slow to arrive and more superficial than would have been the case even ten years ago, which means that they are of less value in improving papers or in giving constructive feedback to rejected authors, something that is particularly important where the author is relatively junior and the editor may wish to support their professional development.

6. Measures could be taken to strengthen peer review: Respondents generally felt that improvements would be difficult without more explicit recognition of the importance of this work by the higher education system globally. At present there was a huge pressure on scholars to publish in peer-reviewed journals in support of personal and institutional goals but this was not matched by an equal commitment to encouraging scholars to act as reviewers: everyone wanted quick decisions from editors with high-quality and personalized feedback, but few people were prepared to contribute their time and effort to providing this! Some respondents canvassed the idea of paying fees for reviews, but it was felt that these would never be large enough to become a real incentive, although it might be noted that some major commercial publishers are now offering free access to their online holdings for a period in recognition of a reviewer’s contribution. Within the UK, there was a specific issue about the extent to which the discretionary time traditionally enjoyed by academics and capable of being put to this kind of community service was being squeezed out by the expansion of regulatory interventions and performance management. Because this work was not recognized and measured, it was particularly vulnerable to these continuing changes in the conditions of academic production.

7. The value and use of peer-reviewed science for advancing and testing scientific knowledge: Respondents saw that this as an essential means of quality control that guaranteed a minimum standard of validity in any research that was published. Without this process, science would be appreciably impeded by the need for every researcher to individually validate every piece of information that they needed to use in their own research. The main concern was the failure of other users to appreciate the nature of this ‘kite-mark’ and to make appropriate discriminations in practice and policy between knowledge that had and had not been tested in this process. There was a particular anxiety that the ‘impact agenda’ could favour work that produced politically acceptable, and hence high-impact, findings over work that was more scientifically-oriented and took a more fundamental and long-term view.

8. Value of peer-reviewed science in informing public debate: Respondents saw a problem in the widespread failure to recognize the quality-assured nature of peer-reviewed publication, particularly when it had passed through the double-blind process that dominates in the social sciences. There are important limitations in a publication process that emphasises the advance of science rather than communication with a wider public audience, but this is not primarily an issue for peer-review so much as for the development of intermediary or translational dissemination formats. Open-access publishing or institutional repositories make a very limited contribution, since they are primarily means of making scientific knowledge more widely available to those with the time, inclination and skills to read specialized communications rather than for promoting public engagement. The specialization of language and format in scientific publication reflects the nature of its objective – to promote precise and efficient communication within that community. Peer review can play a part in this but it cannot, of itself, define the use of knowledge in public debate other than through its role in certifying the validity of some forms of inquiry and publication relative to others.

9. Variation between disciplines and countries: There is a considerable degree of variation globally, although, within the Anglophone world, double-blind peer review predominates within the social sciences. We have already noted that this is not the norm in other areas: the British Medical Journal, for example, does not blind reviewers and authors from each other. Outside the Anglophone world, peer review is less dominant, although practices are tending to converge on this norm. A major exception would be the North American model of law reviews, which are significant journals in their fields, edited by annually rotating student committees and making no use of peer review at all. Even here, the more recently established socio-legal journals have tended to abandon this model and to conform to mainstream social science practice.

10. Procedures by which reviewers are identified: Respondents identified three main sources.

a. The personal networks of editors and editorial boards, normally senior figures with an extensive knowledge of the field and its significant scholars.

b. Databases maintained by the journal over periods of years. These vary in quality but are increasingly used to record reviewer performance as well as their interests and expertise. They can also have names added by people volunteering to referee: this is unusual for junior UK colleagues but not uncommon for US graduate students and early-career academics looking to develop international networks.

c. Google. Editors report an increasing use of both Google and Google Scholar searches, particularly as they are forced to spread the net wider to find willing reviewers. These often throw up talented individuals in less research-intensive environments, like small US colleges, with the time and skills to contribute thoughtful reviews.

11. Impact of IT: This has transformed the administration of the review process in ways that have made it sustainable in the face of the challenges outlined above. Online submission, reviewer databases, online contact with reviewers and review receipt have dramatically reduced the costs, the burden on editors and the speed with which the process can be conducted. They have also facilitated a more global search for potential reviewers, contributing to the greater integration of the social science community and the sharing of standards for evaluation and good reviewing practice. This shift has thrown up some problems, particularly for reviewers who are not technologically literate or who have limited institutional resources – the cost of printing papers for review is transferred from author to reviewer, for instance, which may be an issue for reviewers in developing countries or for the ‘active retired’ in developed countries. It would, though, be impossible to turn papers round within any reasonable speed if postal means were still being used to invite potential reviewers with a minimum turn-round between request and refusal measured in days or weeks rather than hours. Similarly, the online systems allow editors to make decisions from wherever they happen to have internet access rather than from a fixed editorial office. While it is possible that lowering the threshold of access for authors has led to an increase in premature or speculative submissions, this is outweighed by the other benefits. There was some interest in the greater use of plagiarism detection software – the current volume of publication was making it more difficult to trust to reviewers’ knowledge of the literature to detect this, as had historically been the case. However, this would need to be used with discretion because, in the social science, there were justifications for some degree of self-plagiarism and duplicate publication in order to reach different audiences or to analyse the same data from different angles. This had been a cause of difficulty in some interdisciplinary fields where different standards were applied from scientific or biomedical reviewers whose knowledge production worked in different ways.

12. Possible alternatives: Respondents found it hard to identify acceptable alternatives. While they were aware of the claims for ‘crowd-sourced’ reviewing (where items are simply placed into the public domain with a facility for readers to post reviews and comments as in a blog), they were sceptical of the potential benefits because of the work that would be required to differentiate serious critiques from frivolous, biased or ignorant commentary. It might also be noted that, if scholars are feeling under too much workload to review in a peer process, they are unlikely to devote much time to posting comments on each other’s papers rather than in posting their own. Where journal publishers have offered a facility for post-publication responses, the uptake has been fairly small. The result could be a deluge of information with very weak quality signals, which would be a significant source of delay in other people’s work because of the investment required to be personally assured of the validity of work being cited or developed. In the social sciences, it could weigh quite heavily against the cumulation of knowledge as opposed to one-off and rather disconnected studies.

Academy of Social Sciences

4 March 2011