Peer review

Written evidence submitted by the
UK Association for Heterodox Economics (PR 52)

1. I am replying on behalf of the UK Association for Heterodox Economics (AHE) to your request for submissions on the operation and effectiveness of the peer review process. We address our response specifically to the way this operates in economics. This profession is in the public eye following errors of judgement and theory brought to light by the current recession, which lends particular importance to this aspect of the committee’s work.

2. We urge the committee to study our proposals and the evidence that supports them. The AHE speaks for a wide range of economists, particularly those critical of, or dissenting from, the views of the economics mainstream. Our membership includes 500 practising economists, including editors of a range of peer-reviewed journals, and our regular annual conference attracts over 100 scholarly papers. The AHE is a forum for international collaboration and exchange, so that, for example, our 2010 conference in Bordeaux was hosted by our sister organisation, the French Association for Pluralist Economics (FAPE/AFEP).

3. A wealth of experience, and a record of high-quality research, informs our response. We provided input to the international benchmark review of the Research Assessment Exercise (RAE), and to the UK Quality Assurance Authority (QAA) consultation on the revision of the Subject Benchmark Statement for Economics (SBSE). The AHE designated Andy Denis to act as guest editor for a special edition of the International Review of Economics Education dedicated to pluralism in the teaching of economics in Higher Education Institutions, at the invitation of the Economics Network of the UK Higher Education Academy. AHE members conduct and publish systematic scholarly research on the impact of the economics profession’s evaluation, publication, research and teaching procedures. This submission draws on that research.

4. The recent economic crisis has confirmed widespread public and professional concern that the process that produces economic ideas and policies has proved defective. We believe these concerns are legitimate. The peer review system as it operates in economics is, we argue, a significant contributory factor to failures which came to light in the recent recession.

5. Peer review constitutes one aspect of a wider process that has produced what Colander et al (2009) term a ‘Systemic Failure’ of the economics profession. The AHE argues (Freeman 2010) that this arises from the ‘Regulatory Capture’ of the economics profession, a view we share with Lord Turner, chair of the Financial Services Authority. This phenomenon, well-known in economics, occurs when an institution plays the role of regulating, that is to say, forming judgements over what is fit for public purpose, and when a particular, private interest, affected by the outcome of these judgements, arranges systematically to influence these judgements so as to favour itself as against other private interests and thereby, ultimately, against the public interest.

6. Regulatory capture is systemic: no ‘lever’ may be identified that can be pulled to create or dismantle it. The peer review system is however a fulcrum: it determines what the profession ‘judges to be’ good economics. The metric of last resort when deciding if research funding should be allocated, appointments made, or policy recommendations adopted, is publication in approved journals.

7. Economics plays a specific role in society; economic judgements are used by decision-makers to determine which policies are valid and which are not, whether in government or in other institutions with a public role such as banks, trade bodies, regulators of public services, or indeed, credit rating agencies. The economics profession thus plays a de facto regulatory role, whilst enjoying almost total freedom from external scrutiny or even self-regulation. Yet its judgements are frequently unsound, as is now recognised in the wake of the 2008 financial crash and ensuing recession.

8. The process that renders economics vulnerable to capture was termed ‘groupthink’ by the International Monetary Fund (IMF)’s Independent Evaluation Office (IEO 2011), which identified groupthink as a principal cause of serious errors of judgement. It stated that: ‘The IMF’s ability to correctly identify the mounting risks was hindered by a high degree of groupthink, intellectual capture, a general mindset that a major financial crisis in large advanced economies was unlikely, and incomplete analytical approaches.’

9. A prominent outcome of groupthink is the so-called ‘Efficient Markets Hypothesis’ (EMH) – until recently considered the hallmark of good judgement in economics, but which led to the false conclusion that there could be no systemic failure of the financial markets. This doctrine is now rightly discredited. The problem to be addressed is, however, why the economics profession discarded this unproven hypothesis after, not before, the crisis, and why research that dissented from it – now widely accepted as valid – was equally widely dismissed within the profession until a year into the crisis. This oversight led to much harm. Public policy should consider what guarantees the public may be given against further oversights of this scope and nature.

10. The peer review system in economics contributes to groupthink in a number of ways, which re-enforce each other. The effective origin is the imposition of a hierarchy of acceptability as an outcome of the peer review system. Economics journals are informally or formally stratified into so-called ‘Diamond List’ or ‘Elite’ journals defined as those which secure the greatest degree of peer approbation, at the highest levels of the profession.

11. Ranking by acceptability narrows the field of ideas deemed legitimate to a narrow band of received ideas that constitute themselves as a set of doctrines, stifling critical judgment and denying the public access to alternative ideas – notably, ideas that could have foreseen, prepared for, ameliorated or even helped avert the recent crisis.

12. The peer review system interacts perniciously with the allocation of research funds (Lee 2007, Lee and Harvey 1998, Gillies 2011), intensifying the tendency to groupthink. Our research shows that UK Higher Education Institutions responded to the RAE, and its successor, the Research Evaluation Framework (REF), by focussing recruitment and research priorities on scholars publishing in journals that conform to the mainstream and form part of the ‘Diamond List’. This stifles research which might draw conclusions counter to those published in the mainstream journals.

13. Higher Education Institutions react logically and instrumentally to RAE and REF processes in an environment where publication is the main metric of research excellence. They determine both research priorities and recruitment or promotion decisions on the basis of current or expected publication record in mainstream journals. This practice is not always officially acknowledged and on occasions denied. However, our studies show it is very widely practiced. In consequence, University departments of economics have been stripped of critical, dissenting or even original scholars.

14. This creates a vicious circle. The scholars thus selected have been chosen for conformity with current received doctrine, and make up the ‘peer group’ which judges what is to be published, with the result that academic economics is being steadily reduced to a body of people that conform with a narrow range of dominant – and, as it has turned out, frequently wrong – theory.

15. Applied and theoretical research deploying other ways of interpreting economic behaviour is increasingly confined to departments outside of official economics, for example in Business Schools and Departments of International Development; even there, it is under pressure from orthodoxy and, of course, scope for the publication of dissenting or critical work in economics journals is just as restricted.

16. Despite a highly influential public role, economics is uniquely free from external scrutiny and uniquely oblivious of the need for a public standard, code of conduct, or indeed, ethical principles. Economics alone amongst the professions, particularly those with a major influence on public policy affecting large numbers of people, recognises no process of verification beyond peer reviewed publication, other than paid appointment or simple receipt of fees. Its output is not scrutinised for functionality, by applying the simple test of whether they work, as is the output of Engineers or Architects. Nor does any professional body enforce, monitor or even attempt to define good practice, as in Medicine, Accountancy, or the Legal Profession. Finally, there is no code of conduct: the profession has evolved no clearly-stated criteria as to what constitutes good or bad economics, and does not see the need for them.

17. A linked cause for concern is the disproportionate impact of private or external funding from bodies with a significant interest in the outcome of economic research. It is particularly striking that, unlike medical publications, economics editors do not request information on sources of funding, do not publish them, and take no precautions to offset the risk that accompany conflicts of interest.

18. The recent embarrassing resignation of the director of the London School of Economics draws attention to the scale of the potential risk. Substantial concerns have been voiced about the Icelandic Chamber of Commerce’s procurement, for a considerable fee, of a clean bill of health for Iceland’s financial institutions from the then Secretary of the Royal Economic Society, months before these institutions collapsed. We believe it an urgent matter to incorporate procedures into the peer review system to protect against, and to publicly declare, conflict of interest – in line with publication standards in other disciplines.

19. The AHE believes that a review of all practices in the economics profession is an urgent matter. There is no more important place to start than the process which passes judgement, and in the absence of all external or independent scrutiny, thereby defines, what constitutes ‘good economics’ – the peer review process for economics journals. The restriction of economic doctrine to a single paradigm, or a best a small range of paradigms, and to a narrow range of ideas and theories, is in our view the principal problem to address in a much-needed reform of this process.

20. The conclusion that the AHE draws from its research, and the experience of its members, is that the solution is an approach to economic theory which we term ‘pluralism’. This respected approach is normal in the natural sciences but foreign to economics, despite a declaration in support of it published in the American Economic Review in 1992, signed by 44 prominent economists including four Rijksbank Nobel Prize winners. Pluralism informs us that the duty of the economist is not simply to study or present a single answer, certified correct, to those that she or he advises, but to make available the conclusions which flow from the range of legitimate available approaches germane to the problem at hand, to make clear the assumptions upon which each depends, and to devolve to the decision-maker the ultimate right and duty to choose between these conclusions.

21. We have also concluded that pluralism should be assertive; it should be the subject of specific, explicit guidelines, supported by mechanisms of incentive, scrutiny, and where appropriate, sanction. In the absence of such prescription, the natural tendency of economics, arising from its insertion into modern society, is to revert to the practices which give rise to groupthink.

22. In line with this approach, we recommend an assertive pluralist reform of the peer review system in economics with the following objectives:

22.1. to protect against the narrowing of received economic truth to a single trend or small range of trends of thought, to the exclusion of legitimate critical and dissenting views. The public needs specific protection against this distinctive feature of economics not seen (Freeman 2009) in other academic disciplines.

22.2. to provide due place for critical and dissenting judgements, without discriminating so as to define them as de facto inferior by virtue of the sole fact that the theories involved, and their conclusions, are either unpopular with their opponents, or are not widely subscribed to. Specific, prescriptive measures are required to discourage the narrow selectivity to which economics is peculiarly prone.

22.3. to guard against the systematic confusion of quality with conformity. Economics, unusually for science, exhibits a marked tendency to reject scholarship that does not conform to current orthodoxy, no matter how well argued, theoretically coherent, and well-supported by evidence. To guard against the groupthink this produces, the economics peer review process should be required to exhibit transparency, objectivity and to pay due attention to external criteria of judgement that do not arise from within the profession.

22.4. to take prudent precautions against the self-referential use of rankings to determine funding allocations for research whose output is measured by publication in peer-reviewed journals on the basis of self-declared hierarchies of merit unsupported by externally-defined criteria and subject to no external scrutiny.

22.5. to take prudent precautions against distortion arising from conflicts of interest.

23. To this end we propose a number of changes to the peer review system as it operates in economics:

23.1. A standard of acceptability in the peer review procedures of journals is required, covering journals in which publication is taken as a criterion for allocating UK research funds. The simplest, though not the only way to achieve this, would be to set up an independent body to draw up standards and oversee their implementation in consultation with appropriate public and private bodies representing both the profession, and those affected by the quality of its output. The aim would not be to override editorial academic freedom but to provide an independent standard of good practice in economics publishing, above all to inform the allocation of funds.

23.2. Attention should be paid to the following:

23.2.1. There should be published guidelines on acceptable peer review procedures;

23.2.2. Account should be taken of the standard wherever and whenever reference is made to publication in economics journals as a metric of research output and quality, notably in allocating research funding but also in recruitment and promotion.

23.2.3. The standard should include a code of publication ethics in economics, drawing on the ethical standards set by professions such as the Medical and Legal professions, and drawing also on the guidelines offered by the Press Commission for published magazines, particularly concerning accuracy, fair representation, and right of reply.

23.2.4. Clear, transparent, and public criteria are required for the selection of articles for publication. The aim of such criteria is to provide for objectivity, in the specific sense that an independent observer can verify whether the criteria have been applied.

23.2.5. A common core of the criteria should repose on a code of publication ethics common to the profession, for example as specified in 23.2.3 above.

23.2.6. Where a contested range of theory or opinion is relevant to the subject matter of either an individual article, or a field of research or publication, journal readers need access to the range of pertinent views involved.

23.2.7. Published articles should be judged, among other criteria, on whether they take due account of, and recognise the existence of, alternate, dissenting, or critical views.

23.3. Journals should therefore recognise an editorial duty to take due account of legitimate views, meeting normal general academic standards, that dissent from those that they publish, or for which they exhibit a preference unjustified by explicit editorial criteria:

23.3.1. The standard should require journals to protect against misrepresentation. This includes the right of reply, and a duty to publish corrections or retractions in the event that a published article is subsequently proven to contain significant misrepresentation. This is particularly important, since the preservation of an orthodox doctrine and the non-recognition of dissent is, in the main, rendered possible by the systematic misrepresentation or omission of opposing views.

23.3.2. In cases where paradigmatic conflicts are known to exist, the standard should require that review panels are balanced and include reviewers who work within the paradigm adopted by the author. There should be no presumption that a reviewer hostile to an author’s paradigm can make sound judgements on quality of the author’s submission.

23.3.3. The standard should require that journals seek out and publish views that significantly dissent from or criticise views that are heavily represented in their selection of articles, where a body of dissent of high academic quality exists.

23.3.4. The standard should require a right of appeal against improper application of selection criteria or non-conformance with standards.

Alan Freeman

For and on behalf of the UK Association for Heterodox Economics

10 March 2011

References

Colander,D., Föllmer H., Haas, A.,Goldberg, M., Juselius, K.,Kirman, A., Lux,T. and Sloth,B. (2009).‘The Financial Crisis and the Systemic Failure of Academic Economics’,Kiel working paper 1489,Kiel Institute for the World Economy.

Denis, A. (2009) (ed). International Review of Economics Education, ‘Pluralism in Economics Education’, Volume 8, number 2. London: Economics Network.

Freeman, A. (2009) ‘The Economists of Tomorrow: the Case for a Pluralist Subject Benchmark Statement for Economics’. in Denis, A. (2009) (ed). International Review of Economics Education, ‘Pluralism in Economics Education’, Volume 8, number 2. London: Economics Network.

Freeman, A, (2010). "The Economists of Tomorrow,"  American Journal of Economics and Sociology , Blackwell Publishing, vol. 69(5), pages 1591-1613, November

Gillies, D (2010) Economics and Research Assessment Systems. Working Paper, University College London.

Hodgson, G., U. Mäki and D. McCLoskey (originators) (1992) ‘A Plea for a Pluralistic and Rigorous Economics’. The American Economic Review, 82(2).

IEO (2011). ‘IMF Performance in the Run-Up to the Financial and Economic Crisis: IMF Surveillance in 2004–07’. January 10, 2011.

Lee, F. (2007) ‘The Research Assessment Exercise, the state and the dominance of mainstream economics in British universities’, Cambridge Journal of Economics, vol. 31(2), pp. 309–25.

Lee, F. and Harley, S. (1998) ‘Peer Review,The Research Assessment Exercise and the Demise of Non-Mainstream Economics’. Capital and Class, (66), pp. 23–53.

Declaration of interest

The Association for Heterodox Economics is self-funded. It receives occasional donations from other heterodox or pluralist organisations including grants of the order of GBP2000 in 2008, 2009 and 2010 from CPEST, the Cambridge Political Economy Society. Its activities are conducted by volunteer labour and it has no paid officials or employees.