Peer review

Written evidence submitted by Professor Michael J Kelly FRS FREng (PR 09)

1. I am Prince Philip Professor of Technology at Cambridge University. I have been involved in research in solid state physics and semiconductor electronics since 1970. I have published nearly 250 peer-reviewed articles and held over £20M of grants awarded on the basis of peer review. I have worked both in private industry and in academia. I have acted as a peer reviewer for publications and research resource allocations for funding agencies in several countries. I have been both a journal editor and a commissioner of research through the peer review process.

2. I agree with the general notion that the dispassionate peer review ideal is the least bad method of allocating resources and of overseeing the advance of the science canon as through published papers. That said, there are severe problems facing the operation of peer review in both sectors on which I will dwell in my submission.

Peer Review for Publication

3. The sheer volume of research these days means that editors of journals know that they cannot publish all the papers they received even if the science is correct. The editors have to form some view of what their journal is about and then pick only the best papers within that view. This by and large means that certain types of controversial paper simply do not get an airing, which only 20 years ago would have been published and provided the source of a vigorous debate until a new prevailing orthodoxy was arrived at. The sharpening up of the arguments is not happening today in the scientific literature. I shall illustrate this.

4. For more than a decade, I have been interested in the limits of what can be manufactured in the area of nanotechnology. Most nanoscience results in the literature are the reports of one-off experiments. Very few results have made the transition from the lab into a product that is manufactured i.e. is fabricated reproducibly, reliably, to a pre-specified performance within a narrow tolerance, and to a very high yield. Until recently, I have published on the manufacturability challenges of a few specific semiconductor devices. Two years ago I derived a simple theorem that shows that a whole class of nanometre-scale artefacts which might be made individually on a one-off basis with conventional semiconductor nanostructure fabrication tools are intrinsically unmanufacturable using those same tools. The statistical fluctuations associated with dealing with sets of small numbers (of individual atoms in this case) is applied to show that arrays of artefacts will have intolerable feature-to-feature variability, and that elements of arrays will not be able to be addressed or read out from with any acceptable fidelity. I have prepared a short paper describing the theorem (that 3nm design rule technology is intrinsically unmanufacturable using conventional top-down methods of fabrication) and a number of implications that follow from it. In addition I have identified and I quote from 102 papers which include hopeful statements about future manufacturability, which I think are likely not to be met using conventional techniques. Several of my international peers have agreed with the content of my paper and I have given seminars on the topic.

5. The resulting paper in various forms has so far been turned down ten times by seven top international journals devoted to solid state physics, nanotechnology or microelectronics based both in the US and in the UK. These are all journals that publish the positive results including many of the 102 citations described above. I can table correspondence from the editors (who are not peers), but I do not include them here, as such material is considered confidential: see http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5635418&tag=1 The paper has not been rejected as wrong, trivial or derivative, but for a number of reasons which all boil down to ‘we don’t want to be associated with such a profoundly negative paper, and the type of debate it might spur’. Many editors have thought I was offering an opinion, not a theorem. One even suggested I should go to the press! I get no sympathy when I have pointed out in correspondence that it is reprehensible, even corrupt I would submit here, to support research that is claimed to have future commercial potential when the low-cost high-volume manufacture can be excluded as a prospect.

6. I am still trying to get my paper published in the peer review literature, before going to the press on the issue of the integrity of peer review. Until it is published, the paper will always suffer from the unanswerable put-down that it has not been peer-reviewed, a common critique used of much work in other fields such as critical climate change science and impacts. I am concerned that my arguments should first be debated in the scientific literature. There is the chance that a counter-example to my theorem might emerge, and I shall be proved wrong. That would be good for the science. There is a greater chance that the emphasis in nanotechnology might move towards trying to discover unconventional fabrication routes that might circumvent the preconditions of the theorem. That too would be good for nanotechnology. What is not good is that the discussion is not had and that large quantities of public funds continue to be squandered in futile exercises. New orthodoxies get established prematurely without sufficient critical analyses in the literature in the early days. Some of the present problems in the climate science and impacts communities are down to the lack of a healthy sceptical debate within the peer reviewed scientific literature. The unquestioned hyperbole associated with hot areas of science, including nanotechnology, is a further manifestation of the lack of hard-nosed realism.

7. Twenty-five years ago I was the adjudicator on a paper published by a US journal where the new work of one industrial laboratory was being held up by critical and carping reviews from what I think was another industrial laboratory. Other reviewers were more positive. I received a long correspondence to review. The data was new, and the critical reviewer kept pulling new holes in the arguments about the interpretation of that data. I argued that the data should see the light of day, and the explanation should be argued about in the literature until a prevailing view emerged. This is what happened. I also suggested that the carping referee should be censured, and that I would not accept the role of a future referee until this happened. Since my views prevailed and I was asked to do further reviewing I can only hope that the censure took place. The cover story of the 17 February 2011 issue of The Spectator has a revealing discussion of peer review as delaying tactics in the context of climate change. New orthodoxies are not getting the critical review that was much more common 2-3 decades ago.

Peer Review for Research Fund Allocation

8. Research Councils and other public bodies, including the Royal Society, use peer review to rank the quality of proposals for research that they are considering for funding. Nearly every scientist who has had a rejection, and that is pretty well everyone who has ever applied, will have complaints about the peer review system. About 15 years ago, I was the Royal Society Representative on a small panel that reviewed the workings of the peer review process within the EPSRC. We gave the then process a clean bill of health as far as the officials were concerned. The weakness at that time was the small response rate of those who were approached to act as reviewers. It is legitimate to reply from time to time that one is very busy on a timeframe requested by the Council. It is not legitimate to sit on requests to review and not answer at all, including to several reminders. This naturally leads to a narrower set of reviews, and a smaller number of people having influence. A secondary problem was the casual reviewer who gave unhelpful one-word answers to questions as diverse as value-for-money, proposer track record etc. With ever more bids now, the demands for reviewing are getting greater and are often being neglected or given less attention.

9. More recently a number of problems have arisen that are not of the peer review system’s own making. The sheer competition for grants now means that even one mildly critical review (even if constructively critical) out of four or more is enough to kill a grant. This leads to derivative and uncontroversial research being funded in favour of the bold but risky. In the distant past, one knew of pairs of colleagues who had habitually fallen out on the merits of each other’s particular line of research, and one avoided having them review each other’s proposals, or at least discounted them heavily. Now the higher turnover of the number of researchers, and indeed of the number of officials, is such that one cannot be sure that animus is not masquerading behind a critical review.

10. A further problem in peer review is the way that a set of (say) 40 peer-reviewed proposals is then ranked for funding. An ad-hoc panel of peers is called together to comment on and adjudicate among the reviewers’ reports, but members are not allowed to use their own expertise. I have at least once been ruled out of order in such a panel meeting for pointing out, to my knowledge and not picked up by the reviewers, that two other groups were already doing much the same research, and why have three parallel efforts while other areas of research were missing out: we were to only support the best research on the day even if it is the same as going on elsewhere. On another occasion, having agreed to serve on an IT prioritisation panel, I withdrew on receipt of the papers for the meeting. I was being asked to speak on arcane matters of machine language semantics, where I did not understand the proposal, the criticism of the reviewers, or the proposers’ response to the reviewers’ criticisms. I was not in a position to spend three weeks learning all this to be a good panel member. What was worse, there were papers on the physics of semiconductor materials about which I did know a lot being given to panel members who were specialist in machine learning! We were being asked to make fools of ourselves in front of each other, in the name of neutrality and avoidance of bias!

11. As I get older, I am sometimes concerned about the comments of younger peers which exhibit a lack of experience or wisdom. I have seen reviewers’ responses that show ignorance of or hostility to (say) industrial trends, higher-level policy directions, urging of the Councils themselves, etc. It is very hard for Research Council Officials to get such views overridden by panel members when prioritising proposals.

12. Finally there is another real problem, and that is in developing new research fields in mid-career. As the result of three years as Chief Scientific Adviser to the Department for Communities and Local Government I have come to learn about how local authorities go about meeting the demands of central government in terms of matters such as energy efficiency in buildings, sustainability of resource consumption etc. I have seen the clear need for a model trajectory for urban local authorities to get to 2050 in terms the low carbon economy, and have built up a strong local team to construct such a trajectory. My attempts to secure funding to do this have been dismissed by my new peers who think that the proposal is a very good idea, but do not think I have what it takes to deliver given my lack of track record in the new subject. This is a certain catch 22.

Peer Review Problems: Cause or Consequence?

13. The real pressures on peer review are more a reflection of pressures within the multi-billion global scientific enterprise, at a time when competition for precious resources is more intense than ever. The major international journals want to build and preserve their reputation for announcing hot new results, and they don’t want to be harbingers of caution, let alone hard-nosed realism. Funding agencies worldwide see ever increasing competition for funds, often decoupled in the name of scientific excellence from the whole winnowing eco-system that connects a few of the new discoveries to products and services that actually contribute to national health, prosperity and environmental integrity. Seeking an answer to these much bigger questions through tinkering with peer review is a sure recipe for failure. Only if the bigger questions are tackled and solved head-on might peer review be expected to work again as it did when the scientific community was smaller and less intense 20-30 years ago.

14. One real issue to enquire about is the real end-to-end effectiveness of the modern scientific enterprise. The troubles in the drug discovery industry have been well heralded. Many electronics companies downsized their internal research laboratories 20 years ago, effectively outsourcing all discovery research to the universities world-wide. Mainstream inorganic semiconductor electronics has been sufficiently well entrenched to see off most competition from newcomers such as molecular electronics for over 20 years. Have we reached to point of diminishing returns in some of our recent areas of endeavour? Should there be a much greater emphasis on development for delivery of what we have over the dominance of discovery mode research, at least for a few decades? I would argue that the US Mission agencies mean that the overall US funding is better balanced between blue-sky and demand-led research than is the UK funding. Should we at least not openly debate these issues seriously given the scale of resources in play?

15. If one is to take seriously the raft of global neo-Malthusian issues from over-population through profligacy of resource consumption to environmental degradation, one might ask whether the current structure of the scientific enterprise is fit for purpose in the coming decades. There is too much scientific advice and too little translation or solid engineering advice available to governments and corporations. Armed with an indicative global roadmap to 2050 one might devise a quite different role and structure for scientific enquiry, technological development and infrastructural renewal over the coming 40 years. I am not convinced that 40 more years like the last 40 of international scientific and technological endeavour will meet the ambitious technical targets and societal expectations for 2050. We should at least debate this rather than sleepwalk on.

Conclusion

16. Problems in peer review are the symptom, not the cause, of deeper problems in the modern scientific enterprise. It is these deeper problems that should be debated and solved, so that peer view or a timely alternative can do its job again. Since popular discourse sets 2050 as the date by which the world must be transformed, should science and technology be made more directly the handmaiden of such a transformation?

Professor Michael J Kelly FRS FREng

25 February 2011