House of Commons - The Reviews into the University of East Anglia's Climatic Research Unit's E-mails

The Reviews into the University of East Anglia's Climatic Research Unit's E-mails - Science and Technology Committee Contents

3 The programme of work

Timing

28. A criticism, made by the Global Warming Policy Foundation (GWPF), of the review by the Scientific Assessment Panel (SAP) was that it was "rushed and therefore extremely superficial. The body of the report is hardly five pages long. The Panel should have taken more time to arrive at more balanced and more trustworthy conclusions as there was no need to rush the inquiry."[44]

29. The press notice announcing the members of the Panel was issued on 22 March 2010 and indicated that the "panel will meet in Norwich in April and will have the opportunity to see original data and speak to those who did the work".[45] The report was presented to UEA three weeks later, on 12 April 2010. This contrasts with the ICCER, which was announced on 3 December 2009 and reported seven months later, on 7 July 2010.

30. In its report the SAP stated:

The Panel worked by examining representative publications by members of the Unit and subsequently by making two visits to the University and interviewing and questioning members of the Unit. Not all the panel were present on both occasions but two members were present on both occasions to maintain continuity. About fifteen person/days were spent at the University discussing the Unit's work.[46]

31. Lord Oxburgh elaborated in his evidence to us:

I don't think we could have done, usefully, any more than we did to answer the question that we were set. We worked very hard [...] I kept the panel together in Norwich while the report was written and while we went through a series of drafts, so you did not go through the endless iterative procedures [...] of circulating reports, getting a few comments here, getting them back, balancing them with someone else's opposing reports. We did it all around the table. So, actually, that probably saved six weeks over the normal procedures.[47]

32. Lord Oxburgh's explanation for the brisk timetable would be understandable if there was a pressing deadline. In this case, as Lord Oxburgh explained, the urgency came from UEA, the report "had to be done rapidly [...] they [UEA] really wanted something within a month".[48] Lord Oxburgh's statement could also be construed as indicating that the review was not operating wholly independently. Had the SAP been in less of a rush, they could have investigated the integrity of the science with more rigour, particular with regard to CRU scientists' ability to repeat their own experimental work, an issue we discuss in paragraph 60.

33. The disparity in length between the SAP and ICCER reports is striking. When compared to the ICCER, the SAP report—a mere five pages—reads like an executive summary, with none of the detail of the ICCER. From Lord Oxburgh's evidence to us, the report does not appear to explain the detailed work carried out by the SAP. That in itself does not invalidate the SAP report but it does foster an impression that it was not as thorough as the ICCER and was produced quickly in an attempt to be helpful to UEA.

Accompanying documents

34. The SAP's decision not to publish accompanying working documents was also of concern to us, as these would have supplemented the Panel's review and more accurately reflected the work carried out. The working document of one SAP member, Professor Michael Kelly, Prince Philip Professor of Technology at the University of Cambridge, was published online and has received much attention from climate change debate websites.[49] This document highlight the "observations and concerns" of Professor Kelly, including comments such as:

I take real exception to having simulation runs described as experiments (without at least the qualification of 'computer' experiments). It does a disservice to centuries of real experimentation and allows simulations output to be considered as real data. This last is a very serious matter, as it can lead to the idea that real 'real data' might be wrong simply because it disagrees with the models! That is turning centuries of science on its head.

The reading of the papers was made rather harder by the quality of the diagrams, and the description of the vertical axes on a number of graphs. When numbers on the vertical axis go from -2 to +2 without being explicitly labelled as percentage deviations, temperature excursions, or scaled correlation coefficients, there is potential for confusion.

I think it is easy to see how peer review within tight networks can allow new orthodoxies to appear and get established that would not happen if papers were written for and peer reviewed by a wider audience. I have seen it happen elsewhere. This finding may indeed be an important outcome of the present review.[50]

35. When we raised Professor Kelly's comments with Lord Oxburgh, he told us:

With Michael Kelly we discussed all of these things round the table with others, and I think you will see there a perfectly legitimate response of an engineer, a physical scientist, to looking at the work in an area of observational sciences. The language is very different. He, quite legitimately, says, "In our area we wouldn't call these things experiments."[51]

36. Following the oral evidence session, Lord Oxburgh added: "[Professor] Kelly's observations [...] could, taken out of context, have been very misleading. They could be taken as a serious criticism directed at CRU when they are in fact a comment on the language and practice in climate science as a whole. They had no bearing on our inquiry into the scientific integrity of CRU".[52]

37. In analysing the SAP report, we note that Professor Kelly's comments had been taken into consideration. For example, on the concerns raised about noisy data and selection bias, the report stated:

With very noisy data sets a great deal of judgement has to be used. Decisions have to be made on whether to omit pieces of data that appear to be aberrant. These are all matters of experience and judgement. The potential for misleading results arising from selection bias is very great in this area. It is regrettable that so few professional statisticians have been involved in this work because it is fundamentally statistical. Under such circumstances there must be an obligation on researchers to document the judgemental decisions they have made so that the work can in principle be replicated by others.[53]

38. It appears to us that Professor Kelly's comments were considered as part of a process, which included the views of other members of the SAP. The absence of the others' working documents has resulted in attention focussing on Professor Kelly's comments. We asked Lord Oxburgh whether it would have been better, for the sake of supporting what is a short report, to have published the SAP's working documents. He replied: "I actually don't think much would have been added. Again, we were working to time".[54] In our view the effect of pressure of time, which apparently was a factor in the decision by the SAP not to publish supporting documents, was regrettable. In contrast, the ICCER published both written evidence and notes of oral evidence online.[55]

39. In the interest of openness and transparency, supporting documents including the working documents of Professor Kelly and others on the Panel should have been made publicly available alongside the report and should now be made available. Unfortunately, Professor Kelly's comments—which have been published in isolation online—can now be read out of context. Had these been published alongside the comments of the other Panel members with an outline of roundtable discussions we consider that this would not have been a problem. The importance of Professor Kelly's work is that it clears CRU of deliberately falsifying their figures but, as the SAP report put it, "the potential for misleading results arising from selection bias is very great in this area".

Oral hearings

40. Our predecessor Committee recommended that the ICCER's oral hearings or interviews should be carried out in public wherever possible and that it should publish all the written evidence it received on its website.[56] The SAP was set up just as our predecessors published their Report, therefore no such recommendation was made directly to it. While we would have welcomed the openness of oral hearings by the SAP as well, it is unlikely that the rapid nature of the SAP inquiry would have accommodated this.

41. The ICCER written evidence received was indeed published online but the recommendation to conduct interviews publicly was rejected. Sir Muir explained:

What we wanted to do was to get the referenced scientific information down and findable rather than to rely on what people might say on the spur of the moment and have to go through the whole process of writing it up, checking it, modifying it and then going and finding the information. That is a perfectly valid technique for lots of other things, but we thought that this was so scientific, so objective, so much rooted in the references to what people had actually done as scientists, and whether the things that were complained of had influenced what they had done as scientists, that you really had to get after it by going to the record.[57]

42. Our preference would have been, like our predecessors, for evidence to have been taken in public. We accept, however, that Sir Muir's reasons for not doing this were reasonable. He chose to make detailed references of the scientific information relevant to what CRU scientists had actually done, in order to ensure that there was a robust written record. We do not consider, however, that this process would have been hampered by conducting the interviews in public.

Selection of publications

43. The Scientific Assessment Panel examined 11 CRU publications during the course of its investigations. The choice of publications has been widely discussed on climate change debate websites, with questions being raised about the extent to which the Royal Society were involved in the selection and whether or not Professor Jones at CRU was also involved.[58]

44. Lord Oxburgh told us:

We [the Scientific Assessment Panel] didn't choose the 11 publications [...] The publications were suggested to us. They came via the University, but via the University and the Royal Society, I believe [...] There is no suggestion that Professor Jones chose them.[59]

45. Professor Acton and Professor Davies explained that the starting point for the core publications suggested to Lord Oxburgh were the papers listed in the evidence given to our predecessor Committee.[60]They were chosen to address the criticisms of a number of areas of CRU research: the CRU global land temperature records; homogeneity adjustments; urbanisation effects; tree ring density records; and accusations of cherry-picking long records of tree growth.[61] Professor Davies explained the role of the Royal Society in the selection process:

I and Peter Liss, the Acting Director of the Climatic Research Unit, had a verbal discussion with Lord Rees [President of the Royal Society] some time at the end of February, beginning of March. The list was sent to the Royal Society for approval or for further comment on 4 March. The Royal Society responded on 12 March saying that it was content with the list. I am aware of the fact that there are allegations in the blogosphere that the Royal Society responded within 20 minutes. That is not the case. It had the list for a week.[62]

46. Professor Davies made it clear that Professor Jones was not involved in the selection of publications for the SAP.[63] We note that a number of other publications were referenced in CRU's submission to Sir Muir Russell, which the SAP also received.[64]

47. This is at odds with Andrew Montford's submission:

The list of papers for Oxburgh did not include any of the key multiproxy temperature reconstructions. In his evidence, Professor Davies said that he disputed this, but this claim can be shown to be false. CRU has produced three multiproxy temperature reconstructions—Jones et al. 1998, Mann and Jones 2003, and Osborn and Briffa 2006. None were on the list of papers for the Oxburgh panel and Professor Davies offered no evidence to support a claim that they were.[65]

48. Although it did not refer to the three papers identified by Mr Montford, we note that the SAP report did discuss proxy temperature reconstructions, specifically the dendroclimatology work at CRU. The SAP report stated:

CRU publications repeatedly emphasize the discrepancy between instrumental and tree-based proxy reconstructions of temperature during the late 20th century, but presentations of this work by the IPCC and others have sometimes neglected to highlight this issue.[66]

Furthermore, the ICCER discussed the controversy surrounding multiproxy temperature reconstructions following the 1998 publication in Nature by Mann, Bradley, and Hughes (MBH98), who are not scientists at CRU.[67] The MBH98 paper, which sought to reconstruct historic temperatures back to 1400 AD, was first challenged in a peer-reviewed journal by Soon and Baliunas in 2003. We examine the ICCER's work on allegations of subversion of peer review in relation to the Soon and Baliunas paper, and others, in paragraphs 73 to 77.

49. In our view, the debate about the 11 publications examined by the Scientific Assessment Panel (SAP) is frustrating. While there is no doubt that the papers chosen were central to CRU's work and went to the heart of the criticisms directed at CRU, the allegations that certain areas of climate science such as key multiproxy temperature reconstructions were purposely overlooked could have been disregarded if the SAP had set out its process of selection in a more transparent manner.

Publication arrangements

50. Having taken oral evidence from Sir Muir Russell in March 2010, our predecessor Committee was concerned that, upon completing the review, conveying ICCER's findings to UEA in advance of publication would give the impression that UEA was being given an advantage when it came to responding.[68] ICCER responded:

The reason for this proposal appears to be that to do otherwise might put at risk the review's impartiality. There is no question of any contact with the University prior to publication that would influence the review's conclusions, as distinct from any necessary checking of factual matters. The Review was commissioned by the University to report on policies and practices within the University, and should the Review find matters of concern, then it clearly has a duty to inform the University. The Committee will also be aware that natural justice demands that both the University and members of CRU should be informed directly of any critical findings. Finally, it is also common practice in public and Parliamentary life for the subjects of reports to be given embargoed copies of the documents shortly before publication.[69]

51. While we accept that it was not unreasonable for ICCER to inform UEA of the contents of its report in advance of publication, the fact is that this was open to misinterpretation.

44 "Another unsatisfactory rushed job", GWPF press release, 14 April 2010 Back

45 "CRU Scientific Assessment Panel announced", UEA press notice, 22 March 2010 Back

46 SAP report, para 2 Back

47 Q 11 Back

48 Q 8 Back

49 For example, the Climate Audit website, http://climateaudit.org/2010/06/22/kellys-comments/ Back

50 Professor Kelly's notes were available online (pp 81 and following): www.whatdotheyknow.com/request/35907/response/94112/attach/4/David%20Hand%20s%20attachments%20from%20emails%20supplied.pdf; in addition, they were submitted in a memorandum to the Committee-Ev W18 Back

51 Q 27, note 2 Back

52 Note by Lord Oxburgh added to oral evidence at Q 27 Back

53 SAP report, para 6 Back

54 Q 22 Back

55 ICCER, Appendix 4 and www.cce-review.org/Evidence.php Back