Peer review
Written evidence submitted by John McLean,
Professor Chris de Freitas, and Emeritus Professor Robert Carter
(PR 28)
Executive Summary
This submission describes salient events pertaining to a paper of ours that was published in peer-reviewed journal, a subsequent Comment published by that journal and our Response that Comment, which the journal refused to publish. Not only was our original paper subject to peer review but thanks to "Climategate" emails we have information about the review of a subsequent Comment and of course we have information about the review of our Response.
We beg the inquiry's indulgence with a submission that is about 500 words longer than the requested maximum length. We have tried to be brief but we are in the unusual position of being able to discuss three closely related peer reviews - two of our work, one of our critics - and we note numerous failings of those reviews. These failings include misunderstanding about what the task involves and requires, manipulation when authors suggest reviewers, reviewers failing to report inconsistencies or erroneous statements in articles, an editor failing to comply with his journal's review procedures and ultimately ambiguities about recommendations about publishing.
We conclude with a recommendation that we believe has merit and unlike other recommendations that might be made will likely find acceptance among journals.
1 - Introduction
[1] In December 2008 we submitted a paper, with the title titled "Influence of the Southern Oscillation on tropospheric temperature"", to the Journal of Geophysical Research (JGR). The paper was reviewed, as is the practice of that journal, and in February we received an email from the editor containing his cover note and a copy of the reviewers' comments (see Appendix). The reviewers hinted but made no explicit statement that they against the publication of our paper if changes were not made but the editor stated in his cover note "make the necessary changes in your manuscript and respond to me, explaining how you have addressed these comments", which indicates his interpretation of the situation.
[2] We subsequently made the revisions that, for the most part, had been requested in general terms and subsequently the paper was published in July 2009.
[3] Our paper showed that average global tropospheric temperatures closely followed the El Nino-Southern Oscillation of seven months earlier. It implied that man-made emissions of carbon dioxide had negligible impact on average global temperature, which of course is contrary to the view of the IPCC and numerous climate scientists.
[4] Within days of the publication of our paper, a document criticising it appeared on the Internet, formatted in the journal’s template. This document was ultimately published as Comment (i.e. formal criticism) by the journal but it refused to publish our Response to the Comment.
[5] We believe that several aspects of this affair are germane to this inquiry.
2 - Reviewers’ comments on our paper
[6] To set the scene for what follows please note the following extracts from comments from two reviewers of our paper (extracted from an email from JGR, available in the Appendix):
"I found the paper to be well-organized, well-written, and clear on the importance of the research. The abstract is informative, reference section is excellent, and the graphics are of high quality. The findings are likely to be of interest to a wide variety of readers, but I suspect their final sentence will not sit well in some circles."
[7] and
"This very clear and well-written manuscript is an analysis of the relationship between MSU-derived and rawinsonde-based [sic] tropospheric temperature variability and the Southern Oscillation, as modified by major tropical volcanic eruptions. I find few faults with this analysis from a scientific standpoint; my primary concern is the lack of novelty. Climatologists have known about the strong linkage between the SOI (and its cousins) and tropospheric temperature for some time now. The authors acknowledge as much and they include most of the key references on the subject. "
[8] Both reviewers are endorsing the paper, expressing no major reservations and the second reviewer notes that we have included the key references. These points are relevant to subsequent events.
3 - The submission and review of the Comment by our critics
[9] As noted above, criticism of our paper appeared almost immediately on an Internet one web site operated by Grant Foster (under the pseudonym "Tamino") and shortly afterwards on the website of Kevin Trenberth, formatted as if it was already a published JGR Comment. These two plus seven others were listed as authors of the JGR Comment when it was eventually published (see footnote for full reference).
[10] The "Climategate" files, which in November 2009 appeared in the public domain (hacker, whistle-blower, accidental?), contain several emails about the writing, submission and review of the Comment.
[11] On the subject of reviewers one email quotes the journal’s instructions to authors
3) Suggested Reviewers to Include
Please list the names of 5 experts who are knowledgeable in your area and could give an unbiased review of your work. Please do not list colleagues who are close associates, collaborators, or family members. (this requires name, email, and institution).
[12] but it also says
Agree with Kevin that Tom Karl has too much to do. Tom Wigley is semi retired and like Mike Wallace may not be responsive to requests from JGR. We have Ben Santer in common ! Dave Thompson is a good suggestion. I'd go for one of Tom Peterson or Dave Easterling. To get a spread, I'd go with 3 US, One Australian and one in Europe. So Neville Nicholls and David Parker. All of them know the sorts of things to say - about our comment and the awful original, without any prompting. (emphasis added)
[13] Contrary to the journal’s request for reviewers who "could give an unbiased review of your work", these potential reviewers are being considered precisely because they will "know the sorts of things to say".
[14] After submitting the Comment, substantially the same as that which appeared on the Internet, its authors received copies of the reviewers' comments and these appear in another Climategate email.
[15] Reviewer 1 says (in part):
This paper does an excellent job of showing the errors in the analytical methods used by McLean et al. and why their conclusions about the influence of ENSO on global air temperature is incorrect. I have only a couple of suggestions to help clarify their analysis of the methods.
[16] And reviewer 2 says (in part):
I think this comment on McLean et al can be published more or less as is.
I have two comments
First, in the abstract (page 3, line 15), I'm not sure that "inflating" is quite the right verb - the paper itself does not make the point that the filter constructed by McLean et al inflates power in the 2-6 year window. Perhaps "isolating" would be a better verb.
Secondly, I think the points that are being made with Figures 4 and 5 could be strengthened by adding to the right of each plot of a pair of time series, a scatter plot of the pairs of values available at each time. Such a scatter plot would help to clearly illustrate the absence (upper panels) or presence (lower panels) of correlation between red and black values.
[17] Reviewer 3 says in part:
The real mystery here, of course, is how the McLean et al. paper ever made it into JGR. How that happened, I have no idea. I can't see it ever getting published through J Climate. The analyses in McLean et al. are among the worst I have seen in the climate literature. The paper is also a poorly guised attack on the integrity of the climate community, and I guess that is why Foster et al. have taken the energy to contradict its findings.
So the current paper (Foster et al.) should certainly be accepted. Someone needs to address the science in the McLean et al paper in the peer-reviewed literature. But the current paper could be - and should be - done better. That's why I am suggesting major changes before the paper is accepted. All of my suggestions have to do more with the tone and framing of the current paper, rather than ts content.
[18] The first two reviewers seem anxious to help the authors of the Comment even though their role as reviewers is essentially to advise the journal as to the merit of publishing the Comment.
[19] Reviewer 3 starts out by criticising our paper, which is outside the scope of his/her task and makes comments that contrast sharply with the reviewers of our Paper even though both sets of reviewers are supposedly experts. This third reviewer then makes the very partisan comment "The paper is also a poorly guised attack on the integrity of the climate community...", and proceeds, like those before him (or her) to offer advice to the authors of the Comment.
4 - Mistakes in the Comment that the reviewers failed to identify
[20] For all their advice to Foster et al the reviewers failed to identify several flaws in the Comment.
[21] Firstly, the abstract of the Comment says
McLean et al. [2009] (henceforth MFC09) claim that the El Niño/Southern Oscillation (ENSO), as represented by the Southern Oscillation Index (SOI), accounts for as much as 72% of the global tropospheric temperature anomaly (GTTA) and an even higher 81% of this anomaly in the tropics.
[22] But the introduction states:
They argued that more than two thirds of the interseasonal and longer-term variability in global tropospheric temperature anomaly (GTTA) (72% using the 29-year-long MSU satellite record and 68% using the longer 50-year RATPAC-A record), and an even larger 81% of the variation in tropical (20◦S-20◦N) tropospheric temperatures, can be explained by the long-term variations in the Southern Oscillation Index (SOI). (emphasis added)
[23] The first indicates a direct link between ENSO and temperature while the second refers to variability. These are inconsistent in the same way that velocity is not acceleration
[24] In fact in our abstract we said,
"Change in SOI accounts for 72% of the variance in GTTA for the 29-year-long MSU record and 68% of the variance in GTTA for the longer 50-year RATPAC record. ... The results showed that SOI accounted for 81% of the variance in tropospheric temperature anomalies in the tropics." (emphasis added)
[25] These were the only two sentence in our paper in which these figures are mentioned in this manner; in the text we show them as R squared values (i.e. statistical constructs) in the context of a discussion about derivatives based on the change in temperature and the change in SOI. At all points we refer to change in values rather than absolute values, which is not what the Comment's abstract states. So why did the reviewers fail to note the inconsistency?
[26] Secondly, our comments appeared in our Abstract and in the statistical form in our Analysis, but not mentioned in the Discussion or Conclusions. The journal's documentation, readily available on the Internet, says that Comments should be directed at the main substance of a paper, but neither the reviewers nor the journal editor noted the failure of Comment to address this.
[27] Thirdly, the Conclusion of the Comment states
It has been well known for many years that ENSO is associated with significant variability in global mean temperatures on interannual timescales. However, this relationship (which, contrary to the claim of MFC09, is simulated by global climate models, e.g. Santer et al. [2001])
[28] In fact we said:
We also note that Figure 1 of Lean and Rind [2008], based on a different temperature data set to that used here, indicates that their model underestimated temperatures during and immediately following the predominantly El Niño conditions from September 1939 to January 1942, and overestimated temperatures during and immediately following the La Niña dominated period from August 1906 to December 1910. This suggests that their model poorly replicated ENSO effects.
[29] And shortly afterwards we said ...
Chapter 3 of the Working Group I contribution to the Fourth Assessment Report by the Intergovernmental Panel on Climate Change (IPCC) [2007] notes the strong relationship between the ENSO and various climate phenomena, including surface temperature. Chapter 8 reports ‘‘considerable model skill out to 12 months for ENSO prediction.’’
[30] Claims that we said that the ENSO temperature relationship was not modelled are therefore false, and the reviewers should have identified this fact.
[31] It appears to us that the peer-review process has serious problems when reviewers cannot identify clear errors of fact. It can also be argued that a competent editor would have identified these problems for himself, but the publication of the false information indicates that this was not the case.
5 - Our Response to the Comment and the rejection of our Response
[32] In November 2009 we were advised that the JGR had accepted the comment for publication and we were invited to respond.
[33] This invitation came from a new editor at JGR. The Climategate emails show that on 3 Aug 2009, less than a week after our paper was published, one of the Comment authors emailed the JGR to request that the editor be replaced and less than 3 days they were advised that this had been done. (The usual practice is that the same editor that dealt with the paper then deals with any subsequent Comment and the author's Response to that comment.)
[34] We inadvertently copied the editor on an email between we three authors of the original paper shortly after being invited to respond. In that email we noted that substance of the response had been posted to the Internet, in the format of a JGR paper, almost 4 months earlier and therefore was in breach of the journal's regulations.
[35] The editor's responded saying:
The Foster comments were subjected to the same rigorous peer review that your manuscript passed. The fact that similar comments were posted on the web is irrelevant. I have not seen the Hadley emails, which are also irrelevant. The comments will be published whether you submit a reply or not.
[36] His first sentence is doubtful for reasons given above and he is prejudging the contents of unseen emails.
[37] During a subsequent rapid exchange of about 8 emails the new editor tried to claim that posting to the Internet was not publishing, then he said that the posting had used the journal's boilerplate (i.e. formatting template) before quoting a journal regulation that said authors may post their unformatted [sic] papers or their abstracts to their own Web sites or their departmental Web sites according to certain guidelines.
[38] In short the JGR editor refused to acknowledge that the Foster et al Comment had in essence been published on the Internet in JGR format.
[39] In another of his responses during this series of exchanges he said
I repeat, I will NOT reject a manuscript that has passed peer review. I did not reject your manuscript although it was clear after it was accepted that there were serious problems with it. I am not going to hold a double standard.
[40] The second half of his comment is in contrast to those of the reviewers of our Paper (see earlier) and despite another editor of the same journal accepting our Paper for publication. The "double standard" that exists is the reverse because our Paper complied with the regulations for publishing, was reviewed and was duly published. The Comment failed to comply with regulations about prepublication, Climategate emails show that the authors nominated certain reviewers on the expectation of their bias and, as noted above, the Comment contained obvious flaws.
[41] None of the comments above about the editor have a direct bearing on the peer-review process but they do show the attitude of the person who selected the reviewers and subsequently made decisions based on their comments.
[42] We wrote a Response and duly submitted it in January 2010. We said that the Comment focussed on a matter in our Analysis and failed to address the substance of our Discussion or Conclusions. (We did not mention it in our response but this failure put the Comment in breach of documented procedures and policies.) We also pointed out that our finding of a 7-month time lag was not inconsistent with the findings of others, including one of the authors of the Comment. (For more information about our Response see the document given in footnote below.)
[43] We submitted our Response confident that we had addressed the issues of substance. A few weeks later received an email containing the reviewers' comments and editor's decision, and again we found breaches of the journal's documented procedures and a very flawed peer-review.
[44] The journal's procedures say that a Response is sent to one reviewer and the purpose of the review is to confirm that the respondents have addressed the points raised in the Comment. In an abuse of the journal's documented procedures our Response was sent to three reviewers, not one.
[45] Reviewer 1 said (in part) ...
The essence of this response to the comment by Foster is that McLean et al claim that they did clearly characterize their findings as pertaining to the relationship between the derivatives of the Southern Oscillation Index and those of global mean temperature anomalies, rather than to the original unfiltered time series. Unfortunately, I find their defence to be rather disingenuous - they certainly did not go out of their way to make it clear to the reader that their conclusions and interpretations applied only to these derivatives and not to the unfiltered SOI and temperature series, and by omission of these reminders, they implicitly and inappropriately invited the reader to interpret their results as applying to these unfiltered series.
[46] This statement is an opinion about clarity rather than any question of the scientific validity of our response.
[47] Reviewer 2 said that the correlation between time-lagged SOI and temperature was weak, but we had said in our Response that other minor forces (often characterized incorrectly as "noise") influence both the SOI and temperature and that a precise match cannot be expected. This reviewer made four other minor points were made and asked for changes to the text.
[48] Reviewer 3 starts with the comment:
REJECT! The reply by McLean et al to Fea10 (their notation) is not worth publishing. The techniques they use and conclusion drawn from them are largely bogus. How in the heck did the original dog of a paper ever get through the review process. Please check it out and reprimand the appropriate editor.
[49] According to the journal's procedures the review of our Response should only focus on whether it addresses the Comment but this reviewer has decided to criticise our original paper and only peripherally refers to our response to the Comment.
[50] So three reviewers, not one, and two make minor observations that might warrant some changes to the text and the third concentrates instead on the original paper. Despite these relatively minor criticisms from the reviewers, the editor's cover note to the email (16 March 2010) said (in part)
"All of the Reviewers raise very serious objections and recommend against publication. I therefore regret to inform you that based on the Reviewers' recommendations, I am unable to accept your paper for publication in JGR-Atmospheres. Please note that the reviewers are highly respected members of the scientific community. I consider their reviews to be unbiased assessments of the the [sic] scientific validity of your response.
[51] With this extraordinary statement, based on a very flawed peer-review, and dishonestly saying that all three recommend against publication, we were denied the right of reply, which is an extremely rare situation for scientific journals.
6 - Conclusions
[52] Our experience with our Paper shows many faults with the current peer-review process.
[53] Some reviewers seem confused about which document they are to comment on (Paper, Comment or Response?), what issues they should address in regard to the document, what the document itself should be addressing, and the nature of their role (advising authors or advising editors?).
[54] Reviewers are also fallible. We saw here their inability to identify inconsistencies, factual errors and the fact that a Comment did not address the substance of a paper.
[55] Because the process is not transparent we cannot be sure that reviewers who were nominated "because they know what to say" were not in fact those who reviewed the Comment. Strictly speaking that would be a failing of the journal but the fact that it may have occurred highlights a failing that authors can exploit.
7 - Recommendation
[56] It is our view that while radical changes to the peer review process are possibly justified, the various scientific journals may be very reluctant to accept those changes. Bypassing the journals via some new mechanism may not be the answer either because the new system is likely to be considered as somehow inferior.
[57] Likewise disclosure of intended reviewers' names to paper authors (and arguably the wider community) and allowing objection on certain defined grounds would be an onerous burden on journals. (In court the defence does not reject potential jury members because they might favour the accused but this favouritism can arise in peer review, as Wegman et al mentioned in relation to the Hockey Stick temperature graph.)
[58] It is our recommendation that journals introduce a "checklist" approach to review. The checklist would be tailored to match the nature of the article being reviewed – Paper, Comment, Response. It would describe the reviewer's task and list a series of items where each had a simple evaluation scale (e.g. Yes/No, 1 to 5 scale, 1 to 10 scale) and space for relevant comments. The final item on each checklist would be an assessment of whether the article was fit for publication as it was, whether minor changes or major changes were required, or whether the article was entirely unsuitable.
[59] In our opinion this approach is simple, comprehensive and flexible enough to meet the requirements of different journals, moreover its clarity should be of benefit to the journals as well as to the reviewers and the authors.
APPENDIX
Editor’s cover letter and reviewers’ comments re our Paper
Manuscript Number: 2008JD011637
Manuscript Title: The influence of the Southern Oscillation on tropospheric temperature
Dear Dr. de Freitas:
Attached below please find 3 reviews on your above-referenced paper. At least one of the Reviewers has raised questions and made suggestions for important revisions. Please consider the Reviewer reports carefully, make the necessary changes in your manuscript and respond to me, explaining how you have addressed these comments. In your Response to Reviewer letter, please include a statement confirming that all authors listed on the manuscript concur with submission in its revised form.
The Reviews were mixed, but I am prepared to consider publication providing certain doubts are resolved. Reviewer 1 has mainly minor comments. Reviewer 2 ponts out that much of what you present has been shown before. Therefore you need to emphasise what is new about your work, as pointed out by Reviewer 3.
This is a very brief paper and the lack of connection with the previous literature on the subject is of concern to me. This lack of connection even with the conclusions of IPCC concerns me, as it has resulted in the throwaway last sentence which has not been demonstrated in the paper. At the very least this needs to be removed. If I have understood your paper correctly, it does indicate a certain naivety that the variations you see in the SOI are all natural. In the end, your work is unable to demonstrate cause and effect, as you have only used regression. You have shown a relationship between global temperature and SOI, but that is in itself not new, yet the IPCC conclusions remain in contradiction to your last sentence.
(Instructions for resubmission removed by submitters to this inquiry)
Sincerely,
XXXX XXXXXX (redacted by submitters to this inquiry)
Editor, Journal of Geophysical Research - Atmospheres
--------------------------------------
REVIEWER 1
Reviewer #1 (Highlight):
Suggesting that the planetary temperature has not been appreciably influenced by humans could be a headline throughout the world!
Reviewer #1 (Comments):
Thank you for providing me the opportunity to review the paper by McLean et al. on SOI and global and tropical tropospheric temperatures. I found the paper to be well-organized, well-written, and clear on the importance of the research. The abstract is informative, reference section is excellent, and the graphics are of high quality. The findings are likely to be of interest to a wide variety of readers, but I suspect their final sentence will not sit well in some circles.
Actually, claiming that SOI accounts for variation is different from the issue of trend in the temperature data. If the authors could expand this discussion a bit, the paper would be strengthened. The key question in the eyes of many deals with explaining any trend in the data, not just variation in the temperature measurements.
I would suggest some editing, particularly treating "data" and the plural form of "datum". I am also concerned that the authors appear to have made little or no effort to conform to the style of AGU journals. Along these lines, we find things like "Lacis et al; Sato et al" on page 9 (line 21) ... very odd to not see dates of the publications.
REVIEWER 2
Reviewer #2 (Highlight):
Reviewer #2 (Comments):
1) The warming of the entire troposphere to ENSO is well established, if not well understood. (for example, see: Journal of Climate; Sep2002, Vol. 15 Issue 18, p2702, 5p)
2) The current manuscript documents the influence of ENSO on global tropospheric temperatures with additional observed datasets.
3) It is very clear that the authors have spent considerable amount of time in preparing the relevant datasets, to reconfirm the previously well established results. I am not sure if this is enough to deserve publication in JGR.
4) The manuscript would be more acceptable if it leads to an increase in the current understanding of the mechanisms behind the influence of ENSO on global tropospheric temperatures.
REVIEWER 3
Review of "The Influence of the Southern Oscillation on Tropospheric Temperature" for JGR-Atmospheres.
This very clear and well-written manuscript is an analysis of the relationship between MSU-derived and rawinsonde-based tropospheric temperature variability and the Southern Oscillation, as modified by major tropical volcanic eruptions. I find few faults with this analysis from a scientific standpoint; my primary concern is the lack of novelty. Climatologists have known about the strong linkage between the SOI (and its cousins) and tropospheric temperature for some time now. The authors acknowledge as much and they include most of the key references on the subject. The only novelty that I see here is the use of differenced smoothers rather than raw data in identifying the strongest lag correlations, but this method is commonly employed in ARIMA modeling, which is essentially what the authors are doing here. They do make the case that these relationships have not been updated in some time, and that is a valid point, particularly given the lack of recent warming and the lack of a reasonably strong El Niño event since the late 1990s. To convince me that this work is publishable, they need to make a better case for either the novelty of their analysis or the importance of the update, and preferably both.
Major Issues
Volcanic Eruptions
I would not be concerned about where volcanoes are located within the tropics. The key importance of tropical volcanoes is that their ejecta is more easily transported into the lower stratosphere via strong tropical convection. Thus, all volcanoes, but particularly tropical ones, might have global impacts for many months or years depending on the weather conditions at the time of the eruption and the total mass of ejecta. Given this variability, I think your assumption that the influence of volcanoes is 12 months is an oversimplification (you admit as much). I think you can address this by assuming different "influence periods" for each volcano, then reproduce your fitting procedure until you find an optimal fit. This may allow you to identify the range of influence of volcanic eruptions (on temperature) and help you to settle on an optimal fit by using a more objective approach.
Lags
Please include a new figure showing the autocorrelation functions for each of your best fit lines. These would be useful to the readers to help understand the properties of the lagged relationships.
Prediction
I think it would also be useful to see how well your best model works in a predictive mode. You could do this a variety of ways, including withholding some data or jackknifing.
Minor Issues
Page 1, line 22. "SOI index" is redundant.
Page 5, lines 18-20. This sentence is unclear (to your credit, it’s the only unclear sentence in the manuscript)
Page 11, line 19. Change "is" to "are"
Page 12, line 5. Change "global temperatures" to "global temperature anomalies."
Page 12, line 7. Add a comma after GTTA
Page 12, lines 24-25. The reader could mistakenly infer that you are suggesting that greenhouse gases have not influenced global temperatures since 1960. Although you say "temperature variation," I think you could make this more clear. Throughout the paper, you have not dealt with temperature trends, only the short-term variability of temperatures about some mean value. Given that this work on the role of El Niño is not new, a statement noting that these results confirm earlier work that most of the variations in tropospheric temperatures can be accounted for by the SOI seems more appropriate.
References. You may find the following paper to be relevant:
Lean, J. L., and D. H. Rind (2008), How natural and anthropogenic influences alter global and regional surface temperatures: 1889 to 2006, Geophys. Res. Lett., 35, L18701, doi:10.1029/2008GL034864.
John McLean, PhD candidate James Cook University, Australia
Professor Chris de Freitas, University of Auckland, New Zealand
Emeritus Professor Robert (Bob) Carter, Townsville, Australia
7 March 2011
|
