Peer review

Written evidence submitted by the
American Meteorological Society (PR 48)

The following is submitted by the American Meteorological Society (AMS), a scientific and professional society that has 14,000 members world-wide and publishes 10 peer-reviewed scientific journals. This submission has been approved by the AMS Executive Committee. It is based largely on existing AMS statements and policies.

The "AMS Statement on the Freedom of Scientific Expression" (adopted by the AMS Council in February 2006) states the following:

Advances in science and the benefits of science to policy, technological progress, and society as a whole depend upon the free exchange of scientific data and information as well as on open debate.  The ability of scientists to present their findings to the scientific community, policy makers, the media, and the public without censorship, intimidation, or political interference is imperative.  With the specific limited exception of proprietary information or constraints arising from national security, scientists must be permitted unfettered communication of scientific results.  In return, it is incumbent upon scientists to communicate their findings in ways that portray their results and the results of others, objectively, professionally, and without sensationalizing or politicizing the associated impacts.

These principles matter most - and at the same time are most vulnerable to violation - precisely when science has its greatest bearing on society.  Earth sciences and their applications have growing implications for public health and safety, economic development, protection of the environment and ecosystems, and national security.  Thus, scientists, policy makers, and their supporting institutions share a special responsibility at this time for guarding and promoting the freedom of responsible scientific expression.

Independent organizations such as the American Meteorological Society provide multiple avenues for the unfettered dissemination of scientific results, but chief among them are scientific journals that employ peer review as a means to ensure the integrity of the results published in them. The concept of peer review is as old as scientific societies themselves, and it has always represented a means of ensuring that the science disseminated by the society met the standards of the scientific community, namely, that scientists should be held to a high standard of integrity and honesty, and their conclusions should be unbiased and firmly rooted in observations, experimentation, and appropriate scientific methods. Without these core attributes, public confidence in the scientific enterprise will remain elusive. A key element necessary to build public confidence is that of full and open disclosure of scientific evidence, including methods of analysis. Publication of scientific results in respected peer-reviewed journals represents that full disclosure.

Ideas that eventually become part of our scientific knowledge must have supporting evidence, stand up to challenges by other scientists, and be able to successfully predict and explain our world; otherwise they are modified or tossed out. Accuracy and precision are highly valued, and carelessness or half-truths are not tolerated. This approach ensures that science will be self-correcting and converge on a realistic description of nature, even though it may take years or even centuries for this process to be fulfilled.

Science operates within a social context: people ascribing to a set of rules, values, and procedures that have been useful for advancing knowledge. Scientists value the pursuit of knowledge and the opportunity to be pioneers on the frontiers of science. They recognize the power and effectiveness of basing these efforts on objective observations, logical analyses, and the requirement of consistency between various scientific statements. Scientists search for universal truths. But an equally fundamental attitude of the scientist is to be skeptical and self-critical. Scientists know there have been ideas supported by observations that later (with different or more accurate observations) had to be modified or replaced. Scientific knowledge continues to grow because it discards erroneous ideas and substitutes ones that can be shown to be a more complete, accurate, and concise description of reality.

Scientists insist on disclosure of hypotheses, observations, methods, and interpretation of the results through the process of peer review, which allows other scientists an opportunity to evaluate their methods and the logic that led to their conclusions. A published result may not be fully believed until other scientists try out the ideas through re-analysis of their observations, taking new observations, repeating their experiments, or running a numerical model - whatever it takes to test the idea. Because of the skeptical nature of scientists, new ideas are accepted very slowly and only after a great deal of scrutiny. In fact, what authority science achieves is based on the openness by which scientific results are presented for review, evaluation, and additional testing.

For most scientific journals, the peer-review of a submitted manuscript represents a major part of the publication process, and the AMS is no exception. An author may need to revise his or her manuscript several times, as well as carry out additional scientific research, before the work has reached the level of excellence to be approved for publication by those who are part of the scientific community. AMS policies call for all manuscripts submitted to an AMS journal to be overseen by volunteer editors chosen from the community who are given the authority to make decisions with respect to the publication of those manuscripts. The following expectations are part of formal AMS policy.

An editor is expected to give unbiased consideration to all manuscripts offered for publication, judging each on its own merits without regard to the author’s race, gender, religious belief, ethnic origin, citizenship, or political philosophy. All authors should be treated with fairness, courtesy, objectivity, and honesty. The editor has complete responsibility and authority to accept a submitted paper for publication or to reject it. The editor may confer informally with associate editors or reviewers for an evaluation of the work to use in making this decision. The AMS uses a single-blind peer-review process, meaning that the reviewers are aware of the author(s) name(s) but the editor must protect the confidentiality of all reviewers unless a reviewer reveals his or her identity to the author.

The integrity of the journals depends on editors exercising care and judgment in their duties as editor and managing any real or perceived conflicts of interest. Editorial responsibility and authority for any manuscript authored (or co-authored) by an editor and submitted to the editor's journal is delegated to some other qualified person, such as another editor of that journal. Editors are called upon to avoid other situations of real or perceived conflicts of interest , as well . Such conflicts include, but are not limited to, handling papers from present and former students, from colleagues with whom the editor has a close professional relationship, and from those in the same institution. Any financial arrangement with sponsors that could lead to the appearance of an editorial conflict of interest is expected to be disclosed to the Publications Commissioner, who has been appointed by the AMS governance to oversee the journals.

A reviewer of a manuscript is expected to judge objectively the quality of the manuscript and respect the intellectual independence of the authors. In no case is personal criticism appropriate. A reviewer is also expected to be sensitive to even the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer's work in progress or published. A reviewer is not to evaluate a manuscript authored or co-authored by a person with whom the reviewer has a close personal or professional connection if the relationship would bias judgment of the manuscript. A reviewer is charged to treat a manuscript sent for review as a confidential document. It is not to be shown to nor discussed with others except, in special cases, to persons from whom specific advice may be sought; in that event, the identities of those consulted should be disclosed to the editor.

Adherence to the above policies by AMS editors and the individuals they choose to review submitted manuscripts forms a basis for the success of the peer-review process. With multiple reviewers chosen for each submission, and with the ultimate authority resting in the editor, one reviewer who does not live up to his or her obligation for a fair review is less likely to result in an inappropriate decision on whether or not to publish the submitted manuscript. A key component of high-quality scientific journals is a set of clearly articulated procedures governing the peer-review process, as well as multiple layers of oversight to ensure those procedures are consistently followed in all cases.

While critics can, and do, point to specific instances of abuse of the peer-review process in which quality science was reviewed poorly and rejected, those cases appear to be few and far between. Further, the fact is that even in most of those rare cases, the work did eventually get published and has become part of the corpus of scientific literature. That its publication was delayed is extremely unfortunate, but the eventual success shows that authors do have avenues to overcome cases of unfair reviews, and the AMS and many other society publishers have built into their procedures opportunities for authors to challenge decisions coming from peer review.

While certainly far from perfect, the peer-review system as currently administered by nearly all credible scientific journals around the world has done an excellent job of filtering the literature in ways that allow science to progress. This does not mean that no incorrect science is published or that no correct science is rejected, but it has allowed the scientific community to concentrate on replicating and building upon that work which has passed the crucial hurdle of peer review, allowing science to move forward at the maximum possible rate. The poor science that passes peer-review will eventually be shown to be incorrect. The good science that is incorrectly rejected initially has generally been published eventually and has, somewhat belatedly, contributed to the base of knowledge.

Furthermore, almost all peer-reviewed journals provide opportunities for peer-reviewed Comments and Replies. Therefore, even if (unintentional) poor science or (intentional) inaccurate or misleading information is published in peer-reviewed journals, such published material would have a high probability of being identified and commented on by the scientific community in those same journals.

Technologies such as the World Wide Web now provide many avenues for formal and informal publication of information on issues of science that allow broad and rapid dissemination. Society in general, and science in particular, has benefitted greatly from the ability to share information in these ways. It must be recognized, however, that many of the avenues available to share such information have little or no independent quality controls. This has, in a number of cases, resulted in misleading and inaccurate information entering the public deliberations on science-based topics that have affected public opinion and complicated effective policy discussions. The peer-reviewed literature offers the best hope of ensuring that both those public deliberations and the policies coming from them are grounded in the best scientific knowledge that is available at this time.

American Meteorological Society

8 March 2011