UNCORRECTED TRANSCRIPT OF ORAL EVIDENCE
To be published as HC 856-i

House of COMMONS

Oral EVIDENCE

TAKEN BEFORE the

Science and Technology Committee

Peer review

Wednesday 4 May 2011

Dr Nicola Gulley, Professor Ronald Laskey CBE FRS FMedSci, Dr Robert Parker and Professor John Pethica FRS

Evidence heard in Public Questions 1 - 61

USE OF THE TRANSCRIPT

Oral Evidence

Taken before the Science and Technology Committee

on Wednesday 4 May 2011

Members present:

Andrew Miller (Chair)

Stephen Metcalfe

Stephen Mosley

Pamela Nash

Graham Stringer

________________

Examination of Witnesses

Witnesses: Dr Nicola Gulley, Editorial Director, Institute of Physics Publishing Ltd, Professor Ronald Laskey CBE FRS FMedSci, Vice-President, Academy of Medical Sciences, Dr Robert Parker, Interim Chief Executive, Royal Society of Chemistry, and Professor John Pethica FRS, Physical Secretary and Vice President, Royal Society, gave evidence.

Q1 Chair: Good morning to you all. Some of you may not be aware that Parliament was sitting until the early hours this morning, so some of my colleagues are a tad on the tired side. Please bear with us. May I, first of all, ask the four witnesses to introduce themselves?

Dr Gulley: I am Dr Nicola Gulley. I am the Editorial Director at IoP Publishing.

Professor Laskey: I am Ron Laskey. I am here as Vice-President of the Academy of Medical Sciences.

Dr Parker: I am Robert Parker. I am Interim Chief Executive of the Royal Society of Chemistry.

Professor Pethica: I am John Pethica, Physical Sciences Secretary and Vice-President of the Royal Society.

Q2 Chair: Welcome. You are familiar with the nature of this inquiry. Let me start off with some basic questions. If you feel at the end of the session that you would have liked to have responded in more detail to one of the questions, please feel free to drop us a note with any additional comments. Peer review is perceived to be "fundamental to scholarly communications". If it disappeared tomorrow, what would the consequences be?

Dr Parker: You would have to come up with something else with which to replace it. There isn’t anything very obvious to replace peer review with currently. The danger would be to the scientific record, really. The importance of it is laid out in the evidence that has been submitted with great clarity from most people who have submitted evidence in writing to this review. The value and quality of that scientific record is paramount, and peer review helps to keep that in place.

Professor Laskey: In the biomedical sciences there would be a particular problem of sorting the wheat from the chaff and knowing what information could be depended on. This, I think, would corrupt the public understanding of science where a firm basis of trust in scientists is something that we could do with more of.

Dr Gulley: I would add that there is the aspect of the time that it would take for scientists to be able to find information that they really wished to read, because at the moment peer review also adds value in having that filter in place. There is evidence that many authors feel that the peer review does improve the quality of the articles that they publish as well.

Professor Pethica: To add a historical perspective, of course, this has been going on for a very long time. You asked the question of what would happen if it disappeared. Its primary function is to improve the process and the coherence of scientific knowledge and its utility.

Q3 Chair: Taking Professor Laskey’s observation about sorting the wheat from the chaff, in a sense, the opposite of that is something that we have been told in the evidence, that peer review has a tendency towards producing conservative judgments. How big a problem is that for the progression of science and what can be done about it?

Professor Laskey: Some journals have a tendency to believe that things that are already well known to be important have a higher impact. It can be more difficult to establish a novel and completely unexpected new branch of science if editors of journals are not alert to the fact that it is coming. There are one or two recent examples. One that springs to mind is a study in plant sciences which concerned resistance to viral infection in plants. That has given rise to a completely new area of understanding of a group of molecules that turn out to be important in all cells, not just in viral defence mechanisms against plants but because they change fundamentally in certain types of cancer. That was a small niche of advance that has suddenly become a front-line subject, but it would have been very difficult to publish that in a front-line journal at the time the work was being done.

Professor Pethica: To add to that a little, there is always a risk in this process that new ideas may be impeded in the way I have described. That is a risk. It has to be balanced against the fact that the likelihood of radical breakthroughs is, unfortunately, rather smaller than exotic ideas that don’t actually work. It is that balance that is difficult to achieve because there is a tension between those two.

Dr Parker: Knowing the right people to ask about research that looks slightly different is one of the most important things. Having professional people overseeing the peer review process is absolutely paramount, because it ensures that, if something is there that is very different, you could get somebody to look at it who will look at it in an open way. There are different outlets for different sorts of science. Sometimes you can get things published that are a little odd or seem a little odd at the time.

As Professor Laskey said, you don’t always know very quickly what is going to be important in areas that are far away from chemistry. You could have some mathematical proof that is not found to be terribly important until 50 years or 100 years later, and suddenly it is important in finding out something else.

Dr Gulley: There is also a cultural difference in certain research areas as well. There is more conservatism in some of the research areas than there is in other areas. Speaking from the journals’ point of view, some journals like to have articles that they feel are cutting edge. That is, partly, how they approach it. Also, different things need to be taken into consideration. There is peer review within, for example, the conferences where you do get a feel for some of the new areas that are coming up before they go into the journals.

Q4 Graham Stringer: Which areas are conservative and which are bold and radical?

Dr Gulley: It varies considerably. I can only speak from the physics side in which I am involved. There are certain very well-established areas where there would be slightly more conservatism because they are very well established and they want to be sure that before something goes in to be the article of record it is correct. There could be some areas, particularly where you have multidisciplinary areas, where there are more differences of opinion. There is then less conservatism about what gets recorded.

Q5 Chair: You have all acknowledged in different ways that there is a risk or a problem there, but none of you really responded to the second part of the question, which is, what can be done about it?

Professor Pethica: I alluded to that a little bit in my response. Given that there is no perfect system, we have to devise a system which optimises the process, that is to say, one that minimises the risks that have been alluded to but also retains the key advantages of the peer review process in establishing a coherent record. A variety of models have been alluded to in other places. For example, different kinds of publishing models are being evolved all the time. For example, there is the arXiv record in high energy physics which stems from the way that high energy physics actually works. That is another way of establishing the record.

It is also important to recall that peer review, as we are describing it here, is about generating a coherent scientific record efficiently as far as possible, but often it is used for other proxy purposes and assessment. That can, potentially, influence how it is carried out.

Q6 Chair: Following on from that, is something like the PLoS ONE model of publishing anything that is scientifically sound, regardless of potential impact or perceived scientific interest, a better way of doing things?

Professor Laskey: It is an alternative that solves some of the problems. At the moment it is an evolving landscape. The attitude of PLoS ONE to publish irrespective of impact but based solely on the criterion of the quality of the science can prevent a trap that, in biomedical sciences, is becoming increasingly troublesome, namely, that a high proportion of time is spent fending off criticisms from reviewers that may not be on the main theme of the work. The reviews are beginning to dictate the agenda of the science in a way that is not fully productive. That can be frustrating, a waste of time and resource. PLoS ONE provides an alternative to that.

The downside, as Professor Pethica has already said, is that there is now a proxy use of peer review, namely, to judge careers by the calibre of the journals in which people have published and to judge institutions by the Research Excellence Framework, again based on the quality of journals in which people have published. PLoS ONE, of course, is not a major front-line high impact journal.

However, that has been compensated for by the ease of electronic searches of the literature. Now you no longer have to depend on readers of a small number of widely read journals seeing your paper, because your paper will be noticed by electronic search routes as an alternative. Against that changing model, there is an increasing value of archival journals, such as PLoS ONE, which ignore impact.

There are potential downsides to it because of the proxy use of peer review data, but they do offer an alternative. One trend that has been emerging, and it has been a surprise to many people, is that, initially, people envisaged PLoS ONE as a journal they would submit to only if their paper was having severe criticism from other higher impact journals. Now, important research has been submitted to get it on the record quickly before it is scooped by someone else who has a smoother path through the refereeing jungle.

Dr Parker: The PloS ONE-type model or the cascading model also has another advantage in that it can reduce the factor of articles being multiply peer reviewed by different journals. It can save time in the peer review process there. Without those cascade journals, you often have the case where papers that are rejected by one journal are then submitted to another and they are reviewed again completely. They could be, and very often are, scientifically acceptable but they just don’t reach the impact criterion for that particular journal. So the process goes on until they find a home. The cascading-type journal does away with that. We found, from doing studies on the articles that we reject, that most of them end up being published somewhere else. There are very few articles that we receive that are scientifically completely wrong. Usually, there is some merit in them.

Q7 Stephen Mosley: Dr Gulley, Dr Parker and Professor Pethica, could you summarise for the record the peer review methodology that you use in the journals that you publish?

Dr Gulley: For the IoP Publishing journals, we use single-blind refereeing where the referees know who the author is but the referees’ names are kept anonymous.

Dr Parker: We are the same at the RSC. We also use, as I suspect others do, pre-screening as well, so not all papers are sent to referees. Some of them are rejected before they are sent to referees. That is either by internal, qualified editorial staff or by external associate editors.

Professor Pethica: It is the same process for the Royal Society’s journals, which is single-blind.

Q8 Stephen Mosley: The three organisations represented here all use the same methodologies, but there are other organisations that use different methodologies. Why are different methodologies used across the journals?

Professor Pethica: I have two background points to make. One is, of course, that the subject areas vary very strongly. We should keep in mind that we are discussing areas from pure mathematics through to biomedical research. As a result, the review process in those cases needs to be quite different simply because of the nature of the subject and what they are trying to establish. As a result, the various experiments in the forms that you have described vary quite strongly across subjects. What you have heard about is physics and chemistry, which are what I might call fairly traditional mainstream subjects. As you move to the more trials-based medical work, you get a different structure. Also, I have mentioned arXiv already and high energy physics. This is a very large-scale collaborative exercise in which that kind of model of communication is quite important. There is some variability across the subjects and I suspect that is the demand of the users. The need to establish fairly rapidly a sensible view of what works in the science affects the methods used. It is fair to say that all of these things are in a state of steady evolution. There are core principles, but the actual method by which it is used varies. You have evidence from a variety of people submitted to you about things like completely open double-blind processes. You have some responses about that. I don’t want to add to those because they are broadly correct.

Dr Parker: I would add that open review pre-printing, particularly, is not popular with chemistry because there is very often the possibility that an author will take out a patent on what they are producing. Putting your results out there in a pre-printed form is risking losing priority on them.

Another aspect that is different across disciplines is the amount of experimental refereeing that is done. In mathematics there might be very few experimental results, but in chemistry and physics there are a vast amount of experimental results that sometimes need specialist refereeing.

Dr Gulley: Some of the research communities that I work with particularly are very small, so doing double-blind refereeing where neither the author nor the referee knows who each other is defeats the object because, generally, the referees will know who the author is from the subject area that they are working in or from the references and things like that. It varies very much between different subject areas.

With regard to pre-print, this is something within physics that we have worked with for many years now since the arXiv was set up. That form of open refereeing on the pre-print side is very much entered into for a lot of the subject areas with which I work. However, when it comes to submitting to journals, we find that people are happier to referee if they are kept anonymous because they feel they have more options to be able to criticise openly. As a result, there is a mixture within the physics area now.

Q9 Chair: Where you have these very small communities, isn’t there a tendency for the referee to be somebody who is more advanced in their career than the person whose paper they are refereeing? Doesn’t that inhibit the evolution of some of the science because you would feel a little reluctant in criticising previous work?

Dr Gulley: It also works the other way in that you have some of the more junior researchers-this is one of the areas that, perhaps, would have anonymity of the referees- and it enables people to feel a bit more comfortable when they have to criticise work in particularly well-established research areas when the research area is small. Then there is less worry about it impacting on grant applications and things like that.

There is also an element of what the editor or the editorial office of the journal has to take in making sure that there is a balanced opinion. Generally, for IoP Publishing we don’t just have one referee on an article, for example. It is usually balanced with at least two and usually more.

Q10 Stephen Mosley: Professor Pethica, I noticed that you were nodding when the Chairman asked his question. Perhaps you have a slightly different view.

Professor Pethica: No, on the contrary. It is quite commonplace for research students to be trained by asking them to review papers. The question of senior people reviewing junior people is very often reversed.

Q11 Stephen Mosley: Dr Gulley, we have heard from Dr Parker that the pre-print server, arXiv, isn’t really appropriate for the chemistry community. However, it is widely used within the physics community. Could you give us an explanation as to how it works and how authors interact with the system? Do they maintain their own records, etcetera?

Dr Gulley: The arXiv was set up as a pre-print server so it is the authors’ work at a very preliminary stage. They use that process to be able to comment. It originated from the high energy physics area where they had a reason to be able to discuss the results across the international collaborations. A lot of the work that is posted, particularly from areas such as high energy physics, also goes through internal peer review within the research facilities as well before it is posted. There are a number of different stages it has to go through.

As far as linking in with publishing is concerned, a high percentage of articles that are pre-prints are eventually submitted to journals and get published in journals as well, so there is still that requirement for the independent peer review. We link in with that. We make it very easy for authors to be able to submit from the arXiv into our journals, for example, and this is common across many physics publishers, where the arXiv number can be used so it links back as well. Authors are encouraged to update their versions as well. From the publishing side, we encourage them to update the references so that the link goes back to the final version of record once it has been peer reviewed and published.

Q12 Stephen Mosley: From what Dr Parker said earlier, it is probably not appropriate across the entire science community, but are there areas other than physics that might benefit from it?

Dr Gulley: I am not sure that I am the best person to comment on that since my area is very much within physics.

Professor Pethica: Pure mathematics is a good example of an area where the numbers are fairly small. It can take a very long time for the assessment of theorems to become correct. Therefore, effectively, this process is hybrid like that. I would like to draw your attention to the contrast with engineering, for example, where you have an immediate technological impact on what you are doing. Then the question of publishing that Professor Parker raised is rather important.

Q13 Chair: Are the extremes the areas where there is a much more collegiate approach to helping solve a global high energy physics problem working in some of the big science projects versus things that are much closer to potential commercialisation? Are those the extremes?

Professor Pethica: There will be a difference across that. The reason I raised it is that it is not as if there is a lack of collaboration in that area too. It is very extensive in large technological projects. As you know, it is not just in high energy physics. It is the other factors that Dr Parker raised, such as patentability, exploitation and all the issues that concern companies in research that involves collaborative work and pre-exploitation. This boundary has an interesting area that affects what you publish, when and how, and the role of patents as distinct from peer review publication. There is a continuum across the board.

Dr Parker: Also, the speed of reproducibility of results is an issue where you are sharing big resources like synchrotrons or various other things such as in the area of astronomy. It is very good for peer review. There are small numbers of people using big pieces of equipment that are very expensive. It is good for them to share and work together.

Q14 Chair: Even in those areas you get some quite innovative commercial projects emerging. Charge-coupled devices spring to mind and things like that.

Dr Parker: Yes.

Professor Pethica: Silicon technology.

Q15 Stephen Mosley: Professor Laskey, within the Academy of Medical Science evidence you suggest that the dissemination of non-peer reviewed information may be potentially unhelpful, as you describe it. You also go on to say that even things that are kite-marked may not be totally appropriate. Would a kite-mark work or do you still have some concerns?

Professor Laskey: Two worries were voiced in the Academy’s submission on this topic. One is that biomedical sciences are more prone to inaccurate interpretations. Measurements in biology tend to be, by the very nature of biological material, more scattered than the more precise measurements that can be made in the physical sciences. Although I don’t like the terms from the perspective of a biological scientist, I have to admit that there is some truth in the description of hard sciences, meaning the physical sciences, and the soft sciences, meaning the biomedical sciences, in which it can be more difficult to get precise and incontrovertible evidence. There is a worry that, if you extended the pre-publication model to the biomedical sciences without any attempt to peer review, a lot of half-truths would creep into the literature.

The second problem is the appetite of the media for some aspects of biomedical science. Without peer review we would get a storm, frankly, of incorrect headlines. That is something that would also worry us very much.

Chair: The media don’t do badly at doing that, anyway.

Professor Laskey: The scientific community tries to prevent that but not always successfully, I am afraid.

Q16 Stephen Metcalfe: Three of you publish journals. How do you ensure that your editors are selecting the most appropriate reviewers? What process do you go through? Is that process complicated when you are looking at multidisciplinary work which covers a number of different areas?

Dr Parker: The process we go through is that we have internal and external editors. The internal editors are chemists who work within the RSC. The external editors are people who work out in the community, who are largely academics. We ensure that they choose the right referees by having a long period of training for people who do things like that. Building up a knowledge of the community is very important. There are people who work on general journals that cover broad subject areas, but most people will have a specialism within them even if they do work on those broad journals. People do get to know a particular area and the interactions between certain authors and referees very well. You do get to know your community and you get a feel for whether there are any issues between particular people. We also do quite a lot of training of referees. We have a feedback loop where referees always get the feedback on the outcome of the articles that they have refereed so that they can learn whether their refereeing activity is generally in line with what is accepted and what is rejected. We also do straight face-to-face training as well, particularly in China and India, but also elsewhere.

Dr Gulley: It is very similar for IoP Publishing as well in that we have a combination of in-house editors and external editors. We also do quite a lot of training of our referees, particularly within China. We have different programmes across our international offices as well.

Recently, as a result of requests from some post-docs and graduates, we have given them some initial training on what peer review means. We are teaching them about what refereeing means and what we are expecting. There is a lot of literature as well that people are not always aware of so we have been trying to raise the visibility of that. Internally, we also try and match the interests of the referees to the papers as much as possible. Again, that comes from the extensive training that is required for the internal editors, as Dr Parker has mentioned.

Professor Pethica: To add to that, you specifically asked about multidisciplinary situations, which are very broad. The process in the Society is, essentially, to increase greatly the number of referees and reviewers. Six or seven would be common, whereas two or three might be the number you would have within a well-defined subject, to try and ensure you get that coverage for a number of broad views.

Q17 Stephen Metcalfe: Rather than the individuals having a broad knowledge, you expand the number within their speciality and they would look at a part of the particular subject.

Professor Pethica: In general, one is obliged to do that simply because there may be a few people who have the vast and broad knowledge required, but in truly interdisciplinary areas, which really span gaps, you have to get a broad perspective and that means using more people, including from a variety of countries, environments and so forth. What we are describing here is a totally international process.

Q18 Stephen Metcalfe: How do you keep that networking, that knowledge of who is in the community, up to date? How is that managed?

Dr Parker: The editors, whether they are internal or external, are out in the community a lot. They are going to conferences, seminars, doing university visits and industry visits. From the RSC, our editors regularly attend up to 200 conferences a year overall. Our external editors will certainly be attending quite a number of conferences in their own subject areas.

Dr Gulley: Again, it is the same for IoP Publishing in that we attend a number of conferences each year. It is about 300 to 400 because of the broad range of subject areas. The editors are encouraged to go along to become part of the community and understand. Equally, we also attract the trends internally as well. So we look at what sort of subject areas are coming through and work with the researchers to look at how we can make sure that the journals represent that as well.

Professor Pethica: One is looking at the process of the reviewers as well as the editor. Of course, one can keep a record of how effective various reviewers are, which is done by most journals. Some people are more effective than others and are used correspondingly. Also one uses the community to suggest future names of reviewers. It is very common, for example, if a senior scientist is asked to review something and they can’t do for whatever reason, for them to suggest other names of people. This is a productive, rapid and efficient way of connecting the network of scientists. Since you have multiple reviewers in most cases, then of course you can test out the reviewers a little and build up a track record on them.

Q19 Stephen Metcalfe: How do you ensure that the whole system is impartial and that bias does not creep in at any point? The Chairman touched upon this issue. Particularly where you have very small groups, who ensures that that is an impartial process?

Dr Parker: It is synoptic, really. You have editorial boards that oversee the quality of the journals. They review the quality of the decisions that have been made and they oversee the content of the journals in a retrospective sense. There is always the possibility with all journals of appealing any decision. Appeals are dealt with very seriously. They are taken to fresh referees. Usually, you try and pick out particularly senior referees who you really respect. You respect all your referees, hopefully, but there are certain senior referees who you would particularly respect. Sometimes they go to the editorial boards.

Q20 Stephen Metcalfe: Dr Parker, do you allow all appeals through? If people don’t like the outcome of the review, can they just keep sending it back?

Dr Parker: All appeals are dealt with, yes. We would always deal with appeals seriously.

Q21 Stephen Metcalfe: For how many times would you allow that process to go on?

Dr Parker: Not for ever.

Professor Pethica: American journals have fixed rules. The Physical Review, for example, has fixed rules about that. They have two layers through which you can go. If you fail at the top, with the editorial board having thrashed it firmly, then the decision would be no, we are not going to take this any further.

Dr Gulley: We have some similar processes. Having a combination of the internal editors as well as the external editors helps with impartiality. There is also the option for appeals, as Dr Parker has said. We also have an external science council that we can call on as well if we need somebody to assess that we have actually followed the procedures correctly. There are other options that we can look at as well.

Q22 Stephen Metcalfe: Obviously, training is a key tool in this. Can you describe the training that the editors, the referees and the reviewers receive? Is that a continuous process? Is there a continuous professional development, not only just keeping up to date, but is that a structured training programme?

Dr Parker: We don’t have any structured training programme for that. You don’t know how often you are going to use a referee. Some referees get used a lot. Some we will use more than 100 times a year, for example. Some you might only use once a year because of their specific subject area, but you look at their results over a period of time and how accurate their responses have been. If referees need any specific feedback, our editorial staff will give that.

Q23 Stephen Metcalfe: You mentioned that earlier. Again, is that structured? Is it a formal process?

Dr Parker: No. It is ad hoc.

Professor Pethica: We should not lose sight of the fact that we have a large scientific community doing things. I alluded to the fact that PhD students, for example, are trained, as part of their learning process, to understand how to criticise and to find out what is right and wrong with the scientific literature. That process is something they go through and it carries on with post-docs.

I wanted to raise one other issue related to this. It is important to recall that the ultimate test is the data at the end. If a journal repeatedly publishes very unadventurous things, it will soon be left behind by those who are rather keener to publish more exciting things. But those who go too far in that direction, of course, run risks, too. It is a question of how you get that balance right.

Dr Gulley: In regard to training, most journals will offer referees guidelines to which they can refer. IoP Publishing has referee reports where we try and guide the referees through some of the things that they should be looking at, such as the quality, if it is correct, and the methodology. Depending again, on the subject area, it is very much tailored to the research area that we are working with. There is that overall feedback from the editorial boards. Again, at conferences we try and run workshops where we can do the training in refereeing. We explain what it is and what is expected. That is internationally as well.

Professor Laskey: From a referee’s point of view, something that I found extremely educational is to be sent back the referee reports of the other referees. There are several times when I have wanted to kick myself for missing something that the publisher spotted that I had not. Equally, it is not uncommon to find that you are in complete agreement. It can be an educational benefit.

Q24 Chair: Is that a standard practice?

Professor Laskey: It varies. Access online to the views of other referees is quite widespread in my own field. The policy, back in the pre-electronic era, was that you were sent the hard copy from some of the better journals, but not all of them did it.

Q25 Chair: What about in other disciplines?

Professor Pethica: It is fairly common in the physics journals, for example, and certainly in the American ones that I have been involved with, that, if there is some dispute or argument, referees will be circulated with the other people’s views. That is most instructive and rather helpful.

Q26 Stephen Metcalfe: It does sound like the system around selection is quite ad hoc at the moment. Do you think that any of it should be more formalised-that there should be a standard set of guidelines around which you work rather than allowing it all to grow around what feels right? You have mentioned about PhD students taking some training in peer review. Do you think that that should be a prerequisite of gaining a PhD rather than something that is nice to have done?

Dr Parker: It could be. Being a referee is often used as one of the criteria for tenure in the US. We deal with a lot of requests from US referees, young academics, wanting a letter of endorsement saying that they have acted as a referee for the RSC and that they have been reasonably good at it. It will help them to gain tenure. It is also used within RSC potentially as part of the criteria for becoming a chartered chemist through working in academia. So there is a CPD-type element included.

Professor Pethica: Broadly speaking-you are referring to the training in general- the only questions which arise around that are, first of all, the international aspect of this. This is a process that is, essentially, identical across all countries. Arranging for something like that is an international exercise. The other issue is the question that I raised before, which depends very strongly on the subject area in which you are working and the process that they learn how to do. I have referred to the extreme of, say, pure maths, and the technology of silicon and biomedical. They have their own areas. Of course, there is a continuum between those things. They are all interlocking and are interconnected. One can image a process in a journal, for example, on a specific subject area, where you could set out rules like that, but of course we are constantly raising the question of the boundaries between these things. It is more important that the training of researchers in general includes the understanding that they should participate in this process in an open way as an expectation of being a good scientist.

Q27 Pamela Nash: From the evidence that we have received so far, it has been claimed that "the peer review system is in crisis", that academics and researchers have increasing burdens on their time and there are few incentives to participate in the peer review system. Can I ask each of you what your opinion is on that?

Professor Laskey: It is subject-dependent. I take a slightly different perspective in answering, which, hopefully, addresses the point, which is that the complexities and the duration of peer review can impede the publication of science if it introduces too many distractions from the principal research programme, but I wouldn’t say it is in crisis. I would say that the engine is misfiring rather than it has stalled completely.

Dr Gulley: From the surveys that have been conducted over the last few years, most researchers have a very high opinion of peer review. In a recent survey that was done by Sense About Science, about 86% of researchers said they enjoyed reviewing and there are benefits to it in that they get to see papers ahead of time and they get to keep up to date.

From the publishing point of view, we can support that by making the process as easy as possible.

Dr Parker: I do not think it is in crisis particularly. One of the challenges is in building up a core of referees in areas that have a huge growth in the output of the subject area. For us, it is in chemistry. It is also the same for Dr Gulley in physics. There is a massive growth in output from China at the moment. We have been working very hard to build up Chinese referees and the quality of the Chinese referees’ reports that we use. Building up the referee base in a linear fashion with respect to their growth is quite difficult at the moment, because the growth in output is quite extraordinary.

Professor Pethica: Our experience is that, as a publishing of scientific literature exercise, it is not a serious problem at the moment. It is possible to find referees in the way we described. It is not a crisis. The point we made in our submission to you in paragraph 15 is, of course, that, inasmuch as peer review is used as a proxy for other kinds of assessment, that can introduce a pressure on it.

Q28 Pamela Nash: Dr Gulley, you mentioned that it would be helpful to make the process of peer reviewing as easy as possible. Do you think that any incentives for reviewers are needed? What would you do to encourage more reviewers? For example, would you advocate payment to reviewers or a formal recognition of any peer review work?

Dr Gulley: We have different ways. Again, it depends on the community. There are different views from the researchers when we talk to them about this. There are different ways of being able to recognise the work that they are doing. For some of the communities, we publish the names of the referees in the journals, for example, and they get recognition. Some journals have different rewards that they give to their top referees, for example. There are different ways of recognising what they do. Coming back to what Dr Parker was saying earlier, another way, particularly within the US, is that we get a lot of requests to support younger researchers in their applications for green cards. There is different recognition that way. There is also recognition partly in being involved in the community. Certainly a strong aspect that comes out when we talk to researchers is that they feel it is something they do to become part of the community and stay involved with the community as well.

Professor Pethica: At the Royal Society the referee is not paid, but we do publish a list of the referees at the end of the year to formally thank them for their input.

Dr Parker: We have asked our boards often about the whole recognition of referees and remuneration. Remuneration would be a difficult thing because, if you gave any realistic payment for the time that is involved, it would be a huge amount of money and it would have to be recovered from somewhere. It is just moving a financial burden around the whole system. The system relies on the benefits that people see from being involved in peer review. There is a quid pro quo as long as you are someone who publishes as well; you are an author as well as a referee, which is not always the case.

There are some other advantages, some of which have been mentioned, like seeing material in advance and, if you do peer review for high quality journals, then you see some high quality work and some less high quality work, of course. You get a chance to be involved in shaping how a subject develops, which is quite a powerful thing to do.

Professor Pethica: We should not forget that this is not by any means the only method by which scientists communicate. The sort of processes that are being described here happen at conferences all the time. Indeed, as important as it ever was is going to talk to somebody about what is actually happening.

Q29 Pamela Nash: Dr Parker, you spoke about young academics approaching the RSC wanting experience of reviewing to further their careers. Do you think that any formal accreditation for the peer review system or a more formal definition of that work would be helpful to them?

Dr Parker: It might be. It would be quite difficult to do, though, because we have about 33,000 referees all around the world that we use routinely. Doing something for that number of people could be quite challenging.

Q30 Pamela Nash: We have heard evidence of some publications using the cascade system to pass submissions between journals. Do any of you have experience of this process?

Dr Parker: We do, yes. We find it does save in peer review time. Authors are often happy to go along that route. It reduces the time to publication if the article is not publishable in their journal of first choice. It gives them a quick route for publishing in what might be a journal of second choice.

Dr Gulley: It is exactly the same for IoP Publishing as well.

Professor Laskey: That does work reasonably well in biomedical sciences, too.

Professor Pethica: Likewise. You will see in many laboratories, for example, in the eastern part of the world that they have a long list on the wall of the journals they want to publish in. They just go down the list until they get to one that publishes your article.

Q31 Pamela Nash: There is unanimous support, then.

Professor Pethica: It works.

Q32 Pamela Nash: Finally, I would like to ask a question about the Research Excellence Framework. How is peer review going to be used as a benchmark of quality? Are there any takers?

Dr Parker: The REF panel is a peer review panel itself, isn’t it? When it was the RAE before, they always said that they would look at the quality of the papers themselves. They would read the papers themselves and wouldn’t rely on the impact factors of the journals in which they had been published. That was stated publicly, at least, in the chemistry area by the chemistry panel. How they are going to be used in REF, if it changes, I don’t know.

Professor Pethica: If one just describes peer reviewing as it is termed in the broad sense as described here as something that is acceptable or not within a certain journal, then, of course, to some extent one is only looking at peer review submissions as being relevant to REF. I think you are asking a slightly different question, which is that peer review for high impact journals, low impact journals and so forth, as a proxy, as I have alluded to before, for assessment of quality, is a slightly more complicated question. It depends on the individual subject area, the journals, whether certain journals have an assessment of what they consider newsworthy and what others consider is needed to build on to the knowledge base and so forth. That depends, to some extent, on individual journals and how they see themselves and their role within transmitting and building scientific information. It does vary. That is a complication.

Professor Laskey: There is also a problem of matching the expertise of the REF panel to the spread of subjects that they have within that subject area. That often means that there is no one on a particular panel who is expert in the exact area of a particular individual being assessed. So there is a genuine problem. You have to accept a certain amount of breadth and imprecise match of the expertise to the area that they are investigating. It is a difficult issue as to whether or not you can do that by assessing bibliometric criteria or not. They are a very shaky basis for such a fundamental decision. This comes right back to Professor Pethica’s earlier point about the proxy use of peer review for other purposes for which it is not perfectly matched.

Q33 Graham Stringer: How can and can’t the peer review process be used to guard against fraud or misconduct by scientists?

Professor Laskey: In the current electronic age it has become much easier to detect data manipulation. Initially, there was a problem that data manipulation itself became much easier because of photoshot-type programs, but in practice many journals now routinely examine the data files to see how the images were prepared. Certainly, in biological sciences that is becoming increasingly common. That makes scientific misconduct more difficult.

Errors of interpretation are still very much things that a good peer reviewer has to sniff out fundamentally. There has been an attempt, with which you are probably familiar, of establishing a research integrity office within the biomedical sciences. That has attempted to look at incidences of misconduct and to draw up a national code of conduct and a national procedure for investigation of misconduct which can run alongside the peer review process. There has been a problem for that organisation in that it was set up to look at biomedical sciences. Research Councils UK has wanted to extend its remit to all sciences. One of the major funding bodies in setting it up was not happy with it being extended to other sciences. That organisation, which could have an important role to play, is caught in the very uncomfortable position between different remits of the bodies that initially funded it. It could make a useful contribution in that subject in addition to the standard peer review process.

Q34 Graham Stringer: Just on that point and we can then come back to the fraud or misconduct issue, do you think the Government should intervene and put it on a statutory basis? The similar office in the States is on a statutory basis.

Professor Laskey: Yes, it is. It is a difficult subject because, if it is done in too draconian a way, it gets into the difficulty that the initial organisation in the States ran into, which led to very substantial criticism in the courts. The current stance of UKRIO is a more balanced one, but whether or not it should be put on a statutory basis across sciences or just retained for the biomedical sciences, which I believe was the wish of the Department of Health representative, I can’t judge. That is a difficult matter. It is something that does deserve to be looked at. It could perform an important role for British science.

Q35 Graham Stringer: Can I go back to the fraud issue? Does somebody else want to contribute on that?

Dr Gulley: I want to add that, fundamentally, that responsibility lies with the author but things can be done to help this situation, particularly on the international setting. For example, we have ethical policies. Most journals have an ethical policy that they will promote and ask authors to abide by.

Q36 Graham Stringer: Should it be mandatory to have an ethical policy?

Dr Gulley: In certain subject areas there are parts that are mandatory, such as stating a conflict of interest and certain medical procedures that have to be stated. The ethical policy that we have is much more general. It also takes into account, for example, what is ethical or what is viewed as being ethical in terms of plagiarism, for example, which is one aspect that I wouldn’t say has become easier but it is being picked up more frequently now that you have much more electronic access availability.

Dr Parker: There are also the Committee on Publication Ethics Guidelines that are pretty much an industry standard now. The difficulty with fraud is that the whole peer review system relies on people being ethical. That has to be balanced with what happens when you find that someone has not been ethical. In the relatively rare cases when someone has not been ethical, it will usually be picked up by a reader. If it is not picked up by a referee, if it actually gets through to publication, it should be picked up by a reader and then it is usually dealt with either by the reader coming to the editor of the journal or the reader going directly to the author and dealing with the matter.

Q37 Graham Stringer: The most recent fraud-related case is that of Andrew Wakefield, which took 10 years to sort out, even though the journal in which the article had been published had been approached after two or three years showing that there had been bioselection and that some of the figures had been altered. Why did that take so long? Should the process be altered in view of that experience?

Professor Laskey: I don’t know the details of why it took so long. That is not something I am competent to answer. It does illustrate a concern that we expressed in the written submission about it often being more difficult to firmly dismiss incorrect information in the biomedical sciences. That is a further reason why peer review is crucially important. How that can be addressed more rapidly is hard to know. It becomes particularly difficult once the media are involved and everything is scrutinised openly.

I believe there is a tendency, unfortunately, for people to be more reluctant to come forward and speak openly when they think it is something of a major public issue. That, I accept, is not the position that we prefer to see, but there is a tendency of people to be reluctant to enter a public storm.

Professor Pethica: I can give you an example from the physical sciences, which is rather more obvious, which is the case of Schön and the proposals he had for various solid state structures, which, of course, caused a great storm of excitement until people started to do the kind of analysis of the data that had been described, and it all fell apart. That took about a year and a half before people were convinced that it was a fraudulent process which people had tried to repeat.

Here we are faced with a number of diversion paths. One is the question of ethics that institutions, indeed, should have. Many of the research institutions, be it the universities, national laboratories or whatever, do have expected ethical behaviour of their staff. Then there is the question of whether you will catch it by peer review. Of course, the peer review process is designed to try and catch these things, but, by the nature of things being imperfect, something will eventually get through. The numbers are fairly small, as we have seen. Then the question is what happens after that? To some extent, that will depend on the nature of the subject, the complexity and so forth. In a way, it is inevitable that a scientific fraud will eventually be uncovered, as we have seen in other cases. The question is: how do we shorten the time scale and prevent these things getting out before they cause media damage?

Q38 Graham Stringer: Do you think there should be a code? Just using the Andrew Wakefield case-I am sorry if you are not familiar with it in great detail-it was a journalist who was pushing the issue that there had been a fraud. He went to the journal and the journal, effectively, got the co-author to review what was going on and excluded the journalist. Should there be a code of ethics or a process for dealing with external complaints where fraud is suspected with at least some evidence?

Professor Laskey: The UKRIO is attempting to achieve that. It is attempting to provide a first point of call for people who seek advice on how to proceed in examples of suspicion of fraud. It has drawn up a national procedure which has been widely published and distributed to universities and other research institutions. Again, the very nature of fraud is that it is inherently difficult to prove that it has occurred.

Dr Parker: It is something that is also very subject-dependent. The Wakefield fraud relied on clinical trials and statistics. You can understand why that might take a bit longer compared with something in the physical sciences area that could be repeated by someone else relatively quickly and might be right or wrong. It is a different process.

Q39 Graham Stringer: You have given a fairly dry account of what peer review is like. It sounds unexciting. The insight we got into peer review from the leaked e-mails at the university of East Anglia made it look like a pretty tough contact sport where people were taken out on journals and careers were threatened. What is the accurate scenario? Is it the fairly desiccated view that you have been giving us or is peer review a street fighting business where careers are threatened? Where is the better insight?

Dr Gulley: From my experience, it is probably closer to what we have described so far. There are instances where you do get the street fight-type scenario, but that has been very rare in my experience over the last 14 years.

Professor Laskey: I think the rather dry flavour that we have left you with is probably a more accurate description of the majority of cases. There are a minority of exceptions.

Dr Parker: I am sorry to have to agree with that. Yes, it is a rather dry subject, but exactly the number of cases you get that have a big and florid excitement about them are relatively small.

Q40 Stephen Mosley: Does the publication of fraudulent or incorrect papers that have been through the peer review process damage the public perception of peer review as a mark of quality?

Professor Laskey: It damages the public perception of science as a whole and I think that is extremely unfortunate.

Professor Pethica: If a particular journal does that kind of thing, it affects that journal’s reputation within the scientific community, which is a significant matter too.

Dr Parker: I doubt that the general public has much of a perception of peer review. They have a perception of science, that scientists do experiments and that they publish them. The probably don’t really care that much about peer review, although the Wakefield incident and the UEA climate data issue have brought peer review a bit more to the fore.

Professor Pethica: Not to entirely leave it as unexciting, if a lot is at stake, then the peer review process will tend to be more exciting.

Q41 Stephen Mosley: You can also have a situation where a peer reviewed article may disagree with a previously published paper, and that is perfectly legitimate. If you have a situation where there is some perception of doubt against the peer process, I guess it makes it difficult to judge whether this is a proper result or not. Is that the situation? Are you more wary of research that contradicts previous research now than you might have been previously?

Professor Pethica: If you divide it by subject areas, the paper that comes along and tells you that they think thermodynamics is wrong is not likely to get much of a listening. There are such papers, I should stress, that still come in. At the other end, there are problems that are sufficiently broad where the information, the types of experiments and so forth are not sufficiently defined where it is rather difficult to be sure. There is a continuum of those things.

Dr Gulley: On the other side, you also have areas where it is still evolving and different models are going forward. For example, modelling of the universe is a good example of that, where you will have contradictory models that will evolve until you start to get some of the data that can back up some of the theoretical claims. That is part of science discussion.

Professor Pethica: That is what it is.

Q42 Stephen Mosley: When it comes to public perception and to the perception by policy makers, how do you think the perception of peer review and scientific research can be improved?

Professor Pethica: Perhaps I could venture a comment. Peer review has worked in the sense that the scientific literature we have is coherent and it has effects on the world around us which everybody can see. As to the notion that it is a substitute for getting things absolutely right every time, it would be useful if the public becomes aware of the fact that mistakes happen. It is just that we try and minimise their frequency.

Q43 Pamela Nash: I would like to move on to international issues regarding the peer review system. Are there any perceived differences in the quality of peer review dependent on the country where the publisher of a journal is based?

Professor Pethica: I don’t know, basically, to cut a long story short.

Professor Laskey: There are serious attempts to minimise those differences.

Q44 Pamela Nash: Do you see that there are differences to be minimised?

Professor Laskey: No. The harmony outweighs the differences.

Professor Pethica: More than that, it is becoming more coherent. In the past, foreign academies would have certain rules. For example, papers had to be submitted for approval by certain structures. It is unquestionably the case that international competition in this sense, for a consequence of impact of the science and technology, has driven a convergence. It is hard to say that there is any real detectable change. That is, of course, enforced by the fact that journal reviewers themselves are now drawn from across the world, be the journal UK-based, Chinese, Brazilian, in the US or whatever. The process is essentially the same and they all participate internationally in that process.

Q45 Pamela Nash: You mentioned new technology in use in the peer review system. To what extent do different publishing organisations share best practice in terms of using new technology and online systems?

Professor Pethica: There are standard IT packages now.

Dr Parker: A lot of publishers use the same or very similar packages. Publishers collaborate on various things like linking references, but CrossRef as a collaborative publishing group of publishers is also working on anti-plagiarism software and things like that. There are things that are shared across.

Q46 Pamela Nash: How effective are these tools? You have mentioned the anti-plagiarism tools. How is that being developed?

Dr Parker: That is being developed at the moment.

Professor Pethica: It is fair to say, of course, that IT technology advance is a constant battle. People on one side are doing cunning things and on the other side they are advancing the technology. So anti-plagiarism works for certain kinds of things, but it is an arms race, almost, if you like.

Q47 Pamela Nash: Is the technology at a level yet that it is benefiting the peer review system or is it more of a hindrance in that the technology has not caught up yet?

Dr Gulley: My experience on the plagiarism side of things is that we are finding that we are picking things up more before they go out to the referees, for example. It is minimising the burden, so it is not left to the referees to pick up on this. When articles are submitted to us, we can check them against what has already been published. It is definitely helping in that respect.

Professor Laskey: In data manipulation, the software is now picking up cases. You rarely hear about those because the journal simply declines to deal with that author in future. There are cases of data manipulation being detected by software.

Dr Parker: We have done quite a lot of work on, essentially, running macros on the articles that we are going to publish which check the experimental data for consistency. It is the technical detail. You can check that the spectra and the data are consistent with the number of hydrogen atoms that are in the molecule that you have in the reported structure. We do things like that which help to pick things out.

Q48 Pamela Nash: I will go back to my original question to look at review internationally from a different angle. Are you aware of any differences in the quality of peer review carried out through UK-based journals by reviewers from different areas of the world? I realise that this might be a bit of a sensitive question.

Dr Parker: Publishing is so globalised now that there are very few journals that are based within a particular nation or are very isolated. Most of them are globalised. There is very little difference in quality.

Professor Pethica: We referred earlier on to the fact that you look at the review referee’s performance. Of course, I can’t comment on specific cases but that is a factor.

Q49 Pamela Nash: I am not asking just about the skills of reviewers from another country but perhaps the facilities and funding that is available to them in the UK. Do we take that into consideration or should we take that into consideration in choosing reviewers?

Dr Parker: That is an interesting question, isn’t it? There have to be sufficient experimental data in the area that we publish in to justify the conclusions that are being drawn. If there are sufficient experimental data, that is okay. The difficulty comes where, if there is someone working in a developing country somewhere, they don’t have access to specific sorts of technology that would give them definitive experimental data. One of the things that we are able to do as a society is to try and work with people in those areas, to try and develop sustainable clinical research, which we do, for example, in sub-Saharan Africa.

Q50 Chair: You might have some views working in a country with incredibly poor facilities, but there is a glimmer of something special in that person’s work. If you just judged everything by the standards of access to laboratory facilities that you take for granted, doesn’t that squeeze that group of people out of the publishing world?

Dr Parker: Sometimes it is an advantage because you will get referees offering to work with people and it can set up collaborations. If you have a referee who really sees the merit in this bit of research that has come out from someone working under very difficult circumstances, they could offer to set up a collaboration potentially. Those very often are supported through societies as well. They sought collaborations.

Professor Pethica: It is fair to say that that is one of the reasons why people in those circumstances are often involved in refereeing for highly theoretical subjects where that disadvantage does not apply, and that is certainly widely used in mathematical and theoretical areas.

Q51 Pamela Nash: Dr Parker, you mentioned earlier about the growth of scientific research in China. How do you support and develop peer review skills in China and other emerging regions of scientific strength?

Dr Parker: We do a lot of interaction with the Chinese academic market, as it is. We have two offices in China-one in Beijing and another in Shanghai. We have staff out in China. We do regular visits. We set up conferences in China now. We started off doing roadshows of the top chemistry departments in China. All of our roadshows include presentations on how to publish and how to referee. We have built up quite a significant connection with the Chinese academic market. We also involve them on our editorial boards. We get them involved as associate editors on our journals.

Dr Gulley: It is exactly the same for us as well. We have been working with researchers in China for the past 11 years. We have a member of staff who visits universities and gives lectures on how to get published. We run workshops and we visit regularly. Again, we build up those links and liaisons. We also work with a number of partners in China publishing journals. We certainly work closely with them on looking at peer review and internationalising their journals as well.

Professor Pethica: As you might expect, we work closely with the Chinese Academy of Sciences on this.

Q52 Chair: To what extent do you share best practice amongst publishing organisations, particularly in terms of evolving software and so on?

Dr Parker: There are trade associations for publishing. Publishers get together at those trade associations and at events like the Frankfurt Book Fair. They share non-competitive knowledge as much as possible. Publishers are really quite collaborative these days, much more so than they used to be 20 years ago. They work together a lot more on common issues like anti-plagiarism, reference linking and those sorts of things.

Q53 Chair: It is through the trade fairs and conferences.

Dr Parker: Yes.

Dr Gulley: There are some shared guidelines and recommendations that come out of these discussions.

Q54 Chair: My next question is a slightly amusing example that occurred to me when I was congratulated for getting on the Booker shortlist for publishing Oxygen, but it was another Andrew Miller. How big a problem is ambiguity of names? Do you use systems like ORCID to help track authors?

Dr Parker: We are trying to work with ORCID at the moment. That is a developing situation. There will be an author tracking ability in a relatively short time. It is an issue, particularly in places like Korea, where there are only five or six really common surnames. You get an awful lot of people with the same name. For example, we had two people with the same name both in the chemistry department at the university of Oxford. They both had very much UK names. We try very much to keep those people and their records separate. A bigger problem is proliferation of records by the same person. It can be an issue.

Dr Gulley: ORCID is a very good example of the collaboration, where it is required, and it will be a solution to that problem.

Q55 Stephen Metcalfe: Can I ask for your views on post-publication peer review and commenting, whether any of your journals do that and what your experiences are?

Dr Gulley: Currently none of our journals do that. There are experiments within the industry that are trialling this. It will be interesting to see how they progress.

Dr Parker: We don’t do it. It is another layer. It is something in addition to pre-publication peer review. Where there is an issue, you should hear pretty quickly from readers or whoever, anyway, so it is a way of opening that up, I suppose, more generally speaking.

Professor Pethica: We do use it. We also have a system that is called eLetters. Is it useful? Not really, because remarkably few people choose to use it. It is important to keep in mind that this implies that, once something is published in the peer review literature, that is it and it is set in stone. As I alluded to before, much of a PhD student’s training is the process of assimilating over a long period of time the scientific literature, deciding what is good about it and what is bad about it and then allowing them to progress from there. This process is inherent in the entire scientific enterprise, anyway, in the training of people getting involved therein. Most PhD students, for at least a year or a year and a half, try to figure out which way is up in the scientific literature, which is that process.

Q56 Stephen Metcalfe: So you don’t see it growing as a trend?

Professor Pethica: It is already a central part of that enterprise.

Stephen Metcalfe: It is already there.

Professor Pethica: It is nice to have it. It is implicit in the fact that people publish subsequent papers saying, "X was right, Y was wrong, and we did this and produced that." That is implicit in the whole structure of scientific papers and there is a preamble about what has happened so far. To some extent it exists already.

Q57 Stephen Metcalfe: It is not going to change the value of the pre-publication review. It is not going to take away from that because it already exists.

Professor Pethica: In post-publication terms, it is, effectively, the process. That is why at this point scientific literature is supposed to be a coherent structure rather than a series of random samples.

Professor Laskey: In biomedical sciences, the Faculty of 1000 does provide a post-publication assessment of the value of papers, and, if there is a move towards publication in journals such as PLoS ONE and where impact is less important, then a subsequent impact assessment such as the Faculty of 1000 could become increasingly important.

Q58 Stephen Metcalfe: Is the Faculty of 1000 welcomed by the academic community? Is it well supported?

Professor Laskey: I think so. Its use is patchy but it is recognised as providing a valuable service.

Q59 Stephen Metcalfe: Has social media, by which I mean blogs, etcetera, had an impact on this process at all? Are they helpful, or is it just a proliferation of unchecked views?

Professor Pethica: There will be a change of view depending upon the age of the person to whom you are asking that question. With the research students it is quite common. As one gets somewhat older, the utilisation is probably less.

Chair: It is the same in this building.

Dr Parker: People are relatively reluctant to blog on things at the moment, but they like to see what other people are reading. If there is some way of seeing what other people find interesting, that is where the Faculty of 1000 comes in. It is a positive thing. Everybody wants to be read by the best people.

Q60 Stephen Metcalfe: So you don’t see that as having a significant impact at the moment.

Dr Parker: Not at the moment.

Dr Gulley: It can add to something in the future. It is also an aspect that people are starting to explore about how they explain their science as well to a much broader and more general audience.

Q61 Chair: Aren’t blogs used to help promote a piece of work that is being published?

Professor Pethica: Yes.

Chair: We find that modern technology is a very useful way of getting out to the broader scientific community what we are doing.

Professor Pethica: It works.

Dr Parker: We use blogs for trying to promote particular articles that we think would be newsworthy or interesting to a wider audience. Some are more successful than others.

Dr Gulley: It is certainly a way to raise visibility. Again, for some articles, it is more successful than others.

Professor Pethica: The challenge is making it a two-way process, though.

Chair: Graham described peer review as a dry subject. As I said to you at the beginning, some of my colleagues were up into the early hours, but you have kept us awake and interested. Thank you very much for an informative session.