Evidence Check 1: Early Literacy Interventions - Science and Technology Committee Contents


Examination of Witnesses (Questions 1-)

PROFESSOR BOB SLAVIN, JEAN GROSS AND PROFESSOR GREG BROOKS

4 NOVEMBER 2009

  Q1 Chairman: Could I welcome everyone to this session in part of our work as a Committee looking at the evidence behind various elements of government policy. That, principally, is what we are trying to get at this morning and this particular session is really looking at the evidence behind literacy interventions. The Government has spent a very significant amount of money trying to improve literacy standards in English schools and we are anxious to look at the evidence behind that. On our first panel is Professor Bob Slavin—welcome to you, Professor Slavin—the Director of the Institute for Effective Education at the University of York and Director of the Centre for Research and Reform in Education at the Johns Hopkins University, Jean Gross, the Director of Every Child a Chance Trust—welcome to you—and Professor Greg Brooks, the Research Director of the Sheffield arm of the National Research and Development Centre at the University of Sheffield. If you three do not know the answers, then we are lost! I wonder if I could start with you first, Professor Slavin. Early literacy interventions, we are told, are so important for children who are struggling to learn to read. Is that so and why?

  Professor Slavin: Just as background, we did a huge synthesis of research on programmes for struggling readers and looked at every imaginable kind of thing that people have proposed and at studies done in all different countries, and what was remarkable was the degree to which there is a variety of approaches that make a quite substantial difference in the reading of young children. I think it is absolutely clear that a very large proportion of children who are struggling to learn to read can be successful, and perhaps the most important conclusion that I would take away from all of this is that we can argue about what are the most effective ways, what are the most cost-effective ways to bring about this result, but it is quite clear that we can get much, much better results in reading for children who are struggling in reading than we do in ordinary circumstances.

  Q2  Chairman: When you say you looked at all the evidence that was available, what does that mean?

  Professor Slavin: We did what we call a best evidence synthesis, which is a form of something called meta-analysis, where we applied a consistent set of standards to all of the studies. There had to be a comparison of an experimental and a control group over at least 12 weeks, with measures that were not inherent to the treatment—well matched and some other technical requirements—and then computed what is called an effect size, which is the percent of a standard deviation by which the experimental group exceeded the control group on whatever measures were being used. We found 98 studies that met our criteria across all different kinds of interventions and then put them into categories to look at what were the average effects of these different programmes.

  Q3  Chairman: So, in your view, the evidence is there to say early intervention works?

  Professor Slavin: I think without any doubt.

  Q4  Chairman: Jean, do you support that?

  Jean Gross: Yes, I do. I support that completely.

  Q5  Chairman: Professor Brooks?

  Professor Brooks: Yes, it is better if they come in early, when children are first identified as struggling, but there are also programmes for those who are picked up rather later.

  Q6  Chairman: Do you feel, in terms of the Government's policy in this area, that they did take into account that evidence?

  Professor Brooks: I am sorry; did you say they need to or they do?

  Q7  Chairman: Do you feel that they did do; that they took into account the available evidence?

  Professor Brooks: Not all of it, in my view. There are some programmes which have been researched on quite a detailed scale several times, there are others for which there is promising evidence that, in my view, would justify investigating those programmes in more detail, some of which, I think, are struggling to get heard and to get funding for more detailed investigations.

  Q8  Chairman: Do you think, for instance, that phonics is the best way to teach children to read; that that evidence is there?

  Professor Brooks: That would be to put it too simply. I think the evidence on initial teaching of reading and spelling is that systematic teaching of phonics is essential but it needs to be part of a broad and rich language and literacy curriculum to work best.

  Q9  Chairman: Jean, do you feel phonics is essential in terms of teaching children to read?

  Jean Gross: Yes. All the international evidence shows that it is essential in teaching children to read. I would agree with Greg also and, I think, also Bob. Bob's review shows it needs to be embedded in applying that phonic knowledge into reading books in a rich curriculum where you are also developing your oral language skills. Just phonics alone will not meet the wide reading needs of children, but it is the bedrock.

  Q10  Chairman: Would you explain to me the difference between phonics and synthetic phonics?

  Professor Brooks: Synthetic phonics is one variety of that. Phonics is teaching based on the relationships between the sounds of the spoken language and the letters and letter combinations of the written language. Synthetic phonics is a variety in which children are taught to sound out the letters to produce phonemes (sounds) and then to blend them together to produce a whole word sound for reading. The other major form of phonics that has been given the most attention is called analytic phonics, and in that, in its strict form, sounding out is banned and children are taught to try to infer the relationship between letters and sounds by studying families of words.

  Q11  Chairman: Professor Slavin, in terms of phonics, you would also accept that it is essential, in terms of teaching children to read, that you decode using phonics?

  Professor Slavin: Yes, and I would fully agree that of the forms of phonics synthetic phonics is the one that has the strongest support.

  Q12  Chairman: Is there any evidence to support that?

  Professor Slavin: Yes, there is great deal of evidence of all different kinds from laboratory studies to multi-year investigations. I think in the world of reading that is virtually a settled issue at this point.

  Q13  Chairman: What worries us as a Committee is what does "evidence" mean. In terms of evidence most scientists would say you have to have controlled trials, so that you actually compare two different approaches, and you do that in a systematic way. Do we have that evidence, in terms of the use of phonics? What happens if children do not encounter phonics at all?

  Professor Slavin: There are many children who learn to read regardless of the reading approach.

  Q14  Chairman: Precisely.

  Professor Slavin: So you cannot say that phonics is essential for every child, or systematic phonics, taught in school. There are children who do infer the phonetic principle or who are taught at home, or God knows how they learn it, but there is a very large group (let us say, a large minority) of children for whom phonics really is make or break, for whom if they have systematic phonics they will be successful readers, they will never become struggling readers, they will not come to the attention of the authorities in any way, and without it they are much more likely to run into trouble.

  Q15  Chairman: Jean, if you have children who are struggling, particularly in the early years of primary school, Key Stage 1, and phonics is not working, perhaps it is not working because they have got dyslexia and, therefore, phonics does not make any great sense to them. Are there some children with, for instance, conditions like dyslexia (and we will come on to dyslexia as a particular problem in the future) who cannot read because of other conditions and, therefore, phonics is pretty useless to them?

  Jean Gross: The core cognitive difficulty in the brain for children who struggle with reading, all research shows, is a difficulty in hearing and separating the sounds of spoken words in your brain (not in your ears). It is breaking words into sounds and blending them back together. We all agree that that is the core difficulty for children who struggle. There are other difficulties in their lives, but that is the core difficulty. There is perhaps a difference of view of whether that means that you should continue to do more phonological awareness work to attack the difficulty or whether you should go round it and do other methods. My own reading of the research is that in the end you have got to get children phonologically skilled if they are going to be able to spell lifelong and be able to continue to develop their reading. So you have to get there, but, I think, for some children you do have to go down a little bit of a circuitous route to start with. You might teach phonics in different ways.

  Q16  Chairman: That is the skill of the teacher.

  Jean Gross: That is the skill of the teacher. You might have to teach the children other things such as to visually recognise the difference between letters. I think what I would say from the evidence is that there is not one programme or method that will work for every child; you do need to tailor it for the lowest achieving children with the greatest difficulties.

  Q17  Chairman: You all appear to agree, and I would like an affirmative answer, that early intervention and the use of phonics, in whatever form, is really important to developing good readers and the Government is right to concentrate on those issues.

  Jean Gross: Yes.

  Professor Brooks: Yes.

  Professor Slavin: Yes.

  Chairman: All three said, "Yes." Over to you, Tim.

  Q18  Mr Boswell: Thank you. Perhaps I can preface my remarks by saying that I had for some time ministerial responsibility for adult literacy at a time when my wife was actually tutoring the subject, so I took a certain interest in that end particularly. I am interested really in the evidence base for literacy intervention. Professor Brooks, I know you did a study some two years ago, a review of all the available literature. Perhaps I might lead with you but invite the others to comment as well. I suppose, in a way, we are looking for a kind of platonic ideal for a literacy study which would convince everyone that this is what we want to do, and I really would like to ask you first if there have been any studies that meet those rather platonic criteria for an ideal literacy intervention study?

  Professor Brooks: Are we talking about initial teaching or are we talking interventions for children who do not get it first time?

  Q19  Mr Boswell: I have not differentiated it. It might be useful, therefore, if you could tell the Committee just a little bit about both.

  Professor Brooks: On the evidence for the initial teaching of reading and spelling there is a set of a dozen or so randomised control trials that show pretty clearly that systematic attention to phonics at that stage enables many children to make better progress than if they do not get systematic phonics at that stage.

  Q20  Mr Boswell: That, in fact, informs the common view you have expressed.

  Professor Brooks: We have said that to an extent. If we are then talking about children who do not get it the first time and, therefore, need to have extra attention at around about the age of six or seven, the evidence there is that there is a range of programmes that will help. As you imply, they are summarised in this door stop of a report of mine. There is good evidence that many of those that are within that set which are phonologically based—in other words they have an element of phonics in them or they have a strong element phonic in them—work very well. There are others that have a broader approach, like Reading Recovery. Jean will be able to speak in more detail about that. That has some very strong evidence behind it as well. The essential thing at that stage, I think, is to try and match the intervention to the child and his or her needs, and that requires very well trained teachers who know a range of interventions, I think, and know how to tailor the particular intervention to the child.

  Q21  Mr Boswell: That is helpful. Given that you have just said, in effect, that there is a huge responsibility on the individual teacher knowing the individual pupil and maybe trying things out. Clearly, if you are looking at a literacy intervention trial (and we are interested in the evidence base for that), you are looking at an aggregated number or series of approaches, and if we can bring it to what you might call the second wave, although I am not using that technically—those who had not got it first time, as you called them—what are the main problems with trials? We are asking for evidence. It is social sciences; it is not narrowly definable physical sciences. Is it the control groups? Is it assessment methods or data? Where are the problems in tying down what works best?

  Professor Brooks: In my view, the evidence base for that second wave type intervention is less strong than it is for initial teaching. There are precious few randomised control trials in the field, at least in this country. Bob will know better about the evidence from the rest of the world, but when I was doing this study, out of the 121 pieces of evidence that I was able to amass, only nine were randomised control trials, and some of those were so small as to be hardly worth carrying out.

  Q22  Mr Boswell: Does that disturb you, in the sense that we really do not know what we are claiming we know?

  Professor Brooks: We could do with much stronger evidence at that stage for quite a few of the interventions, yes.

  Q23  Mr Boswell: You are showing assent to that, are you, Professor Slavin?

  Professor Slavin: Yes. We found just the same pattern, in the sense that there were very few even quasi experimental (which means matched) studies that took place in the UK, but if you looked at the whole world literature, most of which is US, there is still not as much as one might like but quite a robust set of evidence about what works for the struggling readers.

  Mr Boswell: Thank you.

  Chairman: Can we just ask why?

  Q24  Mr Boswell: Do you mean why do these trials not exist or why could they not be conducted?

  Professor Slavin: They absolutely could be conducted. I think that there is a lack of funding for that kind of research in the UK and, having worked in both places, I think quite a striking difference in terms of the amount of resource that is available for doing these large scale randomised experiments, but they can be done. We have done them in the UK as well.

  Jean Gross: I have been involved in two programmes, Every Child a Reader and Every Child Counts, which is trying to do the same thing for numeracy. We have learned, and I think the Government is more willing to commit resource to randomised control trials. For Every Child a Reader we had a quasi experimental matched group study. Looking back, I wish at the time we had been able to find money to do a randomised control trial, but Every Child Counts is having a £250,000 government-funded randomised control trial, so I think things are moving on, but it is a lot of money, and if the recommendation in the system is that we want more randomised control trials, that is going to cost budgets at a time when budgets are tight.

  Q25  Mr Boswell: In a sense you are saying that policy about early intervention proceeds in parallel with any systematic randomised control trial; it is not tied up before you start.

  Professor Brooks: Things in education are rarely tied up before you start, but, yes, as I have already said, we could do with stronger evidence. I think it gets difficult to get money for randomised control trials because you need to have done various pilot studies beforehand to show that your intervention works, at least for pilot groups, and that you need to do a large field trial before you commit the money to a very large and complicated expensive piece of research, at least expensive in educational terms, which might give you a null result in the end. That is always the risk.

  Q26  Mr Boswell: You mean no outcome?

  Professor Brooks: No, not no outcome. A null result is not no outcome. What I mean is a finding of no difference between the conditions. That will disappoint some researchers. I actually like those because they are contributions to knowledge anyway.

  Q27  Mr Boswell: Can I just shift to a slightly separate issue. We have already touched on dyslexia, but there are also developmental differences between children, as I know myself. They are not necessarily, as it were, pathological or educationally pathological; they may just be people that grow up faster. The NUT has expressed some concern in its evidence about the Government's concept of age appropriate expectations. How do we pick our way through that? Do you think we are paying enough attention to the importance of differential rates of development and does the Government's concept adequately take account of that difference?

  Professor Brooks: Can I start with part of the answer to that? This sounds perilously close to the idea that there are some children who are late developers in literacy and that if you leave them alone they will catch up. I can tell you that contained in here is a lot of evidence that that is a myth. If you just leave children or leave them with the ordinary classroom curriculum, on the whole, they do not catch up, they need the interventions.

  Q28  Chairman: Is that because they start too early, Professor Brooks, in school?

  Professor Brooks: Start school too early? Ah, now we are getting into highly contested territory.

  Q29  Chairman: It is just following on from Tim Boswell's question really.

  Professor Brooks: I agree. I think we induct children into formal school too young in this country. I think there is a case for having a much more play-based pre-school phase that would last from age three to age six in which there would be very little or no formal teaching of literacy and that would start at six. I think at that point most children would get it first time, as they do in other countries where they start at that age, or, indeed, slightly later, such as Finland, and it would not lead to larger numbers of children not getting it first time.

  Q30  Chairman: That would save us a lot of money on Reading Recovery!

  Professor Brooks: Well, not necessarily.

  Jean Gross: If that was to happen, I think there needs to be some good research beforehand. One of the things I am struck by is the fact that, whilst you have a play-based curriculum for all children, those parents who are more affluent are very busy doing magnetic letters on the fridge and reading to their children and actually will be teaching them to read at home, because you cannot stop them, whereas for those from disadvantaged homes that will not happen. There is a risk that just needs to be examined with evidence. Would a later start increase the gap between more affluent children and disadvantaged children?

  Chairman: I am sorry I opened that up. My apologies. It was too good an opportunity.

  Mr Boswell: That is helpful.

  Q31  Dr Harris: Ms Gross, when you were asked about phonics you said "all the evidence". When you use a term like that, does that mean it is equivalent to the earth being round? There is no evidence out there that disagrees on phonics at all, or is that just a term you tend to use to say, "I think, the majority of evidence"?

  Jean Gross: I think it means that research that does overviews of multiple studies, some of which might show "not", some of which might show "yes", some of which might show neither, but overall the conclusion from those high level analyses is that phonics is essential, so it is that level.

  Q32  Dr Harris: So it is not all the evidence, is it? It is a review.

  Jean Gross: Yes.

  Q33  Dr Harris: A good review shows that there is good evidence, or the best evidence, or "it is convincing that". That is different from "all the evidence", is it not?

  Jean Gross: Yes, it is different; so I apologise if that is incorrect.

  Q34  Dr Harris: You also said "everyone agrees" and "all research" in another answer. Do you think it is wise to use terms like that, because people who disagree think that that is unfair, even if they recognise they are in the minority?

  Professor Brooks: What tends to happen then is that the people who oppose the policy or the findings try to cherry-pick the bits that agree with their point of view and systematically ignore the weight of the evidence. What we are saying here is that the weight of evidence is in favour of phonics for all children at the start and for many of those who need early interventions.

  Q35  Dr Harris: Would you all agree it is better to use terms like "the weight of evidence" rather than "all"?

  Jean Gross: I would agree. I apologise and correct what I said to "the weight of the evidence".

  Q36  Graham Stringer: The Reading Recovery programme is the backbone of the Government's strategy with reading. I just want to ask a number of questions about how we got to it being the backbone, what the evidence was for that, and how it is assessed: because we have had some recent evidence saying that it is probably not the best strategy. First of all, if we can go back to your previous answers, it was chosen after a pilot scheme which did not have a control group. That is right, is it not?

  Jean Gross: The process of choosing it, and I was involved in giving evidence to the Government that perhaps contributed to them choosing it from outside—I was not part of the Government; I am employed by the KPMG business—I believe, involved looking at existing international evidence. There was an awful lot about Reading Recovery before the 2005-08 Every Child a Reader pilot in England. So it was not that it was not starting from nothing. I believe they looked at that evidence—and I certainly supplied that evidence to them—and then, on top of that, they had two sources of evidence, one of which is what I would call management data, the data that is routinely gathered on an international database for every child who goes through Reading Recovery, with a teacher entering data at the beginning of their programme and a different teacher testing the child at the end of their programme and entering the data. So that is management data. The third element of evidence was a quasi experimental matched study, which Bob or Greg can comment on, which was not a randomised control trial but would be recognised, I believe, as a reasonable standard of evidence, the randomised control studies being absolutely the gold standard but quasi experimental being widely accepted in overviews of research as being includable and valuable.

  Q37  Graham Stringer: So it would not be fair to characterise it, as some of the evidence we have had has characterised it, as you put more effort into teaching children to read so more of them lead to a higher standard and there is no control against other methods of teaching them to read?

  Jean Gross: No, the study that the Government used was in relation to 21 London schools with Reading Recovery and 21 without. Those other 21 schools were using a range of other interventions for children aged six. They were doing things that schools normally do, programmes like Early Literacy Support and various other things; they were not doing nothing. They were compared with Reading Recovery, the groups were matched, and the effect size was large, using independent tests.

  Q38  Graham Stringer: One of the points you made earlier in the evidence was the cost of randomised control groups. What is the cost of the Reading Recovery programme per child? It is pretty high, is it not?

  Jean Gross: £2,600 per child is the average cost (once) of providing it, and the cost of not providing it is £50,000 per child.

  Q39  Graham Stringer: I understand the cost of illiteracy. The randomised control groups: you are saying a quarter of a million is not available here. That is 100 interventions. We are talking relatively small amounts of money.

  Jean Gross: Yes.

  Q40  Graham Stringer: That just seems to contradict the evidence that you were giving earlier.

  Jean Gross: I simply said that that money has been made available for Every Child Counts. If you multiply that for all the many things that would need to be researched in the education and children sphere, you are looking at a very big bill at a time of economic downturn. I am not saying it should not happen; I am merely posing that as a potential problem for any government.

  Q41  Graham Stringer: Can you explain to the Committee how the Reading Recovery process is assessed and tested?

  Jean Gross: At the start children are, first of all, put forward by their class teachers as being the very lowest achieving children in a school year one class. They are children who have made no progress in reading after a year, or more than year, at school. The data shows that in England the children entering Reading Recovery do not score on a standardised reading test at all; they are at the floor of the test; so they have a reading age well below five and they have had a year's, or more, instruction. So they are put forward as the bottom children. Teachers normally put forward eight to 12 children, and then they all have an individual assessment that includes the British Ability Scales word reading test—a test of single word reading—a test of letter knowledge, tests of how many words they can write. There are a number of different assessments. From that the very lowest achieving children are selected to have the teaching first, assuming that over the year the teacher will be able to teach four children at once. At any one time the teacher teaches four children. They are usually employed on a half-time basis to do this work. When they have finished with those children in year, they will take on another four children. So you start with the very lowest, with no exceptions. I think this is important. You do not rule out children because of any particular special needs, an issue at home or poor attendance. They start and then they are discharged from the programme when they have reached a certain book level, a level of difficulty of book, which gives you an indication that they are ready to be discharged, if you like. They are then given a repeat of the same test that they had at the beginning by a different teacher, a teacher who did not teach the child in Reading Recovery, and the results are then entered on an international database. So you get a gain in reading age, in months of gain of reading age, for every child and a number of other more technical measures.

  Q42  Graham Stringer: Can I read you a criticism, again, in the written evidence we have had, of the assessment methods for Reading Recovery? It says, "It is very costly but Reading Recovery research is notorious for misrepresenting data. In a recent publication by the Institute of Education the same problem appears. Nearly half of the children from 145 strong RR tutoring group were dropped from the study at post-testing while the control group remained in tact. Secondly, the RR group received individual tutoring; the control group got none. The published paper bears the hallmarks of a bona fide scientific journal until a closer inspection reveals it is published by Reading Recovery. No chance for an impartial peer review process here."[20] That is pretty strong criticism when a great deal of money is being spent on a Reading Recovery programme. I think it is only fair to give you the chance to comment on it.

  Jean Gross: Certainly. May I come in first and perhaps, Greg, who was an impartial person on the steering group for that review, might also like to. Firstly, it is absolutely not true that children were dropped from the study. What happened is there were 147 children in the control group. That was in schools without Reading Recovery. In the schools with Reading Recovery there were 145 children of a similar level to the 147, but only 87 of those children had Reading Recovery because the others did not get to it in the year. There was not enough teaching resource for them to get it. They needed it but they did not get it. So what was looked at were the results of those 87 children with the matched 147 children in the control group. There is no question of dropping children—they had nothing—and their progress was also reported on. If you read the full study, the progress of those children who did not have Reading Recovery is also reported on, and what you have is the very big effect for the children who had Reading Recovery compared to the children in comparison schools, and the children who were in schools with Reading Recovery but did not themselves get Reading Recovery made an intermediate level of progress. So there was some knock-on effect by the presence of a Reading Recovery teacher in the school, which we assume to be because of the impact on the whole class and the teaching of literacy in the school. It is part of Every Child a Reader that the Reading Recovery teacher is meant to work with class teachers to help improve the quality of everyday teaching.

  Q43  Graham Stringer: Is it true that there is no peer review process involved in this?

  Jean Gross: There has been a published study of the first year of that which has been peer reviewed.

  Q44  Chairman: By whom?

  Jean Gross: I will let the Committee know the reference and I will let you know who peer reviewed it.[21]

  Professor Brooks: I can vouch that the data are sound, that the rebuttal that Jean has just given of the criticisms you were reading is valid.

  Q45  Dr Harris: Can I ask if you have ever heard of something called an "intention to treat analysis"?

  Professor Brooks: Yes.

  Q46  Dr Harris: Does this study comply with that when you compare the 87 completers rather than the 145 people you were intending to be in the active group?

  Professor Brooks: Ideally, you would have followed all 145 in the treatment group, the Reading Recovery group, until they had all completed their programme.

  Q47  Dr Harris: In order to comply with the intention to treat analysis, which is, would you agree, required in medical research in order to pass peer review?

  Professor Brooks: Then you would have had to allow for the fact that the children in the Reading Recovery group were receiving the post-test at different lengths of time after the pre-test, and that would have complicated it.

  Q48  Dr Harris: I do not want to go into the detail of this; I just want to ask you the concept of the "intention to treat analysis". My understanding (and I am not a full-timer like you, so feel free to correct me if I am wrong) is that you will just get demolished if you try and enter a medical clinical study which only looks at completers and does not do an analysis of those who were selected for the trial at the outset. That is called an analysis of those who you intend to treat rather than the completers.

  Professor Brooks: Yes.

  Q49  Dr Harris: Because I think they know that that is a huge biasing factor. So any research that does not look at the people who enter into the study has to have a huge flag at the very beginning, which I did not hear—perhaps I heard a defence of it—saying that this cannot be seen as robust under those terms. Am I wrong?

  Professor Brooks: You are not wrong if you are talking about randomised control trials which sign up to the full rigour of the agreed protocols for carrying out randomised control trials. This was not a randomised control trial, it was a matched group quasi experiment, and I would say that the analysis that was carried out on the data from that study was appropriate to the type of study.

  Q50  Dr Harris: One could argue that, given it is not a randomised control trial, it is even more important that you do it right. I just wanted to pick up on something that Ms Gross said, firstly, and this is not meant to be hostile, I just want to clarify it for our records, and this is good practice. Ms Gross, because of the work that you do, would it be fair to say that you have an interest in these matters and that, if you were published, you would say that you had that competing interest?

  Jean Gross: I do have a competing interest; of course.

  Q51  Dr Harris: A separate question now. You mentioned that you thought that the cost of randomised control trials is not trivial, it is a significant amount of money, and that when money is tight (and money is always tight, I suppose) that is something to be borne in mind. Would you accept that when money is tight it is particularly important that it is not spent on things that do not work and, therefore, the ratio of good investment in research, £250,000, compared to the spending of tens of millions on a programme that may work but may not be the most cost-effective way of doing it is actually a better investment? If you are rich you do not really need to worry, do you?

  Jean Gross: My personal view is that it is a good investment. I simply said it is something that a government has to weigh up, but the fact that we are doing this for Every Child Counts, I think, is an extremely positive thing and will give us the robust data. The "intention to treat analysis" will meet the absolute gold standard of this.

  Q52  Dr Harris: Do any of you regret that there was not sufficient good quality evidence before tens of millions of pounds were spent on both a pilot, but I think that is fair enough, and then tens of millions of pounds on a roll-out?

  Professor Brooks: It strikes me as a "Have you stopped beating your wife?" question.

  Q53  Dr Harris: Or a statement of the obvious.

  Professor Brooks: Yes.

  Chairman: I am sorry; can I intervene here? I do not think it is a case of "stop beating the wife". We are talking here about a standard of evidence gathering which is significantly lower than you would find in any other area, particularly in terms of the physical sciences or medicine, and I think a genuine question for this Committee to ask is why is it right to actually spend millions and millions of pounds without the evidence base which would be required as absolutely necessary in other branches of government policy?

  Mr Boswell: May I gloss that and say: what is the problem in doing so? Clearly it is the cost; clearly there may be a time factor. Is there something else we have overlooked?

  Q54  Dr Harris: Professor Brooks, do you want to have a go at my question?

  Professor Brooks: I am sorry; I think I am answering a slightly different one here. There are actually about four or five randomised control trials on Reading Recovery in the literature. They are all carried out in the United States. They show a reasonable effect size for that intervention over no intervention or no special treatment. To my knowledge, there are no, or hardly any, comparative studies of Reading Recovery against another treatment. I do know of one that was carried out in Rhode Island, but that was done so many years ago that today's Reading Recovery is significantly different from the variety that was used in that study. It would be wonderful if all educational programmes and interventions were based on randomised control trials that have been carried out before they were implemented. If that were the case, I think we would still be waiting to find out whether Egyptian hieroglyphs were superior to the alternative writing script that was in use in that country.

  Q55  Dr Harris: So you are saying you could not have done a decent quality randomised control trial on this before it started, before it was rolled out, because it would take 3,000 years? I do not understand your last answer.

  Professor Brooks: No.

  Q56  Dr Harris: My question was do you think it would have been better—and it is not, "When did you stop beating your wife?"—to have done a randomised control trial investing £250,000 and only rolling out if that result was positive?

  Professor Brooks: Yes, I agree, and, indeed, I argued for this when I was a member of the advisory group to the Every Child a Reader study, but by the time I was asked on to the committee the design was already set. So, yes, I would agree that the stronger the evidence that you can get before you roll out an intervention the better.

  Q57  Dr Harris: That is fair enough. Professor Slavin, do you have anything to comment on the question of whether the evidence from the United States on this is best for Reading Recovery and that there is no other gain in town on the evidence base?

  Professor Slavin: Our review found a number of randomised control trials just like the ones that you were asking for, a couple on Reading Recovery and numerous studies of other tutoring models, not only tutoring with certified teachers, but also tutoring with teaching assistants, small group instruction and then whole class interventions to try to improve outcomes for all children, and we found from these randomised and also high quality quasi experiments that there is a variety of things that work very well, that some of them work somewhat less well than one-to-one tutoring with qualified teachers but cost so much less that they are worth considering as alternatives for cost-effectiveness, such as the use in well-structured programmes of teaching assistants.

  Q58  Dr Harris: That is the study that I have got a reference to here—you mention it in your written submission to us (I do not like to use the term evidence for written submissions)—where you say that the overall weighted mean effect for 19 qualifying studies on Reading Recovery was plus .38; whereas on a range of others you found the 11 studies of these programmes—auditory discrimination, in depth, et cetera—had a weighted mesne effect size of plus 6.0.

  Professor Slavin: Correct.

  Q59  Dr Harris: Which implies to me, can you say, that that is 30 times—

  Professor Slavin: No, it would not be six, it would be 0.6.

  Q60  Dr Harris: It says 6.0 here?

  Professor Slavin: That would be a misprint if it is there.

  Q61  Dr Harris: So it is nearly double.

  Professor Slavin: Yes.

  Q62  Dr Harris: If the data is sound. I do not understand, therefore, why, Professor Brooks, you gave the impression that Reading Recovery, from the American studies you mentioned, was ahead of the game. Do you contest this?

  Professor Brooks: I did not say that. I said that there were several randomised control trials that had been carried out that showed that it does actually have a decent effect size. I was not commenting on comparisons with other programmes.

  Q63  Dr Harris: I have got one more question. A Cambridge research group published a very expensive and detailed study which included the question of when it is best to start formal teaching, and that said that there was good evidence from international comparators to support starting formal teaching, regardless of what happens at home, in order to encourage socialisation and stuff in schools. The Hungarians, for example, are not so terrible. That was dismissed immediately by some politicians.

  Professor Brooks: Yes.

  Q64  Dr Harris: Do you think it is wise, if you are talking about evidence based policy, to dismiss it like that?

  Professor Brooks: I refer to my answer of some minutes ago. I would prefer there to be a less formal start for children at five, and I think the international evidence supports that.

  Chairman: I am sorry; I have got to move on at that point.

  Q65  Graham Stringer: I have a quick point on the United States evidence. You say there is evidence from the United States in support of the Reading Recovery programme. Is it not the case that there was a letter from the top people in this area with 31 signatures on it sent to the US Congress urging them not to support Reading Recovery because they believed that the research showed that it had no effect and was very expensive?

  Professor Brooks: I have read that letter. I think it had a valid point at the time when it was written. There has been more evidence coming since then that I think makes it legitimate to fund this programme, amongst others.

  Jean Gross: May I simply comment that the US What Works Clearinghouse, which is its organisation for looking at evidence based practice, gave Reading Recovery it its top rating.

  Q66  Mr Cawsey: Earlier on I think Jean said that the cost of Reading Recovery was £2,600 per child, which is expensive.

  Jean Gross: Yes.

  Q67  Mr Cawsey: You then compared that to £50,000 if you did nothing, but, of course, to a certain extent that is a false premise, because that implies those are your only two choices, whereas you could be doing something else.

  Jean Gross: Okay.

  Q68  Mr Cawsey: I see in our briefing Professor Brooks reviewed, is it, 48 different kinds of reading interventions?

  Jean Gross: Yes.

  Q69  Mr Cawsey: So I say, in a Jim Bowen Bullseye way, "Look at what we could have had"! What I am interested in is what other sort of interventions were dismissed by the Government in coming to this conclusion, and then, leading on from that, are you satisfied as a panel that when the Government reached the decision that they did it was done on the evidence and not for any other reason?

  Professor Brooks: I was not privy to the discussions. It seems to me that there is a range of other interventions which are promising, some of which have produced average monthly progress for some children at least as good as that achieved by Reading Recovery. None of them, unfortunately, has been the subject, up to now, of a really rigorous trial. Therefore, it would be a good idea to have some of those done. Indeed, on one of those other programmes, the one called "inference training", there has now been a randomised control trial at the University of York, which I think is due to publish any time. I am not sure what the implication is of your referring to, or asking about, other programmes that the Government sidelined. At the time when they decided to put a lot of money behind Reading Recovery in the form of Every Child a Reader there were not many programmes that had the sort of knock-down convincing evidence that would have led anybody to say do something else rather than that.

  Q70  Mr Cawsey: So you would say at the time the decision was made it was evidence based, on the basis of what evidence was available at that time, but it is a moving feast.

  Professor Brooks: Yes, but there would have been some others that would have benefited from a roll-out and a more detailed study as well.

  Q71  Mr Cawsey: I do not want to put words into your mouth, but do you therefore think that the Every Child a Reader programme ought to be more flexible to allow different strategies to come into place as more evidence becomes available? In other words, have we cornered ourselves off into one aspect only?

  Professor Brooks: It is in itself quite a flexible programme anyway. I do not think they would want to be told to incorporate other programmes within it. What I am saying is that there are other programmes that would benefit from being studied in the same depth as that.

  Jean Gross: May I explain that Every Child a Reader is not just one programme; it is structured to provide Reading Recovery by a specialist teacher for the very hardest to teach children, and that teacher in the school is expected to support, coach and train teaching assistants and volunteers to provide other lighter-touch interventions to children who would be in the next 15 per cent. If we look at the whole of the bottom 20 per cent of children who are struggling with reading, the aim of Every Child a Reader is to meet that 20 per cent but not to provide the most expensive and intensive intervention to all those 20 percent. It differs in this from how Reading Recovery might be used in other countries. There are a number of other programmes already from Greg's research that are already open to schools to choose. Schools choose what they will use in what we call the other layers. One of the key points in my evidence to you that I would really exhort you to bear in mind is that we may be looking for a magic bullet and what is the best buy; there will be not be one best buy, there will be different things that work for different levels of need. My reading of the evidence that I have presented to you in writing is that the children who are hardest to teach, there may well be an alternative to Reading Recovery for those children, and, yes, if it is found and if it is half an hour a day of just phonics teaching, or something else, it should be incorporated, but at the moment my reading of the evidence is that we do not have anything to put in place comparable for those children but that there will be other things for children with lesser needs, and we need an holistic solution, not one. I am a practitioner, not a professor. Last year this programme reached 12,000 children and last year 78 per cent of them got back to the level of their peers after 12-20 weeks of teaching.[22] You are scientists, but are we actually going to wait to do randomised control trials on 40 different interventions, comparing them with each other, for different levels of need? You cannot just do a randomised control trial for all children; you would have to do one for children in the bottom five per cent and one for the children with slightly less needs. How long do you wait before attending to those children's needs?

  Q72 Mr Cawsey: But you are confident that other strategies can come through and that they will not be knocked out by a decision that was made on Reading Recovery.

  Jean Gross: Absolutely not. Well, that is not my decision, is it? That will be government's decision. Our external organisation piloted Every Child a Reader but government are now running it, we are not running it.

  Q73  Mr Cawsey: Do you think that it is inevitable, given that ministers decide, ultimately, and ministers are just grumpy politicians who have the inevitability of working in a very brief political cycle in the world of having to do academic research and make decisions and then review and move on, that you are always going to end up with decisions that are not properly evaluated with all the evidence weighed up, not by people such as yourselves, but by the people who ultimately make the decisions, because they have a political agenda where they must be seen to be doing something before the next time they go back to the electorate? Does not that make all of the work that you do frustrating and irrelevant?

  Jean Gross: I would like to see a nice equivalent, an equivalent for the National Institute for Health and Clinical Excellence for education and children's services that was independent of the political electoral five-year timescale which, in my view, does not lead to long-term best evidence decision-making.

  Chairman: That strikes me as a fantastic note on which to finish. I suspect that might appear in our report at the end of the day! Can I thank you all very much indeed for giving your evidence this morning and being so patient with the Committee. We actually just want the best at the end of the day, but thank you very much indeed.





20   Ev 74, para 3 Back

21   Note by witness: The research report was published in Literacy Teaching and Learning: An International Journal of Early Reading and Writing (ISSN 1538-6805). The report was read by Professor Bob Schwartz and Professor Roger Beard. It was blind reviewed by appropriately selected international members of the Editorial Review Board of the journal. As is customary, the names of blind reviewers were not made available to the authors. Feedback from the peer reviewers was responded to and approved. Back

22   Note by witness for clarity: The Every Child a Reader programme reached 12,000 children. 78% of the children who had full Reading Recovery got back to the level of their peers after 12-20 weeks of teaching. Back


 
previous page contents next page

House of Commons home page Parliament home page House of Lords home page search page enquiries index

© Parliamentary copyright 2009
Prepared 18 December 2009