Examination of Witnesses (Questions 1-)|
4 NOVEMBER 2009
Q1 Chairman: Could I welcome everyone
to this session in part of our work as a Committee looking at
the evidence behind various elements of government policy. That,
principally, is what we are trying to get at this morning and
this particular session is really looking at the evidence behind
literacy interventions. The Government has spent a very significant
amount of money trying to improve literacy standards in English
schools and we are anxious to look at the evidence behind that.
On our first panel is Professor Bob Slavinwelcome to you,
Professor Slavinthe Director of the Institute for Effective
Education at the University of York and Director of the Centre
for Research and Reform in Education at the Johns Hopkins University,
Jean Gross, the Director of Every Child a Chance Trustwelcome
to youand Professor Greg Brooks, the Research Director
of the Sheffield arm of the National Research and Development
Centre at the University of Sheffield. If you three do not know
the answers, then we are lost! I wonder if I could start with
you first, Professor Slavin. Early literacy interventions, we
are told, are so important for children who are struggling to
learn to read. Is that so and why?
Professor Slavin: Just as background,
we did a huge synthesis of research on programmes for struggling
readers and looked at every imaginable kind of thing that people
have proposed and at studies done in all different countries,
and what was remarkable was the degree to which there is a variety
of approaches that make a quite substantial difference in the
reading of young children. I think it is absolutely clear that
a very large proportion of children who are struggling to learn
to read can be successful, and perhaps the most important conclusion
that I would take away from all of this is that we can argue about
what are the most effective ways, what are the most cost-effective
ways to bring about this result, but it is quite clear that we
can get much, much better results in reading for children who
are struggling in reading than we do in ordinary circumstances.
Q2 Chairman: When you say you looked
at all the evidence that was available, what does that mean?
Professor Slavin: We did what
we call a best evidence synthesis, which is a form of something
called meta-analysis, where we applied a consistent set of standards
to all of the studies. There had to be a comparison of an experimental
and a control group over at least 12 weeks, with measures that
were not inherent to the treatmentwell matched and some
other technical requirementsand then computed what is called
an effect size, which is the percent of a standard deviation by
which the experimental group exceeded the control group on whatever
measures were being used. We found 98 studies that met our criteria
across all different kinds of interventions and then put them
into categories to look at what were the average effects of these
Q3 Chairman: So, in your view, the
evidence is there to say early intervention works?
Professor Slavin: I think without
Q4 Chairman: Jean, do you support
Jean Gross: Yes, I do. I support
Q5 Chairman: Professor Brooks?
Professor Brooks: Yes, it is better
if they come in early, when children are first identified as struggling,
but there are also programmes for those who are picked up rather
Q6 Chairman: Do you feel, in terms
of the Government's policy in this area, that they did take into
account that evidence?
Professor Brooks: I am sorry;
did you say they need to or they do?
Q7 Chairman: Do you feel that they
did do; that they took into account the available evidence?
Professor Brooks: Not all of it,
in my view. There are some programmes which have been researched
on quite a detailed scale several times, there are others for
which there is promising evidence that, in my view, would justify
investigating those programmes in more detail, some of which,
I think, are struggling to get heard and to get funding for more
Q8 Chairman: Do you think, for instance,
that phonics is the best way to teach children to read; that that
evidence is there?
Professor Brooks: That would be
to put it too simply. I think the evidence on initial teaching
of reading and spelling is that systematic teaching of phonics
is essential but it needs to be part of a broad and rich language
and literacy curriculum to work best.
Q9 Chairman: Jean, do you feel phonics
is essential in terms of teaching children to read?
Jean Gross: Yes. All the international
evidence shows that it is essential in teaching children to read.
I would agree with Greg also and, I think, also Bob. Bob's review
shows it needs to be embedded in applying that phonic knowledge
into reading books in a rich curriculum where you are also developing
your oral language skills. Just phonics alone will not meet the
wide reading needs of children, but it is the bedrock.
Q10 Chairman: Would you explain to
me the difference between phonics and synthetic phonics?
Professor Brooks: Synthetic phonics
is one variety of that. Phonics is teaching based on the relationships
between the sounds of the spoken language and the letters and
letter combinations of the written language. Synthetic phonics
is a variety in which children are taught to sound out the letters
to produce phonemes (sounds) and then to blend them together to
produce a whole word sound for reading. The other major form of
phonics that has been given the most attention is called analytic
phonics, and in that, in its strict form, sounding out is banned
and children are taught to try to infer the relationship between
letters and sounds by studying families of words.
Q11 Chairman: Professor Slavin, in
terms of phonics, you would also accept that it is essential,
in terms of teaching children to read, that you decode using phonics?
Professor Slavin: Yes, and I would
fully agree that of the forms of phonics synthetic phonics is
the one that has the strongest support.
Q12 Chairman: Is there any evidence
to support that?
Professor Slavin: Yes, there is
great deal of evidence of all different kinds from laboratory
studies to multi-year investigations. I think in the world of
reading that is virtually a settled issue at this point.
Q13 Chairman: What worries us as
a Committee is what does "evidence" mean. In terms of
evidence most scientists would say you have to have controlled
trials, so that you actually compare two different approaches,
and you do that in a systematic way. Do we have that evidence,
in terms of the use of phonics? What happens if children do not
encounter phonics at all?
Professor Slavin: There are many
children who learn to read regardless of the reading approach.
Q14 Chairman: Precisely.
Professor Slavin: So you cannot
say that phonics is essential for every child, or systematic phonics,
taught in school. There are children who do infer the phonetic
principle or who are taught at home, or God knows how they learn
it, but there is a very large group (let us say, a large minority)
of children for whom phonics really is make or break, for whom
if they have systematic phonics they will be successful readers,
they will never become struggling readers, they will not come
to the attention of the authorities in any way, and without it
they are much more likely to run into trouble.
Q15 Chairman: Jean, if you have children
who are struggling, particularly in the early years of primary
school, Key Stage 1, and phonics is not working, perhaps it is
not working because they have got dyslexia and, therefore, phonics
does not make any great sense to them. Are there some children
with, for instance, conditions like dyslexia (and we will come
on to dyslexia as a particular problem in the future) who cannot
read because of other conditions and, therefore, phonics is pretty
useless to them?
Jean Gross: The core cognitive
difficulty in the brain for children who struggle with reading,
all research shows, is a difficulty in hearing and separating
the sounds of spoken words in your brain (not in your ears). It
is breaking words into sounds and blending them back together.
We all agree that that is the core difficulty for children who
struggle. There are other difficulties in their lives, but that
is the core difficulty. There is perhaps a difference of view
of whether that means that you should continue to do more phonological
awareness work to attack the difficulty or whether you should
go round it and do other methods. My own reading of the research
is that in the end you have got to get children phonologically
skilled if they are going to be able to spell lifelong and be
able to continue to develop their reading. So you have to get
there, but, I think, for some children you do have to go down
a little bit of a circuitous route to start with. You might teach
phonics in different ways.
Q16 Chairman: That is the skill of
Jean Gross: That is the skill
of the teacher. You might have to teach the children other things
such as to visually recognise the difference between letters.
I think what I would say from the evidence is that there is not
one programme or method that will work for every child; you do
need to tailor it for the lowest achieving children with the greatest
Q17 Chairman: You all appear to agree,
and I would like an affirmative answer, that early intervention
and the use of phonics, in whatever form, is really important
to developing good readers and the Government is right to concentrate
on those issues.
Jean Gross: Yes.
Professor Brooks: Yes.
Professor Slavin: Yes.
Chairman: All three said, "Yes."
Over to you, Tim.
Q18 Mr Boswell: Thank you. Perhaps
I can preface my remarks by saying that I had for some time ministerial
responsibility for adult literacy at a time when my wife was actually
tutoring the subject, so I took a certain interest in that end
particularly. I am interested really in the evidence base for
literacy intervention. Professor Brooks, I know you did a study
some two years ago, a review of all the available literature.
Perhaps I might lead with you but invite the others to comment
as well. I suppose, in a way, we are looking for a kind of platonic
ideal for a literacy study which would convince everyone that
this is what we want to do, and I really would like to ask you
first if there have been any studies that meet those rather platonic
criteria for an ideal literacy intervention study?
Professor Brooks: Are we talking
about initial teaching or are we talking interventions for children
who do not get it first time?
Q19 Mr Boswell: I have not differentiated
it. It might be useful, therefore, if you could tell the Committee
just a little bit about both.
Professor Brooks: On the evidence
for the initial teaching of reading and spelling there is a set
of a dozen or so randomised control trials that show pretty clearly
that systematic attention to phonics at that stage enables many
children to make better progress than if they do not get systematic
phonics at that stage.
Q20 Mr Boswell: That, in fact, informs
the common view you have expressed.
Professor Brooks: We have said
that to an extent. If we are then talking about children who do
not get it the first time and, therefore, need to have extra attention
at around about the age of six or seven, the evidence there is
that there is a range of programmes that will help. As you imply,
they are summarised in this door stop of a report of mine. There
is good evidence that many of those that are within that set which
are phonologically basedin other words they have an element
of phonics in them or they have a strong element phonic in themwork
very well. There are others that have a broader approach, like
Reading Recovery. Jean will be able to speak in more detail about
that. That has some very strong evidence behind it as well. The
essential thing at that stage, I think, is to try and match the
intervention to the child and his or her needs, and that requires
very well trained teachers who know a range of interventions,
I think, and know how to tailor the particular intervention to
Q21 Mr Boswell: That is helpful.
Given that you have just said, in effect, that there is a huge
responsibility on the individual teacher knowing the individual
pupil and maybe trying things out. Clearly, if you are looking
at a literacy intervention trial (and we are interested in the
evidence base for that), you are looking at an aggregated number
or series of approaches, and if we can bring it to what you might
call the second wave, although I am not using that technicallythose
who had not got it first time, as you called themwhat are
the main problems with trials? We are asking for evidence. It
is social sciences; it is not narrowly definable physical sciences.
Is it the control groups? Is it assessment methods or data? Where
are the problems in tying down what works best?
Professor Brooks: In my view,
the evidence base for that second wave type intervention is less
strong than it is for initial teaching. There are precious few
randomised control trials in the field, at least in this country.
Bob will know better about the evidence from the rest of the world,
but when I was doing this study, out of the 121 pieces of evidence
that I was able to amass, only nine were randomised control trials,
and some of those were so small as to be hardly worth carrying
Q22 Mr Boswell: Does that disturb
you, in the sense that we really do not know what we are claiming
Professor Brooks: We could do
with much stronger evidence at that stage for quite a few of the
Q23 Mr Boswell: You are showing assent
to that, are you, Professor Slavin?
Professor Slavin: Yes. We found
just the same pattern, in the sense that there were very few even
quasi experimental (which means matched) studies that took place
in the UK, but if you looked at the whole world literature, most
of which is US, there is still not as much as one might like but
quite a robust set of evidence about what works for the struggling
Mr Boswell: Thank you.
Chairman: Can we just ask why?
Q24 Mr Boswell: Do you mean why do
these trials not exist or why could they not be conducted?
Professor Slavin: They absolutely
could be conducted. I think that there is a lack of funding for
that kind of research in the UK and, having worked in both places,
I think quite a striking difference in terms of the amount of
resource that is available for doing these large scale randomised
experiments, but they can be done. We have done them in the UK
Jean Gross: I have been involved
in two programmes, Every Child a Reader and Every Child Counts,
which is trying to do the same thing for numeracy. We have learned,
and I think the Government is more willing to commit resource
to randomised control trials. For Every Child a Reader we had
a quasi experimental matched group study. Looking back, I wish
at the time we had been able to find money to do a randomised
control trial, but Every Child Counts is having a £250,000
government-funded randomised control trial, so I think things
are moving on, but it is a lot of money, and if the recommendation
in the system is that we want more randomised control trials,
that is going to cost budgets at a time when budgets are tight.
Q25 Mr Boswell: In a sense you are
saying that policy about early intervention proceeds in parallel
with any systematic randomised control trial; it is not tied up
before you start.
Professor Brooks: Things in education
are rarely tied up before you start, but, yes, as I have already
said, we could do with stronger evidence. I think it gets difficult
to get money for randomised control trials because you need to
have done various pilot studies beforehand to show that your intervention
works, at least for pilot groups, and that you need to do a large
field trial before you commit the money to a very large and complicated
expensive piece of research, at least expensive in educational
terms, which might give you a null result in the end. That is
always the risk.
Q26 Mr Boswell: You mean no outcome?
Professor Brooks: No, not no outcome.
A null result is not no outcome. What I mean is a finding of no
difference between the conditions. That will disappoint some researchers.
I actually like those because they are contributions to knowledge
Q27 Mr Boswell: Can I just shift
to a slightly separate issue. We have already touched on dyslexia,
but there are also developmental differences between children,
as I know myself. They are not necessarily, as it were, pathological
or educationally pathological; they may just be people that grow
up faster. The NUT has expressed some concern in its evidence
about the Government's concept of age appropriate expectations.
How do we pick our way through that? Do you think we are paying
enough attention to the importance of differential rates of development
and does the Government's concept adequately take account of that
Professor Brooks: Can I start
with part of the answer to that? This sounds perilously close
to the idea that there are some children who are late developers
in literacy and that if you leave them alone they will catch up.
I can tell you that contained in here is a lot of evidence that
that is a myth. If you just leave children or leave them with
the ordinary classroom curriculum, on the whole, they do not catch
up, they need the interventions.
Q28 Chairman: Is that because they
start too early, Professor Brooks, in school?
Professor Brooks: Start school
too early? Ah, now we are getting into highly contested territory.
Q29 Chairman: It is just following
on from Tim Boswell's question really.
Professor Brooks: I agree. I think
we induct children into formal school too young in this country.
I think there is a case for having a much more play-based pre-school
phase that would last from age three to age six in which there
would be very little or no formal teaching of literacy and that
would start at six. I think at that point most children would
get it first time, as they do in other countries where they start
at that age, or, indeed, slightly later, such as Finland, and
it would not lead to larger numbers of children not getting it
Q30 Chairman: That would save us
a lot of money on Reading Recovery!
Professor Brooks: Well, not necessarily.
Jean Gross: If that was to happen,
I think there needs to be some good research beforehand. One of
the things I am struck by is the fact that, whilst you have a
play-based curriculum for all children, those parents who are
more affluent are very busy doing magnetic letters on the fridge
and reading to their children and actually will be teaching them
to read at home, because you cannot stop them, whereas for those
from disadvantaged homes that will not happen. There is a risk
that just needs to be examined with evidence. Would a later start
increase the gap between more affluent children and disadvantaged
Chairman: I am sorry I opened that up.
My apologies. It was too good an opportunity.
Mr Boswell: That is helpful.
Q31 Dr Harris: Ms Gross, when you
were asked about phonics you said "all the evidence".
When you use a term like that, does that mean it is equivalent
to the earth being round? There is no evidence out there that
disagrees on phonics at all, or is that just a term you tend to
use to say, "I think, the majority of evidence"?
Jean Gross: I think it means that
research that does overviews of multiple studies, some of which
might show "not", some of which might show "yes",
some of which might show neither, but overall the conclusion from
those high level analyses is that phonics is essential, so it
is that level.
Q32 Dr Harris: So it is not all the
evidence, is it? It is a review.
Jean Gross: Yes.
Q33 Dr Harris: A good review shows
that there is good evidence, or the best evidence, or "it
is convincing that". That is different from "all the
evidence", is it not?
Jean Gross: Yes, it is different;
so I apologise if that is incorrect.
Q34 Dr Harris: You also said "everyone
agrees" and "all research" in another answer. Do
you think it is wise to use terms like that, because people who
disagree think that that is unfair, even if they recognise they
are in the minority?
Professor Brooks: What tends to
happen then is that the people who oppose the policy or the findings
try to cherry-pick the bits that agree with their point of view
and systematically ignore the weight of the evidence. What we
are saying here is that the weight of evidence is in favour of
phonics for all children at the start and for many of those who
need early interventions.
Q35 Dr Harris: Would you all agree
it is better to use terms like "the weight of evidence"
rather than "all"?
Jean Gross: I would agree. I apologise
and correct what I said to "the weight of the evidence".
Q36 Graham Stringer: The Reading
Recovery programme is the backbone of the Government's strategy
with reading. I just want to ask a number of questions about how
we got to it being the backbone, what the evidence was for that,
and how it is assessed: because we have had some recent evidence
saying that it is probably not the best strategy. First of all,
if we can go back to your previous answers, it was chosen after
a pilot scheme which did not have a control group. That is right,
is it not?
Jean Gross: The process of choosing
it, and I was involved in giving evidence to the Government that
perhaps contributed to them choosing it from outsideI was
not part of the Government; I am employed by the KPMG businessI
believe, involved looking at existing international evidence.
There was an awful lot about Reading Recovery before the 2005-08
Every Child a Reader pilot in England. So it was not that it was
not starting from nothing. I believe they looked at that evidenceand
I certainly supplied that evidence to themand then, on
top of that, they had two sources of evidence, one of which is
what I would call management data, the data that is routinely
gathered on an international database for every child who goes
through Reading Recovery, with a teacher entering data at the
beginning of their programme and a different teacher testing the
child at the end of their programme and entering the data. So
that is management data. The third element of evidence was a quasi
experimental matched study, which Bob or Greg can comment on,
which was not a randomised control trial but would be recognised,
I believe, as a reasonable standard of evidence, the randomised
control studies being absolutely the gold standard but quasi experimental
being widely accepted in overviews of research as being includable
Q37 Graham Stringer: So it would
not be fair to characterise it, as some of the evidence we have
had has characterised it, as you put more effort into teaching
children to read so more of them lead to a higher standard and
there is no control against other methods of teaching them to
Jean Gross: No, the study that
the Government used was in relation to 21 London schools with
Reading Recovery and 21 without. Those other 21 schools were using
a range of other interventions for children aged six. They were
doing things that schools normally do, programmes like Early Literacy
Support and various other things; they were not doing nothing.
They were compared with Reading Recovery, the groups were matched,
and the effect size was large, using independent tests.
Q38 Graham Stringer: One of the points
you made earlier in the evidence was the cost of randomised control
groups. What is the cost of the Reading Recovery programme per
child? It is pretty high, is it not?
Jean Gross: £2,600 per child
is the average cost (once) of providing it, and the cost of not
providing it is £50,000 per child.
Q39 Graham Stringer: I understand
the cost of illiteracy. The randomised control groups: you are
saying a quarter of a million is not available here. That is 100
interventions. We are talking relatively small amounts of money.
Jean Gross: Yes.
Q40 Graham Stringer: That just seems
to contradict the evidence that you were giving earlier.
Jean Gross: I simply said that
that money has been made available for Every Child Counts. If
you multiply that for all the many things that would need to be
researched in the education and children sphere, you are looking
at a very big bill at a time of economic downturn. I am not saying
it should not happen; I am merely posing that as a potential problem
for any government.
Q41 Graham Stringer: Can you explain
to the Committee how the Reading Recovery process is assessed
Jean Gross: At the start children
are, first of all, put forward by their class teachers as being
the very lowest achieving children in a school year one class.
They are children who have made no progress in reading after a
year, or more than year, at school. The data shows that in England
the children entering Reading Recovery do not score on a standardised
reading test at all; they are at the floor of the test; so they
have a reading age well below five and they have had a year's,
or more, instruction. So they are put forward as the bottom children.
Teachers normally put forward eight to 12 children, and then they
all have an individual assessment that includes the British Ability
Scales word reading testa test of single word readinga
test of letter knowledge, tests of how many words they can write.
There are a number of different assessments. From that the very
lowest achieving children are selected to have the teaching first,
assuming that over the year the teacher will be able to teach
four children at once. At any one time the teacher teaches four
children. They are usually employed on a half-time basis to do
this work. When they have finished with those children in year,
they will take on another four children. So you start with the
very lowest, with no exceptions. I think this is important. You
do not rule out children because of any particular special needs,
an issue at home or poor attendance. They start and then they
are discharged from the programme when they have reached a certain
book level, a level of difficulty of book, which gives you an
indication that they are ready to be discharged, if you like.
They are then given a repeat of the same test that they had at
the beginning by a different teacher, a teacher who did not teach
the child in Reading Recovery, and the results are then entered
on an international database. So you get a gain in reading age,
in months of gain of reading age, for every child and a number
of other more technical measures.
Q42 Graham Stringer: Can I read you
a criticism, again, in the written evidence we have had, of the
assessment methods for Reading Recovery? It says, "It is
very costly but Reading Recovery research is notorious for misrepresenting
data. In a recent publication by the Institute of Education the
same problem appears. Nearly half of the children from 145 strong
RR tutoring group were dropped from the study at post-testing
while the control group remained in tact. Secondly, the RR group
received individual tutoring; the control group got none. The
published paper bears the hallmarks of a bona fide scientific
journal until a closer inspection reveals it is published by Reading
Recovery. No chance for an impartial peer review process here."
That is pretty strong criticism when a great deal of money is
being spent on a Reading Recovery programme. I think it is only
fair to give you the chance to comment on it.
Jean Gross: Certainly. May I come
in first and perhaps, Greg, who was an impartial person on the
steering group for that review, might also like to. Firstly, it
is absolutely not true that children were dropped from the study.
What happened is there were 147 children in the control group.
That was in schools without Reading Recovery. In the schools with
Reading Recovery there were 145 children of a similar level to
the 147, but only 87 of those children had Reading Recovery because
the others did not get to it in the year. There was not enough
teaching resource for them to get it. They needed it but they
did not get it. So what was looked at were the results of those
87 children with the matched 147 children in the control group.
There is no question of dropping childrenthey had nothingand
their progress was also reported on. If you read the full study,
the progress of those children who did not have Reading Recovery
is also reported on, and what you have is the very big effect
for the children who had Reading Recovery compared to the children
in comparison schools, and the children who were in schools with
Reading Recovery but did not themselves get Reading Recovery made
an intermediate level of progress. So there was some knock-on
effect by the presence of a Reading Recovery teacher in the school,
which we assume to be because of the impact on the whole class
and the teaching of literacy in the school. It is part of Every
Child a Reader that the Reading Recovery teacher is meant to work
with class teachers to help improve the quality of everyday teaching.
Q43 Graham Stringer: Is it true that
there is no peer review process involved in this?
Jean Gross: There has been a published
study of the first year of that which has been peer reviewed.
Q44 Chairman: By whom?
Jean Gross: I will let the Committee
know the reference and I will let you know who peer reviewed it.
Professor Brooks: I can vouch
that the data are sound, that the rebuttal that Jean has just
given of the criticisms you were reading is valid.
Q45 Dr Harris: Can I ask if you have
ever heard of something called an "intention to treat analysis"?
Professor Brooks: Yes.
Q46 Dr Harris: Does this study comply
with that when you compare the 87 completers rather than the 145
people you were intending to be in the active group?
Professor Brooks: Ideally, you
would have followed all 145 in the treatment group, the Reading
Recovery group, until they had all completed their programme.
Q47 Dr Harris: In order to comply
with the intention to treat analysis, which is, would you agree,
required in medical research in order to pass peer review?
Professor Brooks: Then you would
have had to allow for the fact that the children in the Reading
Recovery group were receiving the post-test at different lengths
of time after the pre-test, and that would have complicated it.
Q48 Dr Harris: I do not want to go
into the detail of this; I just want to ask you the concept of
the "intention to treat analysis". My understanding
(and I am not a full-timer like you, so feel free to correct me
if I am wrong) is that you will just get demolished if you try
and enter a medical clinical study which only looks at completers
and does not do an analysis of those who were selected for the
trial at the outset. That is called an analysis of those who you
intend to treat rather than the completers.
Professor Brooks: Yes.
Q49 Dr Harris: Because I think they
know that that is a huge biasing factor. So any research that
does not look at the people who enter into the study has to have
a huge flag at the very beginning, which I did not hearperhaps
I heard a defence of itsaying that this cannot be seen
as robust under those terms. Am I wrong?
Professor Brooks: You are not
wrong if you are talking about randomised control trials which
sign up to the full rigour of the agreed protocols for carrying
out randomised control trials. This was not a randomised control
trial, it was a matched group quasi experiment, and I would say
that the analysis that was carried out on the data from that study
was appropriate to the type of study.
Q50 Dr Harris: One could argue that,
given it is not a randomised control trial, it is even more important
that you do it right. I just wanted to pick up on something that
Ms Gross said, firstly, and this is not meant to be hostile, I
just want to clarify it for our records, and this is good practice.
Ms Gross, because of the work that you do, would it be fair to
say that you have an interest in these matters and that, if you
were published, you would say that you had that competing interest?
Jean Gross: I do have a competing
interest; of course.
Q51 Dr Harris: A separate question
now. You mentioned that you thought that the cost of randomised
control trials is not trivial, it is a significant amount of money,
and that when money is tight (and money is always tight, I suppose)
that is something to be borne in mind. Would you accept that when
money is tight it is particularly important that it is not spent
on things that do not work and, therefore, the ratio of good investment
in research, £250,000, compared to the spending of tens of
millions on a programme that may work but may not be the most
cost-effective way of doing it is actually a better investment?
If you are rich you do not really need to worry, do you?
Jean Gross: My personal view is
that it is a good investment. I simply said it is something that
a government has to weigh up, but the fact that we are doing this
for Every Child Counts, I think, is an extremely positive thing
and will give us the robust data. The "intention to treat
analysis" will meet the absolute gold standard of this.
Q52 Dr Harris: Do any of you regret
that there was not sufficient good quality evidence before tens
of millions of pounds were spent on both a pilot, but I think
that is fair enough, and then tens of millions of pounds on a
Professor Brooks: It strikes me
as a "Have you stopped beating your wife?" question.
Q53 Dr Harris: Or a statement of
Professor Brooks: Yes.
Chairman: I am sorry; can I intervene
here? I do not think it is a case of "stop beating the wife".
We are talking here about a standard of evidence gathering which
is significantly lower than you would find in any other area,
particularly in terms of the physical sciences or medicine, and
I think a genuine question for this Committee to ask is why is
it right to actually spend millions and millions of pounds without
the evidence base which would be required as absolutely necessary
in other branches of government policy?
Mr Boswell: May I gloss that and say:
what is the problem in doing so? Clearly it is the cost; clearly
there may be a time factor. Is there something else we have overlooked?
Q54 Dr Harris: Professor Brooks,
do you want to have a go at my question?
Professor Brooks: I am sorry;
I think I am answering a slightly different one here. There are
actually about four or five randomised control trials on Reading
Recovery in the literature. They are all carried out in the United
States. They show a reasonable effect size for that intervention
over no intervention or no special treatment. To my knowledge,
there are no, or hardly any, comparative studies of Reading Recovery
against another treatment. I do know of one that was carried out
in Rhode Island, but that was done so many years ago that today's
Reading Recovery is significantly different from the variety that
was used in that study. It would be wonderful if all educational
programmes and interventions were based on randomised control
trials that have been carried out before they were implemented.
If that were the case, I think we would still be waiting to find
out whether Egyptian hieroglyphs were superior to the alternative
writing script that was in use in that country.
Q55 Dr Harris: So you are saying
you could not have done a decent quality randomised control trial
on this before it started, before it was rolled out, because it
would take 3,000 years? I do not understand your last answer.
Professor Brooks: No.
Q56 Dr Harris: My question was do
you think it would have been betterand it is not, "When
did you stop beating your wife?"to have done a randomised
control trial investing £250,000 and only rolling out if
that result was positive?
Professor Brooks: Yes, I agree,
and, indeed, I argued for this when I was a member of the advisory
group to the Every Child a Reader study, but by the time I was
asked on to the committee the design was already set. So, yes,
I would agree that the stronger the evidence that you can get
before you roll out an intervention the better.
Q57 Dr Harris: That is fair enough.
Professor Slavin, do you have anything to comment on the question
of whether the evidence from the United States on this is best
for Reading Recovery and that there is no other gain in town on
the evidence base?
Professor Slavin: Our review found
a number of randomised control trials just like the ones that
you were asking for, a couple on Reading Recovery and numerous
studies of other tutoring models, not only tutoring with certified
teachers, but also tutoring with teaching assistants, small group
instruction and then whole class interventions to try to improve
outcomes for all children, and we found from these randomised
and also high quality quasi experiments that there is a variety
of things that work very well, that some of them work somewhat
less well than one-to-one tutoring with qualified teachers but
cost so much less that they are worth considering as alternatives
for cost-effectiveness, such as the use in well-structured programmes
of teaching assistants.
Q58 Dr Harris: That is the study
that I have got a reference to hereyou mention it in your
written submission to us (I do not like to use the term evidence
for written submissions)where you say that the overall
weighted mean effect for 19 qualifying studies on Reading Recovery
was plus .38; whereas on a range of others you found the 11 studies
of these programmesauditory discrimination, in depth, et
ceterahad a weighted mesne effect size of plus 6.0.
Professor Slavin: Correct.
Q59 Dr Harris: Which implies to me,
can you say, that that is 30 times
Professor Slavin: No, it would
not be six, it would be 0.6.
Q60 Dr Harris: It says 6.0 here?
Professor Slavin: That would be
a misprint if it is there.
Q61 Dr Harris: So it is nearly double.
Professor Slavin: Yes.
Q62 Dr Harris: If the data is sound.
I do not understand, therefore, why, Professor Brooks, you gave
the impression that Reading Recovery, from the American studies
you mentioned, was ahead of the game. Do you contest this?
Professor Brooks: I did not say
that. I said that there were several randomised control trials
that had been carried out that showed that it does actually have
a decent effect size. I was not commenting on comparisons with
Q63 Dr Harris: I have got one more
question. A Cambridge research group published a very expensive
and detailed study which included the question of when it is best
to start formal teaching, and that said that there was good evidence
from international comparators to support starting formal teaching,
regardless of what happens at home, in order to encourage socialisation
and stuff in schools. The Hungarians, for example, are not so
terrible. That was dismissed immediately by some politicians.
Professor Brooks: Yes.
Q64 Dr Harris: Do you think it is
wise, if you are talking about evidence based policy, to dismiss
it like that?
Professor Brooks: I refer to my
answer of some minutes ago. I would prefer there to be a less
formal start for children at five, and I think the international
evidence supports that.
Chairman: I am sorry; I have got to move
on at that point.
Q65 Graham Stringer: I have a quick
point on the United States evidence. You say there is evidence
from the United States in support of the Reading Recovery programme.
Is it not the case that there was a letter from the top people
in this area with 31 signatures on it sent to the US Congress
urging them not to support Reading Recovery because they believed
that the research showed that it had no effect and was very expensive?
Professor Brooks: I have read
that letter. I think it had a valid point at the time when it
was written. There has been more evidence coming since then that
I think makes it legitimate to fund this programme, amongst others.
Jean Gross: May I simply comment
that the US What Works Clearinghouse, which is its organisation
for looking at evidence based practice, gave Reading Recovery
it its top rating.
Q66 Mr Cawsey: Earlier on I think
Jean said that the cost of Reading Recovery was £2,600 per
child, which is expensive.
Jean Gross: Yes.
Q67 Mr Cawsey: You then compared
that to £50,000 if you did nothing, but, of course, to a
certain extent that is a false premise, because that implies those
are your only two choices, whereas you could be doing something
Jean Gross: Okay.
Q68 Mr Cawsey: I see in our briefing
Professor Brooks reviewed, is it, 48 different kinds of reading
Jean Gross: Yes.
Q69 Mr Cawsey: So I say, in a Jim
Bowen Bullseye way, "Look at what we could have had"!
What I am interested in is what other sort of interventions were
dismissed by the Government in coming to this conclusion, and
then, leading on from that, are you satisfied as a panel that
when the Government reached the decision that they did it was
done on the evidence and not for any other reason?
Professor Brooks: I was not privy
to the discussions. It seems to me that there is a range of other
interventions which are promising, some of which have produced
average monthly progress for some children at least as good as
that achieved by Reading Recovery. None of them, unfortunately,
has been the subject, up to now, of a really rigorous trial. Therefore,
it would be a good idea to have some of those done. Indeed, on
one of those other programmes, the one called "inference
training", there has now been a randomised control trial
at the University of York, which I think is due to publish any
time. I am not sure what the implication is of your referring
to, or asking about, other programmes that the Government sidelined.
At the time when they decided to put a lot of money behind Reading
Recovery in the form of Every Child a Reader there were not many
programmes that had the sort of knock-down convincing evidence
that would have led anybody to say do something else rather than
Q70 Mr Cawsey: So you would say at
the time the decision was made it was evidence based, on the basis
of what evidence was available at that time, but it is a moving
Professor Brooks: Yes, but there
would have been some others that would have benefited from a roll-out
and a more detailed study as well.
Q71 Mr Cawsey: I do not want to put
words into your mouth, but do you therefore think that the Every
Child a Reader programme ought to be more flexible to allow different
strategies to come into place as more evidence becomes available?
In other words, have we cornered ourselves off into one aspect
Professor Brooks: It is in itself
quite a flexible programme anyway. I do not think they would want
to be told to incorporate other programmes within it. What I am
saying is that there are other programmes that would benefit from
being studied in the same depth as that.
Jean Gross: May I explain that
Every Child a Reader is not just one programme; it is structured
to provide Reading Recovery by a specialist teacher for the very
hardest to teach children, and that teacher in the school is expected
to support, coach and train teaching assistants and volunteers
to provide other lighter-touch interventions to children who would
be in the next 15 per cent. If we look at the whole of the bottom
20 per cent of children who are struggling with reading, the aim
of Every Child a Reader is to meet that 20 per cent but not to
provide the most expensive and intensive intervention to all those
20 percent. It differs in this from how Reading Recovery might
be used in other countries. There are a number of other programmes
already from Greg's research that are already open to schools
to choose. Schools choose what they will use in what we call the
other layers. One of the key points in my evidence to you that
I would really exhort you to bear in mind is that we may be looking
for a magic bullet and what is the best buy; there will be not
be one best buy, there will be different things that work for
different levels of need. My reading of the evidence that I have
presented to you in writing is that the children who are hardest
to teach, there may well be an alternative to Reading Recovery
for those children, and, yes, if it is found and if it is half
an hour a day of just phonics teaching, or something else, it
should be incorporated, but at the moment my reading of the evidence
is that we do not have anything to put in place comparable for
those children but that there will be other things for children
with lesser needs, and we need an holistic solution, not one.
I am a practitioner, not a professor. Last year this programme
reached 12,000 children and last year 78 per cent of them got
back to the level of their peers after 12-20 weeks of teaching.
You are scientists, but are we actually going to wait to do randomised
control trials on 40 different interventions, comparing them with
each other, for different levels of need? You cannot just do a
randomised control trial for all children; you would have to do
one for children in the bottom five per cent and one for the children
with slightly less needs. How long do you wait before attending
to those children's needs?
Q72 Mr Cawsey: But you are confident
that other strategies can come through and that they will not
be knocked out by a decision that was made on Reading Recovery.
Jean Gross: Absolutely not. Well,
that is not my decision, is it? That will be government's decision.
Our external organisation piloted Every Child a Reader but government
are now running it, we are not running it.
Q73 Mr Cawsey: Do you think that
it is inevitable, given that ministers decide, ultimately, and
ministers are just grumpy politicians who have the inevitability
of working in a very brief political cycle in the world of having
to do academic research and make decisions and then review and
move on, that you are always going to end up with decisions that
are not properly evaluated with all the evidence weighed up, not
by people such as yourselves, but by the people who ultimately
make the decisions, because they have a political agenda where
they must be seen to be doing something before the next time they
go back to the electorate? Does not that make all of the work
that you do frustrating and irrelevant?
Jean Gross: I would like to see
a nice equivalent, an equivalent for the National Institute for
Health and Clinical Excellence for education and children's services
that was independent of the political electoral five-year timescale
which, in my view, does not lead to long-term best evidence decision-making.
Chairman: That strikes me as a fantastic
note on which to finish. I suspect that might appear in our report
at the end of the day! Can I thank you all very much indeed for
giving your evidence this morning and being so patient with the
Committee. We actually just want the best at the end of the day,
but thank you very much indeed.
20 Ev 74, para 3 Back
Note by witness: The research report was published in
Literacy Teaching and Learning: An International Journal of Early
Reading and Writing (ISSN 1538-6805). The report was read by Professor
Bob Schwartz and Professor Roger Beard. It was blind reviewed
by appropriately selected international members of the Editorial
Review Board of the journal. As is customary, the names of blind
reviewers were not made available to the authors. Feedback from
the peer reviewers was responded to and approved. Back
Note by witness for clarity: The Every Child a Reader programme
reached 12,000 children. 78% of the children who had full Reading
Recovery got back to the level of their peers after 12-20 weeks
of teaching. Back