5 Research and evidence in DfE
88. This inquiry has scrutinised policy areas
shared between two Government Departments: the Department for
Education (DfE) and the Department for Business, Innovation and
Skills (BIS). However, the focus of our scrutiny has been the
DfE, and as well as examining the DfE's policies we were interested
in the Department's use of research and evidence: the relationship
between evidence and policy continues to be a key theme in our
work as a committee.
Chief Scientific Adviser
89. Carole Willis is the DfE's Chief Scientific
Adviser (CSA) and Director of Research and Analysis. The roles
of CSAs vary between Departments, with some CSAs being externally
appointed experts and others internal civil servants. Ms Willis
falls into the latter category, and described her role as follows:
There are three key aspects to my role, which are
fundamentally all about ensuring that the right robust evidence
is being fed in to inform Ministers' policy decisions. I do that
by ensuring that evidence is generated from within the Department,
and analysis is undertaken. We have a very large set of admin
data around attainment, which we feed in and analyse, looking
at the potential impacts of different policies. I ensure that
information is brought in from outside the Department, both in
the form of external research that is commissioned by others,
and from our own research programme, which commissions different
pieces of work on different policy questions.
I also ensure that we bring in a range of different
external experts to help and advise on particular issues. That
is the external role of gathering the evidence and feeding it
through to Ministers. I am also responsible for 200 professional
analysts within the Department, who work across all the policy
and delivery issues within the Department, helping to ensure that
policy is driven by and informed by the best available robust
evidence. Finally, personally I have a direct role in advising
Ministers and advising senior policy officials around the evidence
base and issues around different policy questions.[225]
Research budget
90. When we asked the DfE for figures on its
research budget over this spending review period, it told us that
its "research budget was £9.5 [million] in 2011-12"
and that it "was £9.5 [million] at the start of 2012-13".[226]
The DfE added:
The budget was reduced during this year as, after
careful review, it seemed highly likely that it would be underspent
this year and the funds were needed for other priorities. We expect
to spend around £6.2 [million] on research this year. In
addition to the research budget, evaluation activity is funded
from a range of different programme budgets across the department,
depending on the need to evaluate new policy initiatives. There
is no fixed budget for this evaluation activity. The Department's
detailed budgets—including the research budget—have
not been decided for the remainder of the spending review period.
They are subject to a business planning process which is currently
underway and will conclude in the New Year.[227]
Ms Willis explained that:
We are just about to go into the next business planning
round, where I will be looking very carefully, including consulting
external academics, at what the evidence gaps are within the Department
that we need to address over the next year or so, and I will be
presenting advice to Ministers about what we should be doing,
what we should be spending money on, and therefore how much we
should be spending on research going forward.[228]
When asked whether she expected the research budget
to increase or decrease, she responded:
I have not done the work yet. It is a demand-driven
approach, including demand from me, so there is no "right
amount" of research. We need to look carefully at the evidence
gaps, and what needs to be undertaken to fill those.[229]
91. We are concerned that the
Department for Education; (i) does not have a clear definition
of what constitutes research spend; (ii) has not planned its research
spend sufficiently in advance; (iii) has no established budget
for evaluation of policies; and (iv) does not protect or ring-fence
its research budget from "other priorities". The desire
to remain flexible and "demand-driven" is not at odds
with strategic, long-term thinking about research and evaluation.
92. The DfE's Chief Scientific
Adviser (CSA), who also holds the post of Director of Research
and Analysis, gave oral evidence on 21 November 2012 and told
us she had "not done the work yet" on future research
spending, yet written evidence from the DfE following the evidence
session stated that business planning would conclude in the New
Year. This suggests to us either that strategic planning of research
spending in the DfE is given very little time or that the CSA
has limited involvement. We would be interested in an explanation
from Government on this matter.
Evidence-based policy
93. While investigating the impacts of Government
changes to vocational education, the Engineering Diploma, the
EBac and University Technical Colleges, we picked up warning signals
about the relative lack evidence on which to base conclusions.
For example, the Royal Society stated "we believe it is too
early to draw conclusions about the few University Technical Colleges
that have opened since September 2010 or to assess the impact
of the English Baccalaureate, introduced in early 2011".[230]
Education for Engineering (E4E), stated:
our concern is that qualifications continue to be
changed without due regard for lessons learned from evaluation
of the past and often without piloting of new qualifications.
Tracing the detail of some engineering qualifications even from
[…] pre-2000 is nigh impossible.[231]
94. We have already touched on some of these
issues previously in the Report. In paragraph 26 we noted that
although the Government provided EBac attainment figures from
the three years between 2009/10 to 2011/12, the decision to knowingly
enter into EBac-relevant GCSEs could only have been taken by the
2011/12 cohort of students and schools. Yet the Government intends
to replace the current GCSE qualifications with the EBCs in 2017.
The Education Select Committee has also expressed concerns about
the retrospective introduction of the EBac, including that there
was a lack of proper consultation and that the EBac was introduced
before the completion of the Government's curriculum review.[232]
It also concluded that "any measure which examines schools'
performance in particular subjects would be better introduced
once the curriculum itself has been defined and finalised".[233]
Similarly, in Chapter 3 we summarised our efforts to obtain evidence
from the Government on the effectiveness of University Technical
Colleges, and the lack of a satisfactory response received. The
change to the Engineering Diploma's GCSE equivalence following
the Government's vocational education reforms was a mistake, as
evidenced by the unanimous outcry it caused, as well as the Government's
subsequent action to redesign the qualification.
95. Our recommendation to the
Department for Education (DfE), based on our experience during
this inquiry, is that greater focus needs to be placed on evidence
before future changes are made, and needs to leave sufficient
time for evidence to be gathered on the effectiveness of its proposed
changes before introducing further change. We recommend
that the DfE conducts a re-evaluation of its attitude towards
the role of evidence in policy and decision-making.
RANDOMISED CONTROLLED TRIALS
96. One method of gathering evidence is to use
randomised controlled trials (RCTs). The Cabinet Office considers
that RCTs "are the best way of determining whether a policy
is working".[234]
RCTs test "which of two or more interventions is the most
effective at attaining a specific, measurable outcome".[235]
Randomisation means that the subjects (be they students or schools)
are randomly allocated to receive a particular intervention, thus
avoiding selection bias. Controlled conditions essentially mean
that the environment around all the subjects is kept as similar
as possible, to minimise the possibility that something other
than the intervention has caused the outcome being tested.
97. A common objection to the use of RCTs in
public policy is that it is "unethical to withhold a new
intervention from people who could benefit from it".[236]
Indeed a similar argument was put forward by Ms Willis during
our predecessor Committee's 2009 inquiry into Early Literacy Interventions.
She stated:
social policy is quite different to medical policy.
You cannot, for example, do double blind randomised controlled
trials as you can in medical practice. It is sometimes difficult
to get people to take part in randomised controlled trials. Some
local authorities or schools perceive it as unfair that some of
their pupils will be getting some sort of intervention that others
are not.[237]
However, the Cabinet Office's June 2012 paper Test,
Learn, Adapt: Developing Public Policy with Randomised Controlled
Trials addressed this objection:
It is also worth noting that policies are often rolled
out slowly, on a staggered basis, with some regions "going
early", and these phased introductions are not generally
regarded as unethical. The delivery of the Sure Start programme
is an example of this. If anything, a phased introduction in the
context of an RCT is more ethical, because it generates new high
quality information that may help to demonstrate that an intervention
is cost effective.[238]
We asked Ms Willis whether her views on the utility
of RCTs in education policy had changed and she told us:
We are looking very carefully at randomised control
trials, and I have been discussing this with the Secretary of
State, who has been very interested in some of the work that the
Cabinet Office has been doing [...] The Secretary of State [...]
is very interested in it, and has been challenging us on whether
we should be doing more RCTs. We are going to build a process
into the research approvals process that I chair, to look in all
cases, for all of our research, every time a question comes up
as to whether we could adopt a randomised control trial approach.[239]
98. We are pleased that the
DfE has warmed to the concept of using Randomised Controlled Trials
(RCTs) in education policy and that the Department is challenging
itself on the use of RCTs. Policy should be backed up by
evidence and although evidence can come from many sources, RCTs
are particularly useful in social policy. The possibility of gathering
evidence from RCTs should be seriously considered every time the
DfE considers an education policy change.
225 Q 161 Back
226
Ev 82 Back
227
Ev 83 Back
228
Q 171 Back
229
Q 172 Back
230
Ev w12 Back
231
Ev w38 para 20 Back
232
Education Committee, Fifth Report of Session 2010-12, The
English Baccalaureate, HC 851, paras 15-20 Back
233
Education Committee, Fifth Report of Session 2010-12, The
English Baccalaureate, HC 851, para 20 Back
234
Cabinet Office, Test, Learn, Adapt: Developing Public Policy
with Randomised Controlled Trials, June 2012 Back
235
Cabinet Office, Test, Learn, Adapt: Developing Public Policy
with Randomised Controlled Trials, June 2012 Back
236
Cabinet Office, Test, Learn, Adapt: Developing Public Policy
with Randomised Controlled Trials, June 2012 Back
237
Science and Technology Committee, Second Report of Session 2009-10,
Evidence Check 1: Early Literacy Interventions, HC 44,
Q 163 Back
238
Cabinet Office, Test, Learn, Adapt: Developing Public Policy
with Randomised Controlled Trials, June 2012 Back
239
Q 162 Back
| 
