3 Research
Overview
27. This chapter focuses on how we have handled research
since 2010, including the research that has been cited in written
submissions to our inquiries and that which we have commissioned.
More than half of the submissions we received in response to our
call for evidence in November 2014 discussed the use of research.[35]
Four issues were identified:
a) the sources of information we draw upon in
our work, including research that we commission ourselves;
b) how we identify relevant information for our
work;
c) how we treat the information we receive; and
d) the role of specialist advisers in our work.
The sources of information we
use
28. Most of the submissions considered that we could
draw upon wider sources of information than we do currently.[36]
Suggestions included consulting organisations such as think-tanks
or the media that regularly scrutinise policy or, making greater
use of information released through Freedom of Information requests.[37]
29. We were also alerted to a number of organisations
working on relevant topics that have been set up explicitly to
provide robust research evidence to inform decision-making and
enable better use of it by governments at all levels. The new
What Works Centres were highlighted in three submissions as a
potential source of information.[38]
Set up in 2013, the What Works Centres aim to improve the way
government and other organisations create, share and use high
quality evidence for decision-making by collating existing evidence
on policy programmes and practices, producing high quality synthesis
reports and systematic reviews in areas where they do not currently
exist, and assessing how effective policies and practices are
against an agreed set of outcomes.[39]
There are currently nine What Works Centres, several of
which focus on areas relevant to our work such as the What Works
Centre on Local Economic Growth.[40]
Other initiatives highlighted include the Local Government Knowledge
Navigator funded by the Economic and Social Research Council (ESRC)[41]
and the Alliance for Useful Evidence, both of which have published
reports that relate to areas of our work for example on the issue
of scrutinising the effectiveness of local authorities.[42]
30. Although
we have not used organisations such as the What Works Centres
or the Alliance for Useful Evidence and cannot comment on their
value, the committee in the new Parliament may wish to explore
how such organisations might usefully feed into the committee's
future work. The committee could, for example, as a pilot ask
one of these organisations to review rapidly the available evidence.
The potential benefit would be the ability to draw upon information
from a range of sources in addition to the submissions received,
and the skills and expertise of such organisations in terms of
assessing the quality and reliability of this information through
methods such as systematic reviews.
31. We were also encouraged to consider broader types
of information, for example making more use of statistical data
in our work,[43] or drawing
upon academic research in the field of social sciences.[44]
It was suggested that research councils such as the ESRC would
be able to assist us in identifying, or even undertaking, relevant
research.[45] Two submissions
suggested that we ask DCLG to provide an overview of the research
and evidence that they use to inform their policies and decisions.[46]
This could be based on the approach taken by the Education Committee
in its 'Evidence check' web forum to ask the Department for Education
to outline its evidence on nine topics, which was then published
by the Committee and views invited on the strength of the evidence
provided.[47]
32. We have
examined the Education Committee's evidence web forum. The committee
in the new Parliament may wish to explore its use with the Department
for Communities and Local Government. During this Parliament we
have pressed DCLG to explain the evidence base underpinning some
of its policies with limited success and an arrangement to elicit
better evidence would have been of assistance. It would also be
helpful if DCLG was more transparent in sharing the business cases
used to inform ministers' policy decisions.
USE OF COMMISSIONED RESEARCH
33. We have commissioned two pieces of research[48]
since 2010:
a) The nature of planning constraints
from the Cambridge Centre for Housing and Planning Research;[49]
and
b) Investigation into the British Retail Consortium's
Voluntary Agreement Regarding the Sale of Electrical Products
from Electrical Safety First.[50]
(In addition, Professor Colin Copus from De Montfort
University undertook, on his own initiative but with our encouragement,
research in relation to our 2013 follow-up work on our report,
Councillors on the frontline;[51]
we found his research useful and published it.[52])
34. Our use of commissioned research was welcomed
by four respondents to our call for evidence.[53]
Commissioning research was seen to be a useful way for a committee
to get an overview of a particular topic and to understand the
key issues within it. Specifically, it was suggested that it could
assist the committee to:
i) avoid drawing conclusions from a small number
of experiences based on the submissions it receives;[54]
ii) help to identify further lines of enquiry
which could inform the call for evidence;[55]
and
iii) provide a basis upon which to question witnesses.[56]
35. We welcome the positive feedback on our commissioned
work. In particular, we agree that such research works best when
it addresses a clearly defined objective:
i) in the planning research—relevance to
the forthcoming inquiry and a starting point for the inquiry setting
out the key issues; and
ii) in the labelling of electrical products—following
up an undertaking given in response to a recommendation in a report.
36. The process of commissioning research requires
greater planning ahead and a longer lead time for inquiries, which
may make it impractical for inquiries that require a short timetable.
We consider that select committees
will be assisted in their work if they commission more research
and they should build time to commission, carry out and review
such research into the timetables for their work.
| Reflection: Simon Danczuk MP
I have really enjoyed serving on the Communities and Local Government Select Committee over the past few years and it has been a real privilege to have contributed to the work of the Committee during this time. I believe we have a really good mix of people on the Committee, with different members bringing unique experience and perspectives. This has contributed to a refreshingly non-partisan work environment with members mixing well as a team and using their different strengths to add breadth and depth to the work of the Committee.
In particular I am pleased that the Committee has been involved in some really high profile issues in the recent past. This ranges from huge and complicated policy areas such as fiscal devolution and the future of local authority funding to more sensitive issues such as the Rotherham abuse scandal. I believe the Committee has acquitted itself well under the pressure that accompanies the profiles of these issues and has made a good contribution to shaping policy recommendations and holding people to account for their failures. I would certainly like to see the Committee spend more on such high profile and popular issues going forward.
I have always been of the opinion that the CLG Committee should focus on accountability as much as on policy recommendations and that this scrutiny should apply to individual local authorities as well as the department. In the future I would like to see the Committee take a more wide-ranging approach to scrutinising local government and exposing failings, especially if more powers are to be devolved downwards.
Finally I would like to thank the clerks for all of their hard work and commitment, they have been an invaluable help and ensured that the whole Committee has run effectively.
|
How we treat the information we receive
37. Several submissions saw greater scope to assess
and challenge the evidence we receive. The British Property Federation,
for example, said that "more could be done to ensure that
the research evidence and statistics submitted to the Committee
are challenged with rigour".[57]
38. Distinguishing between the quality and robustness
of evidence received as part of select committee inquiries has
been highlighted previously.[58]
In 2007, the Science and Technology Committee recognised the importance
of differentiating between submissions that "are firmly evidence-based"
and those that "are primarily opinion pieces".[59]
The Committee recommended asking organisations and individuals
to declare potentially relevant interests when submitting evidence
and give their expertise and experience when stating opinions
about particular matters, and it is for our successor committee
to explore whether to use different terminology when referring
to evidence-based submissions and other submissions.
39. Another approach was to use established checklists
or tools to check the quality and appropriateness of evidence
submitted as part of our inquiries. One example is the Bond Evidence
Principles checklist, designed for use by non-governmental organisations
(NGOs) in the international development sector.[60]
This checklist can be used to examine a wide range of evidence
from statistics, case studies, annual reports and research studies,
and assess it according to the extent that it draws upon the views
of people actually affected by particular policies (such as park
home residents)[61] and
the appropriateness of the methods used for what it is trying
to find out. The checklist scores the evidence on a scale from
1-4 and then gives it an overall score, and colour (amber, yellow,
light green or dark green) based on the quality and robustness
of the evidence. The use of a checklist like this might help us
to better assess the evidence received, provide an additional
tool to help scrutinise and challenge this evidence, as well as
be more transparent about how evidence is handled and weighed
up.[62] Other suggestions
included:
i) holding seminars or meetings to get an overview
of the volume and quality of the evidence available on a particular
topic;
ii) seeking briefings from organisations with
specialist and technical expertise such as the National Audit
Office at the start of inquiries;[63]
and
iii) training committee staff so they are better
able to apply a checklist such as the Bond Evidence Principles
checklist to the evidence received—this checklist is used
for assessing and improving the quality of evidence in evaluation
reports, research reports and case studies.[64]
40. The committee
in the new Parliament may wish to review how it handles research
evidence—or, as we sometimes found, material that purports
to be such evidence. We see scope to improve the process, for
example by asking respondents in the call for written submissions
if they cite research to explain the methodology and what degree
of review the research has been subject to.
41. The committee
may wish to pilot an evidence checklist, for example, based on
the Bond Evidence Principles. This would give a clear indication
to both committee staff and members of the overall quality of
the evidence submitted, based on the score and colour it receives.
Role of specialist advisers
42. Several submissions we received discussed our
use of specialist advisers.[65]
Suggestions included appointing specialist advisers on the basis
of their subject knowledge, specialist and technical expertise
(such as in the areas of research methodology, finance or performance
management) or likely awareness of, and where possible contact
with, relevant research. We were surprised by these comments as
such considerations are already taken into account when we appoint
specialist advisers.
43. Where we did see some scope for change was in
the suggestions for adapting the way that we work with specialist
advisers. Our usual method of operation was to appoint a specialist
adviser for an inquiry with the appointment effectively terminating
when the Government responds to our report. Following the lead
of the Liaison Committee, we have given greater emphasis to following-up
the recommendations in our reports and to monitoring the regular
announcements of initiatives in key policy areas. We therefore
see a case for appointing advisers for longer terms than just
a single inquiry, or appointing a panel of advisers each covering
a key area scrutinised by the committee, to provide us with access
to expertise. But there will likely still be areas of inquiry,
such as procurement, where more specific or in-depth knowledge
or expertise may be required.
44. The committee
in the next Parliament may wish to appoint a panel of advisers
in 2015 to cover the key areas it is scrutinising.
35 Professor Christine Whitehead (WSC 023); Richard
Styles (WSC 001); Professor Jane South (WSC 021); Local Government
Ombudsman (WSC 020); Local Government Ombudsman (WSC 019); Educe
Ltd (WSC 018); Association Of Independent LSCB Chairs (WSC 016);
Centre For Public Scrutiny (WSC 015); ACCA (WSC 014); What Works
Centre For Local Economic Growth (WSC 013); Local Government Knowledge
Navigator (WSC 022); Dr Tim Brown (WSC 011); London Councils (WSC 010);
Brethren's Gospel Trusts Planning Group (WSC 009); Core Cities
UK (WSC 008); Professor George Jones and Professor John Stewart
(WSC 007); Westminster City Council (WSC 006); John Thornton (WSC 005);
British Property Federation (WSC 004); Richard Tracey (WSC 002);
Richard Bate (WSC 012) Back
36
Richard Styles (WSC 001); Professor Jane South (WSC 021); Educe
Ltd (WSC 018); ACCA (WSC 014); What Works Centre For Local Economic
Growth (WSC 013); Local Government Knowledge Navigator (WSC 022);
Dr Tim Brown (WSC 011); London Councils (WSC 010); Brethren's
Gospel Trusts Planning Group (WSC 009); Professor George Jones
and Professor John Stewart (WSC 007) Back
37
ACCA (WSC 014) Back
38
Educe Ltd (WSC 018); What Works Centre For Local Economic Growth
(WSC 013); Dr Tim Brown (WSC 011) Back
39
For more information on What Works Centres see Cabinet Office,
'What Works Network', accessed 11 February 2015. Back
40
What Works Centre For Local Economic Growth (WSC 013). The What
Works Network includes: The National Institute for Health and
Care Excellence (NICE); the Education Endowment Foundation (EEF);
The Early Intervention Foundation; The College of Policing; The
What Works Centre for Local Economic Growth; The What Works Centre
for Wellbeing; The Centre for Ageing Better; What Works Scotland;
and The Public Policy Institute for Wales. Back
41
Professor Jane South (WSC 021); Local Government Knowledge Navigator
(WSC 022); Educe Ltd (WSC 018) Back
42
For example the submission from Educe Ltd (WSC 018) refers to
the Alliance for Useful Evidence report Squaring the Circle: Evidence at the Local Level,
May 2013, while the submission from Local Government Knowledge
Navigator (WSC 022) refers to two reports: Local Government Knowledge
Navigator, Collaboration in action: local authorities that are making the most of research,
June 2014 and Local Government Knowledge Navigator The role of evidence and research in local government,
September 2014. Other examples include Professor Jane South (WSC 021)
reference to the Knowledge Navigator Evidence Review on local
government and public health, which focuses on some of the same
issues that we covered in our Eighth Report of Session 2010-12,
The role of local authorities in health issues, HC 694-I. Back
43
London Councils (WSC 010) Back
44
Professor Jane South (WSC 021) Back
45
Professor George Jones and Professor John Stewart (WSC 007) Back
46
Brethren's Gospel Trusts Planning Group (WSC 009); Richard Styles
(WSC 001) Back
47
Education Committee, '"Evidence check" web forum', accessed
11 February 2015 Back
48
The process for select committees commissioning external research
has several stages:
the committee draws up a specification;
it is assessed in term so of appropriateness,
feasibility, timetable and value for money;
invitation to tender and submit bids
(if above a threshold); and
bids received are considered against
the tender. Back
49
Cambridge Centre for Housing and Planning Research, The nature of planning constraints,
March 2014 Back
50
Electrical Safety First, Investigation into the British Retail Consortium's Voluntary Agreement Regarding the Sale of Electrical Products,
January 2015 Back
51
CLG Committee, Sixth Report of Session 2012-13, Councillors on the frontline,
HC 432-I Back
52
Professor Colin Copus, Report to Communities and Local Government Select Committee: Councillor Workshops: Councillors on the Frontline,
August 2014 Back
53
Centre for Public Scrutiny (WSC 015); Dr Tim Brown (WSC 011);
Richard Bate (WSC 012); Professor Christine Whitehead (WSC 023) Back
54
Centre for Public Scrutiny (WSC 015) Back
55
Richard Bate (WSC 012) Back
56
Dr Tim Brown (WSC 011) Back
57
British Property Federation (WSC 004); Centre For Public Scrutiny
(WSC 015) Back
58
Science and Technology Committee, Thirteenth Report of Session
2006--07, The Last Report, HC 1108 Back
59
Science and Technology Committee, Thirteenth Report of Session
2006--07, The Last Report, HC 1108, para 33 Back
60
Educe Ltd (WSC 018) Back
61
CLG Committee, First Report of Session 2012-13, Park Homes,
HC 177-I Back
62
Professor George Jones and Professor John Stewart (WSC 007) Back
63
We consider the position of the NAO at paragraph 54. Back
64
Bond Evidence Principles, accessed February 2015 Back
65
Professor Christine Whitehead (WSC 023); Brethren's Gospel Trusts
Planning Group (WSC 009); Professor George Jones and Professor
John Stewart (WSC 007); British Property Federation (WSC 004);
ACCA (WSC 014) Back
| 
