Examination of Witnesses (Questions 20-39)
PROFESSOR DAVID
RHIND, MR
RICHARD ALLDRITT
AND MR
MARTIN WEALE
2 NOVEMBER 2005
Q20 Susan Kramer: Picking up on some
of those issues, and there is the whole issue of independence,
and believability. Many countries already have some form of statistics
legislation, to put that structure in place. To what extent do
you think the introduction of that kind of legislation would improve
confidence in national statistics here in the UK?
Professor Rhind: We think in the
longer term it is a necessary condition for having some form of
over-arching legislation. There is quite a lot of legislation
at present, but it is piecemeal, for example, the Census of Population
Act, the Statistics of Trade Act and so on. What we research or
seek is a statistical act which would have some things in common
with those in over a hundred other countries, but which updated
them because the circumstances have changed somewhat. What we
looked for was a greater degree of independence for the National
Statistician. What we looked for was a replacement for this Statistics
Commission as a statutory commission and reporting to Parliament
rather than to Government, and there are a series of other things
that we would like to see embedded in that, including, perhaps,
access by statisticians to data already held elsewhere in Government
so as to minimise the burden on respondents and filling in forms
multiple times. There are some quite complex legal constraints
in that at the moment.
Q21 Susan Kramer: What you are saying
is what implications are there for the ONS in that?
Professor Rhind: I would have
thought that perhaps—you must ask the National Statistician—but
I would have thought wholly beneficial. There is an issue about
how the National Statistician can be both the guardian of quality
across the whole of British Government in statistical terms, and
also head of the management in a major producer organisation and
the Conservative Party produced some proposals before the election
on how to handle that. Our proposals are a little less radical.
Mr Alldritt: The proposals for
legislation that we published last year in May 2004 took a particular
form, after we took extensive legal advice on the way to achieve
the goals within the United Kingdom structure of existing legislation.
The way that we saw things working was to have an act that created
a right for the National Statistician to draft a code of practice
which would be approved by a statutory commission and we saw that
as the most effective way of enhancing the influence of the National
Statistician. It is slightly indirect, but it is very difficult
under English law to give a civil servant in one organisation
authority across different government departments.
Q22 Susan Kramer: Okay. So what kinds
of resources are implied for the Statistics Commission if you
were to go along this path?
Professor Rhind: My judgment is
that actually, if we went along the path recommended in our report,
our report was deliberately constructed to be a minimal change
mechanism. We took the view that radical change in the British
Government to facilitate greater public trust in statistics was
unlikely to be successful, so we looked for ways of achieving
it by minimal change. Our view is that, with a statutory commission,
reporting to Parliament, we would probably need no more resources,
or very little more, because the leverage that that gives to a
commission is actually somewhat greater than the leverage that
we have earned over the last six years.
Q23 Susan Kramer: Given these proposals
that you made, I believe it was in May 2004, on the Statutory
Code of Practice of the Statutory Commission, what response have
you had from Government, or what timetable do you expect to hear
a response from the Government?
Professor Rhind: We have not received
anything formal, except comments to the extent that this was thoughtful
and very interesting, and all the rest of it. It has to be considered,
in my view, in the Framework for National Statistics, the review
which is about to start, I cannot see how some consideration cannot
be given to the legislation question if that same work is going
to be revised.
Q24 Mr Love: Earlier this week, you
published a report on PSA targets and that was characterised in
one of the heavy duty newspapers as suggesting that measures can
be too complex, and rely on data that are not timely and of low,
or unknown, quality. In some cases, departments have even set
targets for figures that do not exist. Is that an accurate assessment
of what is in the Report?
Professor Rhind: It is accurate
as far as it goes. We looked, deliberately, bottom-up, as it were,
at each one of the 102, 109, 120, whatever target—the number
depends upon how they are grouped; some of them are shared by
different departments—and what we found were, some were
very well specified. For example, a number of the Home Office
targets were actually very clearly specified. We saw others where
work was still in progress a year and a bit after the agreement
of the PSA target, to actually find ways of measuring whether
things had been achieved. In addition to that, we saw a number
of others where the targets seemed to have been forced into a
very quantitative framework where it was very difficult to see
how that could be made to work and where, in our view, qualitative
targets were actually rather more appropriate. I think some of
the Foreign and Commonwealth Office ones where the influence or
the control that the FCO has over some of the processes towards
that is pretty limited, and we had a feeling that there had been,
"Well, we have a process; we must carry it forward; we must
have numerical targets" and in some cases we thought those
were inappropriate. We have drawn up some generic conclusions
across the piece but there is a rich variety of different stories
in each one of the individual targets, but the span of them is
the one I have just described.
Q25 Mr Love: I understand you have
released a Report in draft form because you are looking for comments
on it and you suggest a number of recommendations and pick out
in particular that technical notes should be expanded into target
protocols and I think that is an important area. You also talk
about "a robust cross-departmental planning system".
What are the main changes that you think are necessary in order
that we can have the sort of trust and confidence in the PSA and
the figures that go into the PSA targets and therefore the targets
themselves?
Professor Rhind: Forgive me—I
am not going to not answer your question—if I can just put
a real dilemma that I think faces people who are setting targets,
and that came out in a meeting yesterday. To create good statistics—reliable
statistics, if they are not already in existence—is not
a five minute job. The Allsop Report that was published in 2004,
begun in 2003, made recommendations about regional statistics;
to get them to a situation where they are robust, is likely to
take until 2010 or thereabouts. Now, what do you say to a minister—"I
am sorry, you cannot set any targets because we do not have robust
information yet; wait for two governments until we get them."
Clearly, you have in some cases to use what is already there.
I think what we were looking for was to say, "Let us not
pretend that there are ways of measuring some things or let us
not get into the situation of using wholly unsuitable data because
you need to have something for measuring these PSA targets".
Frankly, if there are some targets which are not measurable, let
us make it clear that those things will have to be judged on a
qualitative basis.
Mr Alldritt: There is another
specific point whichwe mention in the report, which is, in quite
a lot of cases, targets are being measured not against regularly
published official statistics but management information coming
from within systems, within government departments, and we think
that where this is happening that is cause enough to require regular
publication of the statistics in an open and routine way.
Q26 Mr Love: Let me come in on that,
because you mention there, that is obviously a major part of the
recommendations you are making, about the independence of the
statistics that are used, and the numbers that are used. You mentioned
earlier that a significant amount of national statistics is not
produced by the ONS; it is produced within departments and you
talked about the National Statistician having greater powers in
order to monitor that situation. We have a system, as I understand
it, of audit based quality reviews and they are supposed to be
carried out on a five-yearly cycle. How is that process getting
on, and should that be strengthened?
Professor Rhind: You are absolutely
right in the sense that there should be a five-year cycle across
all the domains of statistics. In fact, in the first five years
about half of the areas have been looked at, and there are some
areas that have not yet been studied. I would hesitate to call
it an audit based approach; it has been in many cases a very detailed
approach but we think, and we produced a report last week that
said that we really wanted to move towards a much more typically
audit based approach and selection of areas, not on the basis
that we had to do everything every five years, but some assessment
of the risk and of the materiality of there being something wrong
in that. As I say, this came out last week and we are in discussion
with the relevant government departments about how to take that
forward.
Mr Alldritt: I think it is worth
saying that in a letter from the Financial Secretary this year,
the Financial Secretary recognised that the quality review process
had not been entirely successful over the last five years, so
there is a common understanding that there needs to be improvement.
The recommendations we worked up with the National Audit Office
have two elements which are audit. One is that the selection of
the statistics to be reviewed should be done on the basis of risk
and materiality, looking at what the risks of going wrong in that
area would be, rather than just everything every five years and
the other is that, in the actual reviews themselves, there should
be both an audit of how the statistics have been produced—are
you getting what you think you are getting—and a separate
process of looking at the fitness for purpose of the statistics,
in other words, are they meeting a real need from decision-makers;
are they serving a valid purpose? Our proposals are, therefore,
quite different in kind to what is happening at the moment.
Professor Rhind: Central to all
of this is, are we getting the statistics that are needed? Are
they fit-for-purpose? That presupposes that we have, in general,
a much better set of inputs from users about what they need. Now
getting users to specify what they need and anticipate that is
not straightforward but that seems to us to be a crux of the matter.
Q27 Mr Love: Let me press you just
a little bit on that, because you mentioned the National Audit
Office and, if you were to mirror the way the National Audit Office
works their reports are always agreed with the department that
they are auditing. Would you say that is the way forward, that
agreement had to be reached, even though that might be difficult?
Secondly, what sanctions should be available to the National Statistician
or the Statistics Commission in order not only to get the flow
of work running, but also to ensure that we look at the areas
of high risk, we look at the areas that you are particularly interested
in?
Professor Rhind: I think it will
be obvious to everyone that the National Audit Office carries
somewhat more fire power than the Statistics Commission—at
least, as yet. In 150 years' time, perhaps it will be different.
In terms of how do we get what we believe to be the right way
forward brought in, what would we say on that?
Mr Alldritt: I think it is important
to recognise here that we are not proposing external audits that
would parallel the National Audit Office approach exactly. We
are proposing there should be an external element but that the
government department responsible for the statistics with the
Office for National Statistics would lead the work. That may be
necessary in many cases because you need the expertise of, shall
we say, the Department for Education to look at educational statistics
in real depth, so I do not think they could be done fully without
involving the department. Now the extent of the external involvement
in that, is something that is open to discussion. We have said
there should be some, but clearly one way forward would be to
strengthen that progressively, the external component. It could,
on occasion, be the National Audit Office itself that might be
involved.
Q28 Mr Love: Let me as a final question
just ask you; I have some confidence in the newspaper that I was
quoting earlier on, that that might be relatively accurate and
I think the overall report is a relatively accurate summation
of what went into your report. Sadly, that is not always the case.
We know that statistics are, shall we say, used selectively regularly
in our media. Is there anything that we can do in relation to
that, and do you think the Statistics Commission has any role
whatsoever in trying to counter this?
Professor Rhind: Absolutely. I
think one role that we might have of facilitating relates to an
earlier point that was made, which was about the independence.
We see it as our very firm belief that interpretation of statistics
is something that actually statisticians must do more of, and
publish more of, separately from whatever ministers might wish
to publish. We believe there are other mechanisms for independent
assessment, analysis and publication results, there is that particular
element which is very important. Long term, I do believe that
it would help to—this is going to sound very idealistic—enhance
the education of those who write about statistics. I happen to
know that many of the people who go into journalism schools actually
come from arts backgrounds, and many have not a great experience
in quantitative skills. There are ways of fixing that, but it
is not a five minute job, again.
Q29 Ms Keeble: Thank you very much.
Can you say what you actually do to try to counter misrepresentation
of statistics by either the media or by politicians?
Professor Rhind: May I give you
an example? Perhaps the most obvious example is our report on
the revisions to economic statistics, where there was a great
kerfuffle in the press about the revisions to, in particular,
the GDP figures and my colleague here, Martin Weale, is an expert
on it, so he may well wish to come in. What we did—this
was so serious that it was affecting confidence in a central part
of economic statistics. We decided we must take a look at all
of this. We went out and got the evidence, we found out what happened
in other parts of the world, we had a thorough piece of work done
on analysing those economic statistics and we published a report
that said in this particular case, we think the interpretations
in the media, in the main, were inappropriate and incorrect. We
have stood up to the media and to government departments and to
others when it is appropriate. That had a very interesting effect;
I think there has been very little by way of the same strength,
or consistency, or avalanche of criticism about those particular
statistics since then. I think we pointed out to the government
departments concerned a number of lessons that they could learn,
for example, the very terminology made some people think that
these were something quite different to what they actually were.
The fact that it is called "revisions" as opposed to
"updates"—updates because new information comes
in—really made some people think that this was a process
that was, if you like, crudely put, being fiddled. So we have
gone back to the government department and said, "You can
do some things better, but we have published a report which said,
we have studied this particular process over a 10 year period
and it has worked pretty well".
Q30 Ms Keeble: I think you have set
out very clearly what the scale of the problem is, because there
can be real issues about public confidence, if the public feel
that it is thin, that they are being conned. However, in terms
of the example that you have just described, you have obviously
done some fairly weighty work on it and produced a publication.
Professor Rhind: And it took some
time.
Q31 Ms Keeble: Yes, I am sure. The
problem with journalists—and I speak as a former one, before
I came here—is that they can work very selectively and capture
the public imagination. How do you set about capturing the public
imagination with your statistics? Is it a challenge?
Professor Rhind: It is. Statistics
are not widely regarded as the most fascinating subject on the
face of the planet. What we can do, I think, is try and foster—and
we have done quite a lot of work in this—good communications
by government departments, not written in a way that only a PhD
in Statistics can understand but written in plain English, and
trying also to make clear why this is important and what it shows
is important. A 0.1% change may not be really something significant.
So the communications issues are areas in which we believe there
is still substantial scope for improvement. I happen to believe
that meeting with journalists and other intermediaries on a regular
basis is a very good idea because some of the things that happen
are because of misunderstandings and poor communication; for example,
a new development was that the Office for National Statistics
convened a meeting of journalists in the Bank of England, about
six or nine months ago, which was the first time it had happened
and they described some of their economic statistics and that
is not going to fix every problem, but at least those sorts of
interactions are tremendously important. At the end of the day,
there are only a relatively small number of intermediaries who
help shape the public's view of all of this, so it should be possible
to—
Q32 Ms Keeble: You rightly said that
the work is very important and that also means looking at not
just how you measure the results of government policy but the
way that you can determine it and that is key. Looking at, for
example, the whole area of social exclusion and poverty and, there,
policy formulation relies on reliable statistics—in particular,
small scale statistics to select areas if you look at programmes
like the New Deal and estate-based programmes. There is real difficulty,
is there not, about producing those, both to do with poverty and
also to pick up the small-scale areas. What progress are you making
on that, and how reliable do you think is the statistical underpinning
of the programmes that we have currently got? If I can just ask
my supplementary so I get them all off my chest at once, how do
you then feed into the small-scale statistics, statistics associated
with race which, as we have seen, is explosive when you get it
wrong?
Professor Rhind: As a geographer,
you will expect me to be very keen on fine-grained geographical
statistics and indeed I am. The prime source of a lot of fine-grained
statistics is the Population Census and we are involved in discussions
with the Office for National Statistics and the other census-taking
agencies about how that will be carried forward in 2011, what
the questions will be, and so on. Indeed, they are involved in
a very big consultation exercise at the moment. The Neighbourhood
Statistics, which are now widely available on the Web, are, I
think, one of the unrecognised success stories in British statistics.
It came out of a policy need, interestingly enough, social cohesion
policy needs and related areas, about poverty and so on. It became
clear that we could not make any sensible statements about all
of that on a fine-grained geographical, small-area basis and from
that the Neighbourhood Statistics have been created. Now they
had to be created from lots of different sources of information;
so comparability is far from perfect, but it is a triumph in what
has been produced so far.
Mr Alldritt: I did mention that
the Home Office told us this morning about work they are doing
to produce small area police data, for example, which would be
considerably richer than what is available at the moment, but
there is a lot of development work going on of that kind.
Professor Rhind: I am not sure
that answers all your questions.
Q33 Ms Keeble: At the time that there
were disturbances in the north of England, one of the criticisms,
if you recall, was that while there was neighbourhood information,
it was a bit patchy and that one of the factors that it did not
manage to capture was ethnicity. Therefore not misallocation,
but the decisions on allocation of resources, exacerbated some
racial tensions because of all of that I wondered if you would
comment on the way that you see the statistical work is making
sure that decisions are taken properly on allocation of resources.
Professor Rhind: Your point is
very well made. Well over £80 billion per year is allocated
on the basis of those sorts of geographical—
Q34 Ms Keeble: Eighty billion?
Professor Rhind: Eighty billion.
It is more than that, now, because of the money for the NHS, as
well as the money to local authorities, but may I ask my colleague
to fill in?
Mr Alldritt: On the specific point
on race and ethnicity, that is an example of an area that, speaking
as a statistician, is very difficult to get accurately. You have
got to look at the small number of sources that are capable of
delivering that. One is the Census, as the Chairman has mentioned,
and the other, potentially, benefits data, from the Department
for Work and Pensions, if it is coded into that in a reliable
way. The way forward on an issue like that is to look at those
few systems, the Health Service data to some extent, that are
sufficiently comprehensive and rich enough, but there are very
few of them.
Chairman: Now we have got about 10 minutes
left, and I am going to stop when the bell goes. Mark Todd.
Q35 Mr Todd: First, a brief apology;
I had to do a Standing Committee for an hour before coming here.
One of the things that you were just touching on in the exchange
with Sally [Keeble] was over the use of statistics and people's
understanding of them. Do you think—because I am reflecting
on the conversation we had at the Treasury, where it was fairly
evident that civil servants do not naturally turn towards the
use of evidence and statistics as support for policy choices in
all departments—there is not the need for greater training
in the use of statistics within the decision-makers in Government?
Professor Rhind: I think there
is a huge need for greater training, very generally. When I talk
to people outside and find that some of them do not understand
percentages, I get extremely concerned about how people can interpret
things that come back. So I could not possibly comment on whether
the same thing applies to our Treasury colleagues, but I think
there is a very major issue about mathematical understanding and
statistical understanding at all levels of society.
Q36 Mr Todd: And certainly among
politicians! I would have said they are more ignorant than many
in this area. You are recruiting commissioners to fill spaces
on the Commission. What criteria are you using, because there
are two possible perspectives: recruiting highly respected statisticians,
or recruiting users of statistics, who can then discriminate on
what is useful and meaningful?
Professor Rhind: The approach
the Commission was set up to take, really, and has taken ever
since, is that we would not see ourselves as, if you like, world-class
statisticians, professional statisticians. We can get that sort
of expertise, from wherever we want to, whether it is from universities,
whether it is abroad, whether it is the Civil Service, so that
if you look down the Commissioners you will find that most of
them have run biggish organisations and faced all sorts of organisational
challenges and been dealing with evidence and information. That
said, we do have some people: my colleague the Chief Executive
is a professional statistician; I think Martin Weale is a very
well known economist, which has some relevance to statistics;
and in Sir Derek Wanless we have someone who started off as a
mathematician/statistician. But the general principle is, we have
got people who are looking after the user interests and, to that
extent, they come from a range of different communities but ones
in which they have played a senior role.
Q37 Mr Todd: Do you think there are
groups who are under-represented amongst the Commissioners?
Professor Rhind: Yes. We do not
presently have anyone, I think I am right in saying, who has come
from the voluntary sector, or journalism even.
Q38 Mr Todd: Are you seeking specifically
to target those in the recruitment process?
Professor Rhind: We have put out
a rather broad advertisement and on that basis we will be able
to pick something that fits with the skills we already have.
Mr Alldritt: I think it is fair
to say we would like somebody with direct experience with one
of the devolved administrations.
Q39 Chairman: May I just pick up
another couple of points. You expressed concern about the ONS's
modernisation programme last year in a letter to the Financial
Secretary. What is your current view? The risks you were worried
about then, have they been dealt with?
Professor Rhind: The first thing
to say, Chairman, is that we think the modernisation programme
was overdue and was absolutely crucial. There had been substantial
under-investment for 20 or 30 years in systems and in some cases
they were extremely fragile and very labour-intensive and the
advantages of getting something into a common framework where
you could inter-relate the data clearly could not be taken. So,
no problems about the general principle. I do not think, in some
senses, our concern was about the management of that, that is
a management matter, but we are concerned to know when this will
begin to deliver and what it will deliver, and I think the programme
for delivery of the benefits is not quite as clear as, perhaps,
it should be. I have read recently that the ONS have reviewed
it after Phase 1 and are going to do it on a more incremental
basis, rather than the very "big bang" approach that
they initially started out. It seems to us that that is one of
the factors which will determine the success of ONS and, more
widely across government, statistics in the next few years. I
think the new National Statistician has recognised that and indeed
the Financial Secretary to the Treasury in that letter, I think,
says that the modernisation programme remains the first priority
of ONS.
|
