Examination of witness (Questions 63 -
79)
TUESDAY 16 JUNE 1998
DR ALAN
MARSH
Mr Wicks
63. May we start the formal proceedings
of the Committee? I think we should explain that unfortunately
our Chairman, Archy Kirkwood has a very bad back and cannot be
in London. He sends his apologies. In his stead I am taking the
Chair. Can we start by thanking Dr Alan Marsh very much for coming.
As you know, we are doing an inquiry into success measures in
social security, which has been a theme for some time but our
inquiry is very much stimulated by the Government's Green Paper.
May I start by saying you have done some of the most well regarded
research in Britain on issues around social security, employment,
work incentives and Family Credit and you have sent us a very
useful paper which summarises much of that research.[1]
May I ask in a sense the straightforward question? From that detailed
research and from your knowledge of the databases, do you think
that government is right to focus on a search for success measures?
(Dr Marsh) Yes. You have just disposed of the
first five paragraphs of my notes, so I can go straight to a comment,
if I may? I was a little puzzled as to the degree of comment in
previous evidence that people thought that this might be an impossible
job to do. Certainly it is thought to be extremely difficult to
set up success measures, set up a series, follow them through
and then make a judgement after five, ten or 20 years about whether
these measures and policy initiatives have been a success. I say
that I am puzzled because we have had relatively little difficulty
in describing the failures of the past. There is a literature
which you know very well, which has described the change in the
shape of the British income distribution, for example, in ways
which suggest that there has been a tremendous widening of resources
available to British citizens. I suppose if you were to set up
a series which wanted to know at the end of the measurement process
that you had rolled back the gathering tide of social exclusion,
you would have shown that occupation of the lower income strata
was much less predictable from your position this time last year,
for example, that there was much more movement, that it was much
less predictable by a series of markers for social disadvantage,
social tenancy, poor education, the habit of collecting income
tested benefits in and out of work and so on. That it was much
less predictable by your father's position in this distribution
and that those who do occupy the lower stratum of the income distribution
are much less prone to the experience of severe hardship, and
that the risks of hardship have been reduced. All of that is perfectly
possible. We have those measures now and we can continue those
measures. As you say, I work much more in the bespoke end of the
evaluation trade. The work we have done has been directly linked
to certain policy changes. We evaluate policy change and have
set up our own series of studies into Britain's low income families.
I think you are aware of that work so I will not drag you through
it. Although we work at a level of very considerable detail, we
have been able to show the broad direction of the effects and
the outcome of policy, the success if you like, of the use of
in-work means tested benefits. Broadly speaking we showed that
they were performing a lot better than a lot of people expected
them to be performing, myself included to be honest. Yes, they
created high marginal tax rates, but nobody seemed to mind and
it did not seem to affect anybody's behaviour very much. People
moved on to better advantage, if better advantage was to be had.
We showed that yes, lone parents were a lot better off in work
but more importantly those who went into work got better off.
Whatever measures you do choose, they should allow for the detail,
they should allow for the measurement of change. We are fortunate.
We live in a very well documented country and over the last ten
years the Economic and Social Research Council for example has
put a great deal of resources into improving the data environment
exactly for this kind of measure; the investment for example in
cohort studies, such that are done by Professor Bynner at City
University, that followed cohorts of people through. They have
been following through the 1958 cohort every ten years. These
give you a very accurate portrait of social change. Whether you
can then link every aspect of change to every aspect of policy
is quite another matter. This Green Paper is advertised to us
as a contract but it is a contract which perhaps the level of
specification of what the customers can expect in exchange for
their money and their votes is not what you would expect to find
in a contract. But that does not matter. There are other directions
you want to move in and of course we can measure it.
Mr Pond
64. Many of us know your work and that of
the Policy Studies Institute (PSI), as the Chairman indicated,
but just to get it on the record for those reading the transcript
who maybe do not, could you tell us a little about the way that
the PSI is funded and some of the implications of that in terms
of the role of DSS commissioned research?
(Dr Marsh) The Department felt about ten years
ago that it had insufficient research resources, particularly
after the split with the Department of Health and found that a
lot of the research which it was able to commission was either
not of the quality they would like or not of the scope. They set
up two core funded units and one has been at PSI for the last
eight years. The work I do for the Department is wholly funded
by the DSS. It also links to other work in PSI, which also tends
to be funded by other government departments and by research councils
and by various charitable foundations, for example into the New
Deal. Michael White's work into that will be of great significance.
65. Would the PSI see itself now very much
as an internal, albeit cross-departmental research body for government,
or does it see itself as independent of government?
(Dr Marsh) We have always clearly maintained our
independence. I am quite often asked this question and one of
the safeguards of independence of course is that we publish everything
we do. The advice to Ministers is always the same: if you do not
want this published, do not commission it. The Department has
valued and encouraged our independence and I now have the interesting
experience of working through, in a continuous series of research,
a change of government. I was always impressed how much the previous
administration valued our independence. We have a style of working
which is non-partisan and objective.
66. In response to the warning, if you do
not want to know the answers to this do not commission us to do
it, to what extent is the Institute itself able to look into some
of those issues which government might not actually want the answers
to?
(Dr Marsh) We get the money from somewhere else,
usually the Economic and Social Research Council.
67. Is it possible—I am being devil's
advocate here—to do that and perhaps be critical in some
areas of policy without compromising the possibility of raising
further funds from government departments?
(Dr Marsh) Yes, it is. I have had this experience.
We have given both good and bad news to administrations and they,
to my surprise and delight, have said yes, they understand that.
It is a question of being polite.
68. Bringing my final question directly
back to the success measures issue, in terms of the structure
of the way in which the departments, specifically here the DSS,
commission research, is the way they go about that a help to get
at some of the questions we are now addressing in terms of success
measures and are there ways in which those structures of commissioning
research could be improved to focus much more clearly on the success
measures approach?
(Dr Marsh) We often talk about this. We have a
very good relationship with the Analytical Services Division of
the DSS, who I know consider these questions all the time. If
you are to have joined-up policy thinking of the kind advertised
in this document, you are going to have to have joined-up social
science between, particularly, government departments and that
will take this broad look at trends and capture the data which
you need, although individual departments do need their details
seen as well. I stress that one of the surprising experiences
one has while doing this strategic benchmark research for government,
is that the level of detail and the exactness which is required
is very high. Again and again you see that valuable long-term
conclusions flow out of these and they are not always the ones
you expect. If you do very high quality custom built research
into the effects of policy change or the likely effects of policy
change, you have a tool in your hands which is going to help you
evaluate the outcomes. I would think that these are really no
more than a collection of suggestions at the moment of ways you
might go with this and in the white space provided I could give
as many more before the end of the morning. There is an enormous
number of things. If you do the basic research and do it well
then you can go back and revisit the data constantly. Part of
the relationship we have with the Department is that they ask
what the research told us about this, because we are now thinking
of moving in this direction and you can go back again and use
the research for this purpose. I hope that is why they core fund
us because they use us as a continual resource. That does not
make us a wholly-owned subsidiary of the Department.
Mr Wicks
69. On the issue of academic freedom and
independent research, do you have to check out your press releases
with the Department?
(Dr Marsh) Yes.
70. Do they change them?
(Dr Marsh) No. What happens is that they show
us theirs and we show them ours. They ask whether we really have
to say that and we say yes. We say something is wrong and they
say, is it really, and a dialogue carries on. We have just had
this experience. We are publishing Lone Parents on the Margins
of Work this morning and we have politely shown each other the
press release we are going to put out. I find myself counselling
other academic researchers who work for government in this art
and everyone seems to have to reach agreement on a press release.
It is futile because people have different things to say. You
really need separate ones. We do advise the Department on content
and we put them straight if they get it wrong and they query our
emphasis if they want to. No.
71. It would probably be for another institution
to do but I see a study of draft press releases compared with
final press releases and questions of academic freedom. Perhaps
that is not appropriate. You are relaxed about it anyway.
(Dr Marsh) I have done this job full-time since
1990 and I have never at any point felt that the Department put
any undue pressure on me to write my report in a way that would
favour the government position.
Ms Stuart
72. At what stage does the Department have
sight of any of your findings?
(Dr Marsh) As soon as we have completed the report.
We send the draft report. It is examined swiftly in the analytical
services division for technical competence. It then goes to their
customer divisions for comment. I then get a wedge of closely
typed comments which I work quietly through and satisfy their
curiosity about additional points they want to raise, argue with
them about things they want to contest. They are very often right
about things. It is very valuable. Our work is well regarded and
one of the reasons that it is is that it is very accurate. One
of the reasons it is accurate is that we get help and participation
from our colleagues in the Department.
73. So you see the purpose of submitting
a draft report to the Department as one of factual evaluation
rather than of judgement.
(Dr Marsh) Yes. I should stress that, perhaps
unlike a lot of people who work in this field, we at PSI do not
waste much of our time telling people what to do. We tell you
what is connected to what and if you want more of that you dial
that up. We do that quite well but our reports are not characterised
by tendentious recommendations and the sort of things which make
senior civil servants uncomfortable. We do the job. We do not
know enough sometimes as policy researchers to say that means
that and therefore you have to do that. There are all kinds of
other considerations which impinge. We make a scientific contribution.
74. May I suggest something to you? The
problem we have in social sciences is that we are dealing with
behaviour and however much we should like to be rational beings
the empirical evidence tells us that we are not. It is very easy
to analyse in retrospect and it is always easier to be an analytical
pessimist and explain afterwards why things did not work, hence
a lot of people's fear of saying what the success measures are.
Can you give me some indication of how comfortable you feel that
by analysing policy impact retrospectively you can make accurate
forecasts of future behaviour which is really where policy is?
(Dr Marsh) Yes. I could hide in statistics and
we have confidence intervals and we can tell you how much it might
vary but there are other approaches; we use a mix of approaches.
You can road test policies. More and more of this is now being
considered. The Earnings Top-Up pilot for example is a very large
programme which we are running alongside our low income families
research, whereby essentially extending family credit to people
without children is being tested in eight areas and compared with
four comparison areas for a period of three years. We will be
able at the end of that to answer a number of the key questions:
does it lower wages, does it create new jobs? With a reasonable
degree of certainty, hedged round by the normal statistical qualifications,
that will tell us a lot about the impact of this new policy. The
Americans carry out random trials rather in the way that medical
drugs are trialled, that people get a particular kind of help
to help them get into work, welfare to work, intervention randomly:
half the sample do, half the sample do not and you follow them
up. I am sure you are aware of the work of Jim Riccio at MDRC
who worked with us in PSI for a while. That is another way forward
in prospectively testing the likely effects of changes in policy.
You are right, most of the large survey research tells us how
we got where we are, what the antecedents of that were and, if
present trends continue, that is what you will get. Yes, there
are margins of error, of course there are.
75. Something which I have tried to elicit
from previous witnesses and never got a satisfactory answer. If
you are talking about pilots, which we are, there is a maximum
size for pilots for such things as whether our forms are effective,
the way we ask people to fill in a particular piece of paper for
which you actually need a small sample group. What is your idea
of a minimum size for a pilot where you actually assess behaviour?
(Dr Marsh) It depends how homogeneous your target
group is. If it is a run of the general working population then
essentially things smooth out at around 2,000. I have about 1,000
lone parents in my sample because they are such a homogeneous
group. They are all terribly like one another so the sample can
be a lot smaller and still be statistically accurate. About that
level.
Ms Hewitt
76. Thank you for this evidence which is
all very useful. The government is putting enormous emphasis on
getting people into paid work 16 hours a week plus. What would
you recommend as appropriate success measures: overall increase
in the percentage of people in employment, something specific
around exits from benefit into employment of 16 hours a week or
more, reduction in the number of households below half average
income, something else? Give us a sense there of what you think
are useful success measures?
(Dr Marsh) I would put a stress on longitudinal
dynamic measures. Cross section measures are very useful, of course
they are, and I am delighted to see we have restored the General
Household Survey for example, which keeps going a series which
we all value very much. You can do purpose built surveys on what
people respond to and the rate they move into work and so on.
Our lone parent cohort is showing what actually happens to lone
parents as time goes on. It is the length of time people spend
in this which is the most important thing to be concerned with.
Misfortune strikes people often in a life. It is the speed of
recovery which is to me the success measure for a welfare system
which is there to assist people. It is back to the safety net
concept. A lot of the problems complained of in the critique suggested
by this Green Paper can be summed up by the unintended shift of
the social security system away from a safety net towards an enduring
source of income for too many people. It is that shift it seems
to me you have to concentrate on. Accompanying that—if I
do not get another chance to say this may I work it into my answer
to your question?—is that another general tone of this Green
Paper and the success measures you are looking for, has to do
with making the money work harder, not passively unloading £90
billion a year but in those chosen areas where you think improvement
can occur, where success can be obtained, making the money do
other things than simply keeping people out of severe hardship.
If you include in that a greater sensitivity to avoiding deadweight
costs for example, one of the great perils of putting money onto
the proactive side of welfare intervention, which characterises
a great deal of the use of in-work benefits and something we are
terribly sensitive to and why we were road testing the Earnings
Top-Up for example in the way that we were, is to try to avoid
wasting the money, that things would happen anyway. That is the
biggest challenge for research. I am conscious that I am speaking
publicly but it is certainly worth considering that one of the
most welcome initiatives surrounds the additional support for
child care. Certainly as a general aim, making it easier for women
to work by providing high quality access to child care must be
a good aim. I will subscribe to that. What it is going to cost
and what you might have got for the money is something you really
ought to keep in mind. Again I would stress that it is the length
of time people are exposed to the risks of hardship which are
associated with too long a spell of time on income support. It
is not too late to consider in this to what extent income support
levels do avoid the exposure to severe hardship. We find it very
useful in developing these measures to make a distinction between
income, poverty and hardship. That sounds like the sort of thing
only a social scientist would be interested in, but it is vital.
It actually allows us to develop the measures we need. We know
income churns about, cross section income measures are not very
helpful. An awful lot of self-employed people tell us they have
no income but have expenditure which is about average and all
these problems are associated with `income'. A low income will
put you into the `below half average-income' section. The variation
in income among that group tells you nothing about their risks
of hardship, nothing at all. It is extraordinary. What it does
tell you is whether they are social tenants, whether they have
no educational qualifications, whether they have been claiming
means tested benefit in the past even if they are not now. As
you add those markers for disadvantage as measures so their risks
of severe hardship double and double again.
77. How do you define and measure hardship?
(Dr Marsh) We have in our survey. I will not drag
you through it now but I believe there is an account in the paper
I gave you. We have developed a measure of severe hardship designed
specifically to distinguish the poor from the very poor, if you
wish, those who are not coping on benefit, those who are getting
into chronic and unserviceable debt, those who are not able to
afford things which everybody else thinks they ought to afford.
There are over 70 measures which pour into that summary measure.
We find it extremely helpful. It is also helpful in deciding how
successful your measure is because one of the things we were able
to show was that when people get into work, having crossed that
threshold and started claiming family credit, it was commonly
assumed that they were really not any better off to be honest.
They are better off. They are not quite as better off as policy
might hope at first, but they carry on getting better off. When
they cross that threshold out into the point beyond their family
credit threshold the risks of severe hardship disappear altogether.
Most measures which have been used and developed as poverty measures
do not break into that.
78. Are you suggesting from that that one
success measure might be a reduction in the number of people experiencing
severe hardship? Secondly, are you arguing—I think you are—for
continued longitudinal studies so we can actually see the same
people and track them over time. Thirdly, should a success measure
be a reduction in the numbers of people experiencing more than
a certain length of time on income support or income related JSA?
(Dr Marsh) It is the abiding risk of hardship.
At any one time too many lone parents, about one third of them,
are in severe hardship and by anybody's standard that is poor
and not coping. Over a five-year period, 60 per cent of them are.
It is that kind of measure you have to get to grips with. It is
complex but it is the way the Japanese build machines. You put
in a large number of understressed components and you understand
how they work and then look at the pattern across them. There
is in the Green Paper a commitment to very broad measures which
frankly confuse good aims and good measures. It is not very profitable.
It is getting to the detail in the way that I describe.
Mr Wicks: It is an
interesting week to talk about the Japanese model.
Mr Roy
79. I represent a constituency in west central
Scotland which is continually highlighted as in poor health. I
am most interested in the points with regard to the barriers to
work and the relationship between poor health and deteriorating
poor health for lone parents. Could you expand on that and could
you tell us how health targets can intertwine with success measures
in relation to social security?
(Dr Marsh) That is, if I may say so, a most valuable
question. It is an example of perhaps the unexpected coming out
of a long series of detailed studies, that we now feel entitled
to conclude that poor health, below the level of entitlement to
disability benefit but persistent poor health, is a major barrier
to work in this country now. We have seen it in the work I described
in the paper you have, in particular one of our latest lone parents
surveys shows that the proportion suffering long-term limiting
illness which affects the amount of work they can do, rises from
about 15 per cent when we initially meet them to over 30 per cent
five years later. These start as 33-year-old women on average.
They are quite young. There is something about the experience
of lone parenthood which is driving their health down. That is
a combination of the experience of severe hardship, poor health
behaviour, it must be said—a great majority of them smoke
for example—but you put together these measures and you
discover that people who already face considerable difficult entry
to work problems are constantly dragged back by periods of severe
poor health. It is that interaction. That is why we do this research.
You can see that interaction between markers for disadvantage,
persistent low income, rising risks of hardship and then someone
hands them a health bill. Interestingly the health difficulties
sometimes come later. People seem to cope with periods on benefit
initially. Particularly lone parents, who make a lot of courageous
and resourceful responses for one, two or three years and then
just as things are getting better for them, their health lets
them down. That is probably not surprising. It is the use of longitudinal
measures which show you these patterns and how various strands
of disadvantage then interact and have the outcomes which cannot
be pinned down in one particular success or failure outcome but
can be described longitudinally and very accurately by doing this
kind of research.
1 "Lowering the Barriers to Work in Britain",
by Alan Marsh, Social Policy Journal of New Zealand, March
1997. Back
|
