Examination of Witnesses (Questions 200-219)
PROFESSOR ALAN
MARSH, MS
VICKY DAVIES
AND MS
CLARE JOHNSON
WEDNESDAY 16 JANUARY 2002
200. Put bluntly, actually the ONE pilot approach
is not hitting the objective which was set out for it in the beginning
of this dual approach?
(Professor Marsh) This was the evidence that the clients
gave us, and people's memories are not infallible—this was
some five months after the interview so people have struggled
sometimes to remember what was discussed—and since many
of them were still on benefits perhaps benefits was in the front
of their minds. But it was not memorable. This was on average
a 32-minute interview, it would have to be a pretty memorable
32 minutes for them to say, "Yes, they talked about jobs
and this was the job ..."
201. I see that point, but of course in the
research you are doing about the interview, you can tell fairly
quickly to what extent the emphasis was on getting into work and
to what extent it was about the availability of benefits and so
forth.
(Professor Marsh) It did not succeed in emphasising
work as the main thing that was spoken of.
202. Can I just ask you one other point, your
latest published research, Cohort Two, Wave One, suggests there
was an employment effect for sick and disabled people but not
for other groups?
(Professor Marsh) Yes.
203. The In-House Report on the employment effects
of ONE, which is called number 88, uses administrative benefit
records to conclude there was an employment effect for lone parents
but not for other groups. Can you just explain to us, without
getting too technical, what conclusions we should draw from the
different findings there?
(Professor Marsh) You should focus on the size of
the differences. The In-House Report was using every single flow-through,
which would have an end of about 90,000, anything that twitched
would probably have been used. I, myself, am not fully qualified
to study the macroeconomics that were done there, I do surveys.
In this very large sample a closed sample of lone parents moving
through the system were interviewed at five months, and there
was no difference at all between the groups being piloted and
controlled, but there was in the first stage, in the voluntary
stage. We found a difference for the sick and disabled at the
compulsory stage, which was quite noticeable. I think for the
same reasons, if I may say so, novelty, that the lone parents
were introduced to ONE in the voluntary stage at exactly the same
time Working Families' Tax Credit was introduced, and they picked
up on it quicker in the ONE areas. In the second interview the
good news had spread to the control areas too, here is a new novelty.
This is the first time that sick and disabled claims were being
required and confronted with an interview, and that may have had
a kick-start effect.
Chairman
204. Staying for a moment with the question
of the whole evaluation process, if I said to you that I have
quite a strong memory of a representative from Deloittes, one
of the managers there, of one of the pilots, Richard Granger.
He actually said that he very much looked forward to seeing properly
studied technical data that gave him as a businessman some idea
of whether there had been success in the objectives in the pilot
areas. I am getting from you the sense that there are an awful
lot of health warnings with the results that you are bringing
to us this afternoon. That is not a criticism of you, because
you said yourself to do this properly you would have started earlier
and done it in other ways. What value can we actually attach as
a Committee to the professional work that you have done and what
are the limitations? Is Mr Granger likely to be disappointed as
a businessman trying to say, does this add up as a sensible reform
for Government to make? You, presumably, have no input at all
into any cost benefit questions, for example? You would not be
able to measure value for money, if we said it cost £68 million
between 1999 and 2001, social policy researchers would not have
a view about that either way?
(Professor Marsh) That research is being separately
done and it will incorporate some of our research in it.
205. I want you both to tell me to what extent
the hard value of the research that you have done is safe for
us to rely on?
(Professor Marsh) Let me be very firm about this,
what this evaluation lacks in design—necessarily it did
not have an element of random assignment or any of those new tricks
people can do—it makes up for with resources. It is an extraordinarily
well resourced piece of research. We have the numbers which enable
you to understand the process and we have the numbers to be able
to apply post hoc statistical controls. It just makes it
very, very hard work to do all of the extremely complicated statistical
manipulations which we have to do to control out the likely other
sources of variation, comparing pilots and controls, when you
do not have observations before the experiment.
206. Well resourced, quantitative studies I
can understand, what about qualitative? Did you find that you
could make up for some of the design faults, if you like, by the
fact that you had enough depth to do the qualitative stuff?
(Ms Johnson) To reinforce what Vicky Davies said earlier,
from our work it was always intended that by undertaking in depth
interviews with a sample of clients, it was approximately 100
interviews at each round, that our findings would be used alongside
the quantitative findings to help explain why things happened
the way they did for which groups of clients. That is the way
that you need to treat our findings, that you can look to us with
questions about why things happened the way they did, for which
clients, which goes very much hand in hand with the quantitative
data.
207. What is next? What else can we expect by
way of research? Do you know of other pieces of research that
you or other people are doing in the same area that we might have
some access to?
(Ms Davies) We are currently doing fourth round research
of work with ONE clients, particularly the impact of annual triggers
on lone parents, but also taking a more detailed look at some
of the different groups within the sick and disabled category
to find out what distinctions can be drawn from there. That also
includes some follow-up interviews with clients that we spoke
to in the last year and a half, so it will provide you with long-term
impact to see what progression has happened over time, the longitudinal
impact.
208. To be published when?
(Ms Davies) April or May.
(Professor Marsh) We are currently analysing Cohort
Two, Wave Two, and that will be available shortly. For lone parents
there will be Cohort Two, Wave Three and they will be contacted
again and new data gathered on them. After 18 months we really
will have a very clear portrait of what has happened to them.
Miss Begg
209. Just very briefly as a follow-up to the
qualitative research, do you get down to the fine detail of where
the personal adviser came from, whether he came from the Employment
Service or the Benefits Agency, and whether there is a difference
in the quality of their advice and whether that is an explanation
as to why in some cases there is job focus and in other cases
it is benefits and whether it comes from the quality of the training
of the personal adviser?
(Ms Davies) We do not get that information at all
because we only speak to clients, we do not have access to the
staff side at all.
210. Would that not be very useful for the Department
to know, because that obviously must influence the kind of training
that they will need when the thing is rolled out?
(Ms Davies) Yes, it would. Other research has being
going on and Tavistock may be presenting it to you later.
(Ms Johnson) We were not able to systematically know
where the personal adviser came from, but at the same time we
said there were cases when we were talking to clients that they
reported that the PA had said to them that they had previously
worked in the BA and in the ES and because of that they were or
were not in a good position to provide particular sets of advice
about benefits or moving towards work. It is an issue you need
to take account of.
Mr Stewart
211. In its memorandum to this Committee the
Department of Work and Pensions commented on research and said,
"Clients had a strong expectation that ONE would mean that
their benefit claim would be dealt with more speedily, but this
was not generally their experience". Can you tell us what
your research found about benefit processing?
(Ms Davies) Our research itself did not address benefit
processing. We were talking to clients themselves and we were
reliant on them expressing their individual experiences. We did
not talk to the BA staff or to the ONE staff that would be responsible
for processing claims at all. We do not have the broad statistics
which show whether the length of the process has been reduced
as a result of ONE. The main thing we can say is that where there
were delays in receiving benefits clients expressed considerable
frustration, often because they did not know when their money
was coming. Again, it comes back to the financial security issue,
that often prompted clients to go back to their personal adviser
to find out what was going on or, alternatively, going to specialists
that deal with either income support or incapacity benefit to
get further information. As a whole, that is not something that
we can comment on in great depth.
212. Professor Marsh, is that the same for you?
(Professor Marsh) It took longer, 3.4 weeks compared
with 2.9.
213. One of the objectives of ONE was to improve
the assessment and delivery of benefits by providing a more individual
service to claimants tailored to their needs. What does your research
suggest about the quality of information and the advice that claimants
received and the quality of services regarding benefits more generally?
(Professor Marsh) The valuation is higher in the pilot
areas compared with controls. In the pilot areas people generally
found the advice they got more helpful, more likely to be new
advice, rather than stuff they had heard before, advice that left
them feeling more hopeful and advice that treated them as an individual,
not as a number. Generally speaking they felt they had been treated
very well. Some of those differences are quite small. On that
last one I mentioned, for example, among lone parents 35 per cent
thought they had been treated very well compared with 21 per cent
in the conventional system of control.
214. There is quite a considerable difference
there between the ONE and the non-ONE pilot?
(Professor Marsh) More had a meeting, so the aggregate
has increased.
Ms Buck
215. Just going back to the function of the
personal adviser, it has come over strongly, certainly from your
research, that people were expecting the benefits element to be
the key role of the adviser, where did they get their impression
from generally?
(Ms Davies) It was not so much necessarily getting
the impression from somewhere, it was more what was their primary
concern. Why were they making the claim, and the information they
may have been told, did not necessarily sink in.
216. They did not go into a session with a personal
adviser thinking this is going to be different particularly?
(Ms Davies) No.
217. Okay. Did they feel they had enough time?
(Ms Davies) With their personal adviser? It varies
again in individual cases. There is also a sense that clients
often felt rushed or that their personal adviser was incredibly
busy and so was not able to spend longer with them to discuss
more detailed enquiries and to explore work in more detail, again
it is mixed.
(Professor Marsh) At least half thought they had got
all of the information they needed.
218. About benefits?
(Professor Marsh) Yes. About anything.
219. What has come out from what you have been
saying is that the experience is particularly focussed upon benefits.
What I am trying to get at is whether the research tells us anything
about people expecting the process to be about benefits and work
and the extent to which the interview, by not addressing the work
component as much, was driven by the client's expectation and
the adviser's expertise or just sheer lack of pressure and time?
(Professor Marsh) In these flow samples— it
is not often one gets such a large flow sample so you can tease
out the process— a lot are new clients. They have come to
claim benefit; it is an important event in their lives; it is
the first time they have done it in many cases; what else would
they expect to talk about? They have their start-up meeting. They
are told you have to come back in three days and have this meeting
or you will not get any benefits. They assume they are going to
talk about their claim.
|
