Mr.
Hoban: The Minister is right to point out that there is a
dual nature to paragraph 6. It gives the body a reason to report issues
up to the FSA where it feels that they will impact on market confidence
and financial stability. Equally, it provides a broader framework for
the body’s activity. My concern is that a tension could emerge
between the role of the consumer financial education body and some of
the wider objectives that the FSA has set up regarding market
confidence and financial stability. That tension is not a bad thing,
because it will provoke debate on some of the issues to do with the two
bodies and how they
work. There
is a fine dividing line between providing financial education,
information and advice on the one hand and acting as a consumer
advocate on the other. There are difficult boundaries regarding
products and hot issues in the financial services. We have seen that in
the work of some voluntary groups, whose experience of difficulties
that they and their clients have identified, and that arose from their
casework, flowed into a campaigning issue. Some of those difficulties
revolve around financial services. As the consumer body develops its
identity, persona and remit, there is a challenge as to where it draws
the boundaries. It has to be trusted by consumers, and has to be seen
to be credible by them if its work is to have value. We need to be
careful that it does not compromise that by being cautious in how it
approaches its remit. However, having had that debate, I beg to ask
leave to withdraw the
amendment. Amendment,
by leave, withdrawn.
2.15
pm
Mr.
Hoban: I beg to move amendment 48, in
schedule 1, page 54, line 32, at
end insert— ‘(d) the
quantitative and qualitative measures it will use to determine whether
it has met its
objectives.’. This
morning we discussed how we measure or evaluate the performance of the
consumer financial education body. The hon. Member for South Derbyshire
talked about the Treasury Committee evidence session with the FSA, at
which he probed the people from the FSA about how they calculated some
of the measures of success that they had used in evaluating financial
education. The example he gave was that when they talked about the
number of schoolchildren they had reached, it turned out to be the
number of schools to which they had sent packs multiplied by the number
of pupils at those schools. He suggested that the information they sent
may have been put in the bin or left lying on the shelf—there
was no guarantee that, having sent the information to all those
schools, it would have been read, digested and used in lessons. The
risk is that the focus is on input and process rather than on
outcomes. The
hon. Gentleman not only raised the topic at the Treasury Committee but
also expanded on it at our Committee’s evidence session. The
FSA, in its additional memorandum to the Committee, talked about how it
assesses or evaluates the impact of the financial capability work. Four
main areas were referred to, namely reach, content, process and
impact: “Reach—the
number of people reached by a
programme...Content—satisfaction with, and quality of, the
information and guidance received...Process—appropriateness
and efficiency of the intervention; and...Impact—the
intention to act and/or the actions that have been
taken.” Going
back to the hon. Gentleman’s example of packs being sent to
schools, the FSA would have achieved a great deal in terms of reach,
and the content box may well have been ticked—there may have
been a good standard of content—but I am not so sure about
process and, certainly, there would have been very little impact if the
information packs had all ended up in the bin. We need to make sure
that there is good evaluation of the work done by the consumer
financial education body, so that it can demonstrate
progress—ultimately, I suppose, against the baseline study of
2006. People could then see a demonstrable level of progress. It is
important to make sure that the right sorts of measures are being made
of its
success. The
appendix of the supplementary memorandum submitted to the Committee
looks at various activities that the FSA has undertaken as part of its
consumer education programme, and gives some measurements relating to
what people have done as a consequence. Thus as a result of the
“Parent’s Guide to Money”, which I suspect is the
publication given to expectant mothers, 43 per cent. of people took
action, rising to over half of people on low incomes; 67 per cent.
reviewed their monthly spend and income. Some 91 per cent. of those
involved in workplace seminars reported that the seminar made them
better at finding financial information, and 87 per cent. reported that
it had made them better at comparing prices.
The
measurement made of the “Learning Money Matters” pack for
schools, delivered through the Personal Finance Education Group, is as
follows:
“The
majority of teachers are very satisfied with the support provided by
pfeg
consultants”. That
sounds good, although it does not necessarily lead to actions that
young people might take to manage their money. The evaluation
continues: “Involvement
in Learning Money Matters often acts as a catalyst to encourage
teachers to initiate or expand the teaching of Personal Finance
Education in their
schools.” “Money
Doctors”, the higher education programme, is making sure that it
reaches out to students. Interestingly, that demonstrated some success,
in that after attending the seminar, students were more likely to check
their balance before withdrawing cash. The 50 per cent.
of “students
who had not attended a session reported that they were constantly or
usually overdrawn on their main bank
account,” but “this
dropped to 40 per cent. for those students that had been to a
session.” That
sounds like a bit of
progress. My
point is that we need to ensure that there are good measures available
to help us evaluate the work that the consumer financial education body
does, and the measures should be
transparent. Amendment
48 adds to paragraph 8(4) of schedule 1 a requirement
that “the
quantitative and qualitative measures
it” that
is, the
CFEB— “will
use to determine whether it has met its objectives”
be set out in the
annual plan that the CFEB publishes, so that we can see clearly not
only the objectives and their relative priority, and the allocation of
resources between the objectives, but the measures that will be used to
determine whether they have been met. I recognise that some of the
measures are long term. Some 97 per cent. of people might become more
aware of financial matters and might find out financial information
now, but what proportion of people will use that to improve their
pension provision, for example, in 10 or 15 years’ time? How do
we track the long-term benefits of what they have
done? Simply
ensuring that people are more comfortable finding financial information
might not lead to the step change that we want in people’s
preparedness for retirement, or ability to withstand a shock to their
income or unexpected expenses. The real measure of success is whether
it actually changes people’s outcomes. Are they better off as a
consequence of being given financial education? As well as setting out
its objectives, it is important that the consumer financial education
body tells us in its annual plan how it will achieve and measure those
objectives. That will give people more confidence in its work, and will
increase transparency in measuring how successful it has been. Given
the significant sums of public and private money that the body will
benefit from over the years, it is important that there is proper
discipline, so that we know exactly what it is doing and how well it is
doing it.
John
Howell: I want to follow up on a couple of points, but I
spoke earlier and do not want to duplicate what I said this morning. We
need to make a more subtle distinction than we have done so far. We are
talking about two different things. The first is how to
measure the success of the body as a whole, in terms of increasing
financial capability. The additional memorandum from the FSA sets out
five
tests: “being
able to make ends meet, being able to keep track of finances, planning
ahead, staying
informed— that
is, individuals staying
informed— and
being able to choose financial
products”. Those
are all soft targets and difficult to evaluate. It would have been
helpful if the FSA had been able to provide a little more behind that.
I assume that the main way in which it will measure that is through
differences between the baseline survey in 2006 and the next survey,
whenever that takes place. Those surveys are useful, but there comes a
point when we need further qualitative information beyond
that.
As for
evaluating the individual projects that the new body will undertake, my
hon. Friend the Member for Fareham set out four tests: reach, content,
process and impact. How do those four tests for individual projects
relate to the overall tests being set for the organisation as a whole?
We seem to have two sets of tests, and yet there must be a sense in
which the overall success of the body is represented by the cumulative
success of the individual projects that it has taken up. Reach is
clearly an important part of assessing a programme’s value for
money, but the weight given to that in the past has led to suggestions
that there has been an over-claiming of success for the projects
undertaken to date.
On content,
the FSA says:
“satisfaction
with, and quality of, the information and guidance received
satisfaction.” From
whose standpoint is that being asserted? On the basis of the material
that we have been provided with, satisfaction with the content is the
FSA’s view of satisfaction, whereas it should be from the
consumers’ point of view. Process comes down to the
appropriateness of the channel used. Again, it is not clear how that
will be integrated and managed.
Lastly,
impact is defined as
“the intention
to act and/or the actions that have been
taken.” Ultimately,
that is about behavioural change. It is only of short-term interest to
talk of people’s intentions to do something. For example, when I
read something I am often inspired to go off and do
something—that is partly due to the sort of things that I read.
However, in many cases those intentions do not get turned into action
because of other things that occur in life. Intentions are great, but
let us have the actuality. We do not see that happening. Where there
are glimpses of that, they are only short term. For example, if we look
at the money guidance pathfinder, we find that there is only two months
to go back and look at its impact. A longer-term approach would have
been nice.
Earlier this
week, my hon. Friend the Member for Fareham mentioned how much German
he had forgotten. I was very enthusiastic about ancient Greek, but I
can barely understand any now except for the well known Homeric phrase,
“rosy-fingered dawn”, which these days is hard to see. We
forget things, and a repeat process is necessary to determine success.
Therefore, the amendment is very important. It is crucial to have
determinants of the quantitative and qualitative success measures that
will be included for this body. Without them, we will still be asking
what success looks like in a year and two years’
time.
Ian
Pearson: My point about the amendment is that although it
is useful in specifying some things that will aid evaluation, that will
happen anyway. For more than 20 years, I had an interest in evaluation.
Unlike the hon. Member for Fareham and his German, I have kept up an
interest in the literature. I am talking here about the link between
inputs, outputs, outcomes and impacts and how effectively policy
evaluation methodologies can be designed.
On Second
Reading my hon. Friend the Member for South Derbyshire and others
raised a number of concerns about how evaluation has been performed to
date. The Committee will be aware that the FSA has set out the tools
that it uses to address its financial capability work, and the
evaluation methodology—both qualitative and
quantitative—employed to measure the effectiveness of the money
guidance pathfinder. For the money guidance pilot, the FSA and the
Government commissioned an independent evaluator, studying both short
and longer-term effects of the service, including actions taken by
consumers as a result of using the service. They were trying to look at
impacts, but as the hon. Member for Henley pointed out, they need to be
considered over different periods. This is a complex area, but the
baseline work stands comparison with that which has been done
internationally. In evaluation, the game has moved
on.
2.30
pm With
the annual plan and annual report process in place, I am confident that
there will be appropriate mechanisms by which we can judge the new
consumer financial education body’s success and impact. Of
course, we will want to ensure that the new body has clear targets to
meet against its objectives, and we will also want those targets
properly evaluated. I want to highlight the fact that the Bill also
gives the FSA the power to commission an independent review of the
economy, efficiency and effectiveness of the new body in discharging
its consumer education function. That independent review would have the
right of access to documents and information held by the new body that
would be reasonably required. Furthermore, we would expect the National
Audit Office also to play a role, certainly in reviewing the use of
Government resources. Given that the FSA has recently called in the NAO
to review its operation, it is also likely that the new body would be
subject to NAO review at some point in the future.
I do not
think that there is any difference of substance between us on this
issue. We want quantitative and qualitative evaluation of the new body,
which we hope will be successful. We want to examine the new body
closely, but the amendment, which would put that function in the Bill,
is unnecessary.
Mr.
Hoban: I am grateful that the Minister keeps up with the
literature on evaluation. It should reassure us all that the methods
that will be followed by the FSA and the consumer financial education
body will be up to date, modern and efficient. That is good.
The consumer
financial education body will undertake a great deal of work to assess
its effectiveness and to establish to what extent it achieves its
objectives, but one challenge is to ensure that the right type of
measurement is used; earlier, I referred to the example given by the
hon. Member for South Derbyshire. The consumer financial education body
will need to consider very
carefully how it chooses to measure success, because the targets and
goals that it chooses need to be credible if they are to command the
confidence of the people who fund the body. Of course, there is a
distinction between how effective somebody is in undertaking those
tasks and whether the money is being spent wisely. I want to raise in
the stand part debate an issue about funding for the new
body. However, I
take on board the Minister’s remarks about what the new body
will do. Therefore, I beg to ask leave to withdraw the
amendment.
Amendment,
by leave, withdrawn.
Question
proposed, That the schedule be the First schedule to the
Bill.
| 
