Private Finance Projects and off-balance sheet debt - Economic Affairs Committee Contents


Supplementary memorandum by the National Audit Office on Optimism Bias

  1.  The Committee asked for a note on the academic antecedents for the Government's use of optimism bias in its calculations. Optimism bias has its roots in the work of social psychologists and behavioural economists. We set out some of this work below. We also add some comments on the use of optimism bias in forecasting by the UK Government and for PFI projects in particular.

HEURISTICS AND SOCIAL PSYCHOLOGY

  2.  A number of social psychologists have worked on concepts of heuristics (mental shortcuts to make decisions) and how they introduce bias into decision making. Much of this work is based on the 1960s and 1970s research of Amos Tversky and Daniel Kahneman on human judgement. This won Kahneman the Noble Prize for Economics in 2002, and helped launch the academic discipline of behavioural economics.

  3.  One avenue of heuristic research focuses on optimism bias: peoples' tendency to predict better outcomes than actually happen. For example, Weinstein used psychological experimentation to find that college students and undergraduates rated their own chances as significantly above average of experiencing positive events and below average for experiencing negative events.[112] Others applied similar experiments to experts, managers, chief executives, traders and market analysts, finding that these groups are also susceptible to optimism bias. Weinstein argued that people are unrealistically optimistic because they focus on factors that improve their own chances of achieving desirable outcomes and fail to realise that others may have just as many factors in their favour.

  4.  Many of these experiments showed that people were more likely to be optimistic for events where:

    — the issue is within their control, more than random events;

    — a positive outcome if perceived as common and negative outcomes are perceived as rare; and

    — they have little personal experience.

  5.  Psychologists argue that this optimism bias is caused by peoples' tendency to overstate the likelihood of outcomes they desire (motivational reasoning) and overestimate their own abilities because they are comparing themselves to others (focalism) and it is easier to access information about yourself and your own project or task (egocentricism).

  6.  This general concept, it is argued, leads to a planning fallacy: a systematic bias in planning and decision-making whereby the costs, completion times, and risks of planned actions are underestimated, whilst the benefits are overestimated.[113]

  7.  Thomas Gilovich, Dale Griffin and Daniel Kahneman recently brought much of this work together in Heuristics and Biases: The Psychology of Intuitive Judgement (Cambridge University Press, 2002).

FLYVBERG

  8.  The concepts of optimism bias and planning fallacy have been adopted by academic researchers and practitioners outside the field of social psychology. Notably Bent Flyvberg applied the concept to transport infrastructure planning, measuring the uplift in actual costs compared to the original estimates.[114]

  9.  Flyvberg argues that optimism bias has four causes:

    — Technical causes, including imperfect information, scope changes and poor management, leading to inadequacies in business cases;

    — Psychological causes arising from the decision making processes of project promoters and forecasters;

    — Economic causes arising from rational incentives to get the project approved to create work; and

    — Political-institutional causes, particularly the question of whether cost forecasts are biased to serve the interests of project promoters in getting projects funded and started.[115]

  10.  Flyvberg thus highlights that underestimations in forecasts can be either real technical errors, or purposeful deceptions. The latter benefit the forecasters or their clients and stakeholders.

  11.  Flyvberg uses Kahneman's work to argue that planning fallacy stems from actors taking an "inside view" to a task or project—estimating its costs and time by looking at the constituents of the plan. He argues that they should instead take an "outside view"—looking at the results of similar projects. Flyvberg recommends that public policy makers rely less on forecasters' plans and more on comparisons with other similar projects. He calls such comparisons "reference class forecasting".

THE UK GOVERNMENT'S USE OF OPTIMISM BIAS

  12.  The Treasury introduced the concept of optimism bias into its Green Book guidance on appraisal and evaluation in 2003.[116] Its rationale drew directly on the work on Flyvberg. Since 2003, it has been standard practice to apply an uplift to estimated costs to take account of optimism bias.

  13.  To provide an estimate of how high such uplifts should be, the Treasury commissioned Mott MacDonald to undertake a review of optimism bias in public infrastructure projects in the UK.[117] Mott MacDonald found optimism bias to be a prevalent phenomenon in public sector procurement, and provided estimates of optimism bias by sector.

  14.  The Treasury advises public authorities to generate their own estimates of optimism bias based on their own experience, and sector specific studies. For example, the Department for Transport engaged Flyvberg to undertake a sector specific study on optimism bias in Transport infrastructure projects.[118] But where public authorities do not have access to such data, the Treasury advises them to use the generic estimates produced by Mott MacDonald.

THE USE OF OPTIMISM BIAS UPLIFTS IN THE FINANCIAL MODELLING OF PFI PROJECTS

  15.  The Treasury also updated its value for money guidance for PFIs to include uplifts of costs for optimism bias. In its guidance, it argues that optimism bias applies to two stages of contracting. Optimism bias up to the point when contracts are signed, and optimism bias of how costs and time will change after the contracts are signed.

  16.  The Treasury's guidance and template model applies optimism bias to both the counterfactual Public Sector Comparator and the PFI costs up to the point contracts are signed. This allows for how estimates changes between their initial proposal and actual contracting, and especially for how project costs and times develop over the tendering period.

  17.  But the Treasury's guidance and template model only apply optimism bias to the Public Sector Comparator, and not the PFI project, to take account of potential increases in cost and time of the Public Sector Comparator that would arise after the contracts are awarded. The guidance assumes that the fixed price PFI contract will deliver to time and budget once the contract is signed, but that the Public Sector Comparator will not. It does not address the issue of whether there may be optimism bias built into expected private sector performance, including the delivery of social benefits that have a non-cash value.

  18.  The Treasury thus assumes optimism bias will be greater for conventional projects and requires public authorities to adjust their estimates of VFM accordingly. If they did not have these adjustments, it would be less likely that public authorities would estimate that their PFIs would be cheaper than their Public Sector Comparator.

THE NAO VIEW OF OPTIMISM BIAS

  19.  The NAO believes that optimism bias is a sound theory and a good contribution to the Treasury's Green Book. Optimism Bias is a phenomenon that we frequently observe during our audit work. For example, our 2009 study on Building Schools for the Future found that the Department for Children, Schools and Families greatly underestimated how long it would take to renew England's secondary school estate, despite data from existing school projects clearly demonstrating that it would take longer than they were predicting.

  20.  We would like to see optimism bias reduced through better technical planning, estimating and forecasting, combined with better governance and scrutiny over business cases and plans. In particular, public authorities should reference their estimates for how long things will take to the experience of other projects.

  21.  But as the academic literature points out, the causes of optimism bias are not all technical. It thus matters in what context planning and forecasting is being done, and especially how the institutional incentives affect the planning decision. Applying uplifts to optimism bias is itself a subjective adjustment and susceptible to manipulation.

  22.  We have therefore always been professionally sceptical about optimism bias uplifts and especially in the context of Public Sector Comparators. Where optimism bias uplifts have been a material focus of one of our value for money reports we have scrutinised carefully how the optimism bias uplift was applied and its effect on the outcome. For example, the Committee of Public Accounts used our work in their report on the GCHQ building. They pointed out that GCHQ's PFI project relied solely on the highly uncertain assumption that the conventionally procured building would have over-run its budget by 24%.

  23.  We are also sceptical of the way optimism bias is applied to Public Sector Comparators but not to PFI projects. Although Mott MacDonald believed optimism bias in PFI projects was far lower than for conventional procurement, they believed PFI projects suffered optimism bias of around 5% after contracts were signed. However, difficulties in collecting the data meant they could not provide directly comparable figures between PFI and conventional projects.

  24.  Our 2009 study on construction costs showed that 31% of PFI projects were delivered late and 35% were delivered over the estimated price. Most of the price increases were from changes to the scope in the project made by the public authority or a third party. The academic literature would include such price increases in the calculation of optimism bias, on the argument that the forecasters should know the statistical likelihood of making amendments to the scope during the construction period.

  25.  In conclusion, our view is that is fair to consider optimism bias in estimates so long as adjustments for optimism bias are based on robust evidence. We would like to see more evidence behind the adjustments that we see. And we argue for separate disclosure of such adjustments and that public authorities make their subjective nature clear to the models' users.

November 2009











112   Weinstein, N. D. (1980). Unrealistic optimism about future life events. Journal of Personality and Social Psychology, 39, 806-820. Back

113   For example, Delusions of Success: How optimism undermines executives' decisions. Dan Lovallo and Daniel Kahneman, Harvard Business Review (July 2003) Back

114   Underestimating Costs in Public Works Projects: Error or Lie, Bent Flyvberg, Mette Skamris Holm and S<&oslash;>ren Buhl, Journal of the American Planning Association (Vol 68 No 3 Summer 2002). http://flyvbjerg.plan.aau.dk/JAPAASPUBLISHED.pdf Back

115   The British Department for Transport, Procedures for Dealing with Optimism Bias in Transport Planning, Guidance Document, Bent Flyvberg in association with COWI, June 2004, commissioned and used by the Department for Transport. Back

116   The Green Book on Appraisal and Evaluation in Central Government, Treasury, 2003, http://www.hm-treasury.gov.uk/data_greenbook_index.htm Back

117   Review of Large Public Procurement in the UK. http://www.hm-treasury.gov.uk/d/7(3).pdf. Mott MacDonald describes itself as a world-wide management, engineering and development consultancy Back

118   The British Department for Transport, Procedures for Dealing with Optimism Bias in Transport Planning, Guidance Document, Bent Flyvberg in association with COWI, June 2004, commissioned and used by the Department for Transport. Back


 
previous page contents

House of Lords home page Parliament home page House of Commons home page search page enquiries index

© Parliamentary copyright 2010