1.The Committee holds an annual evidence session with the Governor of the Bank of England. One of the topics raised during his appearance in January 2018 was the Retail Prices Index. The Governor told us that as the Retail Prices Index had “known errors”, it should not be further embedded in Government contracts. He said that “if there is anything that this Committee can do, with farsightedness, to advance this process, it would be a real service.”
2.This prompted us to schedule two public evidence sessions into the use of the Retail Prices Index in June 2018. But with strong opinions on both sides of the debate, it became clear a more thorough investigation was required and we launched a full inquiry the following month.
3.This chapter sets the context for the inquiry. It provides background information on consumer price indices. In Chapters 2 and 3 we set out criticism of the Retail Prices Index and explain the steps we believe the Government and statistical authorities need to take to address these criticisms.
4.We would like to thank the Royal Statistical Society for their advice in producing this report.
5.Box 1 sets out the bodies involved in producing and maintaining statistics in the UK.
The origins of the UK’s current statistical system date to the establishment of the Central Statistical Office, a governmental body, in 1941. A series of mergers between various governmental bodies involved in producing and publishing statistics over the rest of the century led to the creation of the Office for National Statistics in 1996. A Statistics Commission was established in 2000 to provide independent advice and oversight.
The Statistics and Registration Service Act 2007 created a statistical system that was independent from Government. The Statistics Commission was abolished.
The Act created the UK Statistics Authority with responsibility for “promoting and safeguarding the production and publication of official statistics”. It was established in 2008 and the statistical system is comprised now of four entities:
The current Chair of the Board is Sir David Norgrove and the Board has eight other non-executive members and three executive members. The executive members are the National Statistician, the Director General of Regulation and a Director General from the Office for National Statistics on a rotational basis.
The National Statistician is advised by two independent advisory panels: a technical panel which advises on technical aspects of the statistics and a stakeholder panel to provide advice on the uses and applications of price indices.
6.Consumer price inflation is the rate at which the prices of goods and services bought by households rise or fall. The Office for National Statistics produces estimates of these price changes every month, usually in the form of the change on 12 months earlier. These estimates—often referred to as the rate of inflation—are important indicators of how the British economy is performing and of the rate at which prices are increasing. The three main estimates produced by the Office for National Statistics are the Consumer Prices Index (CPI), Consumer Price Index including owner occupiers’ housing costs (CPIH) and the Retail Prices Index (RPI).
7.These estimates of inflation are produced by using price indices. A price index works by combining two types of data: the increase or decrease in the price of certain goods and services over a given period and the proportion of household expenditure that is spent on those goods and services.
8.The most accurate way of measuring inflation would be to measure the change in price of every item that is purchased by every household. As this would be impractical, the Office for National Statistics instead tracks the prices of a selection of consumer goods and services which are believed to be representative of purchases by households. This is referred to as the ‘inflation basket’.
9.The present basket contains around 700 goods and services. The contents of the basket are updated every year to reflect changing spending patterns. Box 2 gives a flavour of the considerations that are taken into account. Sample prices for each item are collected monthly from shops, the internet, by telephone or from other sources. Around 180,000 separate price quotations are used each month in compiling the indices.
Items added to the inflation basket in 2018 by the Office for National Statistics included:
Items removed from the inflation basket in 2018 included:
10.Households spend more on some goods and services than others. A 10 per cent increase in the price of petrol will have a bigger effect on household spending than a 10 per cent rise in the price of a punnet of raspberries. The goods and services in the basket are therefore weighted to reflect the relative amounts that households spend on them. The weights are calculated from sales and survey data of consumer spending.
11.These weights are reviewed and updated every year to ensure they remain representative of household expenditure patterns. For example, a large rise in the price of tea may cause consumer spending to switch towards coffee, thus requiring a shift in the relative weights the following year. Changes also reflect the fact that spending on new goods and services replaces spending on older items.
12.The prices of goods and services are compared with their prices in the corresponding month of the previous year. The price changes are all weighted and an overall average price change for the 12-month period can be calculated. Box 3 provides a simple example of one method of calculating an annual average price change from a basket of goods.
For the purposes of illustration, this example uses an inflation basket containing four goods: beer, bread, potatoes and tea.
Between October 2017 and October 2018, the prices of these goods changes as follows:
For the purposes of this example, it is assumed that a household spends 40 per cent of its income on bread, 40 per cent on potatoes, 10 per cent on beer and 10 per cent on tea. These percentages will form the weightings for the index.
To calculate the average rise in all prices, the average expenditure on all four items in October 2017 is compared with the expenditure on all four items in October 2018.
Average price rise = (Beer (14.3 × 10) + Bread (6.2 × 40) + Potatoes (6.6 × 40) + Tea (-3.4 × 10)) ÷ 100 = 6.26
The average rise in all prices from October 2017 to October 2018 is therefore 6.26 per cent.
13.The goods and services which make up the inflation basket differ slightly in each of the three main inflation indices produced by the Office for National Statistics—Consumer Prices Index (CPI), Consumer Prices Index including owner-occupiers’ housing costs (CPIH) and Retail Prices Index (RPI). In addition, the weightings used in CPI and CPIH differ from those used in RPI.
14.There are many ways in which the data on price and expenditure can be combined to produce estimates of the changes in price level. For much of the calculation process where expenditure weights are available, the method used in all three indices are identical. But where expenditure weights are not available, different formulas are used at the first stage of what is termed ‘price aggregation’ (where an average price change is determined across similar types of products), as discussed below.
15.The differences between how the main consumer price indices are calculated at the first stage of price aggregation are at the heart of the arguments which this inquiry has heard for and against the use of RPI. We consider these arguments in the next chapter. The rest of this chapter explains the purposes for which consumer price indices are used today and outlines how and why they were developed.
16.The UK Statistics Authority commissioned a review of British price indices by Paul Johnson, Director of the Institute for Fiscal Studies, in 2013. His report identified five main uses of price indices:
17.The evidence we received has focused largely on the second of these uses—in particular the choice of whether to use CPI or RPI to ensure regular payments are not eroded over time by inflation. As RPI has generally been around one per cent higher than CPI in recent years, the choice of index can have a substantial effect on the value of pensions, index-linked gilts or the cost of any inflation-linked item such as rail fares or certain commercial contracts.
18.The Government summarised uses of price indices in the 2018 Budget:
“The Government uses measures of inflation to uprate some taxes and benefits; to determine changes in rail fares, reflecting industry costs; to uprate the rate of interest on student loans; when setting the inflation target for the Bank of England; and as the reference rate for government bonds linked to inflation. In the private sector, inflation is used in some wage agreements; to uprate certain pension payments, particularly defined benefit pensions; and in financial markets.”
19.Many witnesses to our inquiry have suggested that the Government in recent years has elected to use CPI to uprate regular payments that it makes, such as state benefits, and elected to use RPI to uprate regular payments that it receives, such as the interest on student loans. We will assess the issue of ‘inflation shopping’ or ‘index shopping’ in Chapter 3.
“The inquiry, which has been of a somewhat elaborate description, has occupied several years … So far as is known this is the first attempt to compile continuous records of the retail prices of commodities in the United Kingdom in an official report. The available data are much less abundant than in the case of wholesale prices, and a considerable portion … represents original research.” H. Llewellyn Smith, Board of Trade, August 1903.
20.The first recorded consideration of the change in the general level of prices, and how to measure it, is believed to be a 1707 book by Bishop William Fleetwood. Further attempts to address the problem were made over the 18th and 19th centuries, and economists began to consider different methods for collecting and measuring price changes and the use of weights.
21.By the start of the 20th century, the government had begun to explore recording changing retail prices as interest developed in the living conditions of the working classes. The Board of Trade published a report on wholesale and retail prices in 1903, “with comparative statistical tables for a series of years.” As the quotation above from the deputy comptroller-general of the Board of Trade indicates, this was the first official attempt to make such a record. The importance of wholesale prices to businesses meant that these had been much better recorded than retail prices previously.
22.The information in the 1903 report on retail prices was regarded as incomplete, “notably in regard to the usual rents of workmen’s dwellings and the range of prices most commonly paid by the working classes for food commodities.” The Board of Trade produced further reports in 1908 and 1913 to correct these deficiencies. A further aim of these studies “was to obtain a standard of comparison in regard to the cost of living, which could be applied as between the various districts of the United Kingdom, and also to foreign countries.”
23.Building on this work, the Ministry of Labour began to publish the ‘Cost of Living Index’ from 1914. The inflation basket included food and drink, rent and rates, clothing, fuel and lighting. The weightings for the items in the basket were set in 1914 and never updated.
24.A review of working-class expenditure in 1937/38 found that the 1914 weightings did not accurately reflect household spending, particularly on alcohol and tobacco. After the Second World War, a ‘Cost of Living Advisory Committee’ was established to advise the Minister of Labour and National Service on “the basis of the official cost-of-living index figure and on matters connected therewith”.
25.In an interim report published in 1947, the Advisory Committee recommended that a temporary index using the expenditure spending data from the 1937/38 survey should be introduced to replace the Cost of Living Index. This temporary index, the ‘Interim Index of Retail Prices’, began in June 1947.
26.The Advisory Committee also considered the nature of a permanent replacement for the cost of living index and discussed the “fundamental question” of the purpose for which a revised index should be used:
“The existing index is designed to show the average percentage increase in the cost of maintaining unchanged the standard of living of a selected group of families as at a fixed date. Our discussions have disclosed that the entire conception of an index purporting to measure the changes in the cost of maintaining an unchanging standard of living of a section of the community may be strongly challenged, the description of an index as a ‘’cost-of-living” index being felt to be inappropriate and misleading.”
27.The Advisory Committee said the alternative to an index based on an unchanged standard of living was one where the weights in the index were kept continuously up-to-date. To maintain such an index, it envisaged “inquiries into consumers’ expenditure would be instituted to take place at regular intervals and periodically the weights would be adjusted to take account of changes that had been disclosed.” It felt however that with rationing, spending conditions were not sufficiently stable at that point for a new inquiry into consumer expenditure.
28.The expenditure survey eventually took place in 1953/54 and a new index —‘The Index of Retail Prices’—commenced in January 1956 following further work by the Advisory Committee.
29.The new index, which would become known as the ‘Retail Prices Index’, or ‘RPI’, differed from the Cost of Living Index in a number of ways:
30.By the late 1960s the inflation basket for the index contained around 350 items with 120,000 price quotations captured each month. There were occasional methodological changes, such as the introduction of a ‘meals out’ category and a move to measuring owner-occupier housing costs by mortgage interest costs rather than equivalent, or imputed, rents (the cost of housing in price indices is considered in Chapter 3). Changes were considered and proposed by the Advisory Committee, which met five times in the 1960s and 1970s and made recommendations to government.
31.A new Advisory Committee was convened in 1984 and a wide-ranging report published two years later made many methodological changes. The report set out some principles to be used when evaluating the effectiveness of the index and considering changes:
(a)The value of the RPI arises from its consistency and continuity—change should be made only where necessary;
(b)The RPI is an index of price changes, not a cost of living index;
(c)Maintaining public confidence in the index is important—methods used should be understandable to ‘the man on the street’;
(d)The index should be appropriate to its uses.
32.There were three further Advisory Committees in the early 1990s, with changes made including the introduction of depreciation as an element of housing costs and council tax. The Advisory Committee last met in 1994.
33.When the Government adopted an inflation-targeting regime in 1992, the RPIX index (RPI minus mortgage interest payments) was chosen as the measure of inflation. The justification for choosing RPIX was that if mortgage interest payments were included, the Government could have influenced the achievement of the target directly though the setting of interest rates. There was a target range of between one and four per cent initially, with a fixed-point target of 2.5 per cent introduced in 1995.
34.Between 1995 and 2007 improvements to the RPI were made through “a series of protocols between the Bank of England, HM Treasury and the Office for National Statistics.” The Office for National Statistics described this as a period of “no governance”.
35.The Statistics and Registration Service Act 2007 established new governance arrangements (see Box 1). Under section 21 of the Act, the UK Statistics Authority must compile and maintain the Retail Prices Index and publish it every month. The legislation sets out the process for changing the coverage or calculation of the index:
36.The main change to RPI since the Act has been an alteration to the collection of clothing prices. This instigated a chain of events that led to the downgrading of RPI as a National Statistic. This is discussed in Chapter 2.
37.The 1990s also saw the birth of a new price index: the Harmonised Index of Consumer Prices, later renamed in the UK as the Consumer Prices Index.
38.The Harmonised Index of Consumer Prices (HICP) was developed in the 1990s because of the need across the European Union to develop harmonised statistics for measuring EU economies. This was a result of the launch of the Economic and Monetary Union in 1992, enshrined in the Maastricht Treaty, which aimed for increasing economic convergence across the Union. A harmonised measure of inflation would allow member states to be assessed against the inflation convergence criteria for membership of the Economic and Monetary Union and be the measure of inflation used by the European Central Bank.
39.Member states agreed on a shared approach to measuring inflation and the ‘Harmonised Index of Consumer Prices’ was created, the specifications of which were contained in an EU regulation. Member states were required to compile and publish the index from January 1997.
40.The HICP for the UK was renamed the ‘Consumer Prices Index’ in 2003. That year, the Government switched the inflation target (by then the responsibility of the Bank of England) from RPIX to CPI, reducing the target to 2.0 per cent to reflect the difference in how the indices are calculated.
41.There are similarities between CPI and RPI. They are both based on a fixed basket of goods and services, the vast majority of which are common between the two indices, and the same price quotes are collected for these items. There are however important differences in terms of the population covered by the indices, the composition of the index and the coverage of goods and services.
42.The CPI is based on spending by all private and institutional households and spending by foreign nationals when visiting the UK. The RPI however, as noted above, excludes the highest-earning households (the top four per cent) and pensioner households where three quarters of income is from state pensions and benefits.
43.The CPI includes spending by foreign visitors to the UK but the spending of UK households abroad is excluded. The RPI does the opposite, excluding spending by foreign visitors but including the spending of UK households abroad.
44.The spending data used to calculate weights are taken from different sources. The RPI uses the ‘Living Costs and Food Survey’, a descendant of the Family Expenditure Survey which is conducted by the Office for National Statistics. The CPI uses ‘Household Final Monetary Consumption Expenditure’, which is taken from the National Accounts.
45.For around 60 to 70 per cent of the prices collected for price indices, expenditure weights are not available for the level at which prices are collected. For example, one class of item in the inflation basked is “Loose and pre-packed potatoes—old, new and baking varieties”. Expenditure weights based on surveys will be available only for potatoes as a whole class, not for individual varieties of potato. But as the price quotes collected will be at the individual variety level—for example, the price of King Edward potatoes—a method of aggregating these prices is required so that an average price increase for the class of item can be calculated. Box 4 explains this point in more detail.
46.The RPI uses two methods of calculating an average to aggregate lower level prices where expenditure weights are not available. Both methods use the ‘arithmetic mean’ where an average is derived from the sum of values. Both methods have strengths and weaknesses and the choice of which one to use depends on the nature of price changes for a given item. The two methods are known as the Carli formula and the Dutot formula:
The operation of both methods is shown in Box 4.
47.The CPI however uses a third method known as the Jevons formula. This uses the ‘geometric mean’ where an average is calculated from the product of the values. The price differences (current price compared to the previous price) are multiplied together and the nth root (where n is the number of products) is taken. Its operation is shown in Box 4.
48.The EU Regulation permits the use of the Dutot and Jevons index in the harmonised price index but discourages the use of the Carli index.
For the purposes of illustration, it is assumed that there are four varieties of potato available in shops: Charlotte, King Edward, Maris Piper and Vivaldi. The table below shows the price of each variety in October 2017 and October 2018, and the percentage increase/decrease in price, expressed as a ratio, between the two periods.
The price change for each variety is collected. The proportion of average household spend on potatoes is known but the proportion spent on each variety is not. A method is therefore required by which the price change of the varieties can be combined, so as to produce a single figure for the average price change of potatoes over the 12-month period.
The three formulas discussed above provide three different methods for calculating the average price change for potatoes:
In this example, the Carli and Dutot formulas produce a 1 per cent average price increase in potatoes over the period whereas the Jevons formula produces a 1 per cent decrease.
49.The choice of which formula to use for which class of goods will have a bearing on the outcomes of an index. Table 1 shows the extent to which the three approaches are used in the CPI, CPIH and RPI.
50.Opinion amongst our witnesses was divided about the properties of the Carli formula and its use in the RPI. The debate around the best formula to use is centuries old. These arguments are considered in Chapter 2.
51.One of the largest differences between the two indices is that the RPI includes measures of owner-occupier housing, such as mortgage interest payments, whereas they are largely excluded in the CPI. Figure 1 shows the difference that this can cause: the decrease in interest rates as a response to the financial crisis caused RPI to go negative, whereas CPI remained above one per cent. The chart compares CPI and RPI since CPI was introduced in 1997.
Source: Office for National Statistics, ‘Inflation and price indices’:
52.Figure 1 also includes CPIH. This was developed by the Office for National Statistics to remedy the exclusion of owner-occupier housing costs from CPI.
53.The Consumer Price Index including owner-occupiers’ housing costs (CPIH) was introduced by the Office for National Statistics in 2013. The index is the same as CPI, except that it includes council tax and attempts to measure owner-occupier housing costs. It does the latter through an approach known as ‘rental equivalence’, where the rent paid for an equivalent house in the private sector is taken as a proxy for the costs faced by an owner-occupier. It is updated and maintained in the same way as CPI.
54.The CPIH lost its designation as a National Statistic in 2014 after concerns about the methodology behind rental equivalence. Following a series of improvements, CPIH regained its status as a National Statistic in July 2017. The treatment of housing costs in the CPIH is considered further in Chapter 3.
1 Oral evidence taken on 30 January 2018 (Session 2017–19), (Dr Mark Carney)
2 A contractor (TNS) carries out price collection on behalf of the Office for National Statistics. Price collectors visit a variety of shops in around 150 locations around the UK. Most shops are visited in person although some work is done over the telephone. The price collectors go to the same shops each month. For other goods and services, staff from the Office for National Statistics collect prices by telephone, websites, brochures or catalogues. Office for National Statistics, ‘Consumer price inflation basket of goods and services: 2018’ (13 March 2018): [accessed 21 December 2018]
3 A punnet of raspberries was added to the basket in 2018: “added to rebalance the sample of fruits by including more soft fruit items with an offsetting reduction in the number of stoned fruits. The addition will reduce the weight of other soft fruits such as strawberries and it is expected to reduce the variability in the overall estimate of fruit price movements.” Peaches and nectarines were the casualties amongst the stoned fruits, which remain represented by plums and avocados.
4 Office for National Statistics, ‘Consumer price inflation, updating weights: 2018’ (19 March 2018): [accessed 21 December 2018]
5 See para 45.
6 Office for National Statistics, ‘Consumer price inflation tables’, Table 1 (14 November 2018): [accessed 21 December 2018]
7 HM Treasury, Budget 2018, HC 1629 (October 2018): [accessed 21 December 2018]
8 Board of Trade, ‘Wholesale and Retail Prices’ (6 August 1903): [accessed 21 December 2018]
9 The Bishop had been tasked with determining whether a 15th century stipulation by an Oxford college—that a fellow of the college must vacate his fellowship if his annual income exceeded five pounds—should still be adhered to. He investigated the change in the value of money over the period by assessing changing price levels and found that five pounds in the mid-15th century was worth 25 to 30 pounds in the late 17th century/early 18th century. He concluded that the intention behind the 15th-century stipulation was to allow for the changing value of money. Robert O’Neill et al (2017), Inflation: History and Measurement, Palgrave Macmillan.
10 Board of Trade, ‘Wholesale and Retail Prices’ (6 August 1903): [accessed 21 December 2018]
11 Board of Trade, ‘Cost of Living of the Working Classes’ (1908): [accessed 21 December 2018]
13 Ministry of Labour and National Service, Interim Report of the Cost of Living Advisory Committee, Cmd 7077, March 1947: [accessed 21 December 2018]
14 Ibid. The weightings for food were based on a 1904 survey of household expenditure. Of the 1,038 household records from this survey which were published by the University of Sussex in 2013, only 56 households recorded expenditure on tobacco and only six on beer. Neither alcohol nor tobacco were therefore included in the Cost of Living Index, not being considered necessities. The January 1941 issue of the Ministry of Labour Gazette said of the surveys that there were “indications, however, in some of the budgets received, that expenditure on tobacco and cigarettes was not in all cases fully stated … The experience of previous inquiries into household expenditure has indicated that the particulars given in large-scale collections of family budgets are unlikely to disclose the full amount of expenditure either on tobacco and cigarettes or on alcoholic drink.”
16 Ibid. The notion of a ‘cost of living index’ is considered in Living Costs and Food Survey, Chapter 2.
17 Data were supplied by 12,911 households selected at random. Analysis showed again that alcohol and tobacco was under-reported (see footnote 34) as it did not correspond with figures from Customs and Excise. Adjustments were made accordingly.
18 Expenditure weights in the RPI today are based on the Living Costs and Food Survey, a successor to the Family Expenditure Survey.
19 It was renamed the Retail Prices Index Advisory Committee in 1971. Office for National Statistics, Consumer Prices Indices Technical Manual (April 2010): [accessed 21 December 2018]
20 Robert O’Neill et al (2017), Inflation: History and Measurement, Palgrave Macmillan, p 152
21 Office for National Statistics, Consumer Prices Indices Technical Manual (April 2010): [accessed 21 December 2018]
22 Robert O’Neill et al (2017), Inflation: History and Measurement, Palgrave Macmillan, p 224
23 Andrew Haldane, ‘Targeting Inflation: The United Kingdom in Retrospect’ (2000): [accessed 21 December 2018]
24 Office for National Statistics, ‘History of and differences between the Consumer Prices Index and Retail Prices Index’ (2011): [accessed 21 December 2018]
25 Written evidence from Office for National Statistics ()
26 Statistics and Registration Service Act 2007,
27 Statistics and Registration Service Act 2007,
28 Statistics and Registration Service Act 2007,
29 There are three types of official statistics: National Statistics, which have been assessed by the Office for Statistics Regulation as fully compliant with the Code of Practice for Statistics; experimental statistics; and statistics that have not been assessed as fully compliant with the Code of Practice for Statistics. The Office for Statistics Regulation provides independent regulation of all official statistics produced in the UK. UK Statistics Authority, ‘Types of official statistics’: [accessed 21 December 2018]
30 European Commission, ‘What is the Economic and Monetary Union? (EMU): [accessed 21 December 2018]
31 Council Regulation (EC) No 2494/95 of 23 October 1995 concerning harmonized indices of consumer prices, (, 27 October 1995)
32 Office for National Statistics, ‘History of and differences between the Consumer Prices Index and Retail Prices Index’ (2011): . [accessed 21 December 2018]. It remains the HICP for the UK.
34 Institutional households include educational and health care institutions, institutions for retired or elderly people, military institutions and religious institutions.
35 The Office for National Statistics have estimated that these exclusions represent around 13 per cent of private household spending: Office for National Statistics, ‘History of and differences between the Consumer Prices Index and Retail Prices Index’.
36 Living Costs and Food Survey data is fed into this but combined with other information.
37 For the remaining items, expenditure weights can be calculated at the lowest level of aggregation, for example, gas and electricity bills, car purchases and mobile phone charges: UK Statistics Authority, UK Consumer Price Statistics: A Review (January 2015): [accessed 21 December 2018]
38 The Dutot is used for items where there is relatively little variation in price between products within the same class (for example, food, alcohol and tobacco) as it can produce odd results when products at very different price levels are combined; the Carli is used where there is more variation in prices between products (for example, household appliances or furniture).
39 This method of calculating a price change was developed by Count Giovanni Rinaldo Carli, an 18th century economist who served Leopold of Tuscany. Robert O’Neill et al (2017), Inflation: History and Measurement, Palgrave Macmillan, p 244.
40 Nicolas Dutot was a French economist who had purchased Bishop Fleetwood’s book (see footnote xx) and built on it to produce a price index in 1738 using the ratio of the average of prices. Robert O’Neill et al (2017), p 247.
41 The British economist William Stanley Jevons set out his formula in an 1863 pamphlet in which he argued that the average of price ratios used must be the geometric rather than the arithmetic mean. He gave an example of where the price of cocoa has doubled and the price of cloves has halved. The Carli formula would say that prices have risen overall by 25 per cent but Jevons said this logically inconsistent as if one has doubled and the other halved there has been “no alteration of price whatsoever”. The example is discussed in Chapter 2.
42 This is due to difficulties some EU members had in implementing the agreed common approach.
43 Other differences in coverage: the RPI includes council tax but the CPI does not; the CPI includes university accommodation fees and stockbroker fees but the RPI does not.
44 CPIH was introduced in 2013 but has been calculated back to Q1 2006 by the Office for National Statistics.
45 Office for National Statistics, ‘Introducing the New CPIH Measure of Consumer Price Inflation’: [accessed 21 December 2018]