Statistics: Communicating and Publishing Statistics

Written evidence submitted by Full Fact (4STATS 07)

A. Summary

 Our evidence shows inconsistent standards in communicating official statistics and suggests possible ways standards of communication could be improved, including:

· Excellence in communication must be valued as much as technical excellence.

· Senior statisticians should have a prominent role in promoting public understanding of official statistics, including talking directly to the media.

· More emphasis on non-professional audiences for statistics, including policy-makers, the media and the public.

· More emphasis on providing topical statistics, including publishing all ad hoc releases which provide data not readily available anywhere else.

· More and better use of the internet, websites and interactive online material.

We endorse the role the UK Statistical Authority has carved out for itself so long as it continues to intervene when intervention is necessary. We also recommend more formal coordination between the Authority and other bodies concerned with standards in public life.

Full Fact is an independent factchecking organisation. We research claims, many statistical, made by politicians, journalists and pressure groups. As an organisation dealing in official statistics, we have frequent contact with the Office for National Statistics and the Government Statistical Service.

B. Response

1. How well are the practices for the release of official statistics, and pre-publication access to those statistics ("pre-release access") working?

1.1 Normal statistical bulletins are released satisfactorily as far as we are aware; ad-hoc statistical releases are more problematic. We address this in our response to question 6. We have expressed our concerns about pre-release access in our submission to the first study.

2. Has the right balance been struck between the resources devoted to producing those statistics and those devoted to communicating them?

2.1 Full Fact does not have a view on the current allocation of resources. We do, however, believe that the communication of statistics needs improvement. If more resources must be allocated to achieve this, the resources would be well spent, but communicating better should not necessarily cost more.

2.2 It is important that, as well as technical excellence, excellent communication is valued and nurtured within the Office for National Statistics and the Government Statistics Service. Communication is not a secondary task but crucial to how statistics serve the country.

2.3 All good communication starts from its audience. Increasingly, policy makers, journalists and the public access statistics directly because they are only a click away. Statistics producers need to take this into account in deciding how to communicate.

2.4 To that end, it is encouraging to see that there are clearly people with senior support thinking hard about how information is best presented not just online, but in publications and on the airwaves. The most effective innovations need to be made widespread, and the baseline raised. Much of this, we suspect, is not about spending more money but rather a question of emphasis, culture and to some extent skills.

2.5 We would ask the Committee to consider the role of senior statisticians in this process. As data becomes ever more prominent within public life, we think the National Statistician could usefully have a higher public profile, engaging in and informing debate. This could be done without in any way compromising her neutrality. We would extend this to Heads of Profession in government departments too.

2a. What should be the future of the National Statistics publication hub, particularly in the context of the proposed single Government domain for communicating data?

2.6 The Publication Hub is not perfect or particularly engaging, but it fulfils its basic role as a portal for first releases. The principle of the single portal for discovering all new releases across different producers and across the UK is a good one, and this should not be lost in any government overhaul of statistics communication.

2.7 We would point out that data and statistics are sometimes treated as interchangeable terms – but statistics are the product of professional expertise, careful methodology and review, and necessarily come with important context. For example, raw crime data from police are distinct from the national statistics that are ultimately produced from that data. We would be concerned, therefore, if statistics and data in its broader sense were mixed in a way that eroded the distinction: not all numbers are equal.

3. In what circumstances should the UK Statistics Authority comment on the use or misuse of statistics?

3.1 The UK Statistics Authority should continue to exercise broadly its current approach, monitoring statistics used by a wide range of people including those in politics, the media and business, and reserving intervention for serious misuses of statistics only.

3.2 As Sir Michael Scholar has said, the UK Statistics Authority cannot ‘assume the role of a referee on each occasion that an official statistic is misquoted or misinterpreted.’ [1] It would be impossible for the UK Statistics Authority to intervene every time a statistic is misused, and it is ultimately a matter of judgement when an intervention would be useful. The test for this judgement is whether those who might misuse statistics are conscious of and deterred by the possibility of an intervention from the Authority.

3.3 Civil society must play its part in informing public debate and Full Fact’s work helps to fulfil this role. We research claims, many statistical, made by politicians, journalists and pressure groups. When we find mistakes, we ask for them to be corrected. Ministers, government departments, a Select Committee, MPs, the BBC, most national newspapers and a variety of pressure groups have made corrections when we have pointed out inaccuracies. Where necessary, we turn to appropriate regulatory bodies to ensure that corrections are made.

3.4 Pursuing individual factual claims sometimes exposes wider policy issues. For example, when we directed the UK Statistics Authority’s attention to a series of opaque media reports on benefits, the Authority’s intervention led to the Department for Work and Pensions starting to publishing its ad-hoc analyses systemically online. This principle is now expected to be implemented more widely across government. Similarly, a recent letter from the Chairman of the Authority about the limitations of comparative education statistics used by HM Chief Inspector of Education has roots in concerns we raised with the Authority back in 2010. [2]

3.5 These examples illustrate that it is not necessary any more than it is possible for the Authority to pick up every concern for itself. However, this model of course only works if the UK Statistics Authority can be relied upon to bring its full weight to bear down on more serious statistical misuses. So far we have found that it can.

3.6 The only gap we would point to at the moment is one of cooperation. There are quite a number of authorities broadly concerned with aspects of trust in public life, from the Advertising Standards Authority to the Press Complaints Commission. In March, the UK Statistics Authority criticised the Daily Mail’s coverage of prosecution of offences related to the 2011 riots. [3] Surprisingly, the Press Complaints Commission ruled that the Daily Mail had not breached the Code of Practice, effectively overruling the UKSA.

We feel there would be value in the UK Statistics Authority approachi ng other relevant bodies, to es tablish a working relationship in advance of situations when they might deal with cases involving official statistics. These bodies would then be able to reply with the canonical advice of the UK Statistics Authority. That would probably be a better approach to the worlds outside government and politics than attempts to intervene directly where the UK Statistics Authority has no clear means to make its writ run.

4. To what extent are the requirements of users taken into account in decisions on the communication and publication of official statistics?

4a. How easy is it for users to find the official statistics they need, whether via the Office for National Statistics website, other websites, the National Statistics publication hub or elsewhere?

4.1 The overall experience of finding statistics is one of frustration. Unless you know your sources, it can be very hard to access even basic information.

4.2 The ONS website is notoriously poor and needs to be redeveloped again, this time from first principles. In particular, the search function is hopeless. For example, a search for ‘population’ yields ‘2008-based Subnational Population Projections for England’ as its first result. The site does not consistently make links between different datasets and analyses that may illuminate each other. Since 2011, when the website was replaced, all links to ONS data used in Full Fact articles have broken. This is frustrating, and a breach of basic online norms. We would be concerned if the same mistake is made when the site is overhauled again.

4.3 However, there is good work going on. The ‘key figures’ provided on the front page and theme pages do often answer the kinds of questions we receive from our audience. The interactive material available online is another promising initiative. Eventually this should be expanded and moved to a more prominent location, but first the website’s organisation and search function should be improved.

4.4 There is no standardised or centralised way of locating statistics in the websites of government departments, so finding data is neither consistently easy nor hard. data.gov.uk is messy and has gaps. We do not think an average member of the public would find departmental websites user-friendly (we find most data without huge problems partly because we use statistics outlets frequently); however, it is hard to see how the UK Statistics Authority can fulfil its remit to improve public access to statistics without raising the standard of departmental and other publishers’ sites.

4b. How well do the formats used for releasing official statistics meet user needs, both in terms of publication type (compendia, reports, press releases) and data format (pdf, excel, open data standards), and what changes should be made?

4.5 The goal of publishing official statistics should be to ensure that users can get the information they need, in its full context, in the most convenient way. In particular, it should strive to present a coherent statistical picture in important or contentious areas of public debate. This entails different things for different users, which might be met in some of these ways:

· Graphically-led presentations such as graphs and maps.

· Web pages using links to bring the full picture into view (suited to the increasing proportion of mobile internet users).

· PDFs, easily downloaded and printed, with full background and interpretation in one place.

· Spreadsheets for those who want raw data.

· In due course, interactive tools.

4.6 At the moment, official statistics are primarily presented in PDFs and spreadsheets, which seem geared toward more technical users. Web-native presentation should have a much greater role, being more searchable, less intimidating and quick to use. It also allows the possibility of linking between information to give people a fuller picture in a way that static formats do not.

4.7 Spreadsheets sometimes seem to prioritise appearance over functionality, for example spacer columns obstruct calculations. This should stop. Simply putting data into a format that can be used for calculations can entail hours of work. There are better ways to make spreadsheets easy to navigate and the priority should be making data easy to work with.

4.8 To standardise and improve ONS releases, many of which are poorly written, all should be sent through a desk of sub-editors and the communications office before publication. The Norwegian equivalent of the ONS employs journalists full-time. They work closely with statisticians to produce public-friendly press releases. While this system might not translate perfectly into the UK, the Committee could certainly look to this model as inspiration for what can be achieved when communication is made a first-class part of a statistical office’s task.

4.9 Finally, the rarity of topical publications is glaring. For example, the UK Statistics Authority could coordinate the publication of statistical factsheets to inform the debate on the forthcoming independence referendum in Scotland.

4c. How effectively does the UK Statistics Authority engage with the user community to understand its requirements and what, if anything, should it do differently?

4.10 The Statistics User Forum demonstrates that the UK Statistics Authority has an active programme of professional user engagement. However, policy makers are, in effect, the most important users of statistics, and we share Andrew Dilnot’s concern that senior civil servants sometimes seem to see statistics as ‘what you put in after you have worked out what the policy should be’. We agree that the UK Statistics Authority needs to widen its engagement and work with the Cabinet Secretary and Permanent Secretaries ‘to boost the sense of confidence of analysts and statisticians within the civil service.’ [4] The same argument applies to working with Parliamentarians, their staff, and the staff of the two Houses.

4.11 We also endorse Andrew Dilnot’s view that the UK Statistics Authority should work with journalists to improve the communication of statistics. [5] As we argued in our submission to the first study, this means broadcast first and foremost. Our experience of journalism and journalism education suggests that first, there is an increasing awareness of the importance of statistical competence and second, not enough being done to help trainee or existing journalists develop them.

5. How well are trends in, and limitations of, official statistics (produced by the ONS and elsewhere) explained to users? Do these explanations have the right level of detail and how balanced are they?

5.1 There are examples of ONS and GSS releases which explain statistics clearly, but there is no consistent standard. Other releases are ill-structured and presented less coherently. We have two systemic concerns-

5.2.1 Statistics producers need to monitor how their statistics are used in public debate and adjust how they present them accordingly. In the summer of 2011 Full Fact raised concerns with the ONS about media misrepresentation of foreign workers’ jobs statistics. The ONS repeatedly resisted our request for guidance about the statistics’ limitations to be included, and only after we raised the issue with the UK Statistics Authority did they agree to clarify future bulletins. [6] Part of the ONS’s job is to explain statistics and when it sees statistics being misused it should alter their presentation – for example making caveats clearer – to guard against further misrepresentation.

5.2.2 Producers rarely do enough to explain which changes are statistically significant. This reflects the first point, in that it is one of the most frequent areas of misinterpretation. For example, variations in crime level numbers have been treated as major news regardless of whether those variations are significant or whether those numbers really mean that crime has stayed the same as near as we can measure. If such caveats are not prominent in the original release, they stand no chance of making it into the media, the debate or public consciousness.

6. How well do producers of official statistics respond to ad-hoc requests for data, and what should be the publication policy in response to these requests?

6.1 Ad-hoc requests for data are dealt with inconsistently by the ONS. Sometimes staff will respond with data immediately, sometimes a request will be diverted into the FOI process and at other times we must submit an FOI request to get any headway. From the outside, there appears to be little consistency in, or explanation of, the reasons for these actions.

6.2 When it comes to government departments, it can be hard to get hold of statisticians, and conversations mediated through either press officers or freedom of information officers are less likely to be fruitful. Government Statistical Service personnel should be exempted from the requirement that officials do not speak to the media. Their names and telephone numbers should appear on statistical releases. Of particular concern is that we have had topical requests turned into FOI requests, which are slow to process. It seems to us that requests which would illuminate a current topic of public debate should be prioritised.

6.3 Any request which generates data not readily available elsewhere should be published to give effect to the guiding principle of the Code of Practice for Official statistics of ‘equal access to all’. [7] Much of the point of the Code of Practice is about building trust in official statistics, and by extension in public life. As Onora O’Neill argued, "if we want a society in which placing trust is feasible we need to look for ways in which we can actively check one another's claims." [8] Ad-hoc releases often underpin prominent political claims and need to be released for analysis. Even when they do not, it is the public’s information and in belongs in the public domain.

6.4 For example, a number of articles appeared in the press earlier this year derived from ad-hoc ONS statistical releases, describing these releases as ‘exclusive’, ‘unpublished’ and even ‘secret’. The impression given by these newspapers of the ONS’s approach to releasing data was at odds with the Code of Practice and cannot have helped to foster public confidence in how figures are compiled and obtained. After our intervention in a similar practice by the Department for Work and Pensions in 2010, the Department started to publish all its ad-hoc statistical analyses online. This is surely the right default position. Again with the ONS, Full Fact raised this with the UK Statistics Authority [9] and we have since seen progress toward establishing that principle throughout the ONS and government, but we would welcome the Committee’s support for these moves.

7. How well are opportunities afforded by the internet being exploited in the publication strategy for official statistics?

7.1 The opportunities afforded by the internet include the ability to reach the public directly; ease of searching; speed of publishing; interactivity and engagement; informative visualisation; being responsive to current debates and concerns; easy cross-referencing between information; personalisation; provision of detailed explanations at miniscule marginal costs and the ability to make large datasets easy to interrogate. So far few of these opportunities seem to have been explored, alone exploited.

7.2 In a sense, that is understandable. The state of the art in publishing statistical information online is still fairly rudimentary and government can be expected to be behind the curve. What is concerning is that much of the approach to publishing official statistics online seems to be at odds with web culture, where breaking other people’s links is profoundly anti-social, not linking well between your own information is perverse and not being part of the conversation is simply odd. The ONS has a ‘twitter policy’, which includes the phrase: "We encourage users to follow us." [10] At the least, this is not an organisation which is comfortable online.

October 2012