68.Questions of design are at the heart of how the internet is experienced and regulated. The user experience of a website, search engine or social media platform is defined by the designers of that site. They can influence which posts or images users see, which sites users choose to visit, which news stories they read, and which videos or television programmes they watch. Design affects how privacy and security online are understood, how decisions are made about users by both humans and algorithms, and how users understand these decisions. In short, it affects how technology is used and perceived.
69.Thus, although public concern often focuses on inappropriate content or abusive behaviour, issues around the design of services may be more fundamental. Professor Christopher Marsden said that the internet is “the largest single experiment in nudge regulation that exists”. He added:
“If you want to achieve meaningful results, you have to deal with the way the companies regulate us and persuade them to regulate us differently, which means persuading them to change the way they engineer their software.”
70.In this chapter we explore issues arising from design and how they can be better accounted for in regulation. Different user groups may need specific design ethics applied to them. The internet should also cater for adults with specific needs, older people and children of different ages.
71.Privacy and personal data are protected and regulated by an extensive body of law. In May 2018 data protection rights were significantly strengthened by the General Data Protection Regulation (GDPR). This introduced a number of new rights and obligations, as well as reaffirming existing law (see Box 2). The GDPR requires privacy and security to be incorporated in the design of services: “data protection by design and by default”. Dr Paul Bernal of the University of East Anglia said that the GDPR “has the potential to provide a good deal of support for individual privacy—but only if it is enforced with sufficient rigour and support.”
72.As the GDPR came into force in May 2018, it is too early to judge how effective it will ultimately be. Many witnesses agreed that the GDPR was beneficial and that it would improve the visibility of data protection. However, the scale of concerns are considerable. The Children’s Media Foundation told us that “The collection and exploitation of user data is an ongoing concern. The implications for children are even more significant, as they may not understand the long-term implications of sharing data or have the capacity to make informed decisions.”
73.The Data Protection Act 2018 requires the Information Commissioner’s Office to develop an Age Appropriate Design Code to set out requirements for online services “likely to be accessed by children”. This will create a specific provision in UK law which reflects recital 38 of the GDPR, which states that “children merit specific protection”. This provision requires those processing children’s data to respect children’s rights as set out in the UN Convention on the Rights of the Child, and to take account of their age and development stage.
74.A draft of the Code is expected to be published soon and to include provisions requiring: high privacy by default, geolocation off by default, the upholding of published age-restrictions, content and behaviour rules by online services, preventing auto-recommendation of content detrimental to a child’s health and wellbeing, and restrictions on addictive features, data-sharing, commercial targeting and other forms of profiling. The Code must be laid before Parliament before November 2019 and the enforcement penalties available to the regulator mirror those of the GDPR including fines of up to 4% of global turnover.
75.Personal data is vital to the business model which dominates the digital economy. Dr Jennifer Cobbe and Professor John Naughton described how Google developed this model, which came to be known as ‘surveillance capitalism’. They explained that Google provided a search service which was free to use. In return it analysed phrases which a user entered into its search box (a) to make inferences to predict the user’s wants and (b) to sell to other companies “the opportunity to target those users with advertising based on this prediction”. This business model has made Google one of the world’s richest companies, first through targeted advertising and later “by surveilling user activities elsewhere so as to predict behaviour more generally and maximise opportunities for profit in many other contexts”.
76.Conventional wisdom in the industry is that the more data that a business can gather from different sources the more accurate its analyses. This position forms the bedrock of the modern data science of big data analytics. As a result data is extremely valuable and companies strive to gather and trade in data. Some of these data are supplied directly by the user, but tech companies also gather data about user behaviour by monitoring users’ online activities. For example, in the case of Facebook such ‘behavioural data’ include:
“Data on which pages have been ‘Liked’ by a given user; on which posts have been viewed by a given user; on identifying other users with whom a given user has interacted (including how many times, when, and for how long); on which posts, images, or videos have been seen or watched by a given user (including how many times, when, and for how long); on which advertisers a given user has interacted with (including how many times, when, and for how long).”
77.Internet businesses have accrued massive volumes of data, so called big data, which they cannot process efficiently using traditional digital applications. As a result, many are turning to machine learning to analyse these datasets. Machine learning is a form of artificial intelligence which learns from experience and through this process maximises its efficiency at any task. There are many applications for machine learning: it is already used to detect instances of credit card fraud and it will increasingly be used for healthcare. Not all big data are generated online, but the internet is a major source, giving large tech companies a competitive advantage.
78.The Northumbria Internet & Society Research Interest Group (NINSO) told us that the Internet of Things posed additional risks: “As more and more devices become ‘connected’, and more and more businesses collect data, there is the potential for data protection standards to degrade as a result of hacks, mishaps or simple complacency.”
79.As organisations, including financial and health services providers, increasingly perceive individuals as the aggregation of data gathered about them (sometimes called their ‘data selves’), it is essential that data be accurate, up-to-date and processed fairly and lawfully, especially when processed by algorithm. While the GDPR and the Data Protection Act 2018 provide valuable safeguards, including subject access rights to ensure that data are accurate and up to date and the right to opt out from purely automated processing, there are weaknesses in the regime. For example, a subject access request does not give subjects automatic access to behavioural data generated about them because it is deemed to be the property of the company that acquired it.
80.Users of internet services should have the right to receive a processing transparency report on request. In a model similar to a subject access report under the GDPR users should have the right to request a data transparency report from data controllers showing not only what data they hold on the data subject (which is the currently the case under the GDPR) but also what data they generate on them (behavioural data) and any behavioural data obtained from third parties, including details of when and how they are obtained.
81.Data controllers and data processors should be required to publish an annual data transparency statement detailing which forms of behavioural data they generate or purchase from third parties, how they are stored and for how long, and how they are used and transferred.
82.The incentive to seek and retain users’ attention—to gather more of their data and to target them with advertising—is a key attribute of the ‘surveillance capitalism’ business model. Professor John Naughton explained that companies deploy techniques which they have learned from applied psychology. The services are deliberately designed to be addictive. As a result:
“Somebody goes on to Facebook to check a picture from a family member and an hour later they wonder why they are still there. They are still there, because it is beautiful software that is very cleverly designed.”
83.Subforum, a tech design and research organisation, described one psychological technique used, ‘variable rewards’, which plays off human responsiveness to “unpredictable rewards that are offered on a variable, non-fixed schedule”, which increase the level of dopamine produced by the brain. Subforum compared this technique to a slot machine:
“You put in a coin. You pull the lever. Do the three shapes all match? Nope? OK, pull again. How about this time? That’s the hook: the anticipation of getting a reward (whether or not we actually get one) increases the dopamine levels in our brains, which compels us to keep doing the thing that got us a reward before.”
Table 1 provides examples of how platforms use this technique.
Behaviours the platform wants to reinforce
Variable reward offered
Scrolling Facebook’s news feed or pull to refresh on Twitter
An interesting or amusing update
Posting, commenting or responding
Gratifying likes and other responses
Checking messages or notifications
Receipt of inbound communication
84.Margrethe Vestager, the EU’s Competition Commissioner, said these techniques “are designed to create a form of addiction”. Professor Chris Marsden, professor of internet law at the University of Sussex, said of Margrethe Vestager’s remark, “She pointed out that we allow 13 year-olds to use these platforms perfectly legally in the UK—it differs in different European countries—in a way that we have decided not to do to for alcohol, tobacco or other types of addiction. Those are her words rather than mine. The world is built on addictive substances, from tea and sugar to everything else, but we should be aware that we are doing this.”
85.Professor Sonia Livingstone of the London School of Economics and Political Science suggested that human interaction with technology could be described as “a kind of compulsion and fascination, rather than addiction”. She argued that efforts should be made to intentional use, particularly among children: “It is all about the defaults and finding ways not to maximise eyeballs.” Professor Livingstone predicted that we may be at the early of stage of a differentiation of business models. She suggested that businesses should be required to use “notifications, endless reminders and pop-up reminders to say, occasionally, ‘Have you have been on too long?’”
86.Tristan Harris, a former employee of Google, has championed a backlash to the surveillance capitalism model and has founded the Center for Humane Technology to raise awareness of the need for ethical design. He has warned: “With design as it is today, screens threaten our fundamental agency. Maybe we are ‘choosing’, but we are choosing from persuasive menus driven by companies who have different goals than ours.”
87.Digital service providers (such as hardware manufacturers, operators of digital platforms, including social media platforms and entertainment platforms, and games developers) should keep a record of time spent using their service which may be easily accessed and reviewed by users, with periodic reminders of prolonged or extended use through pop-up notices or similar. An industry standard on reasonable use should be developed to inform an understanding of what constitutes prolonged use. This standard should guide design so that services mitigate the risk of encouraging compulsive behaviour.
88.Online platforms have become the primary interface for internet users, helping them navigate vast volumes of content and sifting for what is most relevant. Dr Shehar Bano explained: “The human brain has limited capacity for processing information and the time span for which their interest is sustained; therefore the order and format in which information is presented to users is crucial.” Online platforms use algorithms (see Box 2) to present content to users based on (depending on the nature of the platform) what they were searching for, data collected about them (‘personalisation’) and factors such as whether an advertiser has paid for content to be prioritised.
An algorithm is a set of rules to be used to make the necessary decisions to complete a given task. While algorithms have been used since antiquity, they have been critical to the development of computer science. In recent years, the word ‘algorithm’ is often taken to mean complex decision-making software. Algorithms are used in artificial intelligence. ‘Reinforcement learning’ allows algorithms to improve and rewrite themselves without further human input. Article 22 of the GDPR protects users from being subject to decisions made by algorithms which have “legal or significant effects”, such as when applying for loans online.
89.Although personalisation is often said to optimise customer interaction, the Internet Society noted that it was not clear what was being optimised:
“Is the content on the platform being shaped to provide content that will increase customer wellbeing, or is it shaped to maximise time spent on the platform and/or number of interactions with adverts even if this is to the detriment of the user?”
90.Personalisation of content determines what people see online. Robert Colvile, Director of the Centre for Policy Studies, said that the algorithms tend to “intensify and radicalise your experience”. He gave the example an experience of “liking” content from UKIP on Facebook, which instantly returned content for the National Front and the BNP. Ultimately, these algorithms can create ‘filter bubbles’ where users see only information related to their preferences and ‘echo chambers’ where their beliefs are reinforced by like-minded or more extreme content. These have been linked to the spread of so-called ‘fake news’. Dr Stephann Makri explained that “they can create ‘distortions’ in information flow (e.g. through misinformation, disinformation) that can undermine the fundamental British value of democracy”.
91.Personalisation may be based on profiling, whereby algorithms analyse a person’s data to identify characteristics about the person such as their interests, personal preferences, health, reliability, behaviour and location.
92.Platforms tend to keep the details of their algorithms secret on the grounds of commercial sensitivity and concern that people might seek to ‘game’ them. Our witnesses generally agreed that full transparency about the computer code containing algorithms would not help users to understand how they work. Microsoft argued that even a detailed understanding of an algorithm would not be useful in understanding its outputs, which were derived from input data from other users.
93.However, the lack of transparency has caused concern. Dr Bano described algorithms as “opaque” and was concerned that they may be “biased, and at times outright discriminatory”. Algorithmic bias may be caused by input data which is biased. This may be a particular problem with machine-learning algorithms which are programmed to spot patterns in large amounts of data. Professor John Naughton said, “Most datasets are not clean; they are coloured in one way or another with all kinds of unconscious and other biases.” He said that many people not involved with developing this technology were “dazzled” by it. This included members of the Government and industry who should be more sceptical.
94.The lack of transparency may conceal instances where algorithms are designed to act in ways which are contrary to the user’s interests. For example, Margot James MP, the Minister for Digital and the Creative Industries, told us that some airlines’ websites use an algorithm which identifies passengers with the same surname and deliberately allocates them seats apart from each other. The airlines can then charge passengers to change their seat to be with their family.
95.Many witnesses called for greater transparency. The Children’s Media Foundation proposed “the publication of the editorial guidelines and values that underpin them”. NINSO recommended that “Algorithms should also be auditable and audited frequently by an independent body.” It is not always possible to audit the technical content of algorithms, as they can rewrite themselves beyond the understanding of their creators. However, impact-based assessments are possible. These consider the decisions algorithms make rather than the processes by which they make them. The Information Commissioner’s Office already carries out impact audits for Data Protection.
96.The Information Commissioner’s Office told us that the Commissioner had started to work with the Turing Institute to produce a framework for explaining algorithmic processes and decisions. They stressed the need for transparent explanations of both data inputs and how data outputs are used and also the difficulties of engaging the average user with technical information.
97.Katie Donovan, UK Public Policy Manager at Google, said: “We have developed our own AI principles to ensure that we use them ethically, that we have transparency about them and that we use them for social good.”
98.The Government has set up the Centre for Data Ethics and Innovation to provide independent, expert advice on measures to ensure safe and ethical innovation in data-driven and AI-based technologies. Following a consultation on the role and objectives of the centre, the Government said that it will “agree and articulate best practice” for companies using data.
99.The Information Commissioner’s Office should set out rules for the use of algorithms based on the principles set out in chapter 2. The ICO should be empowered to conduct impact-based audits where risks associated with using algorithms are greatest and to require businesses to explain how they use personal data and what their algorithms do. Failure to comply with the rules should result in sanctions.
100.The ICO should also publish a code of best practice informed by the work of the Centre for Data Ethics and Innovation around the use of algorithms. This code could form the basis of a gold-standard industry ‘kitemark’.
102.The GDPR prohibits personal data from being processed unless they are specifically permitted under one of six lawful bases. Often online platforms rely on ‘consent’ as the legal basis for processing data. The GDPR has strengthened this legal basis by requiring that consent be freely given, specific, informed and unambiguous. Consent must be uncoupled from other written terms of service.
103.The GDPR requires organisations to explain how they use personal data, whether or not consent is the basis for processing. It includes the right to be informed. The Information Commissioner’s Office (ICO) told us:
“Essentially, the GDPR requires organisations to be clear about what they do with individuals’ personal data, how they do it, on what basis they do it, what data they hold, how long they will hold it for and who they will share it with (this is not exhaustive).”
104.The ICO has published guidance on how organisations can achieve this and encourages them “to be innovative in providing this information—embedding and layering the information as part of the design process, not just in one long notice.” It also argued that openness and transparency around data use were important not only for complying with the law but also “to engender trust and improve relationships with … customers”.
105.Nearly all our witnesses said that there was a lack of understanding about how data were used. Information provided in terms of service did not help. The Children’s Media Foundation said that terms of service and information about data use were not easy to find and were written in a way that is “impenetrable for most people—especially children”. Which? argued that businesses should “provide consumers with more transparency on the impacts of data use and the Government and others must work together to understand these impacts”.
106.The Royal Society said that relying on information alone was problematic because of what it called the ‘transparency paradox’: consent requires information to make it meaningful but “anything too long or complex is unlikely to be broadly understood or read”. On the other hand summarising information to make it more digestible “often discards the details that people care about”. The Royal Society concluded: “It is unreasonable to expect an individual to keep track of what data is collected about them and understand how it will be used, and therefore to give meaningful, informed consent.”
107.NINSO, on the other hand, argued that more could be done to ensure that terms of service and privacy policies were clear and easy to understand:
“Videos and infographics are goods, ways to convey complex information such as this. The keywords should be in bold. The text should be readable, i.e. coefficient 8 Flesch-Kincaid [a reading standard] … Ultimately, the information should be delivered with a level of clarity that is sufficient to enable users to make an informed choice.”
108.Terms of service must be written in a form which is clearly accessible and understandable to internet users. Alongside terms of service statements a ‘plain English’ statement should be published which sets out clearly and concisely the most relevant provisions. These may make use of infographics or video statements where appropriate.
109.Where children are permitted to access or use a service age-appropriate terms and conditions must be provided. These should be written in language clearly understandable to children of the minimum age allowed on the platform.
110.Terms of service are often on an ‘all or nothing’ basis. NINSO explained:
“There is a substantial power imbalance between users and the operators of online platforms. Users frequently have no capacity to moderate terms but instead have the ‘choice’ of accepting all terms (which might include giving away significant amounts of personal data) or simply not using the service. This is not providing a real choice.”
As a result, according to Which?, many consumers “choose not to engage because it does not feel worthwhile”.
111.Dr Paul Bernal argued that what platforms do with data is more important than the question of what information they should provide:
“People will generally simply scroll through whatever information is provided and click ‘OK’ at the end. Regulation of the use of personal data based on information and ‘consent’ is not sufficient: it is more important to set clear and strong rules about what is and is not allowed.”
112.Others argued that users should be given greater control. For example, NINSO suggested users should be given greater control of their data by having the option to pay for a premium service which does not collect data.
113.Jamie Bartlett said that the default setting of whether data are immediately shared or not probably has more effect than any other issue of design.
114.Maximum privacy and safety settings should be included in services by default. The Information Commissioner’s Office should provide guidance requiring platforms to provide greater choice to users to control how their data are collected and used.
116.Ethical issues should be considered and addressed during the design process, reflecting concepts such as ‘rights by design’, ‘privacy by design’, ‘security by design’ and ‘safety by design’. These problems are directly associated with design and so it is more effective to consider them early than to react to problems later on. Dr Stephann Makri told us: “This approach is far preferable to a box-ticking exercise where designers try to demonstrate meeting ethical design guidelines or regulations without considering ethical design from the outset.”
117.Doteveryone argued that developers should conduct “independent impact assessments at an early stage of a technology’s lifecycle”. It also argued that the ‘precautionary principle’ could be applied to internet technology:
“This principle is applied in situations where there are reasonable grounds for concern that an activity is causing harm, but the scale and risk of these issues is unproven. The onus is then on organisations to prove that their practices are safe to a reasonable level.”
118.Dr Ewa Luger said that the culture of the tech industry needs to change. Currently “people do not set out to do harm, but they do not know what the alternative is. Responsible innovation is not embedded in the teaching of computer science, machine learning or AI.” Dr Luger recommended investment in higher education and embedding ethics into teaching.
119.Design principles and standards are a normal part of business life across all sectors. Establishing and enforcing standards that would meet the 10 principles would help to reduce harms to users and society. We recommend that regulation should follow the precautionary principle to ensure ethical design while also recognising the importance of innovation and entrepreneurship.
120.We recommend that the ethical approach outlined in our 10 principles should be embedded in the teaching of all levels of computer science. The Government should promote and support this. The Centre for Data Ethics and Innovation will also have a role in providing guidance which can be incorporated into teaching, as well as educating users on the ethics and risks of the internet.
84 (Rachel Coldicutt)
86 Written evidence from Professor Christopher Marsden ()
87 Regulation (EU) 2016/679 of the European Parliament and of the Council on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (, 27 April 2016), Article 25
88 Written evidence from Dr Paul Bernal (
89 Written evidence from CMF ()
90 Written evidence from Dr Jennifer Cobbe and Professor John Naughton (
92 Written evidence from the Royal Society ()
93 Written evidence from CMF ()
95 Written evidence from Subforum ()
96 ‘EU Commissioner Margrethe Vestager: Facebook is designed to create addiction—like tobacco and alcohol’ Berlingske (7 April 2018): [accessed 5 December 2018]
97 Written evidence from Professor Christopher Marsden ()
99 (Professor Sonia Livingstone)
100 Tristan Harris, ‘Tech Companies Design Your Life, Here’s Why You Should Care’ (7 March 2016): [accessed 26 February 2019]
101 Written evidence from Dr Shehar Bano ()
102 Written evidence from Internet Society UK Chapter ()
104 Written evidence from Dr Stephann Makri ()
105 Article 4 of the GDPR defines ‘profiling’.
106 Written evidence from Dr Paul Bernal ()
107 Written evidence from Microsoft ()
108 Written evidence from Dr Shehar Bano ()
111 Written evidence from CMF ()
112 Written evidence from NINSO ()
113 Written evidence from Her Majesty’s Government ()
114 Article 6. The legal bases are consent, contract, legal obligation, vital interests, public task and legitimate interests.
115 This is mainly covered by articles 13 and 14 of GDPR.
116 Written evidence from ICO ()
117 Written evidence from ICO ()
118 Written evidence from CMF ()
119 Written evidence from the Royal Society ()
120 Written evidence from NINSO ()
121 Written evidence from the Internet Society UK Chapter ()
122 Written evidence from NINSO ()
123 Written evidence from Which? ()
124 Written evidence from Dr Paul Bernal ()
126 Written evidence from Dr Stephann Makri ()
127 Written evidence from Doteveryone ()