7Engagement and evaluation
179.HMCTS recognises the value of communication and engagement with its stakeholders in development of the reform programme and the importance of evaluating the programme’s impact. However, our evidence suggested that HMCTS has not engaged well enough with its stakeholders, and that its approaches to evaluation are unlikely to maximise objective scrutiny.
Stakeholder engagement
180.The NAO’s first report on court and tribunal reforms found that the programme’s tight timetable created challenges for stakeholder engagement; it recommended that HMCTS allow more time to engage with affected parties.289 Developing this theme, the PAC recommended that: “By November 2018, HMCTS should publish plans on how and when it will engage with stakeholders and be clear about how it will act on the feedback received and adjust plans if necessary.”290
181.In response, HMCTS accepted the need for greater, and more active, stakeholder engagement and published a plan in November 2018, which describes the three strands of its approach:
- communication to provide regular information and updates about reform;
- dialogue to enable HMCTS to share its reform plans and exchange and views with representative organisations, particularly among legal professional and public user groups; and
- collaboration with users and stakeholders at a project level to design and develop new services.291
182.A minority of witnesses thought that they had been adequately engaged in the reform programme.292 Much of our evidence indicates dissatisfaction with the quality and frequency of HMCTS’s stakeholder engagement and the transparency of the programme. Many witnesses thought that there had not been enough public consultation on reform proposals, with few mechanisms for those with experience of the court and tribunal system to become involved.293 It was suggested, for example, that self-represented defendants in housing possession cases be consulted about their experiences of making emergency applications for warrants of possession to be set aside.294 Ken Butler from Disability Rights UK reflected:
If you stopped anybody in the street and asked them if they had heard about this, most people would not know about it. I am not even sure that people who are current users of the tribunal have had their opinions asked for. For example, if someone has had their face-to-face appeal hearing for PIP, does anybody ask them afterwards, “What do you think of this? This is what we are thinking of doing.”295
183.Others accepted that engagement was taking place, but thought that it was insufficient. Witnesses from the legal profession were particularly unhappy. The Law Society had not been directly informed about key HMCTS announcements.296 The Chartered Institute of Legal Executives (CILEx) accepted that HMCTS had recently increased its engagement on the progress of reforms, but had experienced this as being limited to strategic and communications elements, not engagement on individual projects.297 The Bar Council was unhappy about engagement meetings scheduled during court hours. Hammersmith and Fulham Law Centre complained about being “left in the dark” on the testing of reforms carried out by the Government, and said that HMCTS “roadshows” had not had sufficient capacity or geographical spread to ensure access for stakeholders. The Public Law Project was among those who commented that much of the design for the programme was taking place “within closed focused groups and similar processes.”298
184.Some witnesses from the legal profession doubted whether HMCTS consultations were of any real value, suggesting that policy decisions had already been made and that consultation was therefore “lip service”.299 Matt O’Brien from the Criminal Law Committee of Birmingham Law Society said defence practitioners doubted whether their views were really considered. He complained of a wider sense of consultation fatigue:
We have had so many consultations from the MoJ, the LAA and the SRA on different issues, not specifically the subject of court reform, where people have become very engaged and submitted detailed responses but there is no evidence that they have been taken into account at all, so I think a sense of fatigue creeps in.300
185.Similar misgivings about HMCTS engagement were expressed by other witnesses, including NGO agencies providing advice and representation. IPSEA said it had not been consulted directly on the reforms, nor had consultation taken place via the local Tribunal User Groups which it attended.301 The Free Representation Unit accepted that there had been improvements in communication with their own organisation, but thought that information was still limited.302 Citizens Advice had experienced communication largely taking place through updates from individual HMCTS project teams:
This methodology has however sometimes made [it] difficult to keep track of so many parallel changes and to get a big picture view of how the reforms will ultimately come together.303
186.The consequences of failure to co-ordinate HMCTS initiatives with police IT systems was illustrated by Detective Chief Inspector Kirby from Thames Valley Police, who commented that it sometimes felt change was being “directed upon us, rather than us being involved in supporting it”. To fulfil its commitment to providing evidence digitally by 2020, his force had invested in new technology—only to find that this system did not connect with the court IT infrastructure.304
187.The PCS Union saw the involvement of court and tribunal staff in the reform programme as key to the programme’s credibility. PCS described Government consultation with trade unions on behalf of staff as “virtually non-existent” and pointed out that staff members “understand the legal system and deal with the most vulnerable of users”. The union expressed concern about the impact of the loss of staff expertise on the quality of justice, as well the effect on communities of job losses arising from centralisation of roles within HMCTS.305
Judicial engagement in the reforms
188.The Lord Chief Justice emphasised that the involvement of the judiciary in the detail of the reform programme was seen as central to its success and confirmed that the senior management of HMCTS was receptive to judicial input. He commented:
We provide the most ready source of expertise and are particularly sensitive to issues such as access to justice and open justice. There is much expertise in HMCTS but in a large project with many strands many who have been drafted in to the reform project have little direct experience of the administration of justice.
He explained that all levels of the judiciary have been engaged with HMCTS from the beginning of the reform programme. Judicial input is overseen by a Judicial Reform Board led by Lady Justice Thirlwall, along with the Senior President of Tribunals and District Judge Tim Jenkins. The Judicial Executive Board discusses reform at almost all its meetings.306
189.He also explained that the senior judiciary’s “Judicial Ways of Working” (JWOW) project has sought to achieve engagement with judicial office holders at all levels. In April 2018, four JWOW documents were published, relating to reform plans for crime, civil, family and tribunals. All judicial office holders were encouraged to respond to documents relevant to their areas of work via an online survey and attend local reform events to express views. Responses were received from or on behalf of 10,000 judicial office holders, and 800 people attended 38 events. Survey responses, along with views expressed at events, were used to inform the senior judiciary’s approach to reform and discussions with HMCTS.
190.In addition to JWOW, a network of judicial groups was set up to ensure judicial input into the design of new products and services, including:
- Judicial Engagement Groups for all jurisdictions, with members are selected for their professional expertise (as opposed to seniority); and
- Judicial Working Groups attached to specific reform projects (for example, the Single Justice Service project and Public Family Law & Adoption project).307
191.The Senior President of Tribunals said that the judiciary had taken a full part in the leadership of change. He has established a Tribunals Change Network that brings together project judges, judges who advise on the Judicial Engagement Groups, leadership judges from the Tribunals Judiciary Executive Board and representatives from judicial associations. The independent Administrative Justice Council, chaired by the Senior President of Tribunals, aims to make the administrative justice system accessible, fair and effective.308 It has set up three expert panels drawn from the academic, pro-bono and advice sectors which are helping to provide external scrutiny for the tribunal reform programme.
192.However, not everyone shared the senior judiciary’s belief that there were adequate mechanisms for consultation with all levels of the judiciary. The Council of HM Circuit Judges said it was “apparent from our own inquiries that there are many differing views among the judiciary as to the impact and success of the reforms to date”; it thought that judicial associations should be involved in ongoing consultation and evaluation, as well as the Judicial Engagement Groups.309 The Legal Committee of HM Council of District Judges (Magistrates’ Courts) thought that judicial office holders had been given adequate opportunities to comment on the reform programme.310 By contrast, the Association of HM District Judges questioned whether there had been meaningful consultation with all levels of the judiciary on court closure proposals:
There is a feeling that whilst MoJ/HMCTS consult, they do not listen but proceed simply to implement the decisions taken by them before any consultation took place. Such an approach makes consultation meaningless.311
193.The quality of communication with magistrates about the reform programme has been criticised. The Magistrates Association has expressed concern about the JWOW consultation process; many magistrates were disappointed by the lack of detail in the published response, leaving them unclear as to whether the consultation process had influenced the reform programme.312 John Bache JP accepted that there were engagement mechanisms, including the Magistrates Engagement Group of which he was himself a member. However, he acknowledged that the answers given by HMCTS to the Group’s questions “are not always what we would like to hear.”313
194.The Judicial Intranet is an important means of communication with magistrates. The Magistrates’ Leadership Executive (MLE) thought that there had been difficulties in keeping this up to date, as a result of which “most magistrates have little idea what reform will look like for them and little appreciation of the value it will bring.” The MLE suggested that the magistracy “has different communication needs to the rest of the judiciary”, and that a new approach was needed to engage magistrates and persuade them of the benefits of reform.314
195.The Lord Chief Justice said that he felt “disappointed and, frankly, concerned” that magistrates felt that their communication needs were not being met. He thought that feedback from magistrates into the Judicial Ways of Working project “has profoundly influenced the product of those exercises”, and stressed the important role of the Magistrates Engagement Group. He said the Judicial Office has a new, dedicated communications team to deal with reform, and in his assessment, much information was available on the intranet. However, he accepted that “the very fact that the concern has been expressed—I appreciate that it has been expressed—leads me to conclude that we have to look at that again.”315
196.Our evidence suggests that HMCTS has struggled to explain its vision for the court and tribunal reform programme. Given the programme’s constitutional, strategic and operational significance, we recommend that the Lord Chancellor and Secretary of State for Justice takes clearer ownership of the programme and assumes the lead in communicating its vision.
197.Consultation and engagement should never be mere “lip service.” Early and effective engagement with stakeholders including judicial office holders at all levels is critical to the programme’s success, because this provides external expertise and detailed scrutiny of untested proposals.
198.We recommend that the Ministry of Justice and HMCTS increase the resources dedicated to stakeholder engagement and adopt a more rigorous approach to analysing and reacting to the feedback received.
Evaluation of the reforms
199.HMCTS has started a significant research and evaluation programme, which “continually seeks the views of the people who use the courts and tribunals system and develops insight from its findings.”316 The purpose of this evaluation will be to understand the effect of the reform programme as a whole by answering three principal questions:
- fairness: has reform altered outcomes (e.g. case/hearing outcomes, sentencing and financial awards)?
- accessibility: has reform changed the ability of users to pursue a case effectively (access to justice e.g. ability and speed at which court users can access and pursue a case)?
- cost: has reform had an effect on costs including those incurred by those who use courts and tribunals (e.g. travel costs, costs of time wasted)?
200.HMCTS expected to have completed scoping work for its evaluation programme by Spring 2019, setting out detailed questions that will underpin an assessment of the three principles; what data is collected and what further information is needed; and how it will evaluate the effect of the reforms on vulnerable users.
201.The MoJ has acknowledged that evaluation is likely to raise two specific challenges:
- establishing baselines of pre-reform performance for new measures; where possible MoJ will rely on proxy measures, but it may be limited to measuring actual performance as new data collections become available; and
- controlling external factors that may also have an impact on the performance of courts and tribunals—e.g. wider policy initiatives within the MoJ and/or other justice partners: “It is likely to be the case that in some circumstances we will not be able to be certain that any performance impacts we identify through the evaluation can be fully attributed to the reform programme.”317
202.Richard Goodman, Change Director at HMCTS, explained that evaluation of the programme had several levels. Overarching evaluation was being undertaken outside HMCTS by the MoJ; this would be supported by an independent advisory panel of academic experts.318 The other aspects of the evaluation “are happening all the time in the background”; for example, the independent evaluation of the fully video hearings in the Tax Tribunal that was conducted by the London School of Economics.319
203.Some who submitted evidence to the inquiry had little confidence in the MoJ’s evaluation of reforms so far, or in its proposals for future evaluation; we have already noted these concerns in relation to video hearings and video links (see Chapter 3). A typical response came from Harriet Bosnyak from Shelter:
There does not seem to have been much evaluation of what has gone on before. What has happened? Are people struggling to make it to the courts because there have been so many court closures? Are people actually using the online processes that are already there? [ … .] If we are pushing forward further reform, how do we know whether it is going to have the effect we want it to have?320
Others expressing lack of confidence in MoJ’s evaluation included the Legal Committee of HM Council of District Judges (Magistrates)321 and the Prison Reform Trust.322
204.Some witnesses expressed a degree of cynicism. Referring to HMCTS’s “Post Implementation Review” of the decision to close all but one county court in Greater Manchester, the Association of HM District Judges pointed out that no local or national impact assessments had been provided to the review panel, and reported that one participant judge had summarised the review’s apparent purpose as being to consider “what further changes might we make …… to achieve greater efficiencies from the diminishing staff and judiciary without spending any money?”323 Manchester Law Society Civil Litigation Committee had seen no evidence that significant steps had been taken to evaluate the reforms and had the impression that “pilots are run and then adopted without any significant change, irrespective of how the [legal] profession believes the pilot operated.”324 The Magistrates’ Leadership Executive thought that pilots benefited from the input of extra resources for short periods, leading to false outcomes.325 There have been reports that the published version of research conducted for the HMCTS Customer Insight Team on user experiences of the justice system omitted findings that suggest people have a more positive experience of the justice system if they attend court in person.326
“Agile” design technique
205.In developing processes for specific projects within the court and tribunal reform programme, HMCTS is using new “agile” design techniques, pioneered in the UK by the Government Digital Service. This involves iterative testing with user groups to support the design of processes, developing and revising them on an ongoing basis in response to continuous user feedback. HMCTS considers that this gives opportunities to test, refine and improve each change, rather than bringing everything together at a single point at the end.
206.Although there was support for the principle of user engagement, some witnesses raised concerns about the “agile” approach.327 Dr Joe Tomlinson cautioned that, in placing emphasis on convenience and what users appear to want, care must be taken not to overlook traditional concerns such as procedural fairness.328 Transform Justice argued that unpublished “user research” of this type, designed to help product design, “does not meet any of the guidelines essential for academic research and does not conform to the protocol on the publication of government social research.”329 The likelihood that people using HMCTS prototypes lack expert legal knowledge, and thus do not know when they need more information, was considered problematic by Amanda Finlay. She was concerned about HMCTS teams developing projects in isolation rather than working on an “end to end” process.330
207.The Senior President of Tribunals, Sir Ernest Ryder, thought it inevitable that an agile design technique would involve evaluation of the constituent parts of the programme. However, we were pleased that he emphasised the importance of evaluating the reforms by reference to access to justice principles—an approach also supported by the Administrative Justice Council.331
The views of academics
208.Dr Joe Tomlinson, then of Kings College London, provided a detailed literature review for the Committee, giving an overview of key recent pieces of academic literature relevant to our inquiry terms of reference.332 His review revealed “a dearth of concrete empirical evidence of the performance of online dispute resolution (ODR) and related technologies (such as video link hearings)”; this meant that much published material was best characterised as “sophisticated speculation and analysis but without an empirical evidence base”—although with some notable exceptions. Dr Tomlinson’s review assisted us in distilling some of the more robust research findings that we refer to in our report. We were particularly interested to hear that there is a growing literature on the evaluation of justice processes and related concepts, including access to justice, including the authoritative handbook produced by the Tilburg Institute for Interdisciplinary Studies of Civil Law and Conflict Resolution Systems.333 We also received submissions from several academics on approaches to evaluating the reforms, five of whom (in addition to Dr Tomlinson) attended an informal evidence session to discuss how the Government should ensure proper and effective evaluation of the programme.
209.Dr Natalie Byrom, on the basis of research conducted on secondment to HMCTS in 2019, makes recommendations for principles for evaluating the impact of reform on access to justice, including:
- ensuring HMCTS’s online processes deliver access to justice according to legal standards established by case law: access to a formal legal system; access to a fair and effective hearing; access to a determination; access to an outcome;
- collecting data about court users’ vulnerabilities, including age, mental and physical disabilities, literacy levels, and gender;
- monitoring outcomes in digital courts, to compare with outcomes under pre-digital processes and evaluate how different groups fare under the new system (for example, represented/unrepresented court users)
- ensuring transparency around ‘nudges’ built into the system, designed to promote or discourage types of behaviour by court users, such as seeking legal advice on a case (say, by positioning of buttons on a screen);
- considering introducing unique identifiers for each court user, allowing researchers and evaluators to have a complete picture of an individual’s experience of the court process; and
- ensuring there are ethical controls over how information is used, to avoid misuse and ensure privacy is protected; publishing HMCTS’s open data strategy, developed in line with legal and ethical principles.
210.We regret that the Ministry of Justice’s plans for evaluating its court and tribunal reforms are not as far advanced as might have been expected at this stage in the programme. We are concerned that reliance on “agile” design techniques in some projects may distract from analysis of implications for access to justice. The Ministry could do much more to evaluate the impact of the reforms on vulnerable and excluded groups.
211.The Ministry of Justice should be as transparent as possible in its evaluation of the reforms. We recommend that the following evaluation approaches are adopted:
- projects within the reform programme, and the programme as a whole, should be evaluated against the access to justice standards set out in law: these can be broadly summarised as (a) access to the formal legal system (b) access to a fair and effective hearing (c) access to a decision and (d) access to an outcome;
- evaluation should prioritise monitoring the impact on access to justice for digitally excluded and vulnerable people, with a particular focus on justice outcomes rather than on processes alone;
- the transfer to digital systems should be used as an opportunity to collect detailed, anonymised data on the operation of courts and tribunals and the experiences of users by reference to their personal characteristics;
- the evaluation of the reforms should be seen as a long-term and ongoing commitment, and one for which the investment of resources is required. Evaluation must be robust and objective, using a comparator group where possible.
289 Early progress in transforming courts and tribunals. National Audit Office, May 2018 (paragraph 21a)
290 House of Commons Committee of Public Accounts. Transforming courts and tribunals, Fifty Sixth Report of Session 2017–19, 20 July 2018, HC 976, P6
291 HMCTS, Engaging our external stakeholders, November 2018
292 For example, Crown Prosecution Service (CTS0074); Victim Support (CTS0003); Women’s Aid (CTS0006); JUSTICE (CTS0068)
293 Including Centre for Justice Innovation UK (CTS0008); Criminal Justice Alliance (CTS0050); Right2Justice (CTS0054); The Criminal Bar Association (CTS0063); Transition 2 Adulthood (T2A) Alliance (CTS0037); Revolving Doors Agency (CTS0073)
298 Public Law Project (CTS0027). Similar complaints were made by other witnesses including Rhona Friedman, Sue James and Simon Mullings (CTS0085) and Transform Justice (CTS0022)
299 Leeds Law Society (CTS0011); Housing Law Practitioners Association (CTS0090); Manchester Law Society - Civil Litigation Committee (CTS0004)
306 Together with the Tribunals Judicial Executive Board, the Judicial Executive Board is the most senior decision-making forums for providing a judicial view on design or implementation questions relating to the reform programme.
308 The Administrative Justice Council is the successor body to the Administrative Justice Forum which was abolished in April 2017.
312 MAG0001. In evidence to this inquiry, Captain Hugh Daglish JP (CTS0001) referred to “plenty of anecdotal evidence of worry and discontent” among magistrates, many of whom fear that the reform programme will have a detrimental effect on justice
316 HM Courts & Tribunals Service (CTS0064). The approach to evaluating the HMCTS reforms is set in the MoJ’s Response to the Public Accounts Committee recommendation 4: Evaluating our reforms, January 2019
317 Response to the Public Accounts Committee recommendation 4: Evaluating our reforms, January 2019
318 To date, the membership of the advisory panel has not been announced.
326 See blog by Transform Justice (March 24, 2019) and report in Buzzfeed News (March 18, 2019). The research was published as HM Courts & Tribunals Service Citizen User Experience Research. HMCTS Customer Insight Team, 2018
Published: 31 October 2019