Memorandum submitted by the Understanding
Risk Team and Collaborators (A17)
The Understanding Risk team and collaborators,
are currently conducting an independent evaluation of the GM
Nation? public debate about the possible commercialisation
of transgenic crops in Britain, 2003.
BACKGROUND
1. The Understanding Risk programme is a
major research initiative based at University of East Anglia and
involving researchers from Cardiff University, Brunel University
and the Institute of Food Research at Norwich. The programme brings
together a team of experienced researchers in the investigation
of the nature of risk attitudes, risk governance and public engagement
in contemporary Britain1.
2. Our research work had involved monitoring
the work of the Agriculture and Environment Biotechnology Commission
(AEBC) prior to its proposal to Government for the GM Nation?
public debate to take place. In September 2002 we were invited
to present a detailed evaluation proposal to the public debate's
Steering Board, and on the basis of our proposed methodology were
invited to act as independent evaluators of the debate.
3. Our evaluation work is funded independently
from the debate process, drawing on grants from the Leverhulme
Trust and from the Economic and Social Research Council.
4. It is clear that GM Nation? was,
for the UK, a highly innovative experiment in participatory democracy,
regarding the possible commercialisation of a controversial technology.
In this respect the debate process itself has been broadly successful
in generating unprecedented levels of interest, participation
and considered discussion about complex matters of science and
policy amongst a relatively large number of the general public.
We recognise that members of the Steering Board, who supervised
the debate, committed considerable time and personal effort in
making this important initiative happen.
5. Throughout the process of evaluation,
we were provided with unique behind-the-scenes access by the Steering
Board, which has set new standards in terms of openness and transparency.
To the Board's credit, this has generated not only an extensive
and publicly available "audit trail" for the debate
process, but will be of benefit as a research resource for others
to use in the future.
6. The process of analysing the voluminous
quantity of data that we have collected is still continuing. We
aim to publish a major report in the near future, which will:
— Provide a detailed evaluation of
the implementation of the debate
— Set the debate results in the context
of our research into public views on both GM and its regulation,
and on the merits of the debate itself
— Set out a number of recommendations
on methodological lessons from the experience of implementing
GM Nation?
As academic researchers our primary interest
is in exploring and developing those lessons for informing future
practice and theory. As our research is ongoing we cannot comment
upon all of the issues we are investigating. However, we can set
out some brief observations on a number of aspects of the debate
process in the light of our preliminary findings.
EVALUATION APPROACH
7. There are two main elements when evaluating
exercises in deliberative democracy such as GM Nation?
First how well did the process match up to set criteria and objectives,
and second whether identifiable and desired outcomes were achieved2.
8. As independent evaluators, the focus
of our work has been on the debate process rather than the merits
or dangers of the technology. In carrying out this work, we have
utilised a range of research methods; details of which are set
out in Box 1. When all our work is complete we will have conducted
one of the most thorough investigations of citizen engagement
ever to have taken place.
9. In order to evaluate the debate process
we are working with three distinct sets of criteria.
— First, the aims and objectives as
set by the Steering Board itself. Here one needs to ask whether
the process could reasonably meet all or some of these objectives,
and in the event whether it did indeed do so.
— Second, a set of generic evaluation
criteria derived from the academic literature on participatory
processes. These focus on whether the debate process was: transparent
to both those involved and interested parties; specified well-defined
tasks; was run in an independent and unbiased way; was inclusive
of all relevant views; had sufficient resources with which to
achieve its objectives; and encouraged effective and fair dialogue.
— Third, from analysis of questionnaire
responses of participants to the debate we can develop a view
of how they judged the success or otherwise of various aspects
of the process.
10. At the time we submitted our proposal
to carry out the independent evaluation in September 2002, the
Steering Board was still in the process of preparing its formal
objectives for the debate process. In that sense, work on the
implementation of the debate began on the basis of a general understanding
of what would be needed. At the time we observed that whilst it
was to be expected that the debate would have multiple purposes,
this ambiguity had the potential to create a sense of uncertainty
about what was going on. We pointed out that the debate could
be seen as: an exercise in communication, designed to inform (and
possibly persuade) the public; a consultation exercise, designed
to chart lay views; an exercise in participation, designed to
engage the lay public in a decision-making process; or as an experiment
in engagement, designed to assist future practice and policy development.
Finally, we noted that ideally all parties to be involved in the
debate should be involved in agreeing the criteria that would
be used for its evaluation. In this case, we suggested that the
Steering Board should give serious consideration to identifying
criteria that could reasonably be thought to be widely acceptable.
Box 1: Methods used to gather evaluation data
on GM Nation?
Observation of various aspects of the debate
planning process
Observation of Foundation Discussion workshops
Observation of a sample of the open events,
including all Tier 1 meetings
Observation of `Narrow but Deep' meetings
Issuing questionnaires to participants in all
Foundation Discussion workshops and `Narrow but Deep' groups,
and to a sample of participants in open meetings
Interviewing key players in the debate organisation,
and representatives of engaged stakeholder organisations
Monitoring media coverage of the debate
Commissioning MORI Social Research Institute
to carry out for us a major survey of general public opinion in
Britain among a representative sample of 1,363 adults on attitudes
to GM food and crops, their regulation, and awareness and beliefs
about the debate process itself. This survey was conducted during
the period July-September 2003.
11. A set of nine objectives were subsequently
agreed by the Steering Board. These are set out in Annex B of
the report on the debate findings. In addition, a further four
indicators of success were identified, addressing: public awareness,
the views of participants, the views of "informed commentators",
and the impact of the debate on Government decision-making. We
note that these four indicators do not closely address the stated
objectives; indeed they sometimes imply other, additional, objectives.
We suggest that it is important to consider the setting of these
criteria for how the performance of the debate might be judged
against the evidence of recent practice in evaluating other deliberative
processes. In these terms, most of the Steering Board's objectives
are too imprecisely articulated in order to make them measurable
in any rigorous manner. We fully recognise that the debate was
an experiment, and it was to be expected that slippage would occur
between any given set of objectives and the emerging practice
of the debate process. Nevertheless, the process of setting objectives
might have been better informed by the now quite extensive body
of research findings in this area3.
FRAMING THE
ISSUES
12. The Debate Steering Board was committed
to allowing the public to determine how the issues around GM were
to be discussed (Debate Objective 1). Towards this end the Board
staged a number of discussion groups in 2002 (known as "Foundation
Discussion workshops") to investigate how a representative
cross-section of the lay public tries to make sense of these issues.
Our preliminary findings suggest that participants in these initial
events generally found them very well run, enjoyable, comprehensible,
unbiased and allowing people to have their say, although nearly
half felt that insufficient time was allocated to discuss all
the relevant issues. In methodological terms, these discussions
included some interesting and innovative techniques and were very
effective in establishing how members of the general public interpret
GM in terms of facets of everyday life, like food, health and
trust in decision-makers.
13. As previous research has also found,
attitudes towards GM are relatively ill-formed, touch upon issues
which go beyond the mere "risk" and "benefits"
of the technology, and are characterised by considerable ambivalence4.
We regard this degree of ambivalence as a significant matter in
the interpretation of the outcome of the GM Nation? debate.
Indeed, our 2003 survey indicated that more than half of the general
public were not sure whether GM food should be promoted or opposed.
14. The Foundation Discussion workshops
also highlighted that there were many factual areas where participants
felt they needed to know more: for example concerning scientific
and technical matters. The Steering Board thought it necessary
to develop stimulus materials to address this shortfall of knowledge.
A video, CD-Rom, and information booklet were produced to support
the subsequent stages of the debate process. Providing access
to some of the evidence needed to inform the debate (Debate Objective
4) set a particularly challenging task for the debate organisers.
It was difficult because of the need to reconcile overlapping
considerations: a diversity of interpretations of the issues;
relatively well-established scientific findings; the various uncertainties
associated with the issues raised; and value-based framings, such
as those based on ethical concerns. As a recent US National Academy
of Sciences Report5 on deliberative processes points out, there
is no easy way to strike a balance in presenting potentially conflicting
frames and information to participants. In the case of GM Nation?
a range of concerns were expressed to us by stakeholders we interviewed
about the way in which the stimulus materials were structured:
in terms of an adversarial for/against framing, and in disconnecting
`facts' from their sources.
THE FORM
OF THE
DEBATE
15. The design of the debate process included
both "open" and "closed" elements. The open
components—including the web-site, the various public meetings
and the feedback questionnaire—were designed to maximise
as far as was possible access to the process (Debate Objective
5). The "Narrow but Deep" groups were conceived as "controls"
on the open processes: purposefully including the perspectives
of those who might ordinarily not otherwise self-select to take
part in such a debate (Debate Objective 2). We believe this combination
represents a particularly innovative aspect of the GM Nation?
design.
16. The debate design can also be viewed
in part as incorporating different research processes (embedded
within the deliberations) to ascertain public views (Debate Objective
9). In this respect the Debate itself draws upon both quantitative
and qualitative methods in both the open and "Narrow but
Deep" elements. It is important to recognise that a range
of different standards of acceptable evidence and process apply
to these very different research methodologies6. In addition,
there are two main approaches to sampling; representative and
purposive. Representative sampling aims to develop a statistically
robust understanding of the distribution of attitudes across the
general population. Purposive sampling aims to capture the spectrum
of distinctive views and positions (or sometimes a subset of these)
within a population. The Steering Board's final report, with its
complex set of findings, needs to be interpreted with these distinctions
firmly in mind.
17. Three "Tiers" of open meeting
took place. These included six large regional events (Tier One)
supported by the Steering Board, about 40 events organised by
local authorities (Tier Two) some of which drew support from the
Steering Board and COI, and a large number of local meetings (Tier
Three). The local meetings were mostly organised by voluntary
organisations, including a range of environmental groups. All
of these events were encouraged to draw upon stimulus material
produced by the Steering Board. We observed a sample of these
events, including all six in Tier One. We also issued questionnaires
to participants in all events observed, as well as to a number
of additional events which we were not able to observe.
18. We have preliminary evidence from respondents
in the Tier One open events to suggest they were, in certain respects,
atypical when compared with the general public. For example, the
self-reported educational level of participants who returned our
questionnaires was considerably higher than the UK average (some
two thirds of respondents claimed to have a degree, compared to
one fifth of the UK population aged between 16-74, according to
the 2001 Census). We observed that these events were often dominated
by discussions characteristic of a knowledgeable and experienced
engagement in the GM issue. In attitudinal terms, participants
in these meetings held less favourable views towards GM food in
comparison to baseline data collected in 20027.
19. In organisational terms, the public
meetings were regarded by those attending as poorly advertised,
with participants tending to find out about them at the last moment.
This is not surprising given the relatively small amount spent
on direct advertising, and the restricted time available for conducting
the main Debate. Although it was the Steering Board's intention
to provide stimulus material to participants in advance of the
meetings, very often the late notice appears to have militated
against this happening. Accordingly, participants were generally
unclear about what the events would involve until discussion began.
20. Our respondents generally felt that
the events were well facilitated, and that they had had an opportunity
to have their say. However they also reported that they had insufficient
information and needed more time to complete the process of deliberation
(eg just over two thirds agreed or strongly agreed with the statement
that "there was not enough time to discuss all the issues").
Participants also felt that the events were enjoyable, but they
were not convinced that their views offered in these events would
be taken seriously by Government.
21. Many participants in the public meetings
expected that they would be able to engage directly with experts
on both sides of the GM debate. Whilst the Tier One events did
not provide this, by contrast many of the Tier Two local authority
events did. The local authority events (according to our small
questionnaire sample) were viewed more positively than the six
"Tier One" events according to a number of criteria.
We asked, for example, whether participants felt that any "hidden
agenda" was present in the meetings, and found that this
did not characterise the Tier Two events to the same extent as
the Tier One events.
22. Turning now to the closed "Narrow
but Deep" discussion groups, these were meetings which involved
a cross-section of participants, deliberately excluding people
with an active interest in the issue. This is appropriate research
recruitment practice given the stated objectives of the "Narrow
but Deep" groups. These groups met twice, with a gap of two
weeks during which participants were invited to explore the GM
issue individually, using Steering Board stimulus materials and
any other information that they could access. We were provided
with observational access to two of these groups, and audio recordings
of the remaining eight. On the basis of our observations, we noted
that the way in which issues were discussed within these groups
was in stark contrast to the kind of discourse typically present
at the public meetings. Rather than demonstrating a knowledgeable
engagement with the issues, it tended to be more open-ended, wide-ranging
and tentative.
23. The Steering Board's final report draws
the important conclusion that "attitudes hardened" as
the "Narrow but Deep" participants learned more about
the issues. We view this as an important empirical claim, which
deserves further detailed analysis. We offer the comment here
that, as an hypothesis, it may be that the distinctive characteristics
of the GM issue mean that styles of everyday reasoning tend to
push not-well informed or uncommitted people to adopt a cautionary
position towards the potential risks of GM technology as applied
to food. This seems to be particularly likely if understood in
the context of everyday beliefs about recent food controversies,
especially over BSE. We note with interest that the debate report
also states that the "Narrow but Deep" participants
tended to become more positive about certain potential applications
of the technology, especially in health care or in relation to
improving agriculture in developing countries.
24. Given the importance of the "Narrow
but Deep" findings to the final overall debate conclusions,
we suggest that a detailed discursive analysis8 of the interactions
and talk that took place at these group meetings is necessary,
in order to more fully understand the transition in attitudes
from uncertainty to caution. In addition, the "engagement
period" during which "Narrow but Deep" participants
gathered and debated other information and views also deserves
further detailed research.
25. In addition to the public meetings the
GM Nation? debate received over 36,000 completed questionnaires;
some 47% of which the final report describes as "implacably
opposed" to GM. A legitimate methodological question arises
over the extent to which this sample is representative, in terms
of attitudes to GM, of a broad cross-section of the public9. Other
survey evidence indicates that attitudes towards "biotechnology"
are complex. If one asks about different applications (medical,
crops, food), attitudes differ with the type. In particular people
are typically more positive about medical than agricultural applications,
and generally more positive when asked about GM "crops"
compared to "food". Regarding specific attitudes to
GM food, one can identify three broad groups among the general
public: first, there is indeed a sizeable proportion of the UK
population, which can vary between one third and one half, depending
upon the question asked, who see significant risks and very few
benefits; second, there is an "ambivalent group" who
are concerned but also believe that GM food offers some benefits
alongside potential risks; and third, there is a smaller group
who are generally positive about GM food10. Of course, engaged
people with clear views on the GM issue had a legitimate contribution
to make in the debate. Moreover, the exploration of such views
was important, in view of their prominence within the political
dynamics of wider debates about GM.
PUBLIC AWARENESS
OF THE
DEBATE
26. One objective of the Steering Board
was to create widespread awareness of the GM Nation? debate
(Objective 5), although we note that this does not specify a target
level of "awareness". This objective can be evaluated
in a number of ways.
27. Both the debate and more general issues
related to GM crops and foods achieved reasonable visibility in
the media11, but the degree of attention varied considerably across
different media outlets. There was considerable coverage in the
national broadsheet dailies and in the mid-market Daily Mail,
which returned to its "Frankenstein food" campaign theme:
coined a considerable time before the debate was even proposed.
The two main tabloid dailies carried practically no coverage.
The amount of coverage in the local and regional press also varied,
being generally greater in areas with agricultural economies or
where GM had become a political issue. A Great Britons style television
series was proposed at one stage. In the event, the debate failed
to attract significant coverage on the main terrestrial television
national news bulletins. It attracted more attention on Radio
news. A range of radio programmes, from The Moral Maze to The
Archers, also provided spaces where the issues could be debated.
The internet played an important role as both an additional source
of information and a forum for participatory debate, with the
BBC website dedicating part of its science message board to GM
for most of the debate period. This suggests that future initiatives
would benefit from developing "joined up deliberation"
by actively looking for ways to integrate broadcast, web based,
and face-to-face modes of information exchange and debate.
28. Our 2003 national survey by MORI included
a question on levels of awareness of the debate12. Using a sample
size of 1,363 people aged 15 years and older, 14% reported that
they had heard about the Debate and knew at least something about
it, while a further 13% had heard about it but knew nothing of
it. These figures can be interpreted in any number of ways. On
the one hand, the bulk of the population (seven in ten) had not
heard of the debate at all. On the other hand, this finding does
suggest that a sizeable minority of the British adult population
had been made aware of its existence. Given the relative lack
of advertising, tabloid and television coverage of the debate,
this figure might be regarded as representing a modest success,
and indeed this performance could usefully be compared to data
on awareness of other Government initiatives in comparable areas
of science and technology. However, on a question of this kind
it should be noted that one can get "false positives":
that is, people claiming to have heard or read about something
they have not.
THE CONTEXT
SURROUNDING THE
DEBATE
29. It is important to recognise that the
organisation and form of the Debate were shaped by a number of
external constraints. Of prime importance here were practical
challenges posed by two inter-related factors: the novelty and
scale of the enterprise, and the availability of resources. Without
a clear template about how best to organise such a process, implementing
the debate itself became a learning exercise resulting, inevitably,
in inefficiencies and tensions. In addition, a difficulty in appreciating
what would be required in order to deliver the debate effectively
led to a serious initial under-estimate by Government of the debate
budget. The resulting shortage of funds led to the Steering Board
having to seek additional funding from Government: a process which
created additional tensions and considerable time pressures. The
need to ensure adequate funding at the outset is clearly an important
lesson to emerge from the GM Nation? exercise as a whole.
30. Distrust in Government is highlighted
in the Steering Board's final report. In commenting briefly on
this important issue, we note the ambiguity in Government statements
about the objectives for the debate. It was not clear whether
the motivation behind sponsoring the process lay in capturing
public perceptions and understanding of GM, providing the public
with information, or finding ways of persuading the public to
accept GM crops and food. The refusal by the Government for the
debate to be extended to await the scientific assessment of the
Farm-Scale Trials on GM herbicide-resistant crops also prompted
much criticism.
31. During the year preceding the debate,
a number of ministers had made clear their support for GM, and
the Prime Minister had taken the opportunity of a speech to the
Royal Society to criticize anti-GM activists. The US government
threatened to take action against the EU through the World Trade
Organisation on the basis of its allegedly anti-free market position
on GM produce. The credibility of the Government's claims regarding
its open-mindedness on the commercialisation of GM crops was further
eroded by reminders of its responsibilities of EU membership;
for example by the constraint that under the directive EC/2001/18,
nation states can only reject GM crops on the basis of scientific
evidence of adverse health or environmental impacts, irrespective
of public preferences.
32. It is difficult to judge whether these
contextual influences made a direct difference to the conduct
and outcomes of the GM Nation? debate. However we recognise
that many of them held the potential to undermine the credibility
of the debate process. The evidence provided by our 2003 national
survey indicates that 68% of adults thought that it does not matter
whether there is a debate on GM or not, as "in the end European
and International laws will determine what will happen".
CONCLUDING COMMENTS
33. To reiterate, the GM Nation?
public debate was, for the UK, an innovative initiative in participatory
democracy. Both its strengths and difficulties need to be assessed
in the light of the inevitable constraints and practicalities
of running a major experiment in public engagement of this nature.
34. Important constraints on the process
were the failure to define an adequate level of funding for the
debate at the outset, and some ambiguity over the purposes of
the exercise.
35. Whilst we concur with the debate's key
conclusion, that significant levels of concern about certain aspects
of GM exist among the British population, we recognise the need
to qualify this finding in the light of the high levels of ambivalence
about the technology, and the need to more fully assess, and critique,
the debate's methodology.
36. We strongly endorse the view of the
debate Steering Board that the evaluation of the debate should
be allowed to continue beyond the formal end of the debate process
in order to fully measure its outcomes. This work should also
investigate the long-term articulation of the GM debate with the
science and economic strands (Debate Objective 7), and its influence
on Government decision-making both on GM issues specifically and
more generally on other areas of science and policy-making, including
science and technology.
37. Whilst we were pleased to be in a position
to conduct the research needed to evaluate the GM Nation?
debate, this was in many respects fortuitous and dependent upon
a major existing funding initiative by the Leverhulme Trust. For
future public debates of this size and importance we would recommend
that independent evaluation is explicitly incorporated, and budgeted
for, at the commissioning stage.
38. Our evaluation work is ongoing. We would
be happy to provide further clarification on any of the above
points if required.
Submitted as evidence by Professor Nick Pidgeon,
Director of the Understanding Risk Programme, who is also the
corresponding author.
Authorship of the document is as follows:
Tom Horlick-Jones (Project Team Leader for the
evaluation exercise and Senior Research Fellow, Cardiff University)
John Walls (Senior Research Associate, University
of East Anglia)
Gene Rowe (Senior Research Scientist, Institute
of Food Research, Norwich)
Nick Pidgeon (Professor of Environmental Sciences,
University of East Anglia)
Wouter Poortinga (Senior Research Associate,
University of East Anglia)
Tim O'Riordan (Professor of Environmental Sciences,
University of East Anglia)
In collaboration with: Graham Murdock (Reader
in Media Sociology, Loughborough University), Ann Bruce (Research
Fellow, INNOGEN Centre, University of Edinburgh) and Joyce Tait
(Director of the INNOGEN Centre, University of Edinburgh)
TECHNICAL NOTES
1 Details of the Understanding Risk research
programme may be found at: www.uea.ac.uk/env/pur. For the purposes
of evaluating the GM debate, the Understanding Risk network has
received invaluable collaborative support from the ESRC-supported
INNOGEN centre at Edinburgh University and by the Department of
Social Sciences at Loughborough University. We are also pleased
to acknowledge a number of colleagues who have assisted us in
collecting data and taking part in helpful discussions: Nick Bailey
(Cardiff University), Karen Bickerstaff (UEA), Mick Bloor (Cardiff
University), Alan Irwin (Brunel University), Kevin Jones (Brunel
University), Irene Lorenzoni (UEA), Carl Macrae (UEA), and Peter
Simmons (UEA).
2 See Parliamentary Office of Science
and Technology (2001) Open Channels: Public Dialogue in Science
and Technology POST, London
3 See Rowe, G. and Frewer, L. (2000) `Public
participation methods: a framework for evaluation' Science, Technology
& Human Values, 25(1), 3-29.
4 See eg Grove-White, R., Macnaghten,
P., Mayer, S. and Wynne, B. (1997) Uncertain World: Genetically
Modified Organisms, Food and Public Attitudes in Britain, Centre
for the Study of Environmental Change, Lancaster; Petts, J., Horlick-Jones,
T. and Murdock, G. (2001) Social Amplification of Risk: the Media
and the Public, HSE Books, Sudbury; Marris, C., Wynne, B., Simmons,
P., Weldon, S, et al (2001) Public Perceptions of Agricultural
Biotechnology in Europe (PABE) Final Report to the European Commission
FAIR CT98-3844 (DG12-SSMI) (see www.pabe.net).
5 National Research Council (1996) Understanding
Risk: Informing Decisions in a Democratic Society National Academy
Press, Washington DC.
6 See eg, Bryman, A. (1988) Quantity and
Quality in Social Research. Unwin Hyman, London
7 For example, we compared levels of concern,
acceptability, importance and the acceptability of current regulations
(all in relation to GM food) from a sample of just over 400 questionnaire
responses from the Tier One events with a nationally representative
sample of 296 from our earlier research. Details of the earlier
2002 survey can be found in Poortinga, W. & Pidgeon, N.F.
Public Perceptions of Risk, Science and Governance. Main Findings
of a British Survey on Five Risk Cases (Technical Report). Norwich:
Centre for Environmental Risk, January 2003 (copies of this report
can be downloaded from the `Latest News' field of www.uea.ac.uk/env/pur)
8 Such discourse analysis would typically go
well beyond the thematic analysis applied to such data by market
research organisations. See, for example, Sarangi, S and Candlin,
C. Eds. (2003) `Categorisation and explanation of risk: a discourse
analytic perspective'. Health Risk & Society, 5(2), 115-228.
9 It is, of course, a misnomer in risk attitude
research to talk of `the public' as if this is a single undifferentiated
entity. In reality civil society comprises a myriad of attitudinal
positions and interests.
10 For poll evidence from Eurobarometer see eg
Gaskell, G., Allum, N., Bauer, M.W., Jackson, J., Howard, S.,
& Lindsey, N. (2003) Ambivalent GM Nation? Public Attitudes
to Biotechnology in the UK, 1991-2002. Life Sciences in European
Society Report: London School of Economics and Political Sciences.
11 Analysis of media coverage of GM issues, and
of the debate itself, during the debate period is being conducted
by Graham Murdock of Loughborough University.
12 The survey was conducted for us by the research
company MORI (Social Research Institute) directly after the end
of the formal Public Debate; ie, between 19 July and 9 September
2003. A nationally representative quota sample of 1,363 people
aged 15 years and older was interviewed face-to-face in their
own homes in England, Scotland and Wales. All data have been weighted
to the known profile of the British population.
10 October 2003
| 
