Scottish Affairs CommitteeWritten evidence submitted by Ipsos MORI

When the Scottish National Party won an overall majority of seats in the Scottish Parliamentary election of May 2011, it was clear that there would be a referendum on Scotland’s constitutional future at some point in the current Holyrood term.

It was the intervention of the Prime Minister in early 2012 which brought the issue into sharp focus in the media and public consciousness. While the UK Government accepts that the referendum should take place in accordance with the manifesto promise made by the SNP, it is clear that there is very little agreement between it and the Scottish Government about any element of the process involved in holding the ballot.

The governments have disagreed on a range of issues. On timing, the Scottish Government has stated its preference for holding the referendum in the autumn of 2014 while the UK Government prefers an earlier ballot. The Scottish Government is open to the inclusion of an additional question in the ballot about what has become known as “Devo. Max”, affording substantial new powers to the Scottish Government while Scotland remains in the UK. The Scottish Government also wants consideration to be given to extending the franchise in the referendum so that those aged 16 and 17 are also eligible to vote while the UK Government does not support this idea.

So, there are a range of issues to be decided at the outset of the referendum process. Of all these issues of current disagreement, the wording of the referendum question itself is likely to be the most contentious. On January 25 2012 the Scottish Government published its referendum consultation paper, “Your Scotland, Your Referendum”1 which included its proposed question for the ballot: “Do you agree that Scotland should be an independent country.” The publication of a preferred question was instrumental in moving the debate forward and was an important moment for those of us in the research industry with an interest in question design and the impact it can have on results obtained.

In any referendum the question and the preamble to the question provide the focal point for the debate, which usually involves groups coalescing around either a “Yes” or “No” campaign in favour of, or opposed to, the proposition being put forward. The question itself can set the tone and context for the campaign, allowing both sides to put their case to voters.

Our view is that the referendum question must satisfy two key criteria, it must be clear so that voters know exactly what they are being asked and it must be fair so that the results of the referendum are undisputed.

The question must be clear and easy for voters to understand. This means that it should be short and straightforward and should not be open to legal or political challenge after the result of the referendum is known. The Scottish Government’s proposed referendum question has generally been accepted as being clear. However, it does assume that voters understand exactly what is meant by “an independent country”, including, for example, questions of currency, borders, the military and nuclear weapons. The importance of voters understanding these issues would be heightened if a second question on “Devo. Max” were included and also needed to be defined for voters.

Previous referenda in the UK have highlighted the importance of simplicity and clarity, ensuring that all sections of the electorate are clear on the question being asked; indeed the Electoral Commission insisted on a change to the question originally proposed by the UK Government ahead of the 2011 AV referendum because they considered that “particularly those with lower levels of education or literacy, found the question hard work and did not understand it”.

The question must also be deemed to be fair and objective, ensuring that the final result is accepted by all sides without accusations of bias in terms of the question wording. Before explaining why this is so important and what can be done to mitigate against question bias, it is worth pointing out the difference between referendums and the opinion polls we regularly conduct.

We know from years of experience, backed up by academic research, that even subtle changes in the wording of an opinion poll question can affect the answers we get, creating a bias in one direction or the other. The effect would normally be much less in a referendum because voters already know what they are going to be asked and have decided which side they are on, so you would not expect the exact way the question is asked (providing it still has the same meaning and is not in any way confusing) to change many people’s votes. The same would apply in a poll about how people will vote in the referendum, once they are aware of the issue and have had a chance to think about it, and even more so once the campaigns have got going. There is much less chance of the wording affecting the outcome of a referendum than of an opinion poll when the question has been sprung on a respondent who is not expecting it.

However, this does not affect the fact that the wording may be biased, only whether that bias will have a big practical effect on the vote. Since it is a fundamental part of the legitimacy of a referendum that the whole process should be fair and should be seen to be fair to all sides, you should therefore avoid using a question wording that tends to persuade people more to vote for one side than for the other, even if that bias is so subtle that it may not have a major effect in practice.

The Scottish Government’s proposed question has been subject to much stronger criticism on grounds of bias, with critics suggesting that the question statement goes only in one direction and invites voters to agree with it.

One key point here is to consider what responses would be if the question was asked the opposite direction, such as “Do you agree that Scotland should stay in the United Kingdom?” Previous referenda have resolved this issue by offering the two alternatives on offer and giving them equal weight; the 1997 devolution referendum provides a useful example as voters were asked to choose between the statements, “I agree that there should be a Scottish Parliament” and “I do not agree that there should be a Scottish Parliament.” Some argue that this type of approach makes the question more balanced, giving voters “equal” choice between the two sides of the debate.

There are a number of practical ways we would recommend of ensuring the referendum question is both clear and fair. These include making sure the question has few words, avoids ambiguous words or statements and steers clear of loaded, leading or double-barrelled questions. Judging whether proposed questions satisfy these criteria is a more lengthy and difficult process. Both the Scottish and UK Governments have established public consultation exercises so that voters can have their say about all matters to do with the referendum. There will also be debates in Holyrood and Westminster where MSPs and MPs can contest different options for wording the question as well as widespread media comment on the issue, all of which can aid voters in understanding the issues.

There will clearly also be a role for the Electoral Commission in ensuring clarity and fairness, as was the case in the most recent UK-wide referendum on the voting system for Westminster. Our view is that there also needs to be significant testing of different question options before a final decision can be made.

This testing can be done in a number of ways. Polling is one way. In our most recent poll, at the end of January 2012, we used the Scottish Government’s preferred question to measure support for independence and found 39% agreed that Scotland should become an independent country.2 This represented an increase of just 1% from our previous poll in December 2011 when we measured support for independence using a different question.3 This suggests that the question wording has not made a huge difference to support for independence although we’ll have a better idea of whether a question is clear and fair once different options have been tested rigorously and analysed against differently-worded questions, with robust sample sizes.

In addition to using evidence from polling, we believe that a phase of “cognitive testing” of various question options would be useful. Cognitive testing is an approach we use to assess how questions are understood and answered by respondents. After a respondent has answered the question, a researcher interviews the respondent to explore how they went about answering it. This process can be used to refine questions before further testing. We have successfully used cognitive testing techniques in a range of settings including testing the question on “national identity” before the 2011 census. In this case our testing informed the wording of the final census question, ensuring that respondents were clearer about what was required and better able to express their identity/identities fully. In the case of the referendum debate, this would be extremely helpful in providing evidence from voters about the wording of different question options.

So, the wording of the referendum question is vital. It can frame the terms of the campaign for all sides of the argument. It must avoid ambiguity to ensure that there is no ambiguity in the result. It must be clear to voters so that they know exactly what they are voting to accept/reject and understand the impact of the result. And it must be seen to be fair so that the losing side in the ballot does not try to discredit or challenge the result on the basis of the question wording.

There are likely to be a range of alternative questions proposed before the referendum is held. Each will be debated vigorously and will be subject to media scrutiny and comment. However, it is important that each is also tested thoroughly and independently, including the use of cognitive techniques, before the final decision is made.

April 2012

1 http://www.scotland.gov.uk/Publications/2012/01/1006/0

2 http://www.ipsos-mori.com/researchpublications/researcharchive/2912/Four-in-ten-Scots-back-independence.aspx

3 http://www.ipsos-mori.com/researchpublications/researcharchive/2891/Scots-support-early-referendum-as-support-for-independence-increases.aspx

Prepared 4th May 2012