Scottish Affairs CommitteeWritten evidence submitted by Ruth Stevenson

Executive Summary

Ruth Stevenson is an experienced researcher and methodology expert based in Scotland. In this paper Ruth sets out the professional standards of questionnaire design outlined in good practice guidance provided by the Market Research Society, and provides commentary on how these standards should be applied to the Scottish referendum question.

In order to ensure that the results to the Scottish referendum are conclusive and stand up to scrutiny, Ruth proposes that the final referendum question(s) should be:

Short, straightforward and clear so as to be accessible to the full spectrum of Scottish adults.

Balanced so as not to lead voters towards a particular response.

Presented in a manner that does not introduce bias.

Reflective of the intended use and outcomes of the referendum, so that voters can give honest responses and the results are conclusive.

This will ensure that the referendum provides a useful foundation upon which to make a genuine evidence-based decision about Scottish independence.

Context

1. I am an independent research expert with ten years of experience in research attained through my consultancy roles at global market research agencies (TNS and MORI) and managing research departments at charities (The Audience Business and Scottish Development Centre for Mental Health). In 2010 I set up Ruthless Research to provide high quality research services for public and voluntary sector organisations, which is something that I am passionate about.

2. I am a full member of the Market Research Society and an award-winning researcher (Royal Society for Public Health), with a thorough grounding in research methodology through a degree in Sociology (University of Edinburgh), the MORI graduate scheme, and the Diploma of the Market Research Society.

3. As an expert in the methodology of questionnaire design and a resident of Scotland, I have a professional and personal interest in the methodology used to collect Scottish referendum responses. At the time that the SNP proposed an initial referendum question in 2007 I managed the Scottish omnibus population survey for global research agency TNS, where I polled the proposed question to track voting intentions over time. When another referendum question was proposed in early 2012 I prepared an article critiquing the methodology which featured on my research methods blog1 and was syndicated by Research Live, Research Magazine and the Huffington Post.

4. As a result of my specific expertise outlined above, I have been asked to present written evidence for the Scottish Affairs Select Committee giving my professional opinion on the referendum question, and the mechanics of this.

5. As a professional researcher I abide by the Market Research Society Code of Conduct2—a set of ethical standards of good practice developed by the Market Research Society over the last 50 years. As the Code states, “The Code of Conduct is designed to support all those engaged in market, social or opinion research in maintaining professional standards. The Code is also intended to reassure the general public and other interested parties that research is carried out in a professional and ethical manner”.

6. As the Scottish referendum is a form of social or opinion research, I believe that it makes sense to judge its methodology against the professional standards of the research industry. I have therefore focused the following paper around relevant elements of the MRS Code of Conduct (specifically section B14 relating to questionnaire design) and provided commentary on the application of these when framing the final referendum question(s).

Key Points

B.14: Members must take reasonable steps to ensure that the design and content of the data collection process or instrument is appropriate for the audience being researched

7. By its very nature, it is intended that the Scottish referendum question will be answered by all adults in Scotland. This means that the audience being researched is very wide and diverse, and comprises people with varying degrees of literacy and people whose first language is not English. It is thus vital that the question is worded in such a way that it can be read and understood by the full spectrum of voters in Scotland. Alex Salmond was reported to say that the early 2012 question was “short, straightforward and clear” and I concur that ensuring that these parameters are applied to the final question(s) should be a priority.

B.14: Members must take reasonable steps to ensure that Respondents are not led towards a particular point of view

8. Questions can be written in such a way that they lead respondents towards particular responses, however doing so is inethical and therefore would be inappropriate for a Governmental exercise which is subject to public scrutiny.

9. The early 2012 question asks us whether we agree that Scotland should be an independent country. This is an unbalanced question, as it leads the voter towards the agree response because it assumes a particular value position. It is essentially asking Do you agree (with us) that Scotland should be an independent country? A more balanced question would be Do you agree or disagree that Scotland should be an independent country? as this question gives equal weight to either potential response and does not assume a value position.

10. However, although marginal, it is also my opinion that the inclusion of the word should also potentially leads respondents towards a positive response as it is not counterbalanced by should not. Furthermore, the use of should may even be considered inappropriate emotive language in this context, as in combination with the assumed value position its inclusion implies that independence is a patriotic duty. This wording may therefore provoke feelings of nationalism, which could in turn inflate the agree response.

11. Linked to this, as well as considering whether the question wording is biased in favour of nationalism, it is also vital to ensure that the ballot paper is free from patriotic imagery that could influence the voter—and that includes the Scottish Government logo because it is a Saltire. It is acknowledged that it will be impossible to remove influences in in campaigning and at the polling station, but influencing elements should be removed from the process where possible.

12. The final question(s) must therefore be balanced and unbiased, and presented in a format that has no intrinsic value position.

B.14: Members must take reasonable steps to ensure that Respondents are able to provide information in a way that reflects the view they want to express

13. It is implied in the process of the referendum that a yes vote is a vote for independence, and a no vote is a vote against independence. Voters will cast their votes based on this knowledge. It is therefore essential that the question wording reflects this.

14. In the early 2012 question voters are asked whether Scotland should be an independent country. Voters are not asked to judge when independence should happen, or whether it will work. Instead they are asked to say if, in an ideal world, Scotland ought to be independent. This is not the same as asking voters whether they would support the SNP immediately pursuing independence. Consequently, voters who do agree that Scotland should be an independent country but don’t believe that independence will work in practice will be forced to lie with a no vote. This is inethical, because it means that the process prohibits some voters from adequately expressing their views.

B.14: Members must take reasonable steps to ensure that responses are capable of being interpreted in an unambiguous way

15. Linked to this, as far as is possible it is important to ensure that the results to the referendum cannot be subject to multiple interpretations. The results from asking whether Scotland should be an independent country give us a view of what the nation would prefer in an ideal world, which has limited use as it would be open to criticism around the practicality of the result. A yes result following this question does not technically answer the implied question would you like us to go ahead and do it? Even so, it is likely that a yes result would be used to proceed with independence negotiations. I would not consider this to be a fair interpretation of the result. The question wording therefore needs to link directly and logically with the intended use of the data.

16. Both of the above points could be addressed by wording the question in such a way that it reflects the intended use of the referendum results, and/or by introducing a second question or more response options to ensure that all potential viewpoints are covered.

Recommendations

17. In conclusion, it is my professional opinion that the question proposed in early 2012 should be improved by taking on board lessons from the good practice guidelines provided in the MRS Code of Conduct.

18. I propose that the final question(s) selected for the Scottish referendum should be:

Short, straightforward and clear so as to be accessible to the full spectrum of Scottish adults.

Balanced so as not to lead voters towards a particular response.

Presented in a manner that does not introduce bias.

Reflective of the intended use and outcomes of the referendum, so that voters can give honest responses and the results are conclusive.

19. I propose that the final question(s) selected for the referendum should be checked against the MRS Code of Conduct, and endorsed by a representative of the industry to ensure that high ethical and professional standards are upheld.

20. In combination, this will ensure that the results to the referendum are conclusive and hold up to scrutiny, thus providing a useful foundation upon which to make a genuine evidence-based decision about Scottish independence.

April 2012