Environmental Audit CommitteeWritten evidence submitted by Dr Jan Eichhorn

Dr Eichhorn is a research fellow in the School of Social and Political Science at the University of Edinburgh. His work focuses on subjective well-being and in particular on how to properly contextualise individual-level influences on subjective well-being using regional-or country-level factors that affect the experiences of individuals. His doctoral thesis explored how economic, demographic and cultural differences between countries affect the experience of unemployment. In his work Dr Eichhorn uses advanced quantitative methods to analyse large scale social surveys. Results of his work have been published in high ranking journals including European Sociological Review and the Journal of Happiness Studies. Dr Eichhorn contributed to the ONS consultation on “Measuring National Well-being” through written submissions, engagement in the British Sociological Association’s Happiness Study Group and public outlets (Eichhorn 2011), eliciting a response from Paul Allin, then in charge of the ONS programme.

Dr Eichhorn currently holds two collaborative research grants under the ESRC programme “The Future of Scotland the UK” acting as co-investigator on both.

0. Executive Summary

This report presents evidence to highlight current weaknesses in the ONS “Measuring National Well-being” project in relation to Indicators of Subjective Well-being (SWB), such as happiness or life-satisfaction. It points out how these weaknesses could be overcome in order to provide better tools for policy development. The key points of the report are:

The way SWB measures are constructed from the data for reporting is partially insufficient, as common features of SWB scales are not taken into account. This leads to inappropriate interpretations about levels of subjective well-being in the UK.

The reports so far are not sufficiently able to reflect people at “extreme” points (for example very old age), although we know from other studies that their SWB patterns differ. Addressing this has important policy implications (for example for hospital care for the elderly).

The comparisons of SWB levels across UK regions have to be considered with caution at the moment. So far it has not been explored whether the meaning of different SWB measures is consistent across all of the UK. If it is not, then comparisons of levels would be a misrepresentation.

So far many influences on SWB have been found at the individual level as well as for different levels of aggregation (eg communities or regions). However, no proper distinction is being made between those different levels—which may lead to wrong conclusions, if results from one aggregate unit are simply assumed to also apply to other aggregations or even individuals.

Some of the mean levels of SWB measures comparing different UK regions have been misleading as they have entered public discourse stipulating substantial differences, where there largely were none.

Until now there has been a lack of considering how societal context factors influence individual-level SWB patterns. This may lead to the misclassification of so-called “deviant” subgroups, although potentially the difference from the norm may be caused by contextual factors.

1. Introduction

The “Measuring National Well-being” project conducted by the ONS is generally highly commendable. It widened the perspective of well-being indicators beyond standard economic measures. Its greatest contribution in the opinion of the author is that it actively sought to incorporate subjective measures of well-being. Such measures allow us to see how changes in certain objective circumstances (for example: income, employment, family relations, etc.) may actually have different effects on an individual’s perception and motivation depending on other factors. However, there are several shortcomings in the use of subjective well-being (SWB) concepts so far in the project. If proper policy is to be developed based on the proposed measures, it is imperative that these shortcomings are addressed.

2. Focusing on Levels of SWB

Many of the analyses presented in ONS papers, including the summary publication from November 2012 (Self et al.) show levels of SWB on different indicators (for example life-satisfaction). These levels are then compared across different groups (for example regions or age groups). This of course is of great interest and provides an overview of which groups of people have higher levels of SWB compared to others. However, there are a number of problems with these analyses potentially meaning that wrong conclusions are drawn.

3. Choosing Measures

People in the UK (and most other Western countries) tend to be overt to choosing SWB measures at the lower extremes. Therefore statements in the reports suggesting that low levels of “completely dissatisfied” people are a good finding (Self et al. 2012, p. 17), for example when looking at household income satisfaction, have only limited merit. We would not expect to find many people valuing themselves at such low levels, partially because there is a certain social desirability for people to consider positives in their lives when making such evaluations. The interpretation of absolute levels on a particular subcategory (such as “completely dissatisfied”) is therefore of rather little meaning. In this light, it is very problematic to group medium and high rankings of life-satisfaction together (as for example done on p. 32 in the same paper). This is where we really see some differentiation that is masked through such an approach. A person valuing their life-satisfaction as only medium (neither satisfied/nor dissatisfied) has to be understood as rather distinct from a person saying they actually are satisfied.

4. Considering “Extremes”: Drops in SWB in Late Life

In analyses that compare many groups the focus usually is—understandably—on the “larger picture”. The most poignant case is the analysis of SWB over the life course. Self et al. (2012, p. 33) replicate a common finding of a U-shaped relationship: Young and old people have the highest levels of SWB, the middle-aged have the lowest levels. However, they overlook a very crucial aspect (which has been overlooked in nearly all ONS papers so far): for the very old, we actually see a dip in SWB again, which is not commented on. It is difficult to analyse those aged 75 and above, because they only make up a limited proportion of a survey sample. However, we have good evidence to believe that we will often find a substantial drop in SWB in the latest stages of life (Brockmann 2010). One of the key factors leading to a drop appears to be hospitalisation—raising highly relevant questions about the role of care and treatment in old age and what that means for individuals. A strong focus on well-being in old age seems highly imperative as a further field of detailed investigation.

5. Regional Variation: Real Differences?

One of the aims of the ONS project stipulated early was to be able to compare levels of SWB across regions in the UK (Waldron 2010) effectively resulting in rankings. Since then a range of mean comparisons across regions has been presented, however one elementary question about this has been largely ignored so far (see for example Tikler & Hicks 2011): Do we measure the same thing when we measure a particular domain of SWB (for example life-satisfaction) in all regions of the UK. In other words, is the understanding of what satisfaction or happiness is largely the same across the UK or are there regional variations in the constitution of the concepts themselves. If the concepts can be seen to be perceived homogeneously on average in each region of comparison, there are no problems in comparing average scores. However, if there is variation in the composition of the meaning of SWB domains, then a comparison would be highly problematic: We would compare values for indicators that measure different things in different regions. There are techniques to address this issue. The key is to check whether the same elements contribute to a particular SWB measure in the same way across regions, before comparing the levels thereof. This can be done through a well-established technique: Multi-group Confirmatory Factor Analysis (MGCFA). While it goes beyond the basic statistical tools employed in the reports so far, it is absolutely imperative to do so, because if we simply assume that the measures are equivalent conceptually we may present rankings of regions that are not actually reflective of differences in the level of a particular SWB domain. Instead they would highlight differences in the meaning of that SWB domain for people—something that should not be ranked.1

6. Distinguishing levels of aggregation: Communities ≠ Regions ≠ Countries

The ONS papers engage with analyses that correlate SWB measures to many other factors at a range of different levels of aggregation. There are analyses that look at relations to SWB on the individual level, but also reports that look at the relationship of SWB to other variables at the level of communities, regions or larger areas. There is a key problem however, because the relationships identified are not properly differentiated based on the level of aggregation we look at (see Beaumont 2011 or Evans 2011 for example). However, this is of great importance: If we find a particular relationship to exist for individuals (for example a “U-shaped” relationship between age and SWB), this does not mean that we have to find the same relationship at a greater level of aggregation (for example, we may not necessarily find a “U-shaped” relationship between average age in cities and the average SWB in the cities). The relationship between two variables can differ at different levels of aggregation—sometimes a relationship that may exist at one level cannot be identified at all at another. Therefore, further reports should be much more careful in considering what level of aggregation they are focussing on—and not simply assume that results found, for example, at the local level will also apply to the national level. Instead this needs to be investigated separately.

7. Ensuring that Differences are Substantial and Significant

Even when deciding on a particular level of aggregation, further analyses can lead to wrong conclusions when comparing levels of SWB across regions but not accounting for problems of substantive and statistical significance. Several mean comparisons of SWB measures (for example life-satisfaction) have been published comparing different sets of regions. While a very detailed breakdown published in 2012 allowed to see differences between certain areas which were substantial and statistically significant (ONS 2012a), this was not the case in all publications. In a different report (ONS 2012b, p. 18) for example the UK is broken down into 12 areas for which mean life-satisfaction scores are presented. The scores range only from 7.2 to 7.6 (on a 10-point scale). Even if results were significant in a statistical sense, the substantive difference is minimal, in particular when considering that 7 of the 12 areas score 7.4 on average. Comparing those regions based on this measure at this level of aggregation has little meaning—apart from finding that at this level there is hardly any variation. Some may argue that there is no problem with publishing these results as the report does not actively encourage a ranking based on this measure, but merely presents a neutral depiction. That however would be naïve considering the public role of the ONS. And indeed several institutes and media outlets picked up this particular table comparing the regions stating that Londoners would be the least satisfied in the country (for example nef 2012), a claim that no serious researcher would make with this data as support, but that permeated the public discourse nevertheless.2 Comparisons and rankings between regions need to be assessed not only with regards to the validity of the measures as outlined in paragraph 6. It also has to be ensured that the differences in levels between regions are substantial and statistically significant (meaning that they do not fall within a certain frame of uncertainty based on sampling variation).

8. Context Matters: Using Multilevel Approaches

It is good to see that the ONS aims to engage more extensively with regression based research (Self et al. 2012, p. 7). This would allow to control for several influencing factors as many of the correlations shown between different variables and SWB measures could be altered as there are a great variety of influences on SWB that also interact with each other. However, this is not sufficient for a comprehensive analysis. All investigations focussing on the individual level effectively assume that the influences on SWB are not dependent on contextual factors. While there are some analyses looking at aggregate factors (for example whether people in a particular type area are more or less satisfied), these two perspectives (focussing on individual or contextual levels) are usually kept separate. This is highly problematic as obviously, individuals always exist in a particular economic, social and cultural context. Multilevel modelling techniques exist to easily integrate multiple layers of analysis and have been applied in social scientific research extensively. So far though, none of the ONS investigations have included such approaches. Even more strikingly the reviews of existing projects effectively do not engage with research employing approaches that systematically contextualise individuals’ SWB constitution with societal factors. Considering that policy is always developed as an intervention from an aggregate and then applied to groups of individuals, ignoring this may lead to ineffective or counter-productive policy, as people’s motivations and perception may be oversimplified as independent of their societal context.

9. Identifying Actually Relevant Processes

Often individual-level influences on SWB are moderated by contextual factors. Consider the following example: It is commonly found that religious people are happier (individual level). However, as the author of this evidence, Eichhorn (2012a), has shown, this effect is highly dependent on what type of society one lives in. When taking into account the average religiosity of a country we find that higher religiosity only has a positive effect on happiness for an individual, when the person lives in a more religious society. The positive relationship with SWB is about being in congruence with your environment, not being religious per se. Another good example points to the effect of unemployment on SWB. As replicated in many of the ONS papers, unemployment is indeed associated with a loss in life-satisfaction. However, the strength of this relationship is dependent on contextual factors. Clark & Oswald showed in 1994 that the negative impact is less pronounced when a person lives in an area with higher unemployment (because they would not be as deviant from the norm). This moderation however is strongly dependent on the level of aggregation. Pittau et al. (2010) for example find this context effect for more regional levels, but not at the country level. The level of aggregation matters strongly and only an integrative multilevel perspective could reveal this. Eichhorn (2012b) showed that at the country level, the key factors that influence how strongly unemployment impacts individual SWB include the age structure of the population and cultural factors such as the perception of autonomy people have, but not so much commonly assumed economic factors, such as unemployment rates or GDP per capita.

10. Implications for Policy Making and further Analyses

The importance of considering contextual influences on individual SWB patterns cannot be overstated. As outlined by the ONS (Self et al. 2012) subsequently emphasis is being placed on identifying groups of people that are considered to be different or deviant from the norm with regards to their well-being patterns. But it is crucial to understand why they are different. Two possibilities exist: Indeed, there could be some substantive differences for these groups. However, it is also possible that differences are not based on individual factors, but contextual influences. If this were the case, but ignored, a group may be labelled as “deviant” and subsequently receiving a policy intervention—but inappropriately so. If the reason for a different SWB pattern stems from the interplay between contextual factors and personal factors, looking at either of them separately will not help us to genuinely understand what the point of intervention actually should be (groups of individuals or a particular contextual influence at a specific level of aggregation that needs to be determined).

11. Conclusion

The ONS project is commendable in many ways, highlighting the need to move beyond traditional, simplistic indicators of well-being. With regards to the work on subjective well-being (SWB), there are several limitations in the work so far.

12. Limitations

Deficiencies in the construction of appropriate measures and interpretations on SWB data, in particular not taking into account an overtness of people to give very low scores and a lack of distinguishing between very high, high and moderate levels of SWB.

A lack of properly accounting for deviance at the extremes of comparative analyses (for example very old age)

Completely missing an investigation into the question of whether conceptions of SWB are constituted in the same way across the UK to confirm the validity of measures.

An underdeveloped degree of differentiating between findings from different levels of aggregation which cannot simply be assumed to be equivalent.

Reporting of SWB measures comparing areas which are not substantially or significantly different from each other, but lend themselves to easy public dissemination.

A lack of engaging with multilevel perspectives, thus making individual-level patterns appear as definitive—lending itself to a misidentification of intervention points.

13. Recommendations

Further reports and research should engage more carefully with SWB scales, identifying where substantial variation lies and differentiating in detail there.

Ways for engagement with individuals at “extremes” need to be developed—for example those in very old age, through booster samples in surveys or separate studies.

There is a strong need for investigations into the constitution of SWB concepts across the UK to ensure that mean comparisons are actually valid. This could be done using multi-group confirmatory factor analysis based approaches.

Findings about correlations between particular factors and SWB measures should be sorted by levels of aggregation (eg individual, local, regional, national). Where gaps exist (ie a relationship has been explored for one level but not another), these should be filled.

It should be ensured that all area or group based mean comparisons establish that the differences found are substantial and statistically significant before reporting them.

Multilevel perspectives and models need to be incorporated into these investigations with the particular aim to distinguish whether subgroups that deviate from the norm do so because of an actual difference or a contextual influence.

14. References

Beaumont J (2011). Measuring National Well-being: A discussion paper on domains and measures. Office for National Statistics working paper.

Brockmann H (2010). Why are middle-aged people so depressed? Evidence from West-Germany. Social Indicator Research 97(1): 23–42.

Eichhorn J (2011). Question marks over the Government’s happiness agenda. Straight Statistics. Available at http://www.straightstatistics.org/article/question-marks-over-government’s-happiness-agenda

Eichhorn J (2012a). Happiness for Believers? Contextualising the Effects of Religiosity on Life-Satisfaction. European Sociological Review 28(5): 583–593.

Eichhorn J (2012b). Unemployment needs context: How societal differences between countries moderate the loss in life-satisfaction for the unemployed. Journal of Happiness Studies doi: 10.1007/s10902–012–9402-y.

Evans J (2011). Findings from the National Well-being Debate. Office for National Statistics working paper.

ONS (2012a). UK experimental subjective well-being estimates. Office for National Statistics. Available at http://www.neighbourhood.statistics.gov.uk/HTMLDocs/dvc34/Well-being_map.html (accessed 12 April 2013).

ONS (2012b). Analysis of Experimental Subjective Well-being Data from the Annual Population Survey, April to September 2011. Office for National Statistics. Available at http://www.ons.gov.uk/ons/dcp171776_257882.pdf (accessed 12 April 2013) .

Pittau M, Zelli R, Gelman A (2010). Economic Disparities and Life Satisfaction in European Regions. Social Indicator Research 96(2): 339–361.

Self A, Thomas J, Randall C (2012). Measuring National Well-being: Life in the UK, 2012. Office for National Statistics working paper.

Tinkler, L, Hicks, S (2011). Measuring Subjective Well-being. Office for National Statistics working paper.

Waldron S (2010). Measuring Subjective Well-being in the UK. Office for National Statistics working paper.

10 May 2013