APPENDIX 8
Memorandum from Professor Nigel Harvey,
University College, London
CONTENTS
1. Witness background
2. Executive summary
3. Estimating risk: Why there is a need
to consult multiple sources of advice
4. Recommendations for advice format
5. Recommendations for number of advisors
6. Recommendations for procedures to integrate
advice from multiple sources
7. Recommendations for use of judgment when
integrating advice
8. References
1. WITNESS
BACKGROUND
1.1 Nigel Harvey is Professor of Judgment
and Decision Research at University College London. His research
into judgment and decision-making has been funded by the Economic
and Social Research Council for the past 15 years. His particular
areas of expertise include the role of judgment in forecasting
and advice-taking. His research is empirical and based on experiments,
simulations, and questionnaire studies. He is co-editor of the
primary reference text of human judgment and decision-making (The
Blackwell Handbook of Judgment and Decision Making, 2004).
He is former President of the European Association for Decision
Making. His current research is funded by ESRC Project Grand R000230114
"Trust in advisors: Origins, effects, and implications
for risk communication", by Leverhulme Trust/ESRC Programme
Grant "Towards a Science of Evidence", and by
the ESRC Centre for Economic Learning and Evolution.
2. EXECUTIVE
SUMMARY
2.1 Advice about risk levels should be obtained
from a number of sources. If advisors are independent of one another,
three to five estimates should suffice. If they are not independent,
more advisors should be consulted.
2.2 Advice about risk levels should be provided
in numerical form. For communicating risk levels to the public,
frequency forecasts are useful.
2.3 Different estimates for the level of
some risk should be combined using some formal procedure. When
no information about the relative quality of advice from different
sources is available, the median or mean value can be taken. (There
are arguments to support the view that the median is often preferable.)
When information about the quality of advice from different sources
is available, advice can be integrated by using a weighted average.
2.4 To be realistic, it is important to
recognise that decision-makers often integrate advice by using
their own judgment rather than a formal procedure. This introduces
error into the final risk estimate. Decision-makers using their
judgment to integrate advice from different sources should be
cautious about using advisors' experience in the domain or the
cost of their advice as proxies for their expertise. If they have
sufficient advice from sources that they have consulted, they
should not include their own estimate or opinion among those they
integrate. They should also take care to ensure that characteristics
of their advisors (eg whether they are internal or external) that
are irrelevant to the quality of the advice produced do not influence
the weight that they place on advice received from different sources.
3. ESTIMATING
RISK: WHY
THERE IS
A NEED
TO CONSULT
MULTIPLE SOURCES
OF ADVICE
3.1 There is a body of scientific research
into factors that determine the influence of advice on decision-makers.
Much of it is specifically concerned with advice about risk. Results
of this research may be of interest to the inquiry for two reasons.
Primarily, they are relevant to government use of advice. However,
some are also relevant to the influence that government advice
has on decisions and behaviour of members of the public.
3.2 A distinction is often made between
risk, where the probability of an undesirable event is known,
and uncertainty, where that probability is unknown. For example,
the probability of death when driving a car may be regarded as
known, whereas the probability of death from eating British beef
in the 1990s may be regarded as unknown. According to this view,
the former situation concerns risk and the latter uncertainty.
3.3 In practice, as Slovic1 has pointed
out, there are disputes about levels of risk associated with hazardous
activities even when data about the frequencies of death or injury
associated with those activities are available. This is because
of disagreements about how outcomes should be categorized (eg
different views about what constitutes serious injury or about
how long after an event death can be associated with it), because
of sampling variations (eg samples vary in size and in when and
where they are obtained), and because of doubts about the relevance
of certain samples to the risk assessment at hand (eg concerns
about whether risk estimates obtained from adults can be generalized
to cover children). Thus there is always room for differences
in opinion about the level of risk associated with any activity.
3.4 To deal with these differences in opinion,
people making decisions and formulating policies need to obtain
risk estimates from a number of different advisors. The different
risk estimates that they receive must then be integrated in some
way.
3.5 The reason that combining advice from
different sources improves accuracy of the final risk estimate
is as follows. Each estimate from an advisor can be regarded as
comprising the true value of the risk, a bias, and some random
error. Combining estimates from advisors improves accuracy by
producing some cancellation in the random error and, if the direction
of the bias varies across advisors, some cancellation in the bias.
4. RECOMMENDATIONS
FOR ADVICE
FORMAT
4.1 Advice about risk levels should be provided
in a numerical form. There are two reasons for this. First, it
is easier to combine numerical estimates than verbal ones. Second,
different people use the same verbal labels (eg quite high, fairly
low) to refer to different numerical levels of risk. 2 For example,
whereas one person may use the term "quite high" in
a consistent manner to refer to a 1% to 2% chance of an undesirable
outcome, someone else may use the term "fairly low"
in a consistent manner to refer to the same level of risk.
4.2 Numerical estimates of risk can be expressed
in various ways. Risks expressed as frequencies (eg 10 out of
1,000 cases) appear to be easier to appreciate than those expressed
as probabilities (eg a probability of 0.01). 3 This may be particularly
important when communicating risks to the general public.
5. RECOMMENDATIONS
FOR NUMBER
OF ADVISORS
5.1 Simulation studies have shown that,
when advisors are independent, most of the improvement in accuracy
that arises from taking advice from multiple sources can be obtained
with relatively few advisors. Integrating advice from just three
to five independent sources greatly improves accuracy of the final
estimate. 4
5.2 When advisors are not independent (ie
they are influenced by each other or by some common source), more
advisors are required to produce the same gain in accuracy in
the final estimate. 4 For example, a decision-maker who uses three
to five advisors when they are independent may need to use five
to seven advisors when they are not, in order to obtain the same
benefit from using multiple sources.
6. RECOMMENDATIONS
FOR PROCEDURES
TO INTEGRATE
ADVICE FROM
MULTIPLE SOURCES
6.1 Estimates obtained from different advisors
should be integrated by using some formal ("mechanical")
procedure. Possibilities include taking the average, the median,
and the mid-range. Many studies have shown that formal procedures
outperform judgment in most situations in which information from
different sources has to be integrated. 5, 6, 7 This is because,
unlike judgment, such procedures do not add random error during
the process of integration.
6.2. What formal procedure should be used
to integrate advice from different sources when decision-makers
have no records of the past quality of the advice from those sources?
The answer to this appears to depend on the format of the advisory
estimates. Probabilities are estimates that are bounded at both
ends of the scale (0 and 1). Likelihoods expressed as percentages
are also bounded in this way (0% and 100%). However, odds ratios
are not bounded. With bounded estimates, the simple average will
generally provide a reasonably good means of integration. With
unbounded estimates, the median or a trimmed average (excluding
one or two of the most extreme estimates) is to be preferred.
8, 9 Technically, the reason for this is that distributions of
human responses have thick tails10 and so even a small sample
of such responses is relatively likely to include a very extreme
value that could distort the average.
6.3. If decision-makers do have records
of the quality of past advice that they have received from different
sources, they should first check to see whether some advisors
are noticeably better or worse than others are. If none are, they
can proceed as before (para 6.2). However, if accuracy of different
advisors varies considerably, their advice should be weighted
according to their past accuracy. For example, a weighted average
could be used to integrate it.
7. RECOMMENDATIONS
FOR USE
OF JUDGMENT
WHEN INTEGRATING
ADVICE
7.1 Although formal procedures provide the
best means of integrating various estimates of risk received from
different advisors, we have to accept that many decision-makers
prefer to use their judgment to resolve differences between advisors.
11
7.2 Consider first the situation in which
a decision-maker has not formed his or her own opinion about the
level of risk before receiving advice. Studies have shown that
without information about past accuracy of advisors, such a person
will base their judgment on the median of the advisors' estimates.
12 However, their judgment will include some random error and,
hence, will not be as accurate as if the median were calculated
formally. With information about the past accuracy of advisors,
the decision-maker's final judgment will weight the better advisors
more heavily. 12, 13 However, again, their judgment will not be
as accurate as a weighted average calculated formally would be.
In this case, there are two reasons for reduced accuracy. First,
some random error will again be included in their judgment. Second,
the difference in how heavily good and poor advisors are weighted
when judgment is used is not as large as it should be. 13, 14
So decision-makers using their judgment to integrate advice should
be encouraged to place more emphasis on their better advisors.
One way of helping them to do this is to provide them with a continuously
updated record of the advice that they have received from different
sources. 15
7.3. In many cases, information about the
past accuracy of advisors is unavailable. In other words, there
is no explicit information about advisors' expertise in the domain
of interest. In these circumstances, decision-makers may consider
their advisors' experience in a domain as a proxy for their expertise
in it. Hence, they place more trust in their more experienced
advisors (ie they weight advice from more experienced sources
more heavily when coming to their final judgment). 14 However,
as with expertise, they fail to differentiate sufficiently between
their advisors. They are influenced too little by highly experienced
advisors and too much by less experienced ones.
7.4. It is worth emphasizing that expertise
does not increase with experience in all domains. (Researchers
have had some success in identifying the characteristics of domains
in which the relation fails to hold. 16, 17) Hence, relying more
on more experienced advisors is not always a good strategy.
7.5. Advisors are also influenced more by
advice that they have paid for. 18 Amount paid for advice may
act as a proxy for expertise in the same way that advisor experience
often does. Again, however, decision-makers should exercise some
caution. Useless advice often costs a lot. 19
7.6. Consider now the situation in which
a decision-maker has formed an opinion of their own about the
level of risk before receiving advice. Should this decision-maker
include their own opinion as well as those of their advisors in
the set of opinions that they integrate into a final risk estimate?
Many studies14, 20, 21, 22 have shown that doing so results in
decision-makers being too influenced by their own views and insufficiently
by those of their advisors. On the other hand, when very few opinions
are being integrated, the addition of an extra one can be expected
to be particularly beneficial. 4 Thus, it is reasonable to suggest
that decision-makers should not include their own opinion among
those that they are integrating when advice from a fair number
of sources is available. If they do include their own opinion,
they should make every effort to ensure that it does not receive
greater weight than those of their advisors.
7.7. There is evidence that decision-makers
are influenced by characteristics of their advisors that should
not influence them. For example, people say that they trust advisors
more when those advisors share their (moral) values. 23, 24 Decision-makers
should do all they can to ensure that they do not place greater
weight on advice from sources who are more similar to them unless
they have evidence that advice from those sources has been better
in the past than that from other sources. (This recommendation
may be relevant to how advice from internal and from external
sources influences decision makers.)
January 2006
8. REFERENCES
1. Slovic, P (2001).
The perception of risk. London: Earthscan.
2. Wallsten, TS, Budescu,
DV and Zwick, R (1993). Comparing the calibration and coherence
of numerical and verbal probability judgments. Management Science,
39, 176-190.
3. Gigerenzer, G (2002).
Reckoning with risk: Learning to live with uncertainty. London:
Allen Lane, The Penguin Press.
4. Johnson, TR, Budescu,
DV and Wallsten, TS (2001). Averaging probability judgments: Monte
Carlo analyses of asymptotic diagnostic value. Journal of Behavioral
Decision Making, 14, 123-140.
5. Einhorn, HJ (1972).
Expert measurement and mechanical combination. Organizational
Behavior and Human Performance, 7, 86-106.
6. Sawyer, J (1986).
Measurement and prediction, clinical and statistical. Psychological
Bulletin, 66, 178-200.
7. Dawes, RM (1979).
The robust beauty of improper linear models in decision making.
American Psychologist, 34, 571-582.
8. Wilcox, RR (1992).
Why can methods for comparing means have relatively low power,
and what can you do to correct the problem? Current Directions
in Psychological Science, 1, 101-105.
9. Streiner, DL (2000).
Do you see what I mean? Indices of central tendency. Canadian
Journal of Psychiatry, 45, 833-836.
10. Micceri, T (1989). The
unicorn, the normal curve, and other improbable creatures. Psychological
Bulletin, 105, 156-166.
11. Kleinmutz, B (1990).
Why we still use our heads instead of formulas: Toward an integrative
approach. Psychological Bulletin, 107, 296-310.
12. Harries, C, Yaniv, I
and Harvey, N (2004). Combining advice: The weight of a dissenting
opinion in the consensus. Journal of Behavioral Decision Making,
17, 333-348.
13. Harvey, N, Harries, C,
and Fischer I (2000). Using advice and assessing its quality.
Organizational Behavior and Human Decision Processes, 81,
252-273.
14. Harvey, N and Fischer,
I (1997). Taking advice: Accepting help, improving judgment and
sharing responsibility. Organizational Behavior and Human Decision
Processes, 70, 117-133.
15. Fischer, I and Harvey,
N (1999). Combining forecasts: What information do judges need
to outperform the simple average? International Journal of
Forecasting, 15, 227-246.
16. Bolger, F and Wright,
G (1992). Reliability and validity in expert judgment. In G Wright
and F Bolger (Eds), Expertise and Decision Support, London:
Plenum Press, pp 47-76.
17. Shanteau, J (1992). Competence
in experts: The role of task characteristics. Organizational
Behavior and Human Decision Processes, 53, 252-266.
18. Sniezek, JA, Schrah,
GE and Dalal, RS (2004). Improving judgment with prepaid expert
advice. Journal of Behavioral Decision Making, 17, 173-190.
19. Sherden, WA (1998). The
fortune sellers: The big business of buying and selling predictions.
Chichester: Wiley.
20. Lim, JS and O'Connor,
M (1995). Judgmental adjustment of initial forecasts: Its effectiveness
and biases. Journal of Behavioral Decision Making, 8, 149-168.
21. Yaniv, I and Kleinberger,
E (2000). Advice taking in decision making: Egocentric discounting
and reputation formation. Organizational Behavior and Human
Decision Processes, 83, 260-281.
22. Yaniv, I (2004). Receiving
other people's advice: Influence and benefit. Organizational
Behavior and Human Decision Processes, 93, 1-13.
23. Earle, TL and Cvetkovich,
G (1999). Social trust and culture in risk management. In G Cvetkovich
and RE Lfstedt (Eds), Social Trust and the Management of Risk.
London: Earthscan, pp 9-21.
24. Twyman, M, Harries, C
and Harvey, N (2006). Learning to use and assess advice about
risk. Forum: Qualitative Social Research, 7, Article 26
(available electronically, http://www.qualitative_research.net/fqs/fqs-eng.htm).
| 
