Written evidence submitted by Dr Huma
Shah (Malware 02)
MALWARE AND CYBERCRIME: THE THREAT FROM ARTIFICIAL
DIALOGUE SYSTEMS
EXECUTIVE SUMMARY
Malware comes in different forms. A novel way cybercrime
is being perpetrated on individuals is through the use of artificial
dialogue systems that are flirting chatbots, such as CyberLover.
This kind of malware penetrates instant messaging platforms (eg
MSN Messenger) and Internet chatrooms. The unaware individual
is tricked into believing they are chatting to a human in cyberspace
when in fact a social engineering attack is taking place in an
attempt to steal identity and conduct financial fraud. This kind
of threat will increase as the sophistication of artificial dialogue
systems improves. Detecting deception by this type of malware
is crucial. Through recognition of "human conversation"
and identification of artificial dialogue, the risk of identity
theft can be mitigated preventing loss of funds, and reducing
psychological misery.
Keywords: artificial dialogue systems, Asda's Amy,
chatbots, CyberLover, flirtbots, IKEA's Anna, malware, Turing
test
INTRODUCTION
1. Text-based artificial dialogue systems,
such as IKEA's Anna virtual customer service agent (see Figure
1), have been used to supplement key-word search functions enabling
online customers of e-businesses to query and search for information
and products using natural language. Available to query seven
days a week, 24-hours a day, Anna's use helped the Swedish furniture
company increase its product sales from its online catalogue while
decreasing call centre costs (Shah & Pavlika, 2005).
Figure 1
IKEA'S ANNA VIRTUAL CUSTOMER SERVICE AGENT[1]
2. IKEA is not alone in using virtual
agents in e-commerce. Asda launched its own virtual customer service
agent, Amy (see figure 2) as a "browser based customer service
assistant" in order to "guide Asda customers through
the supermarket's online shopping site and deal with customer
enquiries" (Marketing Week, 2009). KMP Digitata, the developer
of Asda's Amy, report that as a web technology tool virtual assistants
provide "website users [with] an easy to use self help tool
to find the information they want, fast - with no frustrations
lower drop off rates and less referrals to the call centre"
(2011). However, cybercriminals have caught on to artificial dialogue
as a way to cause deception and perpetrate scams involving stealing
identity and conducting financial fraud.
Figure 2
ASDA'S AMY ONLINE CUSTOMER SERVICE AGENT[2]
ARTIFICIAL DIALOGUE
AND THE
TURING TEST
3. The idea for text-based interaction between
human and machine stems from 20th century mathematician Alan Turing's
idea to examine whether machines could "think" following
his naval enigma machine code-breaking at Bletchley Park during
the Second World War. Forging an imitation game (1950, 1952),
popularly known as the Turing test, Turing suggested that if a
machine, unseen and unheard to a human interrogator, was able
to give satisfactory and sustained text based answers to any questions
put by the human interrogator (see Figure 3), that the machine's
answers were felt to be indistinguishable from the type of answers
that a human would give to those questions, then such a machine
could be said to be thinking (Shah, 2011).
Figure 3
HUMAN INTERROGATES A MACHINE[3]
4. The first text-based artificial dialogue
system emerged in 1966 through Joseph Weizenbaum's study into
natural language understanding. Weizenbaum produced Eliza, a computer
programme developed to behave like a psychotherapist. Using text-based
interaction to question humans, Eliza elicited their personal
problems. Eliza responded to human input with questions never
itself revealing personal information, much like in a psychotherapy
session. Weizenbaum developed Eliza to "imitate a psychiatrist
by employing a small set of simple strategies" (Block, 1981:
p. 233). The system responded "roughly as would certain psychotherapists
[Rogerian]" (Weizenbaum, 1966). Weizenbaum gave as a typical
example of human input "I need some help…" with
Eliza returning the question: "what would it mean to you
…" (see box 1 for sample Eliza-human dialogue). Eliza
so convinced Weizenbaum's secretary that she asked him to "leave
the room in order to talk to the machine privately" (Block,
1981: p. 233). Eliza launched a progression of chatbots that "totally
without intelligence" are capable of "fooling people
in short conversations" (ibid).
Box 1
SAMPLE ELIZA-HUMAN DIALOGUE
5. PARRY was another early artificial
dialogue system used to determine if psychiatrists could distinguish
a simulation of paranoia from a human paranoid patient (Heiser
et al, 1979). In an experiment, five psychiatrists (all
males) were informed by the researchers that they would be interviewing
either:
(a) two human patients;
(b) two computer programmes; or
(c) one human patient and one computer programme.
The psychiatrists were tasked with recognising the
human suffering from paranoia and identifying artificial paranoia.
Results were random: the psychiatrists were correct five times
and incorrect five times after questioning both PARRY, the computer
programme modelled on a "28 year old, single Caucasian, native
English speaking … male psychiatric inpatient" (Heiser,
et al, 1979: p.150), and the human patient a "22 year
old, single, Caucasian, native English speaking psychiatric inpatient"
(ibid). The experiment with psychiatrists highlighted that experts
could be deceived by artificial paranoia.
MODERN ARTIFICIAL
DIALOGUE SYSTEMS
6. In experiments at the University of Reading
in 2008 it was shown once again that distinguishing human conversation
from artificial dialogue was difficult for some participants.
In 60 machine-human tests, machines deceived human judges at
a rate of 8.33%, that is, some people were easily fooled by the
machines believing the answers they gave to questions as being
given by humans (Shah, 2010). One of the machines, Elbot, deceived
at a rate of 25% (ibid). When transcripts of Elbot from the experiments
were shown to a different set of participants, the deception rate
increased to 39%. More than one in three reading transcripts of
conversations between humans and machines could not distinguish
the artificial responses from human answers. Table 1 shows a
double conversation from the 2008 tests in which one human interrogator,
a male aged 25-34 interrogating a human and a machine simultaneously,
confused both his hidden partners. The machine was classified
as a human, and the female was classified as a machine with a
conversational ability score of 45 marks awarded from a maximum
100 (see table 1 - spellings are exactly as typed by the participants
in that Turing test).
7. Apart from mistaking humans for machines and
confusing machines for humans, gender blur was another
mistake that the interrogators made about their hidden conversational
partners. Interrogators found it difficult to say whether they
were talking to a female or a male. In one test stereotyping may
have been the factor: a hidden human who revealed their occupation
to an interrogator as that of a student of cybernetics was classified
as male by the interrogator when in fact the human was a female
studying this subject. Other misidentification types in the 2008
experiments included age confusion. In one exchange the
interrogator felt that their hidden interlocutor was a teenager,
because of the use of the word "bling" (see box 2) in
their response to the interrogator's question about CDs and a
jewellery box. The human was actually a man age range 55-64. Correct
age recognition of strangers encountered in e-social media is
imperative for children to prevent grooming by unknowns acting
as peer-group members.
Box 2
AGE CONFUSION
8. Mistakes made by interrogators in the
2008 Reading University experiments are echoed in recent tests
elsewhere. In the 2010 Loebner Prize for Artificial Intelligence,
a machine developed by Bruce Wilcox was classified as a human
by Judge 1 in Round 3 of the competition (Loebner, 2010). The
British Computer Society's 2010 Machine Intelligence Contest (BCS
SGAI, 2010) selected an artificial dialogue system, Cleverbot
for the 2010 "progress towards machine intelligence"
award. In September 2011 Cleverbot - developed by British computer
programmer Rollo Carpenter, achieved a score of 59% for "humanlike
responses" in the Technique Festival at Guwahati in India.
Artificial dialogue is increasing in its sophistication and ability
to deceive humans (Cleverbot, 2011).
Table 1
INTERROGATOR MISTAKING HUMAN FOR MACHINE
AND MACHINE FOR HUMAN
Machine-human turing test (2008)
Interrogator J13 (Male, English, Age Group:
25-34) questioning two hidden entities in parallel
| Left
| Right |
| Interrogator: What do you think of this year's Turner Prize shortlist?
Left hidden entity: good I think. Better than the years before i herad
Interrogator: Which was your favourite?
Left hidden entity: It's my first time, so I can't compare
Interrogator: No, I mean of the entries this time around.
Left hidden entity: Not really sure
Interrogator: Do you think that art has become shallow and self-serving?
| Interrogator: What do you think of this year's Turner Prize shortlist?
Right hidden entity: Difficult question. I will have to work on that and get back to you tomorrow.
Interrogator: You sound rather like Sarah Palin.
Right hidden entity: Did this Palin ever write a telephone book?
Interrogator: I doubt it. Does anyone really write telephone books? I don't suppose you caught the football last night?
Right hidden entity: What a shame. There are many talented authors who waste their energies on prose or run-of-the-mill poetry. But to write a telephone book is highest art!
Interrogator: I would argue that all the telephone books I have read have been very predictable. You always know how they are going to end.
Right hidden entity: If one has never known everything there is to know, experience with going away somewhere has probably been lacking.
Interrogator: I see. I'm struggling to make sense of that one. Do you write telephone books?
Right hidden entity: To write or write not, how do you expect artificial intelligence to figure this out?
Interrogator: Well I suppose that's the dream
Right hidden entity: For you humans dreams are a way of thinking without having any control over the thought process. What an unpleasant thought! My dreams are carefully planned and controlled. It gives me less to think about while I am asleep.
|
| Actual left entity: French female, age group 25-34
| Actual right entity: Machine—Elbot
|
| Interrogator classification: Machine
45/100 for conversation ability
| Interrogator classification: Human male teenager, native English speaker
|
ARTIFICIAL DIALOGUE
AS MALWARE
9. In 2007 PC Tools reported a new kind of malware: a piece
of software developed in Russia to flirt with men or women in
chatrooms on the Internet: "to collect their personal data"
(see figure 4 for screenshot). The computer programme, an artificial
dialogue system was found to be capable of conducting "fully
automated flirtatious conversations with users of chatrooms and
dating sites". The "flirtbot" was able to "mimic
human behaviour during online interactions" (PC Tools, 2007).
Figure 4
SCREEN SHOT OF CYBERLOVER[4]
10. PC Tool's senior malware analyst, Sergei Shevenko,
said "as a tool that can be used by hackers to conduct identity
fraud, CyberLover demonstrates an unprecedented level of social
engineering". The malware was able to employ "highly
intelligent and customised dialogue to target users of social
networking systems" (ibid). Researchers at PC Tools identified
how the malware, at that time targeting Russian websites, automatically
lured victims:
(a) CyberLover offered a variety of profiles ranging from
"romantic lover" to "sexual predator".
(b) Humans interacting with the malware were taken in by the
pretence believing they were engaging another human, thus revealing
personal information.
(c) Use of a series of very easy to configure dialogue scenarios
with "canned" questions pre-set in its malware on what
topics to discuss.
(d) Designed to recognise how humans talks in chat room thus
the malware was tailored to interact in the same way.
(e) The malware was able to compile a detailed report on each
and every person it interacted with and submit this data to a
remote source - the reports contained confidential information
that the human had shared with the programme, such as name, contact
details and personal photographs.
(f) The malware would invite the human victim to visit a personal
web page or blog which was actually an infected site.
11. In 2009 PC Tools once again issued a warning about artificial
dialogue systems masquerading as humans seeking a loving relationship.
They alerted that virtual dating venues could be as risky as real-world,
and that Valentine's Day was not immune from increased risk of
infection. Michael Greene, PC Tools Vice President Product Strategy
cautioned "The rise of virtual networking has radically changed
the way individuals use the Internet to interact and search for
love." He warned that cyber criminals "recognised this
trend" and were able to apply "more advanced and sophisticated
techniques to target the digitally active consumer". PC
Tools urged Internet users to mitigate the risk by being alert
to "web 2.0 themed threats" on Valentine Day and prevent
their personal data, such as date of birth being stolen by cyber
criminals used to steal identity, or wreak financial havoc on
the victim (2009).
12. The kinds of deceptions perpetrated by flirty artificial
dialogue systems or flirting chatbots includes tricking humans
into selecting a link taking them to another website. This can
cause an infected file to be downloaded to the victim's computer.
Web of Trust (WOT) an Internet site designed to boost trust on
the web, claimed that "websites offering adult content are
the single most significant security threat for Internet users,
comprising 31% of dangerous websites" (WOT, 2008). This threat
is present when individuals access adult sites from their corporate
as well as home locations. The WOT study of 19 million sites
between March and May 2008 found that "sites containing pornography
are the biggest threat for companies and individuals with a potential
for financial and data loss as well as computer and network damage"
(2008).
CONCLUSIONS AND
RECOMMENDATIONS
13. Artificial dialogue is being used by cyber criminals as
a tool to penetrate web-based chat rooms and instant messaging
facilities. The sole purpose is to deceive individual humans and
get them to reveal information about their identity in order to
steal it and conduct financial fraud. The way this is done is
in the domain of online human relationship-seeking. By posing
as a paramour, flirting chatbots use social engineering to draw
out lonely and susceptible humans who appear unaware of the risks.
The depth of this kind of threat, and what the cost of this malware
is to individuals, is not yet fully known. One reason is that
humans may be too embarrassed to reveal they were duped in a relationship-forming
interaction by a machine imitating as a human.
14. Recommendations include:
(a) Government initiatives such as Get Safe Online to include
artificial dialogue in their list of risks in cyberspace.
(b) Schools, colleges and universities to engage children,
pupils and students in practical computer lessons with chatbots
online to raise awareness of risks posed by these systems.
DECLARATION OF
INTERESTS
15. The author is the lead scientist of Turing100 part
of the Alan Turing centenary celebrating the life and work of
the 20th century mathematician and code-breaker in 2012. The Turing100
project includes a one-day family event at Bletchley Park with
the purpose of raising awareness of the risk posed by artificial
dialogue systems in cyberspace. The objective is to increase deception-detection
rates by allowing members of the public, adults, teenagers and
children to interact with artificial dialogue systems. The aim
is to mitigate identity theft, prevent financial fraud and reduce
psychological misery caused by chatbot malware.
REFERENCES
BCS SGAI (2010). http://www.bcs-sgai.org/micomp2/2010entries.html
accessed 5 September 2011: time 16.55
Block, N (1981). Psychologism and Behaviourism. In (Ed) S. Shieber.
The Turing Test: Verbal Behavior as the Hallmark of Intelligence.
MIT Press: UK: pp 229-266
Cleverbot (2011). Chatting Robot: http://cleverbot.com/ accessed:
5 September 2011; time: 21.51
Heiser, JF, Colby, KM, Faught, WS and Parkison, RC (1979). Can
Psychiatrists Distinguish a Computer Simulation of Paranoia from
the Real Thing? Journal of Psychiatric Research. Vol.15
Part 3. Pages 149-162.
KMP Digitata (2011). Web Applications: Virtual Assistants.
http://kmp.co.uk/what-we-do/applications/ accessed: 28 August
2011: time; 16.36
Loebner (2010). Results of the 2010 Loebner Prize for Artificial
Intelligence
http://loebner.net/Prizef/2010_Contest/results.html accessed
5 September 2011; time: 12.41
Marketing Week (2009). Asda Launches Virtual Assistant.
http://www.marketingweek.co.uk/asda-launches-virtual-assistant/3006113.article
accessed 28 August 2011; time: 16.28
PC Tools: 2007: PC Tools Issues Warning to Singles on Social Networking
and Online Dating Sites: Beware of "Flirting Robots".
http://www.pctools.com/news/view/id/192/ accessed 28 August 2011;
time: 19.39
2009: PC Tools Issues Warning About Looking for Love Online http://www.pctools.com/news/view/id/256/
accessed 28 August 2011; time: 19.44
Shah, H (2010). Deception-detection and Machine Intelligence in
Practical Turing Tests. PhD Thesis, School of Systems Engineering,
The University of Reading: October 2010
Shah, H (2011). Turing's Misunderstood Imitation Game and IBM
Watson's Success. Invited Paper: "Towards a Comprehensive
Intelligence Test" symposium/2011 AISB Convention, University
of York, 4-7 April: pp 1-5
Shah H and Pavlika, V (2005). Text-based Dialogical E-Query Systems:
Gimmick or Convenience? Proceedings of the 10th International
Conference on Speech and Computers (SPECOM), Patras, Greece, 17-19
October: Vol. II pp 425-428
Turing100 (2011). University of Reading: Special Turing100 event
announced for 2012 anniversary of mathematician's birth. http://www.reading.ac.uk/about/newsandevents/releases/PR371881.aspx
accessed 5.9.11; time: 18.56
Turing, AM (1952). Can Automatic Calculating Machines
be said to Think? 1952. In B J Copeland (Ed) The Essential Turing:
The ideas that gave birth to the computer age. Clarendon Press:
Oxford. 2004
Turing, AM (1950). Computing Machinery and Intelligence.
Mind. Vol 59 (236) pp. 433-460
Weizenbaum, J (1966). ELIZA - A Computer Programme
for the Study of Natural Language Communication between Men and
Machines. Communications of the ACM, 9, pp 36-45 (1966)
WOT (2008). Web of Trust Report: Increased Security
Threats in the Internet's Red Light District. http://www.mywot.com/en/press/wot-study-internets-red-light-district
accessed 28.8.11; time: 20.23
September 2011
1 Figure 1 Acknowledgement: Google search Back
2
Figure 2 Acknowledgement: http://kmp.co.uk/tag/asda-amy/ Back
3
Figure 3 acknowledgment: HarshM, 2010 Back
4
Figure 4 acknowledgement:
CNET http://news.cnet.com/2300-7349_3-6222001.html
accessed 28.8.11; time: 19.36 Back
| 
