Malware and cyber crime - Science and Technology Committee Contents

Written evidence submitted by Dr Huma Shah (Malware 02)



Malware comes in different forms. A novel way cybercrime is being perpetrated on individuals is through the use of artificial dialogue systems that are flirting chatbots, such as CyberLover. This kind of malware penetrates instant messaging platforms (eg MSN Messenger) and Internet chatrooms. The unaware individual is tricked into believing they are chatting to a human in cyberspace when in fact a social engineering attack is taking place in an attempt to steal identity and conduct financial fraud. This kind of threat will increase as the sophistication of artificial dialogue systems improves. Detecting deception by this type of malware is crucial. Through recognition of "human conversation" and identification of artificial dialogue, the risk of identity theft can be mitigated preventing loss of funds, and reducing psychological misery.

Keywords: artificial dialogue systems, Asda's Amy, chatbots, CyberLover, flirtbots, IKEA's Anna, malware, Turing test


1.  Text-based artificial dialogue systems, such as IKEA's Anna virtual customer service agent (see Figure 1), have been used to supplement key-word search functions enabling online customers of e-businesses to query and search for information and products using natural language. Available to query seven days a week, 24-hours a day, Anna's use helped the Swedish furniture company increase its product sales from its online catalogue while decreasing call centre costs (Shah & Pavlika, 2005).

Figure 1


2.  IKEA is not alone in using virtual agents in e-commerce. Asda launched its own virtual customer service agent, Amy (see figure 2) as a "browser based customer service assistant" in order to "guide Asda customers through the supermarket's online shopping site and deal with customer enquiries" (Marketing Week, 2009). KMP Digitata, the developer of Asda's Amy, report that as a web technology tool virtual assistants provide "website users [with] an easy to use self help tool to find the information they want, fast - with no frustrations lower drop off rates and less referrals to the call centre" (2011). However, cybercriminals have caught on to artificial dialogue as a way to cause deception and perpetrate scams involving stealing identity and conducting financial fraud.

Figure 2



3.  The idea for text-based interaction between human and machine stems from 20th century mathematician Alan Turing's idea to examine whether machines could "think" following his naval enigma machine code-breaking at Bletchley Park during the Second World War. Forging an imitation game (1950, 1952), popularly known as the Turing test, Turing suggested that if a machine, unseen and unheard to a human interrogator, was able to give satisfactory and sustained text based answers to any questions put by the human interrogator (see Figure 3), that the machine's answers were felt to be indistinguishable from the type of answers that a human would give to those questions, then such a machine could be said to be thinking (Shah, 2011).

Figure 3


4.  The first text-based artificial dialogue system emerged in 1966 through Joseph Weizenbaum's study into natural language understanding. Weizenbaum produced Eliza, a computer programme developed to behave like a psychotherapist. Using text-based interaction to question humans, Eliza elicited their personal problems. Eliza responded to human input with questions never itself revealing personal information, much like in a psychotherapy session. Weizenbaum developed Eliza to "imitate a psychiatrist by employing a small set of simple strategies" (Block, 1981: p. 233). The system responded "roughly as would certain psychotherapists [Rogerian]" (Weizenbaum, 1966). Weizenbaum gave as a typical example of human input "I need some help…" with Eliza returning the question: "what would it mean to you …" (see box 1 for sample Eliza-human dialogue). Eliza so convinced Weizenbaum's secretary that she asked him to "leave the room in order to talk to the machine privately" (Block, 1981: p. 233). Eliza launched a progression of chatbots that "totally without intelligence" are capable of "fooling people in short conversations" (ibid).

Box 1


5.  PARRY was another early artificial dialogue system used to determine if psychiatrists could distinguish a simulation of paranoia from a human paranoid patient (Heiser et al, 1979). In an experiment, five psychiatrists (all males) were informed by the researchers that they would be interviewing either:

(a)  two human patients;

(b)  two computer programmes; or

(c)  one human patient and one computer programme.

The psychiatrists were tasked with recognising the human suffering from paranoia and identifying artificial paranoia. Results were random: the psychiatrists were correct five times and incorrect five times after questioning both PARRY, the computer programme modelled on a "28 year old, single Caucasian, native English speaking … male psychiatric inpatient" (Heiser, et al, 1979: p.150), and the human patient a "22 year old, single, Caucasian, native English speaking psychiatric inpatient" (ibid). The experiment with psychiatrists highlighted that experts could be deceived by artificial paranoia.


6.  In experiments at the University of Reading in 2008 it was shown once again that distinguishing human conversation from artificial dialogue was difficult for some participants. In 60 machine-human tests, machines deceived human judges at a rate of 8.33%, that is, some people were easily fooled by the machines believing the answers they gave to questions as being given by humans (Shah, 2010). One of the machines, Elbot, deceived at a rate of 25% (ibid). When transcripts of Elbot from the experiments were shown to a different set of participants, the deception rate increased to 39%. More than one in three reading transcripts of conversations between humans and machines could not distinguish the artificial responses from human answers. Table 1 shows a double conversation from the 2008 tests in which one human interrogator, a male aged 25-34 interrogating a human and a machine simultaneously, confused both his hidden partners. The machine was classified as a human, and the female was classified as a machine with a conversational ability score of 45 marks awarded from a maximum 100 (see table 1 - spellings are exactly as typed by the participants in that Turing test).

7.  Apart from mistaking humans for machines and confusing machines for humans, gender blur was another mistake that the interrogators made about their hidden conversational partners. Interrogators found it difficult to say whether they were talking to a female or a male. In one test stereotyping may have been the factor: a hidden human who revealed their occupation to an interrogator as that of a student of cybernetics was classified as male by the interrogator when in fact the human was a female studying this subject. Other misidentification types in the 2008 experiments included age confusion. In one exchange the interrogator felt that their hidden interlocutor was a teenager, because of the use of the word "bling" (see box 2) in their response to the interrogator's question about CDs and a jewellery box. The human was actually a man age range 55-64. Correct age recognition of strangers encountered in e-social media is imperative for children to prevent grooming by unknowns acting as peer-group members.

Box 2


8.  Mistakes made by interrogators in the 2008 Reading University experiments are echoed in recent tests elsewhere. In the 2010 Loebner Prize for Artificial Intelligence, a machine developed by Bruce Wilcox was classified as a human by Judge 1 in Round 3 of the competition (Loebner, 2010). The British Computer Society's 2010 Machine Intelligence Contest (BCS SGAI, 2010) selected an artificial dialogue system, Cleverbot for the 2010 "progress towards machine intelligence" award. In September 2011 Cleverbot - developed by British computer programmer Rollo Carpenter, achieved a score of 59% for "humanlike responses" in the Technique Festival at Guwahati in India. Artificial dialogue is increasing in its sophistication and ability to deceive humans (Cleverbot, 2011).

Table 1


Machine-human turing test (2008)

Interrogator J13 (Male, English, Age Group: 25-34) questioning two hidden entities in parallel
Interrogator: What do you think of this year's Turner Prize shortlist?

Left hidden entity: good I think. Better than the years before i herad

Interrogator: Which was your favourite?

Left hidden entity: It's my first time, so I can't compare

Interrogator: No, I mean of the entries this time around.

Left hidden entity: Not really sure

Interrogator: Do you think that art has become shallow and self-serving?

Interrogator: What do you think of this year's Turner Prize shortlist?

Right hidden entity: Difficult question. I will have to work on that and get back to you tomorrow.

Interrogator: You sound rather like Sarah Palin.

Right hidden entity: Did this Palin ever write a telephone book?

Interrogator: I doubt it. Does anyone really write telephone books? I don't suppose you caught the football last night?

Right hidden entity: What a shame. There are many talented authors who waste their energies on prose or run-of-the-mill poetry. But to write a telephone book is highest art!

Interrogator: I would argue that all the telephone books I have read have been very predictable. You always know how they are going to end.

Right hidden entity: If one has never known everything there is to know, experience with going away somewhere has probably been lacking.

Interrogator: I see. I'm struggling to make sense of that one. Do you write telephone books?

Right hidden entity: To write or write not, how do you expect artificial intelligence to figure this out?

Interrogator: Well I suppose that's the dream

Right hidden entity: For you humans dreams are a way of thinking without having any control over the thought process. What an unpleasant thought! My dreams are carefully planned and controlled. It gives me less to think about while I am asleep.

Actual left entity: French female, age group 25-34 Actual right entity: Machine—Elbot
Interrogator classification: Machine

45/100 for conversation ability

Interrogator classification: Human male teenager, native English speaker


9.  In 2007 PC Tools reported a new kind of malware: a piece of software developed in Russia to flirt with men or women in chatrooms on the Internet: "to collect their personal data" (see figure 4 for screenshot). The computer programme, an artificial dialogue system was found to be capable of conducting "fully automated flirtatious conversations with users of chatrooms and dating sites". The "flirtbot" was able to "mimic human behaviour during online interactions" (PC Tools, 2007).

Figure 4


10.  PC Tool's senior malware analyst, Sergei Shevenko, said "as a tool that can be used by hackers to conduct identity fraud, CyberLover demonstrates an unprecedented level of social engineering". The malware was able to employ "highly intelligent and customised dialogue to target users of social networking systems" (ibid). Researchers at PC Tools identified how the malware, at that time targeting Russian websites, automatically lured victims:

(a)  CyberLover offered a variety of profiles ranging from "romantic lover" to "sexual predator".

(b)  Humans interacting with the malware were taken in by the pretence believing they were engaging another human, thus revealing personal information.

(c)  Use of a series of very easy to configure dialogue scenarios with "canned" questions pre-set in its malware on what topics to discuss.

(d)  Designed to recognise how humans talks in chat room thus the malware was tailored to interact in the same way.

(e)  The malware was able to compile a detailed report on each and every person it interacted with and submit this data to a remote source - the reports contained confidential information that the human had shared with the programme, such as name, contact details and personal photographs.

(f)  The malware would invite the human victim to visit a personal web page or blog which was actually an infected site.

11.  In 2009 PC Tools once again issued a warning about artificial dialogue systems masquerading as humans seeking a loving relationship. They alerted that virtual dating venues could be as risky as real-world, and that Valentine's Day was not immune from increased risk of infection. Michael Greene, PC Tools Vice President Product Strategy cautioned "The rise of virtual networking has radically changed the way individuals use the Internet to interact and search for love." He warned that cyber criminals "recognised this trend" and were able to apply "more advanced and sophisticated techniques to target the digitally active consumer". PC Tools urged Internet users to mitigate the risk by being alert to "web 2.0 themed threats" on Valentine Day and prevent their personal data, such as date of birth being stolen by cyber criminals used to steal identity, or wreak financial havoc on the victim (2009).

12.  The kinds of deceptions perpetrated by flirty artificial dialogue systems or flirting chatbots includes tricking humans into selecting a link taking them to another website. This can cause an infected file to be downloaded to the victim's computer. Web of Trust (WOT) an Internet site designed to boost trust on the web, claimed that "websites offering adult content are the single most significant security threat for Internet users, comprising 31% of dangerous websites" (WOT, 2008). This threat is present when individuals access adult sites from their corporate as well as home locations. The WOT study of 19 million sites between March and May 2008 found that "sites containing pornography are the biggest threat for companies and individuals with a potential for financial and data loss as well as computer and network damage" (2008).


13.  Artificial dialogue is being used by cyber criminals as a tool to penetrate web-based chat rooms and instant messaging facilities. The sole purpose is to deceive individual humans and get them to reveal information about their identity in order to steal it and conduct financial fraud. The way this is done is in the domain of online human relationship-seeking. By posing as a paramour, flirting chatbots use social engineering to draw out lonely and susceptible humans who appear unaware of the risks. The depth of this kind of threat, and what the cost of this malware is to individuals, is not yet fully known. One reason is that humans may be too embarrassed to reveal they were duped in a relationship-forming interaction by a machine imitating as a human.

14.  Recommendations include:

(a)  Government initiatives such as Get Safe Online to include artificial dialogue in their list of risks in cyberspace.

(b)  Schools, colleges and universities to engage children, pupils and students in practical computer lessons with chatbots online to raise awareness of risks posed by these systems.


15.  The author is the lead scientist of Turing100 part of the Alan Turing centenary celebrating the life and work of the 20th century mathematician and code-breaker in 2012. The Turing100 project includes a one-day family event at Bletchley Park with the purpose of raising awareness of the risk posed by artificial dialogue systems in cyberspace. The objective is to increase deception-detection rates by allowing members of the public, adults, teenagers and children to interact with artificial dialogue systems. The aim is to mitigate identity theft, prevent financial fraud and reduce psychological misery caused by chatbot malware.


BCS SGAI (2010). accessed 5 September 2011: time 16.55

Block, N (1981). Psychologism and Behaviourism. In (Ed) S. Shieber. The Turing Test: Verbal Behavior as the Hallmark of Intelligence. MIT Press: UK: pp 229-266

Cleverbot (2011). Chatting Robot: accessed: 5 September 2011; time: 21.51

Heiser, JF, Colby, KM, Faught, WS and Parkison, RC (1979). Can Psychiatrists Distinguish a Computer Simulation of Paranoia from the Real Thing? Journal of Psychiatric Research. Vol.15 Part 3. Pages 149-162.

KMP Digitata (2011). Web Applications: Virtual Assistants. accessed: 28 August 2011: time; 16.36

Loebner (2010). Results of the 2010 Loebner Prize for Artificial Intelligence accessed 5 September 2011; time: 12.41

Marketing Week (2009). Asda Launches Virtual Assistant. accessed 28 August 2011; time: 16.28

PC Tools: 2007: PC Tools Issues Warning to Singles on Social Networking and Online Dating Sites: Beware of "Flirting Robots". accessed 28 August 2011; time: 19.39
2009: PC Tools Issues Warning About Looking for Love Online accessed 28 August 2011; time: 19.44

Shah, H (2010). Deception-detection and Machine Intelligence in Practical Turing Tests. PhD Thesis, School of Systems Engineering, The University of Reading: October 2010

Shah, H (2011). Turing's Misunderstood Imitation Game and IBM Watson's Success. Invited Paper: "Towards a Comprehensive Intelligence Test" symposium/2011 AISB Convention, University of York, 4-7 April: pp 1-5

Shah H and Pavlika, V (2005). Text-based Dialogical E-Query Systems: Gimmick or Convenience? Proceedings of the 10th International Conference on Speech and Computers (SPECOM), Patras, Greece, 17-19 October: Vol. II pp 425-428

Turing100 (2011). University of Reading: Special Turing100 event announced for 2012 anniversary of mathematician's birth. accessed 5.9.11; time: 18.56

Turing, AM (1952). Can Automatic Calculating Machines be said to Think? 1952. In B J Copeland (Ed) The Essential Turing: The ideas that gave birth to the computer age. Clarendon Press: Oxford. 2004

Turing, AM (1950). Computing Machinery and Intelligence. Mind. Vol 59 (236) pp. 433-460

Weizenbaum, J (1966). ELIZA - A Computer Programme for the Study of Natural Language Communication between Men and Machines. Communications of the ACM, 9, pp 36-45 (1966)

WOT (2008). Web of Trust Report: Increased Security Threats in the Internet's Red Light District. accessed 28.8.11; time: 20.23

September 2011

1   Figure 1 Acknowledgement: Google search Back

2   Figure 2 Acknowledgement: Back

3   Figure 3 acknowledgment: HarshM, 2010 Back

4   Figure 4 acknowledgement:

CNET accessed 28.8.11; time: 19.36 Back

previous page contents next page

© Parliamentary copyright 2012
Prepared 2 February 2012