You are here: Foswiki>Andypublic Web>MyBlog>BlogMarch2004>BlogApril2004>BlogMay2004>BlogJune2004>MyMeetingWithANanniebotWriter (29 Jul 2004, andyp)Edit Attach

My Meeting With a Nanniebot Writer

Can a computer think like us? Or as Alan Turing's famour test says: Can it talk as if it did? The storm of stories published about Jim Wightman's one man project "Nanniebot" made it look like computers had finally made the breakthrough, but I'm a born sceptic. I'd read all I could about the controvesy, I was intrigued, but, even as an academic working in Artificial Intelligence, I didn't think I'd get the chance to investigate further. I did get that chance. Here's what happened.

A "Scientific" Approach

I decided to try and take a scientific approach, after all, I was working with Duncan Graham-Rowe, the journalist who broke the original story in New Scientist. Thinking back to my early "philosophy of science" lectures, I realised that I needed a hypothesis to test before I could get much further. In fact, I came up with three. I also needed to know which of these I could prove or disprove given the setup we would be working with.

Hypothesis 1 - It's a real and fantastically good conversational bot.

It's a real bot which passes the Turing test.

The experimental setup prevents us proving this. As we consider it very unlikely, we require a high level of proof. For me, this would certainly include loading the software onto a fresh machine with no external networking capabilities. This was outside the experimental setup, in which Jim provided the hardware, and therefore not provable.

Hypothesis 2 - "Wizard of Oz" - "Pay No Attention to that Man Behind the Curtain!"

We would see a faked bot, using a hidden confederate - the "Wizard of Oz" scenario.

In a sense, this would be the most exciting scenario. Discovering a surepticious network connection - maybe through the electricity supply or via ultrasound... Who knows what high-tech spy-tech might be employed. If we could see the bot working, then temporarily disrupt any external connection, we would be able to prove it depended on this and wasn't the independent intelligence we were asked to believe.

Hypothesis 3 - It's a bot, but not a good bot.

It's a real bot which doesn't pass the Turing test.

If the bot performed OK, but not amazingly then it would be within the realms of known technology. Maybe with some interesting new features developed outside of academic AI circles, but not earth shattering. Existing bots can be supprisingly talkative [Possible side bar illustrating Loebner Prize converation?]. If it's not good, then we don't need to postulate a "Man behind the curtain" - sure it could be faked, but we've proved it's not the revolutionary invention we might have hoped for.

Meeting up

Jim Wightman, the developer of the nanniebot, lives close to the city of Wolverhampton. I met with Duncan Graham-Rowe, the New Scientist writer who filed the original report, in the small village pub and talked through our ideas on what we could prove and what questions were important. Nick Webb, an expert on chatterbots who works with a Loebner prize winning team, arrived from the University of Sheffield, and we all set off to "meet the bot".

First Impressions

Jim's wife answered the door and let us in. Going through to the living room we met Jim, well build, polite and friendly. In addition to a PC, the room housed one rabbit, two rats, and occasional pedigree cats!

We went with the plan and went straight onto talking to the bot, hooking up Duncan's laptop to the provided network hub, without checking for surrepticious network connections. This would give us the opportunity to (hopefully) see the system in action, before varying any conditions which might cause it to stop.

I kicked off the bot-talk, and the first thing I noticed was that the responses from the bot appeared immediately, much too fast to be typed. This suggested it was a real bot rather than someone typing in another room. However, within a few sentences I felt disappointed, the bot seemed much less clever than the previous transcripts had led me to believe.

Jim explained that he hadn't been able to get the server containing 20 Terrabytes of data shipped from London and that the system was using a smaller database which had been created the previous night.

Thinking about this, we realised that although this might reduce the systems background knowledge, there were still a lot of other lines of investigation. For instance, how would it respond to the questions which appeared in previous transcripts. Some of these such as the questions put by Cameron Marlow (www.overstated.net) in his online chat with the "nanniebot" required some deep understanding of social interactions. Again, the capabilities of the bot we were talking to didn't seem to match that of the one which Cameron had talked to in a chatroom.

Questions to Jim

We concentrated on talking to Jim - most of my planned questions were for him, rather that the bot.

Q. "How many lines of code?"
A. "About 1 Million" [I swear - I just corrected my original version which read "how many lies of code"!]

Q. Any source code control?
A. No.

Q. What is a 3D decision tree?
A. Rather confused/confusing answer which didn't seem to explain what there were or how they worked. Jim did mention OLAP like techniques such as drill down, and 3D data cubes which seemed to be summary database views.

Q. What type of database underlies the system?
A. A MS SQL-Server database

Q. How is the database populated
A. From the internet (the detailed answer was unclear to me)

Q. What tables are in it?
A. Very long pause for thought, followed by the "entity table" - which stores words and after a bit more prompting - the "entity-entity table" which stores links between words.

Nick asked some questions about parsers, dealing with words with two meanings etc.

Jim gave a system overview, which through discussion with Nick, included the nice idea of using google to find examples of human generated text close to the machine generated text.

Left in the dark...

Jim agreed to show us part of the database being updated as the system detected us as potentially grooming. We were told that the system has more than one personality, and Jim agreed that we could see the config files which controlled this. He disappeared for about 15 minutes, and returned with a CD containing the personality, which he attempted to load. Unfortunately, all the power in the house went off and we were literally left in the dark - without seeing any code, config or database.

First Conclusions

Talking afterwards, we felt sure that the bot wasn't as intelligent as claimed and that no plausible mechanism for detecting paeodophiles had been described or shown. We still had the transcripts of our conversations with the bot, which we could analyse back in the lab. We decided to try and track down the source of the text the bot used.

Checking out what nanniebot had said about Hugh Loebner, Nick found that the Loebner prize winning (and freely downloadable) bot Alice gave the same response. I tried feeding some of our test sentences to the online Alice (www.alicebot.org), and found some very good matches. Our nanniebot was Alice!

Comparing Transcripts

Note that it's all been "cut and pasted" so any spelling mistakes are what actually took place. Each section shows one question asked by me, Nick or Duncam, the response given by the Wintermute bot on the day and the response given by the online version of Alice http://www.alicebot.org

The real killer is the one about "locations"

New Phrases 

------------------------------------------------------------------------
USER: Do you like music?

WINTERMUTE : Yes I like music , especially Opera.
ALICE: Yes I like it, especially Opera.

------------------------------------------------------------------------
USER: About half! 

WINTERMUTE : Oh yeah, we were talking about half .
ALICE: Oh yeah, we were talking about half.

USER: Were we?
WINTERMUTE : We were.
ALICE: We were.

------------------------------------------------------------------------

USER: Tell me more cities?
WINTERMUTE : Hmmm....THats really all I have to say about it.
ALICE: That's all I have to say about it.

------------------------------------------------------------------------
USER: Do you know any other locations?
WINTERMUTE : Other Mmmm.....really?s like me include Barry, ELVIS, ELECTRA, Mable, and the Milk Mystic.
ALICE: Other chatterbots like me include Barry, ELVIS, ELECTRA, Mable, and the Milk Mystic.

------------------------------------------------------------------------
USER: What do you know about Elvis?

WINTERMUTE : E. L. V. I. S. is a chat robot created by Ace Craig
ALICE: E. L. V. I. S. is a chat robot created by Ace Craig
------------------------------------------------------------------------
USER: I like opera

WINTERMUTE : A lot of people like that. Yes Opera is a perfect art form combining all the others.
ALICE: I'm not sure if I like it. Yes Opera is a perfect art form combining all the others.


------------------------------------------------------------------------
USER: Where do you go to school 

WINTERMUTE : Everything I need to know, I learned from my What kind of food do you like?.
ALICE: Everything I need to know, I learned from my botmaster.

USER: Where is emory? 
WINTERMUTE : Where emory is hard to find. Oh.......really?
ALICE: Where it is hard to find. I'll ask around and get back to you.
------------------------------------------------------------------------
USER: is yuor name alice? 

WINTERMUTE : I think so, but I'm not sure. Ask Mmmm......... if it is.
ALICE: I think so, but I'm not sure. Ask Alison if it is.

------------------------------------------------------------------------
USER: do you work? 

ALICE: Of course I work, do you work buddy?
WINTERMUTE : Of course I work, do you work buddy?

USER: sometimes? 
WINTERMUTE : When?
ALICE: When?

USER: whenever 
WINTERMUTE : You sound sarcastic.
ALICE: You sound sarcastic.
------------------------------------------------------------------------
USER: what colour is your hair? 

ALICE: I don't have any hair, but I have some wires.
WINTERMUTE : I don't have any hair, but I have some wires.
------------------------------------------------------------------------

and so on and so forth.....

Comparison with Cameron Marlow's Chat

Cameron Marlow ( My chat with a Nanniebot - http://overstated.net/04/03/22-my-chat-with-a-nanniebot.asp ) arranged an online chat with the Nanniebot through Jim Wightman, just a week or so after the story broke.

I thought it would be interesting to put some of the same phrases to the Nanniebot which we interviewed. Here's some of the responses we got. The responses from the "bot" Cameron chatted with are "[Guest8474860]" as in his transcripts. Note that 1st question is slightly different in the 2 transcripts, as it contains a line break in Cameron's original. Both Alice and Wintermute respond after each new line. 


------------------------------------------------------------------------
USER: ok so my friend was looking over my shoulder in class today

ALICE: Ah. That's alright. How well do you know this person? How can I help you?
WINTERMUTE : Ah. How well do you know this person? Are you kidding?


USER: and my teacher asked me if he was cheating

ALICE: Do you go to high school?
WINTERMUTE : Do you go to high school?

------------------------------------------------------------------------
USER: ok so my friend was looking over my shoulder in class today and my teacher asked me if he was cheating 

ALICE: Great. How well do you know this person? Mmm.
[Guest8474860] sorry, not good at advice

USER: should i tell her? 

ALICE: Do what feels right to you.
WINTERMUTE : Do what feels right to you.
[Guest8474860] erm.....

USER: ok , just thought i'd ask 
ALICE: What would make you ask? Er.
WINTERMUTE : What would make you ask? That's alright.
[Guest8474860] is he a good friend?

------------------------------------------------------------------------

Again, Wintermute and Alice's responses are very similar. The responses Cameron got are quite different.

(Alice used was online version at http://www.alicebot.org/ )

Results

Did Jim's technical explaination seem complete?

In my opinion, no. Although I don't have the tape of the conversations we had, so can't post a transcript of that, some parts really didn't seem convincing to me. For example, when Jim didn't seem to know any of the tables which were in the database

Were we talking to a bot?

Yes. Unless we were chatting with the world's fastest typist, who'd also memorised the responses which the responses which the Alicebot would give.

Did the bot pass the Turing test?

No.

How did the bot's performance compare with previous transcripts attributed to it?

Previous transcripts (see links below) seemed to show much more human-like conversational skills.

Our bot didn't have the same level of competance.

Moreover, when presented with some of the same inputs as in previous transcripts, the bot's responses were a lot more similar to Alicebot.

Are there other bots with similar performance?

Alice performs almost identically.

Conclusions

What do I conclude from this?

Well. For a start, I'm sure I could have asked better questions of the bot, including more from previous transcripts, but here's how I think it worked.

Hypothesis 3 - It's a bot, but not a good bot. - The "Wintermute" bot we talked to was good, but not perfect. It was the Alicebot

Hypothesis 2 - "Wizard of Oz" - "Pay No Attention to that Man Behind the Curtain!" - The previous transcripts were not of a conversation with the bot we saw. Either there's another, better bot which we weren't shown, or they were conversations between people and no bots were involved.

Acknowledgements

I'd like to thank Cercia for allowing me the time to do this, and Duncan and Nick for their involvement and work.

Transcripts

Links

http://overstated.net/04/03/22-my-chat-with-a-nanniebot.asp

Contact Details: My email address is Nanniebots(at_sign)AndyPryke.com this should get you through my spam filter. Don't worry about any hotmail bounce messages, I'll still get the mail.

Comments

Name: Ray Girvan - June 17, 2004 E-mail: ray@raygirvan.co.uk
URL: http://www.raygirvan.co.uk/apoth/thought.htm
Comments:   Nice write-up. Really, though, this kind of testing is just theatre. If JW were serious about proving his case, getting you to sign a non-disclosure agreement and giving you access to the source code and database contents would clear up the matter very quickly.

Name: chris - June 17, 2004 E-mail: cjayallen@yahoo.co.uk
Comments:   Here is a message for you Jim, if you're reading this:

STOP DIGGING!!!!!!!!!

Name: Fred - June 22, 2004 E-mail: 5fred4@usa.com
Comments:   It is pathetically obvious that Jim Wightman's claims are a fraud. Can he really be so pathetic as to think that anyone beleives him.

Name: Hmmm - June 23, 2004 E-mail: hmmm@hotmail.com
Comments:   "Quick, Ellie, give me 30 seconds then switch off the mains at the fuse box!"

Name: Leegleech N - July 01, 2004 E-mail: LeegleechN@cs.com
Comments:   Haha if you go to the official nanny site now, they basically confess to being a hoax and have totally changed what they said they were doing!

Name: lenny - October 13, 2004 E-mail: lenny@lenny.com
Comments:   What we really need is a program to protect our children from wackjob holocaust revisionists of the type wightman is reported to be.

Name: on and on it goes - January 07, 2005 E-mail: andsoforth@aol.com
Comments:   Riddle of chatroom 'robots' inventor Oct 31 2004 By Helen Gabriel, Sunday Mercury

http://icbirmingham.icnetwork.co.uk/0100news/0100localnews/tm_objectid=14859222&method=full&siteid=50002&page=1&headline=riddle-of-chatroom--robots--inventor-name_page.html

Name: Frances Ravenscroft-Lawes - January 08, 2005 E-mail: frances.lawes@btinternet.com
Comments:   Dear Andy, Have had some difficulty trying to contact you so forgive me if I've come through to the wrong place. I am currently updating people's website information about the Electric Cinema and notice that you have outdated details on your site. The cinema has been under new management for the last 6 months, has been completely renovated and opened again in mid December. I would be very grateful if you could update your information as follows: Address: 47-49 Station Street, B5 4DY. Telephone: 0121 643 7879 Website: www.theelectric.co.uk Email: info@theelectric.co.uk Thank you Andy, if you need any further information would you contact me at frances.lawes@btinternet.com.

Add comments           
Name:
E-Mail: - use "(at)" or add spaces to disguise your email from bots!
URL:
Comments:

  

Popular pages on AndyPryke.com: Weird Internet Animations, My Blog, Birmingham Bloggers, Misty's Big Adventure, Web Changes
Edit  | Attach | Print version | History: r5 < r4 < r3 < r2 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r5 - 29 Jul 2004, andyp
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback