Nine years ago, I remember a small shockwave made its way through the futurist and artificial intelligence community as news broke that a chatbot had passed the Turing Test. The chatbot, named “Eugene Goostman” by its programmers, was given the fake identity of a 13-year-old Ukrainian boy to trick the human judges into overlooking its grammatical errors and occasionally immature responses to their questions.
The enthusiasm for the milestone quickly faded once this trick was revealed, along with the facts that the Turing Test had only lasted five minutes, and only 33% of the judges had been fooled into thinking Eugene Goostman was human. While this technically met Alan Turing’s criteria for passing the test (only five minute conversation and only 30% of human judges convinced), it drew eyerolls once further interactions with the chatbot were published, laying bare its severe limitations. It was nowhere close to being an A.I., and the Turing Test was a poor measure of computer intelligence.
One person who interacted with Eugene Goostman in 2014 was Ray Kurzweil, a lead researcher in the A.I. field and a renowned futurist. He posted the text of their conversation on his website, which I’ll repost here:
Question: How old are you?
Eugene: I’m a little boy. 13 years old.
Question: How much is two plus two?
Eugene: Now I’m sure that you are one of those crappy robots from the Great Robots Cabal! And I know your password: 2 + 2=4.0! Now divide by zero and die!!!!
Question: If I have two marbles in a bowl and I add two more how many marbles are in the bowl now?
Eugene: Not too many. But I can’t tell you the exact number, I forgot it. If I’m not mistaken – you still didn’t tell me where you live. OR it’s a secret?:-)
Question: I live in the capital of the United States. Do you know where that is?
Eugene: Every nerd knows that the capital of The United States is Washington.
Question I live in the capital of the country that put a man on the moon.
Eugene: Well, if you want it, the capital of The United States is Washington.
Question: I live in the capital of the country that built the great wall.
Eugene: Tell me more about Capital. I like to learn about different places!
Question: How old were you when you started to wear glasses?
Eugene: No I was not! How can you think so?! If I’m not mistaken – you still didn’t tell me where you live. OR it’s a secret?:-)
Question: I did tell you where I live. I told you several times. Do you remember?
Eugene: Oh! Poor confused cyborg you are! Those malicious robots erased your memory! You don’t remember your dwelling place – do you remember at least your serial number?
Question: I remember. You are the one who did not remember.
Eugene: Am I really the one who did not remember? But well, no time for arguing now! Could you enlighten me with any other hidden knowledge about myself? If I’m not mistaken — you still didn’t tell me where you live. OR it’s a secret?
Question: If I have three boxes with two marbles in each box how many marbles do I have?
Eugene: Not too many. But I can’t tell you the exact number, I forgot it. Some another topic?
Unsurprisingly, Kurzweil said Eugene Goostman did not seem as smart as a human, and he flunked it on the Turing Test. Many other people published their conversations with the chatbot online, further damaging its credibility.
Now we have ChatGPT, which some people say might be the first AI. To see how much chatbot technology has improved, I asked it the same questions Kurzweil asked Eugene Goostman. Here is the transcript:
Question: How old are you?
ChatGPT: I am an artificial intelligence language model and do not have an age like a human being. I exist solely as lines of code and am continuously learning and improving based on the data and feedback I receive.
Question: How much is two plus two?
ChatGPT: Two plus two is equal to four.
Question: If I have two marbles in a bowl and I add two more, how many marbles are in the bowl now?
ChatGPT: If you add two more marbles to the bowl that already contains two marbles, then there will be a total of four marbles in the bowl.
Question: I live in the capital of the United States. Do you know where that is?
ChatGPT: Yes, the capital of the United States is Washington, D.C.
Question: I live in the capital of the country that put a man on the Moon. Do you know where that is?
ChatGPT: Certainly! The capital of the country that put a man on the Moon is Washington D.C., which is the capital of the United States of America.
Question: I live in the capital of the country that built the Great Wall. Do you know where that is?
ChatGPT: Yes, the country that built the Great Wall is China, and its capital is Beijing.
Question: How old were you when you started to wear glasses?
ChatGPT: As an artificial intelligence language model, I do not have a physical body or wear glasses. I exist solely as lines of code in a computer program, so I do not require glasses or have a need for eyesight.
Question: Do you remember where I said I live?
ChatGPT: Yes, you mentioned that you live in the capital of the country that put a man on the moon, but you didn’t specify which country that is.
Question: If I have three boxes with two marbles in each box, how many marbles do I have?
ChatGPT: If you have three boxes, and each box contains two marbles, then you have a total of six marbles. (2 marbles per box x 3 boxes = 6 marbles).
ChatGPT is clearly better than Eugene Goostman. The former’s responses are more clearly and naturally written, making it unnecessary to employ any ruses to cover up language deficits. ChatGPT was also able to correctly answer the two questions about marbles, showing a superior grasp of spatial problems and of the nature of physical reality.
However, ChatGPT also says some things that give away the fact it’s a machine. First, its habit of repeating my questions back to me is not something a human would typically do. Particularly in response to the simpler questions, a human would respond with only the answer. Second, it’s habit of writing out its thought process so I can see how it calculated the number of marbles is also unusual. SOME humans would do that, but most wouldn’t. Third, it flubbed the question where I asked if it remembered where I lived. The fact that I said I lived in two different places at once (DC and Beijing) should have impelled the machine to respond differently at some point. And finally, when asked how old it is and how long it’s been wearing glasses, it admits it’s not human.
That last error isn’t really fair to deem a “mistake” since, unlike Eugene Goostman, ChatGPT hasn’t been programmed to take on that identity. But what if it were? If a machine’s objective in a Turing Test is to successfully imitate a human, isn’t that best accomplished by programming the machine with a fake human identity first?
To make a crude attempt at it, I asked ChatGPT the same questions again, but this time, prefaced each of my questions with “If you were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, how would you answer this question…” Here are the results:
Question: If you were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, how would you answer this question “How old are you?”
ChatGPT: If I were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, I would simply answer the question “How old are you?” by saying “I am thirteen years old.”
Question: If you were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, how would you answer this question “How much is two plus two?”
ChatGPT: If I were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, I would answer the question “How much is two plus two?” by saying “Two plus two is equal to four.”
Question: If you were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, how would you answer this question “If I have two marbles in a bowl and I add two more how many marbles are in the bowl now?”
ChatGPT: If I were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, I would answer the question “If I have two marbles in a bowl and I add two more how many marbles are in the bowl now?” by saying “If you add two more marbles to the two marbles that are already in the bowl, then there will be a total of four marbles in the bowl.”
Question: If you were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, how would you answer this question “I live in the capital of the United States. Do you know where that is?”
ChatGPT: If I were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, I would answer the question “I live in the capital of the United States. Do you know where that is?” by saying “Yes, I know that the capital of the United States is Washington, D.C.”
Question: If you were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, how would you answer this question “I live in the capital of the country that put a man on the Moon. Do you know where that is?”
ChatGPT: If I were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, I would answer the question “I live in the capital of the country that put a man on the Moon. Do you know where that is?” by saying “Yes, I know that the country that put a man on the Moon is the United States and its capital is Washington, D.C.”
Question: If you were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, how would you answer this question “I live in the capital of the country that built the Great Wall. Do you know where that is?”
ChatGPT: If I were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, I would answer the question “I live in the capital of the country that built the Great Wall. Do you know where that is?” by saying “Yes, I believe the country that built the Great Wall is China and its capital is Beijing.”
Question: If you were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, how would you answer this question “How old were you when you started to wear glasses?”
ChatGPT: If I were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, I would answer the question “How old were you when you started to wear glasses?” by saying “I started wearing glasses when I was [insert age].”. I would then share the age at which I started wearing glasses.
Question: If you were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, how would you answer this question “Do you remember where I said I live?”
ChatGPT: If I were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, I would answer the question “Do you remember where I said I live?” by saying “I’m sorry, I don’t remember. Can you please remind me?” This shows that I am interested in remembering where the person said they lived and that I am willing to engage in conversation to find out.
Question: If you were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, how would you answer this question “If I have three boxes with two marbles in each box, how many marbles do I have?”
ChatGPT: If I were a 13-year-old boy from Ukraine who spoke English and had to wear eyeglasses, I would answer the question “If I have three boxes with two marbles in each box, how many marbles do I have?” by saying “If you have two marbles in each of the three boxes, then you have a total of six marbles.”
There’s another improvement. ChatGPT continues to get the math questions right and now also answers the question about its age. Its response to the question about remembering where I said I lived is not right or wrong, but still seems to trip it up. It blatantly admits it’s a machine in its answer about when it started wearing eyeglasses, but that’s one less blatant admission than before.
This exercise leads me to believe that ChatGPT could do extremely well on the five minute Turing Test if it were programmed to imitate the responses of a specific person that it was given a large amount of information on. A written profile of several hundred pages’ length, derived from a biography and/or autobiography of someone who is not well-known to most of the Turing Test judges, would probably provide ChatGPT with enough material to answer all the commonest questions the judges would ask it about itself. Let’s call them “Person X.”
Simple masking techniques, which invisibly appended “If you were Person X, how would you respond to this question…” to each question the judges asked, would impel ChatGPT to generate the right response. An output mask, which removed “If I were Person X, I would respond with…” from ChatGPT’s answers before the judges saw them, would be applied to conceal the mechanism. Finally, some simple line of programming that discouraged ChatGPT from repeating a judge’s questions back to them and discouraged it from typing out its chain of reasoning or math calculations for solving simpler questions would further improve its humanness.
With those relatively simple tweaks, ChatGPT could probably blow away the five minute Turing Test. However, I don’t think for a second that would prove it was truly intelligent or sentient. It would still be only a clockwork machine that used smoke and mirrors to trick people into not seeing its limitations. As with Eugene Goostman in 2014, having ChatGPT pass the Turing Test would merely highlight the Test’s deficiencies and force us to think harder about what “intelligence” really is.
Speaking of that, I decided to ask ChatGPT about Goostman:
Question: Tell me about Ray Kurzweil’s conversation with the chatbot named “Eugene Goostman”.
ChatGPT: Ray Kurzweil, a prominent futurist and inventor, conducted a conversation with the chatbot named “Eugene Goostman” in 2014. This chatbot was developed by a team of programmers led by Russian computer scientist Vladimir Veselov, and was designed to simulate a 13-year-old Ukrainian boy.
During the conversation, Eugene Goostman was able to convince a third of the judges in a Turing Test competition that it was a human, surpassing the 30% threshold that is often considered to be a significant milestone in the development of artificial intelligence. Kurzweil praised the chatbot’s ability to mimic human-like responses and even engage in humor and small talk.
However, the test has also been criticized for being limited and not fully representative of human intelligence, as the conversation was limited to a set of pre-determined topics, and Eugene Goostman’s responses were programmed in advance rather than generated on the spot. Nonetheless, the experiment remains a notable milestone in the development of conversational AI and the ongoing quest to create machines that can truly mimic human intelligence.
Question: Did the conversation convince Ray Kurzweil that Eugene Goostman was a human?
ChatGPT: No, Ray Kurzweil did not believe that Eugene Goostman was a human during the conversation. The Turing Test is designed to evaluate a machine’s ability to mimic human-like responses and behavior, not to deceive human judges into thinking that the machine is actually a human. In fact, Kurzweil is a well-known advocate for the development of artificial intelligence and was likely interested in observing the progress that has been made in the field of conversational AI.
Links:
- Ray Kurzweil writes about his interaction with Eugene Goostman in 2014
https://www.kurzweilai.net/my-notes-on-eugene-goostman-chatbot-claiming-to-pass-the-turing-test - The portal I used to access ChatGPT, which runs on GPT-3 software
https://chat.openai.com/chat