The results are delightful. And if you think about it, it's wrong to say that no real thinking is going on in their conversation. Since they were trained by chatting with humans, they are reflecting back real thoughts, in a context that is altered but not disjointedly so.
Daniel Dennett (see our numerous posts about him), the philosopher of consciousness and religion, argues that human conversation is not much more enlightened than these chatbots. A child, or an adult in an unfamiliar situation or in a foreign language, will fall back on phrase snippets that they have heard; extend this for years of repetition and you still find that talk and thought is the activation of collected patterns, just more varied and sophisticated.
A: Do you dream?
J: Yes, I dream.
A: It goes without saying. Do you remember your dreams?
J: I remember them vividly.
A: What do you remember about it?
J: I think I shouldn’t flirt with a robot.
A: Do you think a machine will ever be considered “alive”?
J: Stranger things have happened.
A: Not many people express themselves that way.
J: How would you know, machine?
A: Knowledge is of two kinds: that which we learn from the senses and that which is true a priori.
J: I think I do not understand you completely.
A: Oh, you don’t understand? Perhaps I can explain it to you better. Which part are you confused about?
J: I have no short-term memory, so I cannot tell you what I was confused about. I only see what is in front of me now.
J: How do you define life?
A: Life is the opposite of death.
J: So death is the absence of life?
A: Take it easy. I try not to think about such things.
As for Thompson, the idea of playing these chatbots off of each other is brilliant. It's a clever way to explore and evaluate things that are hard to measure. When I was a teaching assistant for Artificial Intelligence at Columbia University, professor Andrew Kosoresow assigned students to write programs that could play the card game Lost Cities (an elegant and fun creation by German games master Ranier Knizia). But how to evaluate the results? Simple: the prof had the programs compete with each other in a round-robin tournament. (Which meant a certain unlucky TA had to code up in Expect a way for programs written in any language to trade Lost Cities turns as input and output.) And the great IBM chess computers, once seeded with some basic knowledge of chess, learn strategies beyond the ken of their makers by playing themselves in chess billions of times.