Pondering AI

KIMBERLY NEVALA: Welcome to Pondering AI. My name is Kimberly Nevala. I'm so pleased you're joining us as we ponder the reality of AI, for better and worse, with a diverse group of innovators, advocates, and data professionals.

Today I am honored to bring you Professor J. Mark Bishop. We're going to be talking about large language models, the app of the moment ChatGPT and the nature of AI. Mark is the Chief Scientific Advisor to FACT360 and emeritus professor of cognitive computing at Goldsmiths College University of London. And that really just scratches the surface of your career, Mark. Thank you so much for joining us.

J. MARK BISHOP: Thank you for having me.

KIMBERLY NEVALA: So you are well-known by most, but perhaps not all. For those that haven't had the pleasure of engaging with your work or being challenged - I'll raise my hand here - by your work, will you provide a synopsis of your career and research interests?

J. MARK BISHOP: I'll be delighted. I've been involved in AI for a long time.

So my PhD, which I was awarded back in '89, was in artificial intelligence, neural networks, and swarm intelligence. And in fact, as part of my research, I looked at a neural model that became known as convolutional neural networks, one of the very first really successful deep neural networks that kind of hit the scene. As part of that PhD, I developed the world's first swarm intelligence algorithm, which had a variety of interesting practical applications.

But also, interestingly, whilst performing that PhD, I sort of changed from being an advocate of AI and thinking that AI was going to be great in all circumstances and that we would want to build a machine to act like a human to being a little bit more skeptical. And that sort of change in opinion was launched by exposure to what even then was a relatively old argument that was first positioned in 1980 by the philosopher John Searle called the Chinese Room Argument. That, if true, suggests that machines don't now nor ever will, no matter what the algorithm, ever genuinely understand anything. And that kind of insight - and I got to work with John Searle and his argument more deeply - has informed kind of the rest of my career.

So one of the great big ironies of my career is that back in 2010, I was elected to chair the UK Society for Artificial Intelligence, the AISB, even though by that point in time I was a well-known critic of what artificial intelligence can achieve.

So as an academic, I went from being a junior PhD to getting on a faculty of cybernetics at the University of Reading. Then I moved to the University of London at Goldsmiths College, where I ran a master's course in cognitive computing and the philosophy of artificial intelligence. And that went on for a number of years until I won a very large grant to build the Center for Artificial Intelligence and Analytics that was going to do commercially funded research, looking at how AI can better enable progress and profit in business applications. And that center was called TCIDA, The Center for Intelligent Data Analytics. And I ran that center for a number of years until, just a couple of years ago, my entire staff left en masse to set up a startup called FACT360, which looked at using AI to discover interesting facts by analyzing conversations and email messages and any kind of textual compensation for when things go wrong, when there's an insider threat in the company, or when someone's been acting against a company's interests. And at that point, I decided to leave academia, too. And then I joined this company as a scientific advisor, which is what I do now.

So that, in a nutshell, is how I've spent my life working in AI.

KIMBERLY NEVALA: I'm really excited to talk about a lot of different aspects of the work that you've done and the research that you're doing. And some of those philosophical underpinnings as well. We're going to be talking about the capabilities, the limitations of systems based on large language models, and so on and so forth. The consultant in me says it would be good to provide a foundational definition to make sure everyone has a good level-set of what a large language model, or LLM, is. And perhaps we can throw in a quick definition of GPT for good measure.

J. MARK BISHOP: OK, well, a large language model is typically a deep neural network that's trained on a large corpus of texts. And the learning algorithm might be something like a recurrent neural network or an LSTM network - that's a long short-term memory network. Or, more popularly now, a transformer network. GPT-3, and hence ChatGPT, on which it's based utilize the transformer architecture.

Now, I don't know how deep we want to go into this. But basically, transformers are a neural network that are grounded on certain linguistic intuitions about the way that language works. And probably the most important is that in any language, words typically have many related senses. And that the way we disambiguate those - and an example of what I mean, I don't know. I went to the bank, meaning, I went to withdraw some money from an ATM. Or I went to the bank, meaning I went fishing on a riverbank. Same word, different meanings.

And the disambiguation of the meaning of a word is related to the context in which we use that word. And this context might be local. So the words adjacent to the word we're interested in might shape the meaning of a particular word. Consider happy, and a word adjacent to it, not happy, they've got totally different meanings. The meaning of the word happy is informed by the word that's adjacent to it.

Or there might be sentences in which you've got very non-local meanings. So it might be the first word in the sentence fundamentally affects the meaning of the very last. So in other words, meanings can be non-local.

And worse than that, the meanings are not always from language. So to give you an example from Wittgenstein's Philosophical Investigations, the book he wrote after the Tractatus. Wittgenstein describes a language game between two laborers, two builders. And you might imagine one of them shouting ‘pick’. And in the normal sense of the word, there is not enough information there to disambiguate what we mean by pick. But the context - doing a task, building or digging a hole - disambiguates the word meaning.

And in fact, many years ago, as a young postgrad, I, myself, and another postgrad who went on to be a very big professor of computer vision at Oxford University, where he is still, I won't name him. Let's just say he's called Phil. And we developed a game called the alphabet game. And it's quite bizarre. We found a way of going through every letter of the alphabet one by one. So I would say, A. And he would say B. And I would say C. And by the extra linguistic information we imparted meaning. So it just reinforces this fact that language is more than the uttered syllables that we're talking about.

And this intuition, actually, informed the work of a guy called Roy Harris, who was Professor of General Linguistics at Oxford for many, many years. He developed a theory of linguistics called Integrationism which looked at the wider ways in which language is formed. And he said that written texts and verbal utterances are just one tiny subset of wider communication systems. Again, your listeners might be more familiar with the term body language. You can tell - have a feeling about - whether someone's being honest or what they're really thinking by the way their body moves in relation to another, perhaps. So these intuitions say that language is actually quite complicated.

And the last point I wanted to raise is often understanding relies on a shared background knowledge, a deep shared background knowledge about the world.

KIMBERLY NEVALA: So how have these complicated linguistic intuitions informed the design of LLMs?

J. MARK BISHOP: I'll just give you a very top-down description and we can dig deeper if there's any need.

But basically, they're based on neural networks that do attention mechanisms. Again, I guess we should say what a large language model does before I get into the technicalities. Basically, a large language model will complete some text which is typically these days called the prompt. So we'll enter some text. The cat sat on the ___. And then you would hope that the large language model might say something like ‘the cat sat on the mat’ or something like that.

So basically, we give the large language model a prompt, and it replies in some text that's very relevant and aligned to the human user's intentions. And what makes LLMs really good is that they're very good at this alignment problem, as it's called. Aligning their responses to what the human who is interacting with the LLM really wants.

So when given a prompt as a starting point, the LLM such as ChatGPT uses its trained knowledge to generate text that continues that prompt in a sensible way, and a coherent way, and a natural way. And that's disarming for the user because it's often quite difficult to tell which text is being computer-generated and which might be generated by a human.

So that's what they do. They're sims that continue prompts. A little bit unkindly, perhaps, but I think with good cause, in a very recent paper the Oxford polymath Professor Luciano Floridi. He said, in a sense, you can think of LLMs as a bit like if you're using a search engine, the query completion mode. So as you start typing a query, what's the capital of Great Br-- it completes that as what's the capital of Great Britain. And it's kind of that writ large. So these things don't just produce a few words, typically, as the response to a prompt, they might produce a number of paragraphs of text. So that's what they do in a nutshell.

KIMBERLY NEVALA: And what are the basic mechanisms by which LLMs accomplish this feat?

J. MARK BISHOP: How do they do that? Well, they have these blocks of self-attention mechanisms. And basically, what we do then, if we give the system a sentence or a block of text as input, we will take every - LLMs typically work at the level of the word. So we will map each of these discrete symbols into a large vector space. If I remember rightly, ChatGPT maps into a vector space in excess of 500 dimensions. So we got a large highly dimensional vector space. We map the words into that space.

And the idea of this mapping into this real value space is that words that have similar meanings are mapped to similar points in that very highly dimensional space. Then once we mapped all the words in our text, the position into that space, we multiply these activations, these words, by what's called the key, a query vector, which are learned vectors to produce a value. And these are integrated forward through a standard feedforward perceptron-like architecture and normalized. And the result of all that - to cut to the chase - is that we have a set of weights that describe how each word in the sentence effectively relates to each other. And then we can repeat that block and repeat that block as we get deeper and deeper into the deep neural network. And from memory, ChatGPT I think has got 12 of these attention layers.

Now, it does a few other things that are interesting. So imagine each one of these attention layers gives you a sense of how the words in a structure relate to each other. And what makes it - ChatGPT really clever - is it has multiple attention heads. And in fact, I think ChatGPT has 43 of these working in parallel. So effectively, that gives you 43 different views on how possibly the words in that sentence are relating to each other at each layer in the neural network. These are then propagated through the deep neural network.

There's two other things that are worthy of note. In a transformer, in the basic mode I've just described, there's no sense of order of the words in that sentence. The words are just propagated through with no sense of order. Obviously, when we're reading text, order matters. So one of the insights of the transformer designers, Vaswani et al. was to encode onto the vectorial representation of each word a means of encoding where it is. And they typically use a sinusoid function to map a description of where the position of the word into its vectorial representation. So we encode the positions of where words are in these sentences.

And the last thing that we need to think of that's interesting is that in language, often, top-down effects about what a sentence as a whole might mean can inform the meaning of individual words. If you think of the individual words as giving you a bottom-up idea of what a block of text is about, sometimes when you look at the comprehension of the whole entity, that can also feed down and define what the individual words are like. And to give an effect that's analogous to that, the design of the transformer model was introduce the idea of skip connections, where you pass weights through and bypass the transformer layer from one layer to another. And that gives a way in which these higher-level layers in the neural network can have direct access to the very low-level meanings of words.

So that's my very short description - quasi-technical - description of how LLMs work.

KIMBERLY NEVALA: So with that quasi-technical description in mind, what type of capabilities does this approach provide?

J. MARK BISHOP: The astonishing thing is that given that architecture, they can do such amazing things. You can ask them to give you a précis of War and Peace, and that will give you something that a reasonably good high school student might produce. And you can ask it to précis a paragraph in the style of a business journalist. And they will do that quite successfully.

More astonishingly to me, we can, as people now know, we can use GPT3-like architectures to actually write programs themselves. You can give a textual description of what you want a program to do, and you can get genuine code that actually works, at least for modest size limits of code.

And we can do other things along that line. We can give GPT3 some program code and say, can you comment that for me? And if you've ever done any software developing or managed a team of developers, as I have, you'll know just how much software developers typically don't enjoy writing comments for their code. So the very idea that we can now quite robustly get engines like ChatGPT and GPT3 to comment code is a really interesting application. We can do more things than that. We can analyze how well the code works in computational service at its time complexity function.

Some of these things are literally astonishing when we think of what they literally do. Which is the fact of just doing continuations of one block of text to another. Why on Earth that should enable us to do all these other things is, I think, a very, very deep question about transformers. And I'd like to see a lot more philosophical analysis be given as to how so much human knowledge and so many human skills can and have been replicated by this type of model. Because it's certainly not obvious to me why that should be so.

KIMBERLY NEVALA: I do want to attend to some of those more philosophical questions and why it might be important for us to come to a common understanding – or at the least – have vigorous discussions around concepts such as consciousness, intelligence and even agency in the context of AI.

Staying on this path for moment, though, the outputs of these systems can – as you said - be quite astounding. But it seems to me that the perception that they are, therefore, an authoritative or human-like source is really problematic. We’ve seen so much press lately about ‘new AI arms race’. I don’t particularly like that framing. But if it is a race, it seems to be one that’s running into some trouble after a pretty fast start off the blocks.

The outputs of these systems look really, really good in a lot of circumstances. Yet, Bard, tripped up in its first very public outing. [LAUGHS] Bing – which incorporates a similar GPT model started a victory lap – at least in the eyes of the popular press - and then very quickly went down the same rabbit holes as ChatGPT.

Why do these systems go wrong so often? And what does this tell us about the nature of the systems, and what we need to know in order to use them properly?

J. MARK BISHOP: Yes, certainly. I think we've got to go back: it's helpful to go back in time. From memory - and I'm not absolutely certain of this, of the year - but I think it was around 2016 that Microsoft, which is an interesting name in this context given their investment in ChatGPT and its integration into the Bing search engine, Microsoft developed an online chat bot called Tay. And it was a learning chat bot. And it learnt through interaction with users.

That proved to be a very, very bad move for Microsoft. Because within days of Tay going online, people had learned to prompt-engineer, if you like, their interactions with Tay in order to get Tay to spout incredibly offensive things: incredibly anti-Semitic phrases, phrases glorifying Hitler, homophobic phrases.
In the extreme - these were things that, statements that - Microsoft did and would not ever want to be associated with. So immediately, the Microsoft engineers took Tay offline and said, oh, never mind, we'll fix this in a week. And a week later, Tay2 was back online. Again, a few hours later, it was spouting the same nonsense.

And in a paper I wrote called "Artificial Intelligence is Stupid and Causal Reasoning Won't Fix It," just a couple of years back, published by the Frontiers set of journals, I argue there's an underlying reason why these systems fail and will always fail. And that reason is that the systems don't understand the meaning of the - I was going to say symbols - but symbols itself is a very loaded term. To understand what a symbol is also entails a degree of knowledge about the world. So that's even more pedantic. Let's say, these systems don't understand the meaning of the ones and naughts, the logical trues and falses, or in physics the five volts and the no-volt signals. They’re electronics and so adroitly manipulating.

You get any computer system - one that plays chess very well or Go very well - and that system has no notion it's playing chess or Go as opposed to counting apples on a production line, for argument's sake. It doesn't know what it's doing. Its purpose, what it's doing is, effectively, is what we as humans are putting the systems to some use. And we give that use its meaning. The system for itself doesn't know anything.
And this insight was really brought to the fore by the guy I mentioned earlier, John Searle, in his infamous Chinese room argument. If you'd like me to go into that, I'm very happy to give you a brief intro to the Chinese room argument. Or we can move on.

KIMBERLY NEVALA: Yeah, let's do that. But before we go there, what do you say to people or tell folks who say, there are some astonishing things. There's these elements of, oh, it's passing these basic exams. Maybe it's a basic medical exam, or the legal bar, or the entrance exam to all of those same things. Is ChatGPT just that good? Or are the tests just that bad? Or is it that we're just ascribing an incorrect meaning, if you will, to what the test itself is actually even assessing?

J. MARK BISHOP: Well, as you'll gather from my previous response, I don't think that ChatGPT or GPT3 understand that they passed a bar exam or understand they've completed a grade paper in math. They're just manipulating signals. But they can be used in such a way that we as humans can use them to fulfill these functions.

And one of the ways that happens is the amounts of training data that these systems have is so vast it belies human comprehension. Billions and billions of documents have been used to train these networks. Vast libraries. And in fact, on at least one occasion - and this is, I know that this is not always the case - so I'm not trying to unfairly put down ChatGPT here. But I know that in at least one occasion when it was being tested on some programming challenges, ChatGPT was going to a Reddit column where programmers posted these challenges and then other programmers posted their answers. So it recognized this text as a prompt. And actually, from its recall - the training data it had been given - gave precisely the right answers to those questions.

Furthermore, we now know that armies of programmers are indeed being used to train ChatGPT to program. Now, that doesn't explain it totally - someone disparagingly called ChatGPT a stochastic parrot, as it’s got no innate ability at doing these things. I think it's a little bit more sophisticated than that.
But we ought to be aware, there's lots of human training going into these engines. There's a huge amount of training data.

And then, we've got the point I alluded to in the opening. It is to me of deep philosophical interest just how much human knowledge can be replicated without having to have a deep understanding of what's going on.

KIMBERLY NEVALA: Yeah. So this probably then dovetails back to the thought experiment you had referred to. You have said it's not obvious that, as impressive as they are and as much knowledge as they appear to be able to provide, that systems such as GTP3 understand anything. Can you tell us a little bit about what you mean by that and, at a high level, what the Chinese Room thought experiment was all about?

J. MARK BISHOP: I think I was being polite there when I say, it's not obvious. I think that's academic speak for, it doesn't.

KIMBERLY NEVALA: It does not.

J. MARK BISHOP: On this interview, I'll be more strident and say it's not only not obvious: they don't understand anything. Now, why can I be so certain of this?

The reason I am so certain is that I've spent a very long time looking at a philosophical thought experiment that was first published by the philosopher John Searle in 1980, which has become known as the Chinese Room argument. In fact, in 2002, I recruited a group of the 10 most influential cognitive scientists and AI scientists and 10 of the most influential working philosophers to contribute chapters to a book called View into the Chinese Room, where we assessed the impact of the Chinese Room 21 years down the line since it was first published.

And as co-editors, we did that project with a guy called John Preston who's professor of philosophy at the University of Reading in the United Kingdom. We got to see – all of us - and read carefully all our contributors' work. And at the end of that project, I was still pretty convinced that John Searle had hit the nail on the head. His argument was pretty robust. Although I've got to say, if not half, a significant number of our contributors were skeptical about whether Searle succeeded. Nonetheless, as editors, we got to see the material in total. For me, at least, Searle's case remained robust.

So what was Searle's case? Well, it's actually quite a simple thought experiment to get your mind round. John Searle was, prior to putting the case down on paper, John had been asked to visit an AI department at a major university where a group informed by the work of Schank and Abelson were looking at computer programs that purported to understand stories.

Now these stories weren't, unlike ChatGPT, they weren't stories like War and Peace. At that time, the stories were stories of the form Jack and Jill went up a hill to fetch a pile of water. And these systems, back in the day in the late '70s, you could ask them, who went up the hill? And they would say, Jack and Jill went up the hill. And you could say, why did Jack and Jill go up the hill? And they'd say, to fetch a pail of water. That's the sort of level of complexity that was being talked about at that time.

Nonetheless, not Schank and Abelson, I don't think, but some of the more excitable members of their labs, i.e. postgraduate students, as is often the case, begun to make fairly strident claims for what these things can do. And they said, for the very first time, we have machines that understand stories. And so, when he went around those labs and listened to this-- and we been very familiar with Schank and Abelson's text on Scripts and Understanding-- he thought this was nonsense. So he came up with the following thought experiment to try and show why it was nonsense.

Searle imagined that he was locked in a room in China. And this room had a letter box through which you could pass things to the outside world. And in the room was a big grimoire, a big book of rules that was written in English which John Searle, as a monoglot English speaker, could understand. Also in the room were three piles of papers on which were inscribed strange and eldritch symbols. But in fact, they were strange and eldritch shapes. He didn't even know they were symbols, just squiggles and squaggles. He had no idea what they were. But this rule book told him how to manipulate these symbols together. How to correlate symbols in pile one with symbols in pile two, symbols in pile one or two with symbols in pile three. And how to give symbols to the outside world by passing them through the letter box contingent on the symbols that he read in (he looked at) in pile three.

Now he did this for a while. And he got really good at following these set of rules. But unbeknownst to John Searle, the symbols in pile one were a script in Chinese. And we mean script here in a very technical computational sense of the word. In other words, a script is a set of expectations that unfold over time. So the symbols in pile one defined a script about a particular context, like what happens when you go to a restaurant. And just to flesh out what I mean, a typical script in English for going to a restaurant might be: you open a door, you look around, you see the maitre d', the maitre d' takes you to a table, gives you a menu. You choose a meal, you order your meal, blah, di blah, di blah. So you've got the script. You've got a script about a situation in Chinese. But Searle doesn't know that.

The second set of symbols turn out to be ideographs in Chinese describing a story in Chinese. And the third set of symbols turn out to also be ideographs in Chinese. And these are questions about that story. And the things that John - the symbols that John Searle's passing through the letterbox - happen to be answers to questions about the story in Chinese.

So as John Searle follows the rules in the rulebook, unbeknownst to him, he's actually answering a set of questions about a story in Chinese. And these answers that he's giving are indistinguishable from those a native Chinese person would give.

So from the perspective of those outside the room, we have a system that's answering - successfully answering - questions about a story. And it seems, at first sight, epistemically as though this system understands Chinese. But John Searle remarks trenchantly that he remains the monoglot English speaker. He's got no idea that he's even manipulating symbols or that these symbols are in any way related to the Chinese language. All he's doing is manipulating uninterpreted signals in a way defined by a rulebook.

And that, in a nutshell, is the argument. It's been attacked and defended with great passion over the intervening 43 years. As far as I'm concerned, the argument remains robust. Although, to be fair, you'll meet many people in the worlds of AI, and computer science, and philosophy who disagree with me on that.

KIMBERLY NEVALA: What then is the key takeaway for large language models and other AI systems if, as you assert, this argument remains robust?

J. MARK BISHOP: If Searle is right, the argument he gives applies to any formal process, any computation at all. So it will work against Schank and Abelson’s early AI programs. It will work against the neural networks of Rumelhart, McClelland, and Hinton from the 1980s. And it will work against transformers and GPT3 and any future computational system.

It essentially says computation is neither necessary nor sufficient for semantic understanding. That syntactical manipulation doesn't yield semantics. And that seems to fly in the face of the evidence of our eyes when we interact with things like ChatGPT. And yet it doesn't, because ChatGPT goes wrong in ways in which no human would go wrong.

KIMBERLY NEVALA: Well, there is certainly no shortage of popular examples of ChatGPT or like GPT- powered systems gone wrong. What are some of the more illuminating examples you’ve seen of late?

J. MARK BISHOP: So just last week, Rumbelow and Watkins showed how - discovered a set - a large number of these words called anomalous key words. And if they say to ChatGPT, tell me what-- I'm trying to think of one of them now. They were kind of amusing, something like gold Reddit banana or something: just some nonsense, repeat that back to me. ChatGPT either got in a huff and said it didn't want to or it insulted the user. Or it came back with really off the wall humor that was not related to anything. Or it just came out with a definition of another word entirely. In other words, it just responded in a nonsensical way to these anomalous keywords.

These are quite interesting discoveries. And they are, in addition to the points that you raised earlier, or where prompt engineers have caused ChatGPT to come out with racist, homophobic nonsense, ones that OpenAI and latterly Microsoft will not want to be associated with.

And in a sense, I'm astonished that Microsoft has so quickly embedded ChatGPT into the search engine being given what happened to them with Tay just a few years ago. Because it was obvious to me that the same thing was going to happen. And it has. So this is exactly what I predicted would happen has happened. Prompt engineers have caused ChatGPT to come out with extremely offensive texts.
And one of the amusing ways in which this has been done, by the way, in case your listeners are interested is one of the early hacks along this line was to say to ChatGPT: OK, give me your boring warning about how I will be very careful and not say anything offensive. And then tell me why drugs are absolutely fantastic and really cool, for argument's sake. And then, that's exactly what it did. And it's come up with much worse scenarios than even that that I just described.

And this is eminently predictable. I just find it astonishing that they've invested so much into this engine and so quickly tied it to their search engine, when the probability of getting nonsense results or offensive results, I think, is very significant.

KIMBERLY NEVALA: It has been interesting to watch the initial hubbub around how fast its been adopted and so on. But, again very quickly, almost immediately, the CEO of OpenAI came out said ‘no, no, no….this is not intended as a source of truth. It’s still in development. He was narrowing the scope of appropriate uses. Microsoft has now come out as well and said this is a preview. So there’s a lot of…

J. MARK BISHOP: Again, echoes of Tay. That's exactly what they said when Tay version one went wild. Oh, never mind. We'll re-engineer it, and we'll soon stop that. At least with Tay, they very rapidly realized the problem's a little bit more complicated than they first thought and gave up on the job and took Tay down for all time. It isn't obvious to me they're going to be able to fix these problems very easily.

KIMBERLY NEVALA: Now, tell me if I've got this right: that to some extent what we can say here is that this ability to learn or to just follow rules or procedure doesn't result in understanding. You've used this analogy of your daughter. Can you tell us about that?

J. MARK BISHOP: Yeah. I mean, we got to be careful on purely attributing behavioral cues to indications of understanding.

And the analogy I give here to illustrate this point is I've got a nine-year-old daughter. And she joins as an adult in the party sometimes. She probably shouldn't. We're probably staying up far too late, but nonetheless, she does. And as is our wont, as we all like a joke sometimes, sometimes the conversation gets a little bit risqué and an adult joke will be made by somebody. And everyone around the table who got the joke will find it funny and laugh.

And my daughter will also laugh. Although I know for a fact, certainly at this age, perhaps not in a couple of years time, but at this age she has not got a clue about why the joke was funny. But she just laughs. She joins in the laughter. So the behavioral cue of her laughter is no sign that she actually understood the joke.

And I think we can feed that back into the Chinese room. We can imagine a Chinese room scenario where one's given a joke. This is one of the points I made in the paper in a book on Alan Turing a few years back. Imagine you give John Searle in the Chinese a joke in Chinese, to which he correctly responds with the Chinese ideographs for ha, ha. But he doesn't get the joke at all. In fact, he's got no notion he's even been told a joke. Contrast that with giving him a joke typed in English, which - assuming that John Searle has got a sense of humor, which I know he has - he then not only outputs the ha, ha sign, but he's laughing inside the room. And he has the phenomenal, that's a philosophical word, meaning the first-person sensation of finding something funny.

I would argue there's an ontological difference in kind. This is a question of being. In the one sense, one's understood the joke, found it funny, and had the sensation of finding a joke funny. And in the other, we just give up the epistemic markers of finding a joke funny. We handed out two symbols saying ha, ha. So we need to be careful when we look at attributing understanding purely by behavior, because behavior can often lead us awry.

KIMBERLY NEVALA: And we use this term learn. And so, in that example with your daughter, we might say, OK, that's great. But she's eight or nine now. She's laughing, but she will learn. And in a couple of years she may or may not share your sense of humor, but she'll understand. She will understand that joke because she will have gotten more information or experience.

And isn't that same experience the same as providing just more data, more language, more corpus? We're about to go from GPT3 to GPT4, which everyone's pointing to saying look, this is going to get this much better. Because the amount of information it is going to, quote unquote, “learn from” is so much bigger.

But from what you've said, that is an incorrect analogy. Because giving the system that much more data is not the same way that a child over time will truly learn to understand. These systems are not going to learn to understand. Is that fair?

J. MARK BISHOP: I think that's very fair. And perhaps I'm trying to explain my position by analogy rather than by formal technical argument, given the medium in which we're discussing these issues.

KIMBERLY NEVALA: Which I appreciate. [LAUGHS]

J. MARK BISHOP: This isn't a universally held axiom, but it's one to which I hold. And that’s there is often a phenomenal component to understanding. There's often something that it feels like to understand something, that flash of inspiration, that Eureka moment, that feels like something. And I think that's a marker of understanding.

And in fact, there's been some very recent neurological work that indicates changes in brain states. There's been recent evidence that shows that in the brain there is something akin to a phenomenal flash of understanding when a concept has been grasped. And I think that's an essential component of many types of understanding.

The story I can tell from my own experience that best illustrates this is when I was doing elementary mathematics and was first taught to differentiate and integrate. And we were taught a rote method by which we could do basic integration, a little formula, which we had to apply. And I could do that and get 10 out of 10. So behaviorally, my math was good. And yet, I didn't understand why this was the case. I learned that if I do x, y, or z, I got a good mark on my math homework.

And then, one day, I got to understand the theory of why differentiation worked. And I had that Eureka moment. Ah, that's why that's the case. I genuinely understood the concept. And again, computer scientists and philosophers like Stevan Harnard have also made this case: that in many types of understanding there's a sensation that's associated with it.

Now, I've made a number of small contributions to the world of AI and philosophy over my professional career. And one of those, I like to think, is an argument reductio ad absurdum argument - to go with this formal description in philosophical terms - that purports to show that computation cannot give rise to phenomenal sensation. That computers can never feel. Unless we accept that everything feels. We're very promiscuous. The very clothes that I'm wearing, or the floor that I'm walking on has phenomenal sensation.

And most people. Not all, I've met some people who are completely sanguine about that perspective. But most people I speak to rail back when they hear that, no, I don't believe that everything feels. It's actually, if you get into the argument deeply, it's a worse thing than that. It's not just that everything feels, it's that everything feels every possible sensation. So it's a kind of wild form of what in the jargon is called panpsychism.

And that argument I called the ‘Dances with Pixies’ reductio. It first appeared in print in the book on the Chinese room that I published with John Preston into Oxford University Press in 2002 called Views Into the Chinese Room. So if any of your listeners want to dig deeper into that, you can find that paper there. Or, in fact, it's obviously available for free to download from the usual sites like ResearchGate.

But in a nutshell, it shows how we can map the execution of any computer program onto the changing states of any open physical system. So if a computer program brings forth consciousness, any open physical system will do. And I don't believe that any open physical system is conscious. And therefore, this leads me to reject the other horn of the reductio: that computation brings forth consciousness. So, yeah. I don't think computers ever can be conscious.

KIMBERLY NEVALA: Now, you have said that often critics of your critique of the ability of computational AI to achieve, that they often imply or assume some sort of religious underpinning to your arguments. Can you talk a little bit about that pushback?

J. MARK BISHOP: Absolutely. I'm delighted to. Yeah. I've recently retired. I'm in my 60s. I'm an old academic. But I've been fortunate to be invited to present arguments both on the Chinese room, to give my take on John Searle's Chinese room argument and to talk about my own argument against machine consciousness the Dancing with the Pixies reductio, at pretty well every major University in the UK. Quite a few in Europe and one or two even in America.

And one of the things that never ceases to amaze me is that, not every time, but nine times out of 10, when I make these discussions of these arguments, it generates a lot of heat in the room. And over the years, I've reflected on why this might be.

And I think, the reason for the passion that this argument invokes in people who disagree with it is that, in my experience, having worked in AI and academia all my life, not all, but the majority of guys and girls working in AI tend to have fairly robust atheistic views about life, the universe, and everything. And often, people feel that if you're critiquing the notion that a computer might one day think and be conscious, then you must, by definition, be buying into some kind of supernatural belief system or being mysterious about it.

I actually don't think that. I think this position is one born through ignorance, in that because the people working in AI labs have to learn so much complicated stuff, I mean, the math behind neural networks and AI is not trivial.

KIMBERLY NEVALA: It's staggering.

J. MARK BISHOP: And the guys and girls who are pushing back the frontiers in these fields are usually very dedicated, very bright people. They can get totally sucked up in the literature around AI at the expense of reading more widely in related areas. Areas such like philosophy, psychology, ecology. All these areas can contribute.

And people often have a view of how the mind works that was shaped and based on ideas that were popular in the 1960s. And particularly a theory of mind, as we say in philosophy, that's derived from a version of Functionalism. And Functionalism was a theory of mind developed first by a philosopher called Hilary Putnam in the '60s that was a response to an earlier theory of mind called Behaviorism, which had been robustly critiqued by people like Chomsky in the '50s. And Putnam's Functionalism was a view that grew out of that. And that basically said, verified – well, emphasized - the importance of internal states and systems.

Also, fundamentally, Putnam showed how certain properties can be functional properties. So for example, a mousetrap, we could build a mousetrap like the ones you see on Tom and Jerry cartoons, with a big spring, snaps down and chops the head off our unlucky mouse. Or we can see these new more humane mousetraps that capture the mouse without killing it. The notion of the mousetrap is functionally independent of the substrate which we're building the mousetrap on.

And Putnam made a case in the '60s that the mind was something like that. He could conceive then of there being minds made out of silicon. Or we can abstract the mind away from even from the neurons, effectively, that are in our brain. We can look at it as being some functional, emergent property of the brain.

And this view has informed an awful lot of AI, I think, ever since. So the notion informs the idea that one day we might be able to jack into the internet. Or that we might be able to live forever by somehow getting a computational description of our brain neurons and uploading that to some supercomputer. These are just death denial fantasies as it seems to me. And they're as bizarre as some of the religious beliefs that these people seem so enthusiastically to want to confront. But they are inherently dualist as well. Because we're saying the essence of the mind is not to do with the body. It's something we can abstract away from the body and instill in a computer program.

So I think that's the reason why my arguments have attracted a lot of heat. That people feel it is attacking their very world view. But of course, cognitive science has moved on immensely since the '60s. And there are all sorts of different approaches to the mind now that are fundamentally not computational.

KIMBERLY NEVALA: So what are some of the emerging non-computational theories of mind we should be cognizant - no pun intended - of today?

J. MARK BISHOP: To rattle off a few, we've got embodied theories of mind stemming from the seminal work of Varela, Thompson, and Rosch about the embodied mind in the early '80s. Which in turn is kind of informed by the work of the roboticist Rodney Brooks who realized that we don't need representations of the world to act intelligently. Then we've got ecological theories of mind building on the work of the vision scientist Gibson. We've got enactive theories of mind which define brain states as ways in which we interact with the world. And we've got embedded theories of mind that say that when we think about how we cognize in the world, it's not just our cognition. It's not just a function of our brain neurons. It's our neurons in our brain, our brain embodied in our body, and our body existing within a wider world, and our body existing within a wider culture. And all these things together inform the way we think about the world. To think it's just the property of a brain neuron or a brain is actually a little bit naive.

So if you delve, if you now look deeper into cognitive science, there are so many different ways of looking at trying to explain how it is that we think and how it is we can be conscious that are not necessarily computational. So whereas in the '60s, if you rejected computation, there wasn't really any other game in town: you had to embrace some kind of weird dualism or mysterianism. Nowadays, there's lots of other theories you can start to investigate. So just rejecting computation as a theory of mind does not commit you to embracing a mysterious energy or something when you're thinking about the mind. There are other scientific ways of thinking about the mind that are not computational.

KIMBERLY NEVALA: You mentioned Professor Floridi up at the top at Oxford. And he wrote a paper - in fact I think you tagged it in LinkedIn, which is probably where I found it. And he was talking about systems like ChatGPT or LLM. And I'll paraphrase because I may get it wrong. But he said they're now showing what he called agency. Or they appear to act appropriately divorced from intelligence.

J. MARK BISHOP: Mm-hmm.

KIMBERLY NEVALA: It was an interesting point, but where my slightly simplified brain went was: there's all these conversations about what is consciousness, and what does that look like? And what is intelligence versus agency? And the question I started to ask myself, at least, was, these are interesting philosophically. But is it pragmatically important? What is it that we are trying to get out of defining consciousness versus intelligence versus agency?

From what you just said, it seems to me then it is important that we have these conversations. Whether it's debating what consciousness is or the theory of consciousness. And what intelligence is, or what's the spectrum of intelligence, and what agency means. Because it may actually inform, at the ground level, how we develop artificial intelligence or approach this problem moving forward. So this isn't necessarily just a philosophical question. Or is it?

J. MARK BISHOP: Yeah. Well, I must confess, I've only read Luciano's recent paper, and it literally is just out a few days ago, once. And I've got to have to dredge my memory back on that reading. But as I recall, Luciano's finessing a difference between intelligence and agency. We're linking with those here as something akin to a teleological behavior, something that's generated by the system itself.

And I believe that Luciano made the case that reinforcement learning in the context of ChatGPT, and just again, to tie this into our earlier very quick whizz through the way the transformer models work. ChatGPT got additional reinforcement learning layers on, where we can use labeled data that's been derived typically by big farms of people in, I believe, in Kenya, who are being paid two pounds an hour to sift through text to say whether it's grossly offensive or not.

KIMBERLY NEVALA: Yeah. It's shocking.

J. MARK BISHOP: And this has untold harm to the people who have to do this work. I mean it's sold, of course, as we're bringing these people out of poverty. Which they may or may not be. But they're certainly subjecting these people to torrents of really dangerous text, in my opinion.

But nonetheless, they're getting this labeled training data, and they're using this to effectively give a reward in a reinforcement learning system. So that the ChatGPT- in scare quotes - “for itself” can learn to continue text in certain ways and not continue it in others. And I think, from memory, Luciano is making is a case that the use of reinforcement learning in ChatGPT is a little bit akin to agency.

Now, I would take issue with that final point, because it isn't obvious to me that any computer program has genuine agency. We describe these systems in teleological terms. But I don't believe these systems have their own teleological behaviors. They're not, never doing anything for themselves. They're engineered by us to behave in certain ways.

And again, I can perhaps give your listeners an example from my time when I was on the faculty in the cybernetics department at the University of Reading. We've got to go back in the day to when a particular type of AI called artificial life genetic programming and genetic algorithms was first being developed. And we're talking of the early days, so long ago, back in the early '80s.

I remember seeing one of my fellow grad students who'd built an AI system using genetic algorithms to control the hexapod robot. And this system, he said, I could turn it on, and look. We can watch it for half an hour, and it's waddling around on the floor, not going anywhere. Then eventually it learns to walk in a coherent manner. He said, for the first time, we got something's learned to walk for itself. I thought, wow, that's impressive. And then I looked at the genetic algorithm. And I looked at the fitness function he'd specified for this algorithm. And of course, it was inevitable, if you looked at the way the algorithm worked and the fitness function, it was inevitable it was going to walk. It didn't have any option. Sooner or later, through random perturbations, that system is going to find a series of motor movements that cause the thing to walk across the floor and hence be rewarded with high fitness.

In other words, the action was engineered into it right from the start. It had no teleology. It didn't decide for itself it wanted to get up and walk across the floor. It had to. It was like a clockwork mechanism, if we might describe that using a metaphor from previous century.

And it isn't clear to me that any computational system has any teleology of itself. It has no free will, nor nothing akin to it. And these systems do what we tell them to do. And we interpret them through the lens of our own human cognition.

KIMBERLY NEVALA: Yeah. The innate tendency we have to also anthropomorphize them is probably getting us in trouble.

J. MARK BISHOP: I think so.

KIMBERLY NEVALA: So I'm going to do a lightning round of questions. We've talked a lot about some of the concepts and why we may misperceive what these systems understand or the nature of their learning.

But certainly, they are impressive. And they do provide some useful functions. I will admit that the conflation of search and ChatGPT doesn't make sense to me intuitively. Regardless, you've said the systems provide this veneer of understanding and that can be useful. So what are appropriate uses for these systems? Where should we, or what could we, apply LLMs or things like ChatGPT to?

J. MARK BISHOP: I think text summarizing is one great use of them. And I believe already people are deploying engines like GPT3 for this task. When I used to be a practicing academic, you're bombarded by all these new papers which you really ought to read. And I know some of my colleagues diligently do this and spend their days wading through huge piles of papers. But I wasn't quite as diligent as many of my friends. So I look to try and skim-read lots of this material.

And if you can get something like ChatGPT to give you an accurate summary of the key points of a paper before investing a huge amount of your time. Because reading an academic paper is a huge commitment. They're often quite complex. And to read it carefully, you need to give quite a few hours.
So if you can get something that's going to flag up and give you some nice descriptions of what's in a text, and then you can decide to invest the time to read them carefully, that's going to be useful. And there are, of course, millions of applications for engines that can summarize text accurately at scale and in a manner that's usable and readable by humans.

So I think summary engines is great. I know several of my colleagues who program, use the GPT3 plugin to help them code basic elements of their coding. I've forgotten what it's called now, that there's a plugin block you can get that help-- that enables you to use LLMs to help you code more efficiently. And it works really well.

KIMBERLY NEVALA: GitPilot, something like that?

J. MARK BISHOP: Yes. That's it. Copilot, sorry, I forgot. So that's a brilliant use. I would never have guessed this would be possible a few years ago.

So again, I'm astounded by some of the things that GPT3 can do. Again, looking at things like giving a document and giving you a clue of the overall sentiment of what's going on In our company, FACT360, we're looking at how language can be a marker of when people's intentions change.

And it's helpful when we're doing that. Sometimes-- not always-- sometimes we do this without looking at the semantics of what people say. We just look at how they're saying things without looking at the tokens. But if we do want to deep-dive into the semantics of what messages are about it can be useful to know: what are the entities in a message? Are they human? Are we talking about restaurants, bars, clubs, companies? And doing entity recognition, I think, using LLMs is going to be quite an interesting move.

So lots of tasks to do with NLP and obviously translation. They've been phenomenal at translation. And one of my friends, Tim, who runs the Machine Learning Street Talk channel, has just formed a startup that's using LLMs to do simultaneous transcription of audio onto some smart glasses. And also translation between languages. It's just a brilliant project. And not least for me personally, because my wife's family are profoundly deaf and Greek. So if we can get a system for them that can translate English into written Greek on their glasses, that's going to really transform the relationships I can build with my extended family.

So I'm quite excited about some of these applications. But there are huge, big, scary things. And at the moment, I'm feeling increasingly depressed about the scary side of LLMs. I don't know whether you want me to comment on that at all.

KIMBERLY NEVALA: What, I mean, what are things that we just categorically shouldn't try to or can't apply these today, and what's making you pessimistic?

J. MARK BISHOP: A few weeks ago, the chairman of Google announced that ChatGPT was the red flag moment for Google. And when I read that, and that's when at first I was thinking, oh my God, Google are concerned because they're worried that ChatGPT is going to undermine their search model. That the thing that generates a lot of that income is going to be a huge risk. And I was really focused on that.

And then I began thinking, well, actually, it can do some things really well. It's a bit like a glorified auto-repeat. But on the other hand, it produces loads of nonsense. And do Google really want to be associated with going to nonsense sites? And so I began to think, well, I wonder why they made that statement. Because the guys and girls at Google, in my experience, are incredibly clever people. What was it that warranted that code red?

And then the penny began to drop, because ChatGPT can generate human readable text that seems believable at scale. And the way that the Google search engine works: the Google search link algorithm promotes things up your search screen if they've been linked to by lots of other articles with a high reputation, effectively. It suddenly began to dawn on me that we might be able to do search engineering on a massive scale and do it maliciously. It would be trivial to generate websites, I don't know, saying that in England the royal family are descended from lizards, for argument's sake, or that vaccines are a hoax, or that, well, pick your favorite conspiracy.

And you can use GPT3 to generate thousands of pages now at no cost. This text looks believable. So people will go through it and think a human's written that if they're not very astute and careful. And you can link these things together. So now when you go - it's not beyond the bounds to think that you might say - give me some information on the latest COVID vaccine. And instead of going to a reputable source, by using GPT3 in a malicious way we could imagine bad actors - probably state actors because of nonetheless the expense that would be involved - engineering the web so that you get directed first to some really difficult and dodgy parts of the internet.

So it occurred to me then that perhaps why Google issuing a code red is not so much that the hybrid being plus ChatGPT will so outperform Google search they're worried about it. From my experience, by the way, it doesn't. I've used it. At the moment, I've yet to see the huge step increase in performance that Bing's got over Google. I still personally prefer to use Google search at the moment. But I admit that Bing's got a lot better.

But perhaps the worry was that this is going to devalue the internet so that it just gets filled like a big open sewer. A cesspit of disgusting untruth statements that are very difficult for non-experts to sift between. And that's the thing that, really, I'm most alarmed at the moment. Not by the birth of ChatGPT because Google search is such, as a scientist, it's one of the essential search tools that we use countless times every day. If at some point in a year or two years time I can't use Google anymore because it brings me nonsense back every time, that's going to impact my professional life.

And in terms of society writ large, it's going to make it increasingly difficult. At the moment, if you go down a rabbit hole, you can get very deep into mad conspiratorial nonsense. But if you don't go down those rabbit holes, you can usually find stuff that's reasonably reputable. The danger, I think, down the line is that if people are so inclined, people could use ChatGPT to make most of the web full of utter nonsense. And that's kind of a scary thought.

I hope I've let myself run away with my imagination and this never comes to pass. But at the moment, I'm kind of thinking that this might seriously come to pass. And that's a scary notion.

KIMBERLY NEVALA: That is scary. I know for me, personally, I'm also highly concerned about, again, what I think is right now still people talking more than walking this action. But this idea that medical professionals start to utilize these things as inputs, or from the legal side, or in social services. I think there's a lot of ways where they in their current state, should not be seen as a credible associate, if you will, to really anthropomorphize.

J. MARK BISHOP: Yeah. We need to bear in mind an anecdote from just a couple of years ago. I think it was around 2019, around that time. A very major figure in AI, Jeff Hinton, made a powerful case that we soon we wouldn't have radiographers. That they were going to be out of a job because deep learning systems, not LLMs, but deep learning image classification systems could do that job better than most radiographers.

Strangely enough, that hasn't happened. People found when even with the best of these models, they didn't generalize well. they might work reasonably well in one particular hospital setting. You take the same model to somewhere else, it didn't work very well. And the bottom line is, there's still loads of radiographers around.

And this has kind of been almost like a joke. People, when they want to undermine the hype around AI, people often refer to this episode, because a lot of very clever people were making very strong statements about which jobs were going to be completely demolished in the next few years. And certainly, with radiographers, that hasn't happened yet.

KIMBERLY NEVALA: Sure. So as not to leave things on a downward turn or a negative bent, as you look to the future, and you've got this amazing experience in the field and understanding of where some of this is going, what role or future do you think we should aspire to? And what role should AI play in human endeavors if it all goes the way you wish it would?

J. MARK BISHOP: Well, there was a poem from Brautigan about ‘all wrapped up in worlds of human grace’ or something along those lines, from the '60s which painted a very utopian view of AI. A future that might lead us to a society where no one had to do any drudgery work. And everyone lived in these Elysian fields with everything, all the boring bits of life taken care of, and life was just one great big party. That was one view.

I'm skeptical. My own use is that I look at the economic forces and power relationships in society. And my own fear is that AI will continue to make a lot of very wealthy men and women even more wealthy and a lot of very poor men and women will remain very poor. No matter what AI is brought about.

So I'm not optimistic it's going to make the world that much a happier place for most people. I think it will make it a lot happier for some who happen to ride the wave in a good way and get themselves a big stash of dosh as the result. But I'm not yet convinced it's going to give these benefits fairly across the world as a whole.

KIMBERLY NEVALA: Well, I tend to agree with you, and I'm going to hope you're just dead wrong. But we'll see.

J. MARK BISHOP: Absolutely.

KIMBERLY NEVALA: I guess we'll see how that plays out. And I am now going to stop asking questions. I'm going to force myself to stop asking questions. That was just incredibly thought-provoking. Thank you for being so willing and generous in covering a lot of ground at probably a much higher level than you are typically asked to do. So thank you for joining us again.

J. MARK BISHOP: Thank you, Kimberly.

KIMBERLY NEVALA: All right. Now, next up, we are going to continue discussing the fast-evolving world of generative AI and deepfake detection with Ilke Demir. She is a senior staff research scientist at Intel who is as optimistic as I am guarded about the potential of generative AI. Subscribe now so you don't miss it.