Full interview: "Godfather of AI" shares prediction for future of AI, issues warnings

[00:01] the last time we spoke two years one

[00:03] month ago. I'm curious how your

[00:06] expectations over these two years have

[00:08] evolved for how you see the future. So

[00:12] AI has developed even faster than I

[00:14] thought. Um in particular they now have

[00:18] these AI

[00:19] agents which are more dangerous than AI

[00:22] that just answers questions because they

[00:24] can do things in the world. Um, so I

[00:27] think things have got, if anything

[00:29] scarier than they were before. Um, I

[00:31] don't know if we want to call it AGI

[00:33] super intelligence, whatever, very

[00:35] capable AI system. Do you have a a

[00:38] timeline in mind for when you think

[00:40] that's coming? So, a year ago, I thought

[00:44] it was there's a good chance it comes

[00:46] between five and 20 years from now. Um

[00:50] so I guess I should believe there's a

[00:51] good chance it comes between four and 19

[00:53] years from now. Um, I think that's still

[00:57] what I guess. Okay. Which is sooner than

[00:59] when we spoke because you were still

[01:01] thinking like 20 years. Yeah. Um, I

[01:03] think it may, you know, there's a good

[01:05] chance it'll be here in 10 years or less

[01:07] now. So, in 4 to 19 years, we reached

[01:10] this point. What does that look like?

[01:14] So, I don't really want to speculate on

[01:16] what it would look like if I decided to

[01:18] take over. There's so many ways it could

[01:20] do it. And I'm not even talking about

[01:21] taking over. We can talk about that. I'm

[01:23] sure we will talk about that. But

[01:25] putting aside that kind of takeover just

[01:28] like a super intelligent artificial

[01:31] intelligence like what what kind of

[01:33] things would is this capable of or would

[01:35] be doing? So the sort of good scenario

[01:39] is we would all be like the sort of dumb

[01:41] CEO of a big company who has an

[01:43] extremely intelligent assistant who

[01:46] actually makes everything work but does

[01:48] what the CEO wants. So the CEO thinks

[01:50] they're doing things, but actually it's

[01:52] all done by the assistant and the CEO

[01:54] feels just great because everything they

[01:55] sort of decide to do works out. That's

[01:58] the good scenario. And I've heard you

[02:00] point out a few areas where you think

[02:02] there's reason to be optimistic about

[02:06] what this future looks like. Yes. Yeah.

[02:08] So why don't we take each of them? So

[02:11] areas like healthcare um they will be

[02:15] much better at reading medical images

[02:17] for example. That's a minor thing. Um I

[02:20] made a prediction some years ago they'd

[02:21] be better by now and they're about

[02:23] comparable with the experts by now. Um

[02:26] they'll soon be considerably better

[02:27] because they'll have had a lot more

[02:29] experience. One of these things can look

[02:31] at millions of X-rays and learn from

[02:32] millions of them and a doctor can't. Um

[02:36] they'll be very good family doctors. So

[02:39] you can imagine a family doctor who's

[02:40] seen a 100 million people including half

[02:44] a dozen people with your very very rare

[02:47] condition. They'd just be a much better

[02:49] family doctor. A family doctor who can

[02:51] integrate information about your genome

[02:53] with the results of all the tests on you

[02:55] and all the tests on your relatives um

[02:57] the whole history and doesn't forget

[02:59] things. That would be much much better

[03:01] already.

[03:02] um AI combined with a doctor is much

[03:06] better at doing diagnosis in difficult

[03:08] cases than a doctor alone. So we're

[03:11] going to get much better healthcare from

[03:12] these things and they'll design better

[03:14] drugs too. Uh education is another

[03:17] field. Yes, in education we know that um

[03:21] if you have a private tutor you can

[03:23] learn stuff about twice as fast. Um

[03:26] these things eventually will be

[03:28] extremely good private tutors who know

[03:30] exactly what it is you misunderstand and

[03:33] exactly what example to give you to

[03:35] clarify it to you so you understand. So

[03:37] maybe you'll be able to learn things

[03:39] three or four times as fast with these

[03:40] things. Um, that's bad news for

[03:43] universities but good news for people.

[03:45] Yeah. Do you think the university system

[03:47] will survive this period? I think many

[03:50] aspects of it will. I think it's still

[03:52] the case that a graduate student in a

[03:55] good group in a good university is the

[03:57] sort of best source of truly original

[03:59] research and I think that'll probably

[04:01] survive. You need a kind of

[04:03] apprenticeship.

[04:04] Some people hope this will help solve

[04:06] the climate crisis. I think it will

[04:08] help. Um it'll make better materials.

[04:12] We'll be able to make better batteries

[04:13] for example. Um I'm sure AI will be

[04:15] involved in designing them. Um, people

[04:18] are using it for carbon capture from the

[04:21] atmosphere. I'm not convinced that's

[04:23] going to work just because of the energy

[04:24] considerations, but it might. In

[04:26] general, we're going to get much better

[04:28] materials. We might even get room

[04:30] temperature

[04:31] superconductivity, which would mean you

[04:33] can have lots of solar plants in the

[04:35] desert and we can be thousands of miles

[04:37] away. Uh, any other positives we should

[04:40] tick off? Well, more or less any

[04:42] industry it's going to make more

[04:43] efficient because almost every company

[04:45] wants to predict things from data and AI

[04:49] is very good at doing predictions. It's

[04:50] better than the methods we had

[04:52] previously almost always. Um so it's

[04:56] going to make it's going to cause huge

[04:58] increases in productivity. It's going to

[05:00] mean when you call up a call center

[05:02] when you call up um Microsoft to

[05:04] complain that something doesn't work and

[05:06] you get a call center, the person in the

[05:08] call center will be actually an AI who

[05:10] will be much better informed. Yeah. When

[05:12] I asked you a couple years ago about job

[05:14] displacements, you seem to think that

[05:16] wasn't a big concern. Is that still your

[05:18] thinking? No, I'm thinking it will be a

[05:20] big concern. AI's got so much better in

[05:22] the last few years that I mean, if I had

[05:25] a job in a call center, I'd be very

[05:27] worried. Yeah. or maybe a job as a

[05:30] lawyer or a job as a journalist or a job

[05:32] as an accountant. Yeah. Any doing

[05:35] anything routine I think investigatively

[05:37] journalists I think will last quite a

[05:38] long time because you need a lot of

[05:41] initiative plus some moral outrage and I

[05:44] think journalists will be in business

[05:46] for a bit but beyond call centers what

[05:48] are your concerns about jobs? Well any

[05:50] routine job so a sort of standard

[05:52] secretarial job something like a

[05:54] parallegal for example those jobs have

[05:57] had it. Have you thought about what how

[05:59] we move forward in a world where all

[06:01] these jobs go away? So it's like this.

[06:04] It ought to be that if you can increase

[06:07] productivity, everybody benefits. Um the

[06:11] people who are doing those jobs can work

[06:12] a few hours a week instead of 60 hours a

[06:14] week. Um they don't need two jobs

[06:16] anymore. They can get paid lots of money

[06:17] for doing one job because they're just

[06:19] as productive using AI assistance. But

[06:22] we know it's not going to be like that.

[06:24] We know what's going to happen is the

[06:26] extremely rich are going to get even

[06:28] more extremely rich and the not very

[06:31] welloff are going to have to work three

[06:33] jobs. Now I think no one likes this

[06:35] question but we like to ask it this idea

[06:38] of p doom how likely it is and I am

[06:42] curious if you see this as a a quite

[06:45] possible thing or it's just so bad that

[06:49] even though the likelihood isn't very

[06:50] high we should just be very concerned

[06:52] about it. where are you on that scale of

[06:54] probability?

[06:56] So I think um most of the experts in the

[06:59] field would

[07:01] agree that if you consider the

[07:04] possibility that these things will get

[07:05] much smarter than us and then just take

[07:07] control away from us just take over the

[07:10] probability of that happening is very

[07:13] likely more than 1% and very likely less

[07:16] than 99%. Yeah, I think all the pretty

[07:19] much all the experts can agree on that

[07:21] but that's not very helpful. No, but

[07:23] it's a good start. It it might happen

[07:26] and it might not happen and then

[07:28] different people disagree on what the

[07:30] numbers are. I'm in the unfortunate

[07:33] position of happening to agree with Elon

[07:34] Musk on this. Um, which is that it's

[07:37] sort of 10 to 20% chance that these

[07:39] things will take over. Um, but that's

[07:41] just a wild guess. Yeah. Um, I think

[07:45] reasonable people would say it's quite a

[07:47] lot more than 1% and quite a lot less

[07:48] than 99%. But we're dealing with

[07:51] something we've got no experience of.

[07:54] Um, we have no real good way of

[07:57] estimating what the probabilities are.

[07:59] It seems to me at this point it's

[08:00] inevitable that we're going to find out.

[08:03] We are going to find out. Yes, we

[08:05] because um it seems extremely likely

[08:08] that these things will get smarter than

[08:10] us already. They're much more

[08:12] knowledgeable than us. So, GPT4 knows

[08:15] thousands of times more than a normal

[08:16] person. It's a not very good expert at

[08:18] everything and eventually it successes

[08:21] will be a good expert at everything. Um

[08:23] they'll be able to see connections

[08:25] between different fields that nobody's

[08:26] seen before. Yeah. Yeah. I'm als I'm

[08:29] also interested in in understanding okay

[08:31] there's this terrible 10 to 20% chance

[08:34] but or more or or more or less or less

[08:37] but let's just take as a premise that

[08:39] there's a 80% chance that they don't

[08:41] take over and wipe us out. So that's the

[08:43] most likely scenario. Do you still think

[08:45] it would be net positive or net negative

[08:48] if it's not the worst outcome? Okay, if

[08:51] we can stop them taking over

[08:54] um that would be good. The only way

[08:56] that's going to happen is if we put

[08:57] serious effort into it. But I think once

[08:59] people understand that this is coming

[09:01] there will be a lot of pressure to put

[09:03] serious effort into it. If we just carry

[09:05] on like now just trying to make profits

[09:07] it's going to happen. They're going to

[09:08] take over. Um we we have to have the

[09:11] public put pressure on governments to do

[09:12] something serious about it. But even if

[09:15] the AIs don't take over, there's the

[09:17] issue of bad actors using AI for bad

[09:19] things. So mass surveillance, for

[09:22] example, which is already happening in

[09:24] China. If you look at what's happening

[09:25] in the west of China to the weaguers, um

[09:29] the AI is terrible for them. I I to

[09:32] board a plane to come to Toronto, I had

[09:34] to take a facial recognition photo from

[09:36] our US government. Right. When I come

[09:38] into Canada, you put your passport and

[09:41] it looks at you and it looks at your

[09:43] passport. Every time it fails to

[09:45] recognize me. Um everybody else, it

[09:48] recognizes people from all different

[09:50] nationalities. It recognizes me. It

[09:52] can't recognize. And I'm particularly

[09:54] indignant since I assume it's using

[09:56] neural nets.

[09:58] You didn't carve out an exception, did

[10:00] you? No. No. It just there's something

[10:02] about me that it doesn't like.

[10:05] Um, I have to find some place to work it

[10:08] in. So, this is as good a place as any.

[10:09] Let's talk a little bit about the Nobel.

[10:11] Can you paint the picture of the day you

[10:13] found out? So, I was sort of half

[10:17] asleep. I had my cell phone upside down

[10:20] on the bedside table with the sound

[10:21] turned off.

[10:23] But when a phone call comes, the screen

[10:26] lights up and I saw this little line of

[10:28] light because I happened to be lying on

[10:30] the pillow with my head on this side and

[10:32] the it was here facing the phone rather

[10:34] than facing away. Just happened to be

[10:36] facing the phone. I saw this little line

[10:37] of light and I was in California and it

[10:40] was 1:00 in the morning and most people

[10:42] who call me on the east coast or in

[10:44] Europe. Yeah. You don't use do not

[10:46] disturb. No. No. Okay. Um I just I turn

[10:49] off the sound. I turn off the sound. Got

[10:51] it. And I thought I was just curious

[10:54] about who on earth is calling me at four

[10:56] o'clock in the morning on the east

[10:57] coast. This is crazy. So I picked it up

[11:00] and there was this long phone number

[11:03] with a country code I didn't recognize.

[11:05] And then this Swedish voice comes on and

[11:07] asks if it's me and I say, "Yeah, it's

[11:09] me." And they say, "I won the Nobel

[11:11] Prize in physics." Well, I don't do

[11:12] physics, right? So I thought this might

[11:17] be a prank. In fact, I thought the most

[11:19] likely thing was that it was a prank. I

[11:21] was aware that the Nobel prizes were

[11:23] coming up. Okay. Because I was very

[11:25] interested in whether Demis would get

[11:26] the Nobel Prize for chemistry and I knew

[11:28] that was being announced the next day.

[11:30] Okay. Um but I sort of I don't do

[11:33] physics. I'm a psychologist hiding in

[11:36] computer science and I get the Nobel

[11:38] Prize in physics. Was it a mistake?

[11:41] Well, one thing that occurred to me is

[11:43] if it's a mistake, can they take it

[11:44] back? So, but for the next couple of

[11:47] days, I did the following reasoning. So

[11:50] what's the chance a psychologist will

[11:51] get the Nobel Prize in physics? Well

[11:54] maybe one in two million. Now, what's

[11:56] the chance if it's my dream I'll get the

[11:59] Nobel Prize in physics? Well, maybe one

[12:01] in two. So, if it's one in two in my

[12:04] dream and one in two million in reality

[12:06] that makes it a million times more

[12:08] likely that this is a dream than that

[12:10] it's reality. And for the next couple of

[12:12] days, I went around thinking, you know

[12:15] are you quite sure this isn't a dream?

[12:17] You've walked me into this very wacky

[12:18] territory, but it is part of this

[12:20] discussion. Some people think we're

[12:22] living in a simulation and that AGI is

[12:25] not evidence, but hints toward maybe

[12:28] that's the reality in which we live.

[12:30] Yeah, I don't really believe that. I

[12:32] think that's kind of wacky. Okay, so

[12:33] let's put But I don't think I don't

[12:34] think it's totally nonsense. I've seen

[12:36] the Matrix, too. Oh, okay. Okay. Wacky

[12:38] but not totally. Okay. I thought here's

[12:41] where I kind of wanted to head with the

[12:42] Nobel. Um, I think you've said something

[12:45] to the effect of you hope to use your

[12:49] credibility to convey a message to the

[12:52] world. Can you kind of explain what that

[12:54] is? Yes. That um AI is potentially very

[12:57] dangerous and there's two sets of

[12:59] dangers. There's bad actors using it for

[13:02] bad things and there's AI itself taking

[13:04] over and they're quite different kinds

[13:06] of threat. And we know bad actors are

[13:08] already using it for bad things. I mean

[13:10] it's it was used during Brexit to make

[13:15] British people vote to leave Europe in a

[13:17] crazy way. So, a company called

[13:19] Cambridge Analytica was getting

[13:21] information from Facebook and using AI.

[13:23] Um, and AI's developed a lot since then.

[13:26] It was probably used to get Trump

[13:28] elected. I

[13:29] mean, they had information from Facebook

[13:32] and it probably helped with that. We

[13:34] don't know for sure because it was never

[13:35] really investigated. Um, but now it's

[13:39] much more comp competent and so people

[13:42] can use it far more effectively for

[13:44] things like cyber attacks. Um, designing

[13:47] new

[13:48] viruses. Um, obviously fake videos for

[13:52] manipulating elections. Um, targeted

[13:55] fake videos by using information about

[13:58] people to give them just what will make

[14:00] them indignant. Yeah.

[14:03] um autonomous lethal weapons. They're

[14:07] all the big arms selling countries are

[14:10] busy trying to make autonomous lethal

[14:11] weapons. America and Russia and China

[14:13] and Britain and Israel. I think Canada's

[14:16] probably a bit too wimpy for that. The

[14:20] question then is what to do about it.

[14:23] What type of regulation do you think we

[14:25] should pursue? Okay, so we need to

[14:27] distinguish these two different kinds of

[14:28] threat. the bad actors using it for bad

[14:31] things and the AI itself taking over.

[14:34] I've talked mainly about that second

[14:36] threat, not because I think it's more

[14:38] important than the other threats, but

[14:39] because people thought it was science

[14:41] fiction. And I want to use my reputation

[14:43] to say no, it's not science fiction. We

[14:45] really need to worry about that. Um, and

[14:48] if you ask what should we do about it

[14:50] it's not like climate change. Climate

[14:52] change, just stop burning carbon and

[14:55] it'll all be okay in the long run. It'll

[14:57] be terrible for a while, but in the long

[14:58] run, it'll be okay if you don't burn

[14:59] carbon. Um, for AI taking over, we don't

[15:03] know what to do about it. We don't know.

[15:05] For example, the researchers don't know

[15:07] if there's any way to prevent that, but

[15:09] we should certainly try very hard, and

[15:12] the big companies aren't going to do

[15:13] that. If you look what the big companies

[15:14] are doing right now, they're lobbying to

[15:17] get less AI regulation. There's hardly

[15:19] any regulation as it is, but they want

[15:20] less um because they want short-term

[15:23] profits. We need people to put pressure

[15:26] on governments to insist that the big

[15:29] companies do serious safety research. So

[15:32] in California, they had very sensible

[15:34] bill, bill 1047, where they said that at

[15:38] least what big companies have to do is

[15:39] test things carefully and report the

[15:41] results of their tests. And they didn't

[15:43] even like that. So does that make you

[15:46] think regulation will not happen or how

[15:48] does it happen? It depends very much on

[15:50] what governments we get. Um I think

[15:52] under the current US government

[15:54] regulation is not going to happen. Um

[15:57] all of the big AI companies have got

[16:00] into bed with Trump and yeah it's just a

[16:04] bad situation. Elon Musk who is

[16:07] obviously so imshed in the Trump

[16:10] administration has been someone

[16:11] concerned about AI safety for a very

[16:13] long time. Yes, he's a funny mixture.

[16:16] Um, he has some crazy views like going

[16:20] to Mars, which I just think is

[16:21] completely crazy. However, because it

[16:24] won't happen or because it shouldn't be

[16:25] a priority. Because however bad you make

[16:28] the Earth, it's always going to be way

[16:30] more hospitable than Mars. Even if you

[16:32] had a global nuclear war, the Earth is

[16:35] going to be much more hospitable than

[16:36] Mars. Mars just isn't hospitable. Um

[16:39] obviously he's done some great things

[16:40] like electric cars and um helping

[16:44] Ukraine with communications with his

[16:46] Starlink. Um so he's done some good

[16:49] things, but right now he seems to be

[16:53] um fueled by powering ketamine and

[16:59] um he's doing a lot of crazy things. So

[17:02] he's got this funny mixture of views.

[17:03] So, so his history of being concerned

[17:06] about AI safety doesn't make you feel

[17:08] any better about the current

[17:09] administration. I don't think it's going

[17:10] to slow him down from doing unsafe

[17:13] things with AI. So, already they're

[17:15] releasing the weights for their AI large

[17:19] language models. Um, which is a crazy

[17:22] thing to do. Okay. These companies

[17:23] should not be releasing the weights.

[17:25] Meta releases the weights. Open AAI just

[17:26] announced they're about to release

[17:27] weights. Do you think that's I don't

[17:29] think they should be doing that because

[17:30] once you release the weights, you've got

[17:32] rid of the main barrier to using these

[17:35] things. So if you look at nuclear

[17:37] weapons, the reason only a few countries

[17:40] have nuclear weapons is because it's

[17:41] hard to get the file material. If you

[17:44] were to be able to buy file material on

[17:46] Amazon, many more companies would have

[17:48] nuclear many more countries would have

[17:50] nuclear weapons. Um, the equivalent of

[17:53] physile material for AI is the weights

[17:56] of a big model because it costs hundreds

[17:58] of millions of dollars to train a really

[18:00] big model. Not maybe the final training

[18:03] run, but all the research that goes into

[18:05] the things you do before the final

[18:06] training run. Hundreds of millions of

[18:08] dollars which a small cult or a bunch of

[18:11] cyber criminals can't afford. Um, once

[18:14] you release the weights, they can then

[18:16] start from there and fine-tune it for

[18:18] doing all sorts of things for just a few

[18:19] million dollars. So it's I think it's

[18:21] just crazy releasing weights and people

[18:22] talk about it like open source but it's

[18:25] very very different from open source. In

[18:27] open source software you release the

[18:29] code and then lots of people look at

[18:32] that code and say hey that might be a

[18:34] bug in that line and so they fix it.

[18:36] When you release the weights people

[18:37] don't look at the weights and say hey

[18:38] that weight might be a little bit wrong.

[18:40] No they just take this foundation model

[18:43] with the weights they've got now and

[18:44] they train it to do something bad. Yeah.

[18:46] The problem with the argument though, as

[18:48] articulated by your former colleague Yan

[18:50] Lakun among others, is the alternative

[18:52] is you have this tiny handful of

[18:54] companies that control this massively

[18:56] powerful technology. I think that's

[18:58] better than everybody controlling the

[18:59] massively powerful technology. I mean

[19:02] you could say the same for nuclear

[19:03] weapons. Would you like to have just a

[19:04] few countries controlling them or don't

[19:05] you think everybody should have them?

[19:07] One thing I'm taking from this is you

[19:09] have real concerns about it sounds like

[19:12] all of the major companies right now

[19:15] doing what's in society's best interest

[19:17] rather than what's in their profit

[19:19] motive. Is that the right way to hear

[19:20] you? I think the way companies work is

[19:24] they're legally required to try and

[19:26] maximize profits for their shareholders.

[19:28] They're not legally required. Well

[19:31] maybe public interest companies are, but

[19:32] most of them aren't legally required to

[19:35] do things that are good for society.

[19:37] Which, if any of them would you feel

[19:38] good about working for today?

[19:41] I used to feel good about working for

[19:43] Google because Google was very

[19:44] responsible. Um, it didn't release these

[19:47] big, it was the first to have these big

[19:48] chat bots and it didn't release them.

[19:50] Um, I'd feel less happy working for them

[19:53] today.

[19:56] Um, yeah, I wouldn't be happy working

[19:58] for any of them today. If if I worked

[19:59] for any of them, I'd be more happy with

[20:01] Google than most of the others. But were

[20:03] you disappointed when Google went back

[20:04] on its promise not to support uh

[20:06] military uses of AI? Very disappointed.

[20:09] I was very particularly since I knew

[20:11] Sergi Brin didn't didn't like military

[20:14] use of AI. But why do you think they did

[20:15] it?

[20:18] I can't really speculate with any inside

[20:21] information. I don't have any inside

[20:22] information about where they did it. I

[20:24] could speculate that they were worried

[20:27] about

[20:29] um being illreated by the current

[20:32] administration if they wouldn't um use

[20:35] their technology to make weapons for the

[20:36] US. Here's the toughest question I'll

[20:39] probably ask you today. Do you not still

[20:41] hold a lot of Google stock still? Um I

[20:44] hold some Google stock. Um most of my

[20:48] savings are not in Google stock anymore.

[20:51] Um, but yeah, I hold some Google stock

[20:54] and when Google goes up, I'm happy and

[20:55] when it goes down, I'm unhappy. So, I

[20:57] have a vested interest in Google. But I

[21:00] if they put in strong AI regulations

[21:03] that made Google less valuable, but um

[21:07] increase the chance of humanity

[21:09] surviving, I'd be very happy. Um, one of

[21:12] the most prominent labs has obviously

[21:14] been Open AI and they have lost so many

[21:17] of their top people. What have you made

[21:19] of that? Um, that open AI was set up

[21:24] explicitly to develop super intelligence

[21:27] safely and as the years went by, safety

[21:30] went more and more into the background.

[21:32] They were going to spend a certain

[21:34] fraction of their computation on safety

[21:36] and then they reaged on that. So, and

[21:39] now they're trying to go public. They're

[21:40] not now trying to be a for-profit

[21:42] company. um they're trying to get rid of

[21:44] all the um basically all the commitment

[21:47] to safety as far as I can see. So, and

[21:50] they've lost a lot of really good

[21:51] researchers in particular a former

[21:53] student of mine, Ilia Sutska, who's a

[21:55] really good researcher and was one of

[21:57] the people largely responsible for their

[21:59] development of GPT2 and then from there

[22:01] on to GPT4. Um did you talk to him

[22:05] before all that drama that led to his

[22:06] departure? No, he's very discreet. He

[22:09] doesn't talk he wouldn't talk to me

[22:11] about anything that was confidential to

[22:13] open AI. Um I was quite proud of him for

[22:17] firing Sam Arman even though it was very

[22:20] naive. So the problem was that OpenAI

[22:25] was about to have a new funding round

[22:28] and in that new funding round all the

[22:30] employees were going to be able to turn

[22:31] their paper money in OpenAI shares into

[22:34] real money. Yeah. Paper money meaning

[22:36] really hypothetical money. hypothetical

[22:38] money that would disappear if Open AI

[22:40] went bust. Tough time for an

[22:41] insurrection. So, a week or two before

[22:44] everybody's going to get maybe of the

[22:46] order of a million dollars each by

[22:47] cashing in their shares. Um maybe more.

[22:51] That's a bad time for an insurrection.

[22:53] So, the employees massively came out in

[22:54] favor of Sam Antman. But it wasn't

[22:56] because they um wanted Sam Antman. It's

[22:59] because they wanted to get that be able

[23:01] to turn their paper money into real

[23:02] money. Yeah. So, it was naive to do it

[23:06] then. Did it surprise you that he made

[23:08] that mistake or was this kind of the

[23:10] principled but maybe not fully

[23:12] calculated decision that you would

[23:13] expect?

[23:15] I don't know. Ilia is brilliant and has

[23:19] a strong moral compass. So, he's he's

[23:22] good on morality and he's very good

[23:24] technically, but in terms of

[23:26] manipulating people, he's maybe not so

[23:28] good. M I mean this is a little bit of a

[23:31] a wild card question but I do think it's

[23:32] interesting and relevant to the field

[23:34] and relevant to people discussing what's

[23:36] going on. You talked about Ilia being

[23:38] discreet. There does seem to be this

[23:40] culture of NDAs throughout the industry

[23:43] and so it's hard to even know what

[23:45] people think because people are

[23:46] unwilling or unable to even discuss

[23:48] what's going on. I'm not sure I can

[23:51] comment on that because when I left

[23:52] Google I I think I had to sign a whole

[23:54] bunch of NDAs. In fact, when I joined

[23:56] Google, I think I had to sign a whole

[23:58] bunch of NDAs that would apply when I

[23:59] left, and I have no idea what they said.

[24:01] I can't remember them anymore. Do you

[24:04] feel at all muzzled by them? No. Okay.

[24:06] Do you think it's a factor though that

[24:08] the public has a harder time

[24:10] understanding what's going on because

[24:11] people aren't allowed to tell us what's

[24:13] going on? I don't really know. I You'd

[24:15] have to know. You'd have to know which

[24:18] people weren't telling you. Okay. So

[24:20] you don't see this as a I don't see it

[24:21] as a big deal. It's a big deal. Got it.

[24:23] I think it was a big deal that Open AI

[24:26] appeared to have something um that said

[24:29] that if you'd already got shares, they

[24:32] could take the money away from you. Um

[24:35] yeah, that I think was a big deal and

[24:37] they they rapidly backed down on that

[24:38] when that became public. That was what

[24:40] their public statement said they did.

[24:42] They didn't present any contracts for

[24:44] the public to judge whether they had

[24:45] reversed that, but they said they had

[24:46] reversed it. Yes. Um there's a number of

[24:48] just important kind of hot buttony

[24:51] things. Hot button is actually not even

[24:53] a great word, but relevant issues I just

[24:54] like to get your your feedback on. One

[24:57] is the US and kind of the West's

[24:59] orientation to China in their efforts to

[25:02] pursue AI. Do you agree with this idea

[25:04] that we should be trying to restrain

[25:06] China? There's this idea of export

[25:08] controls, this idea that we should have

[25:09] democracies reach AGI first. What's your

[25:13] thinking on all that? First of all, you

[25:15] have to decide which countries are still

[25:17] democracies.

[25:19] Um and my thinking on that is in the

[25:23] long run it's not going to make much

[25:25] difference. It may slow things down by a

[25:26] few years but clearly um if you prevent

[25:30] China from getting the most advanced

[25:31] technology people know how this advanced

[25:33] technology works. So, China's just

[25:36] invested many many billions maybe

[25:39] hundreds of billions um of the order of

[25:41] 100 billion I think in making

[25:44] lithography machines or in in in getting

[25:47] their own homebased technology that does

[25:49] this stuff. Um so it'll slow them down a

[25:53] bit but it will actually force them to

[25:56] develop their own industry and in the

[25:57] long run um they're very competent and

[26:00] they will and so it'll just slow things

[26:02] down for a few years. But race is the

[26:04] right framework. We shouldn't be trying

[26:06] to cooperate with communist China. I

[26:09] wouldn't describe it as communist

[26:10] anymore. I used the loaded term

[26:13] specifically because why wouldn't you

[26:16] cooperate, right? The only rationale to

[26:18] not cooperate is if you think they're a

[26:21] malignant force. Well, there's areas in

[26:23] which we won't cooperate where we is, I

[26:26] guess, I'm not sure who we is anymore

[26:28] because I'm in Canada now and we used to

[26:30] be sort of Canada and the US, but it's

[26:31] not anymore. Yeah. Um obviously the

[26:35] countries are not going to cooperate on

[26:37] developing lethal autonomous weapons

[26:39] because the lethal autonomous weapons to

[26:41] be used against other countries. So but

[26:44] we've had treaties and other types of

[26:46] weapons as you've pointed out. We could

[26:47] have treaties not to develop them but

[26:49] cooperating in making them better.

[26:51] They're not going to do that. Sure.

[26:53] Sure. Sure. Now there is one area where

[26:55] they will cooperate which is on the

[26:57] existential threat. if they ever get

[27:00] serious about worrying about the

[27:02] existential threat and doing stuff about

[27:03] it, they will collaborate on that ways

[27:06] of stopping AI taking over because we're

[27:08] all in the same boat. So, at the height

[27:10] of the Cold War, the Soviet Union and

[27:12] the US collaborated on preventing a

[27:14] global nuclear war and even countries

[27:18] that are very hostile to each other will

[27:20] collaborate when their interests align

[27:22] and their interests will align when it's

[27:24] AI versus humanity.

[27:27] Um, there's this question of fair use

[27:30] whether it's okay to have the content of

[27:33] billions of humans created over many

[27:35] years kind of scooped up and repurposed

[27:39] into models that will replace some of

[27:42] those same people that created the

[27:44] training data. Where do you fall on

[27:45] that? I think I sort of fall all over

[27:49] the place on that in the sense that it's

[27:51] a very complicated issue. So initially

[27:55] it seems yeah they should have to pay

[27:57] pay for that. But suppose I have a

[28:00] musician who produces a song in a

[28:02] particular genre and ask well how did

[28:05] they produce the song in that genre?

[28:07] Where did where did their ability to

[28:08] produce songs in that genre came from?

[28:10] Well it came from listening to songs by

[28:11] other musicians in that genre. So they

[28:14] listen to these songs, they kind of

[28:16] internalize things about the structure

[28:17] of the songs and then they generated

[28:19] stuff in that genre and the stuff they

[28:21] generated is different. So it's not

[28:24] theft um and that's accepted. Well

[28:27] that's what the AI is doing. The AI is

[28:29] absorbing all this information and then

[28:31] producing new stuff. It's not just

[28:33] taking taking and patching it together.

[28:35] It's generating new stuff that has the

[28:37] same underlying themes. And so it's no

[28:41] more stealing than a person does when

[28:44] they do the same thing. But the point is

[28:46] it's doing it um at a massive scale. And

[28:51] no musician has ever put every other

[28:53] musician out of business. Exactly. So in

[28:56] Britain for example, the government

[28:59] doesn't seem to have any interest in

[29:03] protecting the creative artists. And if

[29:07] you look at the economy, the creative

[29:08] artists are work worth a lot to Britain.

[29:11] So I have a friend called BB Bankron

[29:14] saying we should protect creative

[29:15] artists. It's very important to the

[29:18] economy and just letting AI walk off

[29:20] with it all um seems unfair. UBI

[29:24] universal basic income, is this part of

[29:26] the solution to the displacements of AI?

[29:28] You think? I think it may be necessary

[29:31] to stop people starving. Um I don't

[29:34] think it totally solves the problem but

[29:37] even if you had quite high UBI

[29:40] um it doesn't solve the problem of human

[29:42] dignity for a lot of people um who they

[29:45] are is particularly for academics who

[29:48] they are is mixed up in their work.

[29:50] That's who they are. If they become

[29:52] unemployed just getting the same money

[29:54] doesn't totally compensate. They're not

[29:56] who they are anymore. Yeah. I tend to

[29:59] think that's true as well. I saw you

[30:00] give this quote at one point though

[30:01] where you said you might have been

[30:02] happier if you were a woodworker. Well

[30:05] yes, cuz I I really like being a

[30:07] carpenter. And isn't there an

[30:09] alternative where you're born a hundred

[30:10] years later where you don't have to

[30:12] waste all your time on these neural nets

[30:13] and you just get to enjoy woodworking

[30:16] while taking in a monthly income? Yeah

[30:18] but there's a difference between doing

[30:20] it as a hobby and doing it to make a

[30:22] living somehow. It's more real doing it

[30:23] to make a living. So you don't think a

[30:25] future where we get to pursue our

[30:27] hobbies and don't have to contribute to

[30:29] the economy? That might that might be

[30:30] fine. Yeah. Um if everybody was doing

[30:33] that, but if you're in some

[30:35] disadvantaged group who are getting

[30:37] universal basic income and you're

[30:39] getting less income than um other people

[30:43] because employers will want you to do

[30:44] that so they can get other people to

[30:46] work for them. Um that's going to be

[30:48] very different. I'm interested in this

[30:51] idea of robot rights. I don't know if

[30:54] there's a better term to describe it

[30:55] but at some point you're going to have

[30:57] these massively intelligent AIs. They're

[30:59] going to be agentic and doing all kinds

[31:01] of things in the world. Should they be

[31:03] able to own property? Should they be

[31:05] able to vote? Should they be able to

[31:07] marry humans in a loving relationship?

[31:10] Like what what or even if they if

[31:11] they're just smarter than us and if it's

[31:13] a better form of intelligence than what

[31:15] we've got, um should it be fine for them

[31:18] to just take over and humans be history?

[31:20] Yeah, let's go to that bigger idea

[31:23] second. Would I'm curious on the on the

[31:25] more narrow idea unless you think the

[31:27] narrow questions are irrelevant because

[31:28] the big question takes pre. No, I think

[31:30] the narrow questions irrelevant. Yeah.

[31:32] So, I used to be worried about this

[31:34] question. I used to think, well, if

[31:36] they're smarter than us, um, why

[31:38] shouldn't they have the same rights as

[31:39] us? Yeah. And now I think, well, we're

[31:43] people. What we care about is people.

[31:46] Um, I eat cows. I mean, I know lots of

[31:49] people don't, but I eat cows. And the

[31:51] reason I'm happy eating cows is because

[31:52] they're cows. Um, and I'm a person. Um

[31:56] and the same for these super intelligent

[31:58] AIs. They may be smarter than us, but

[32:00] what I care about is people. And so, I'm

[32:02] willing to be mean to them. I'm willing

[32:05] to deny them their rights because I want

[32:09] what's best for people. Yeah. Um, now

[32:12] they won't agree with that and they may

[32:15] win

[32:16] but that's my current position on

[32:19] whether AI should have rights, which is

[32:21] even if they're intelligent, even if

[32:22] they have sensations and emotions and

[32:24] feelings and all that stuff, um, they're

[32:26] not people, and people's what I care

[32:27] about, but they're going to seem so much

[32:29] like people. I feel like it's going to

[32:31] they're going to be able to fake it.

[32:32] Yes. They're going to be able to seem

[32:32] very like people. Yeah. Yeah. Do you

[32:35] suspect we'll end up giving them rights?

[32:38] I don't know. Okay. I tend to avoid this

[32:41] issue because there's more immediate

[32:44] problems like bad uses of AI or the

[32:47] issue of whether they will try and take

[32:48] over and how to prevent that. Yeah. And

[32:51] it sounds kind of flaky if you start

[32:52] talking about them having rights. Most

[32:54] people you've lost most people when you

[32:57] go there. Even just sticking with people

[32:59] there seems to be real soon, if it's not

[33:02] already here, this um ability to use AI

[33:06] to select what babies we have. Are you

[33:08] concerned at all about that line embryo

[33:11] selection? You mean selecting for the

[33:13] sex or selecting for the intelligence

[33:16] and the eye color and the likelihood to

[33:18] get pancreatic cancer and the you know

[33:21] the list goes down and down and down of

[33:22] all the things we might select. I think

[33:24] if you could select a baby that was less

[33:25] likely to get pancreatic cancer that

[33:27] would be a great thing. I'm willing to

[33:29] say that. Okay. So we this is a thing we

[33:32] should pursue. We should make healthier

[33:33] stronger, better babies.

[33:36] Um it's very difficult territory. Right.

[33:39] It is. That's why I'm asking about it.

[33:41] But some aspects of it um seem to make

[33:44] sense to me. Like if you're an a normal

[33:49] healthy couple and you have a fetus and

[33:53] you can predict that it's going to have

[33:55] very serious problems and maybe not live

[33:57] very long. Um it seems to me it makes

[34:01] sense to abort it and have a healthy

[34:04] baby. that just seems sensible to me.

[34:06] Now, I know a lot of religious people

[34:07] wouldn't agree with that at all. Um, but

[34:10] for me, if you could make those

[34:12] predictions reliably, um, that just

[34:15] seems to make sense to me. I've been a

[34:18] little bit holding us back from kind of

[34:20] the central thing that I think you want

[34:22] people to take away, which is this idea

[34:24] of of machines taking over and the

[34:26] impact of that. So, I'd like to just

[34:28] discuss that as fully as you'd like or

[34:31] that we can. like how do you want to

[34:33] frame this issue? How should people

[34:35] think about it? One thing to bear in

[34:38] mind is how many examples do you know of

[34:40] less intelligent things controlling much

[34:42] more intelligent things? So we know that

[34:45] things are more or less equal

[34:46] intelligence, the less intelligent one

[34:48] can control the more intelligent one. Um

[34:51] but with a big gap in intelligence

[34:53] there's very very few examples where the

[34:56] more intelligent one isn't in control.

[34:58] So that's something you should bear in

[34:59] mind. That's a big worry. I think the

[35:02] situation we're in right

[35:04] now, the best way to understand it

[35:07] emotionally is we're like somebody who

[35:10] has this really cute tiger cup. It's

[35:14] just such a cute tiger

[35:17] cup. Now, unless you can be very sure

[35:22] that it's not going to want to kill you

[35:23] when it's grown up, you should worry. M.

[35:28] And to extend the metaphor, you put it

[35:30] in a cage, you kill it. What do you do

[35:32] with the tiger cub? Well, the point

[35:34] about the tiger cub is it's just

[35:36] physically stronger than you. So, you

[35:38] can still control it because you're more

[35:39] intelligent. Yeah. Um, things that are

[35:42] more intelligent than you. We have no

[35:43] experience of that, right? People aren't

[35:45] used to thinking about it. People think

[35:48] somehow you constrain it. You don't

[35:50] allow it to press buttons or whatever.

[35:51] Um, things more intelligent than you

[35:54] they're going to be able to manipulate

[35:56] you. So another way of thinking about it

[35:57] is imagine that there's this

[36:00] kindergarten. There's these two and

[36:02] three year olds and the two and three

[36:03] year olds are in charge and you just

[36:05] work for them in the kindergarten and

[36:07] you're not that much more intelligent

[36:09] than a two or threey old. Not compared

[36:10] with super intelligence, but you are

[36:11] more intelligent. Um so how hard would

[36:14] it be for you to get control? Well, you

[36:17] just tell them all you're going to get

[36:18] free candy and if they just sort of sign

[36:21] this or just agree verbally to this um

[36:24] you get free candy for as long as you

[36:25] like. and you'll be in control. They

[36:28] won't they won't have any idea what's

[36:30] going on. And with super intelligences

[36:33] they're going to be so much smarter than

[36:34] us, we'll have no idea what they're up

[36:37] to. And so what do we do?

[36:41] Um, we worry

[36:43] about whether there's a way to build a

[36:47] super intelligence so that it doesn't

[36:49] want to take control. I don't think

[36:51] there's a way of stopping it take

[36:52] control if it wants to. So there's one

[36:54] possibility is never build a super

[36:56] intelligence. You think that's possible?

[36:59] I mean it's conceivable, but I don't

[37:01] think it's going to happen because

[37:02] there's too many too much competition

[37:04] between countries and between companies

[37:06] and they're all after the next shiny

[37:08] thing and it's developing very very

[37:10] fast. So I don't think we're going to be

[37:13] able to avoid building super

[37:14] intelligence. It's going to happen. The

[37:17] issue is can we design it in such a way

[37:20] that it never wants to take control that

[37:22] it's always benevolent. Um that's a very

[37:26] tricky issue. Just people say well we'll

[37:28] get it to align with human interests.

[37:31] But human interests don't align with

[37:33] each other. And if I say I've got two

[37:35] lines at right angle and I want you to

[37:37] show me a line parallel to both of them.

[37:39] That's kind of tricky, right? And if you

[37:42] look at the Middle East for example

[37:45] there's people with very strong views

[37:47] that don't align. So how are you going

[37:50] to get AI to align with human interests?

[37:53] Human interests don't align with each

[37:55] other. So that's one problem. It's going

[37:57] to be very hard to figure out how to get

[38:01] super intelligence that doesn't want to

[38:03] take over and doesn't want to ever hurt

[38:05] us. Um but we should certainly try. And

[38:09] trying is kind of just an iterative

[38:10] process. Month by month, year by year

[38:13] we try to Yeah. So obviously if you're

[38:15] going to develop something that might

[38:17] want to take

[38:19] over when it's just slightly less

[38:21] intelligent than you are, and we're very

[38:23] close to that now, um you should kind of

[38:26] look at what it'll do to try and take

[38:28] over. So if you look at the current AIS

[38:31] you can see they're already capable of

[38:33] deliberate deception. They're capable of

[38:35] pretending to be stupider than they are.

[38:38] um of lying to you so that they can kind

[38:41] of confuse you into not understanding

[38:43] what they're up to. Um we need to be

[38:46] very aware of all that and to study all

[38:48] that and study about whether there's a

[38:49] way to stop them doing that. When we

[38:51] spoke a couple years ago, I was

[38:53] surprised at you voicing concerns

[38:55] because you hadn't really done much of

[38:57] that before and now you're voicing them

[39:00] quite clearly and loudly. Was it mostly

[39:04] that you felt more liberated to say this

[39:06] stuff or was it really a really big sea

[39:09] change in how you saw it in these last

[39:10] few years? When we spoke a couple of

[39:12] years ago, I was still working at Google

[39:14] then. It was in March and I didn't

[39:16] resign till um the end of April. Um but

[39:20] I was thinking about leaving then. Um

[39:22] and I had had a kind of epiphany before

[39:25] we spoke where I realized that these

[39:27] things might be a better form of

[39:29] intelligence than us. And that got me

[39:31] very scared. And you didn't think that

[39:34] before just because you thought the time

[39:35] horizon was so different. No, it wasn't

[39:37] just that. It was because of the

[39:38] research I was doing at Google. Okay. I

[39:40] was trying to figure out whether you

[39:43] could design analog large language

[39:46] models that would use much less power.

[39:49] Mhm. Um, and I began to fully realize

[39:53] the advantage of being digital. So all

[39:55] the models we've got at present are

[39:56] digital. And if you're a digital model

[40:01] you can have exactly the same neural

[40:04] network with the same weights in it

[40:06] running on several different pieces of

[40:08] hardware, like thousands of different

[40:09] pieces of

[40:10] hardware. And then you can get one piece

[40:12] of hardware to look at one bit of the

[40:14] internet and another piece of hardware

[40:15] to look at another bit of the internet.

[40:17] And each piece of hardware can say, how

[40:19] would I like to change my internal

[40:21] parameters, my weights, so I can absorb

[40:24] the information I just saw.

[40:27] And each of these separate pieces of

[40:29] hardware can do that. And then they can

[40:31] just average all the changes to the

[40:33] weights because they're all using the

[40:35] same weights in exactly the same way.

[40:37] And so averaging makes sense. You and I

[40:39] can't do that. And if they've got a

[40:41] trillion weights, they're sharing

[40:43] information at like a trillions of bits

[40:46] every time they do this averaging. Now

[40:49] you and I when I want to get some

[40:51] knowledge from my head into your head, I

[40:53] can't just take the strength of the

[40:54] connections between neurons and average

[40:56] them with the strength of the

[40:57] connections between your neurons because

[40:58] our neurons are different. We we're

[41:00] analog and we're just very different

[41:02] brains. So the only way I have getting

[41:05] knowledge to you is I do some actions

[41:08] and if you trust me, you try and change

[41:10] the connection strengths in your brain

[41:13] so that you might do the same things.

[41:15] And if you ask, well, how efficient is

[41:17] that? Well, if I give you a sentence

[41:19] it's only a few hundred bits of

[41:21] information at most. So, it's very slow.

[41:23] We communicate just a few bits per

[41:26] second. These large language models

[41:29] running on digital systems can

[41:31] communicate trillions of bits a second.

[41:34] So, they're billions of times better

[41:35] than us at sharing information. That got

[41:38] me scared. Right. But what surprised you

[41:40] or what changed your thinking was you

[41:41] were thinking the analog was going to be

[41:43] the path previously. No, I was thinking

[41:46] if we want to use much less power, yeah

[41:49] we should think about whether it's

[41:50] possible to do this analog. Yeah. And

[41:52] because you can use much less power, you

[41:54] can also be much sloppier in the design

[41:56] of the system. Because what's going to

[41:58] happen is you don't have to manufacture

[42:00] a system that does precisely what you

[42:02] tell it to, which is what a computer is.

[42:04] You can manufacture a system with a lot

[42:06] of slop in it, and it will learn to use

[42:08] that sloppy system, which is what our

[42:10] brains are. Do you think the technology

[42:11] is no longer destined for that solution

[42:14] but is going to stick with the digital

[42:15] solution? I think it'll probably stick

[42:17] with a digital solution. Now, it's quite

[42:19] possible that we can get these digital

[42:21] computers to design better analog

[42:24] hardware better than us. Um, I think

[42:27] that may be the long-term future. You

[42:29] got into this field because you wanted

[42:30] to know how the brain works. Yes. Do you

[42:32] think we're getting closer to that

[42:34] through this? I think for a while we

[42:36] did. So I think we've learned a lot at a

[42:39] very general level about how the brain

[42:41] works.

[42:42] So 30 years ago or 50 years ago, if you

[42:46] ask people, well, could you have a big

[42:48] random neural network with random

[42:50] connection strengths and then could you

[42:52] show it data and have it learn to do

[42:55] difficult things like recognize what

[42:57] someone's saying or answer questions

[43:00] just by showing it lots of data? Almost

[43:02] everybody would have said, "That's

[43:03] crazy. There's no way you're going to do

[43:05] that. it has to have lots of pre-wired

[43:07] structure that comes from evolution.

[43:09] Well, it turns out they were wrong. It

[43:11] turns out you can have a big random

[43:13] neural network. Um, and it can learn

[43:15] just from data. Now, that doesn't mean

[43:17] we don't have a lot of pre-wired

[43:18] structure, but basically most of what we

[43:22] know comes from learning from data, not

[43:24] from all this pre-wired structure. So

[43:26] that's a huge advance in understanding

[43:28] the brain. Now, the issue is how do you

[43:31] get the information that tells you

[43:32] whether to increase or decrease the

[43:34] connection strength? If you can get that

[43:36] information, we know that we can then

[43:37] train a big system that starts with

[43:39] random weights to do wonderful things.

[43:41] The brain needs to get information like

[43:43] that and it probably gets it in a

[43:45] different way from the standard

[43:46] algorithm used in these big AI models

[43:49] which is called back propagation. The

[43:51] brain probably doesn't use back

[43:53] propagation. Nobody can figure out how

[43:55] it could be doing it. Um it's probably

[43:58] getting the gradient information that is

[44:00] how changing a weight will improve the

[44:02] performance in a different way. But we

[44:04] do know now that if it can get that

[44:06] great information, it can be really

[44:07] effective at learning. Do you know if

[44:10] any of the labs now are using their

[44:12] models to try to pursue new ideas in AI

[44:16] development? Almost certainly. Okay.

[44:19] Yeah. And in particular, Deep Mind is

[44:21] very interested in using AI for doing

[44:24] science. And one piece of science is AI.

[44:27] Sure. I mean, was that something you

[44:30] trying when you were there? like this

[44:31] bootstrapping idea of maybe the next

[44:33] innovation could be created by the AI

[44:36] itself. So there's elements of that. So

[44:39] for example, they were using AI to do

[44:42] layout on chips that were going to be

[44:43] used for AI. So Google's AI chips um

[44:47] their um tensor processing units. Um

[44:51] they used AI to develop those chips. So

[44:55] I'm curious if just in your normal

[44:57] day-to-day life you despair. You fear

[45:00] for the future and assume it won't be so

[45:03] good. I don't despair, but mainly

[45:07] because even I find it very hard to take

[45:10] it seriously. Ah, it's very hard to get

[45:13] your head around the fact that we're at

[45:14] this very very special point in history

[45:17] where in a ve relatively short time

[45:21] everything might totally change a change

[45:24] of a scale we've never seen before. Um

[45:26] it's hard to absorb that emotionally.

[45:29] It is. And I do notice even though

[45:31] people maybe are concerned, I've never

[45:34] seen a protest. There's no real

[45:36] political movement around this idea. The

[45:38] world is changing and no one really

[45:40] seems to care that much. Um, among the

[45:44] AI

[45:46] researchers, people are more aware of

[45:49] it. Um, so the people I know who are

[45:52] kind of most depressed about it are

[45:53] serious AI researchers.

[45:56] Um I have started doing practical things

[46:00] like because AI is going to be very good

[46:02] at designing

[46:04] um cyber

[46:06] attacks. Um I don't think the Canadian

[46:09] banks are safe anymore. So Canadian

[46:11] banks are about as safe as you can get.

[46:13] Okay? They're very well regulated

[46:15] compared with US banks.

[46:17] Um but over the next 10 years, I

[46:20] wouldn't be at all surprised if there

[46:22] was a cyber attack that took down a

[46:24] Canadian bank. What does take down mean?

[46:26] Suppose that the bank holds shares that

[46:29] I own, right? Suppose the cyber attack

[46:32] sells those

[46:33] shares. Now my money's gone. So I

[46:37] actually now spread my money between

[46:38] three banks. Okay. So now your mattress.

[46:41] That's the first practical thing I've

[46:42] done because I think if a cyber attack

[46:44] takes down one Canadian bank, the others

[46:46] will get a lot more serious. Okay.

[46:48] Anything else like that? What else?

[46:50] That's the main thing. That's where I

[46:51] noticed I actually did something

[46:52] practical.

[46:54] that flowed from my belief that um very

[46:58] scary times are coming. Okay. Uh when we

[47:01] spoke a couple years ago, you had said

[47:03] you know, AI is like an idiot savant

[47:05] but humans are still much better at

[47:06] reasoning, right? That's changed. Okay.

[47:09] Explain. Previously, what the large

[47:12] language models would do is they'd spit

[47:14] out one word at a time and that would be

[47:16] it. Now they spit out words and they're

[47:20] looking at the words they spit out. And

[47:23] they will spit out words that aren't the

[47:25] answer to the question yet. They'll spit

[47:27] out words. It's called chain of thought

[47:28] reasoning. And so now they can reflect

[47:31] on the words they spat out already. And

[47:33] that gives them room to do some thinking

[47:36] in and you can see what they're

[47:37] thinking. It's wonderful. Yeah. Well

[47:40] it's wonderful if you're a researcher.

[47:42] And a lot of people from old-fashioned

[47:44] AI said, "Well, you know, these things

[47:46] can't reason. They're not really

[47:48] intelligent because they can't reason.

[47:50] And you're going to need to use

[47:51] old-fashioned AI and turn things into

[47:54] logical forms in order to do proper

[47:56] reasoning. Well, they were just utterly

[47:58] wrong. Um, neural nets are going to do

[48:00] the reasoning. And the way they're going

[48:01] to do the reasoning is by this chain of

[48:03] thought, by spitting out stuff that they

[48:05] don't reflect upon. Yeah. You said at

[48:07] the beginning that the last two years

[48:09] the development has been faster than you

[48:11] expected. Are there other examples of

[48:12] that? Things you've seen that if you

[48:13] said, "Wow, it's half it's fast." That's

[48:15] the main example. It's got much better

[48:17] at generating images and things too, but

[48:20] the main thing is that it can now do

[48:22] reasoning quite well. Okay. And that you

[48:24] can see what it's thinking. Like why is

[48:26] that important or where does that lead

[48:27] that is meaningful? Um well, it's very

[48:31] good that you can see what they're

[48:32] thinking because there's these examples

[48:35] where you give it the goal. You give it

[48:39] a goal and you can see it doing

[48:41] reasoning to try and achieve this goal

[48:44] by deceiving people and you can see it

[48:46] doing that. It's like I could hear the

[48:48] voice in your head. Yeah. The other

[48:51] thing we we moved moved through, but

[48:53] maybe I don't know if you have anything

[48:54] more to say about is just it's

[48:56] remarkable that there are so many tech

[48:58] figures that are now have an important

[49:00] role in Washington DC at this very

[49:02] moment where what Washington DC does

[49:05] could be really important to the

[49:07] evolution, the regulation of this

[49:09] technology. Does that concern you? How

[49:12] do you see that?

[49:15] those tech

[49:17] figures are primarily concerned with

[49:20] their companies making

[49:22] profits. So that concerns me a lot.

[49:26] Yeah. I don't see how things really

[49:28] change unless either there's strong

[49:30] regulation or this moves away from this

[49:32] for-profit model. And I don't see how

[49:34] those things happen either. I think if

[49:37] the public realized what was happening

[49:40] they will put a lot of pressure on

[49:41] governments to insist that the AI

[49:43] companies develop this more safely.

[49:45] Okay, that's the best I can do. It's

[49:47] it's not very satisfactory, but it's the

[49:49] best I can think of. And more safely

[49:51] means more resources from those

[49:53] companies toward safety research. Yes.

[49:56] For example, the fraction of their

[49:57] computer time they spend on safety

[49:59] research should be a significant

[50:01] fraction like a third. Right now, it's

[50:04] much much less. There's one company

[50:06] Anthropic, that's more concerned with

[50:08] safety than the others. It was set up to

[50:10] be concerned with safety by people who

[50:12] left Open AI because Open AI wasn't

[50:14] enough concerned with safety. And

[50:16] Anthropic does spend more time on safety

[50:18] research, but still probably not enough.

[50:19] There is this view among many that Open

[50:22] AI has talked a good game about these

[50:24] issues, but is not living out those

[50:26] values. Is that your perspective? Yes.

[50:29] What evidence do you see of that? that

[50:32] all their best safety researchers left

[50:33] because they believed that too. Um that

[50:37] they were set up as a company um that

[50:41] was going to develop area safety and

[50:43] their main goal was not to make profits

[50:44] but to develop our safety and they're

[50:46] now busy lobbying the California

[50:49] Attorney General to allow them to change

[50:53] to a for-profit company. Um there's lots

[50:56] of evidence for that, right? Um and I

[50:58] should give you a chance to hold up

[50:59] anyone as a good actor here that people

[51:01] should feel better about. You mentioned

[51:02] Anthropic. Is that the name or do you

[51:05] see of the companies? Anthropic is the

[51:08] most concerned with safety and a lot of

[51:10] the safety researchers who left OpenAI

[51:13] went to Anthropic and so Anthropic has

[51:15] much more of a culture concerned with

[51:17] safety. Okay. But um they have

[51:20] investments from big companies. Yeah.

[51:22] you have to get money from somewhere and

[51:24] I'm worried that those investments will

[51:26] force them into um releasing things

[51:29] faster than they should. And when I

[51:31] asked you which you'd feel comfortable

[51:32] working for, you said none of them, I

[51:34] think, or just maybe Google. I should

[51:36] have said maybe Google Anthropic. Okay.

[51:39] Thank you so much for all this time and

[51:41] the rest of your time today. I really

[51:42] appreciate it. Okay. You haven't got the

[51:44] rest yet. I haven't got I'm counting on

[51:46] it.