[0:01] the last time we spoke two years one
[0:03] month ago. I'm curious how your
[0:06] expectations over these two years have
[0:08] evolved for how you see the future. So
[0:12] AI has developed even faster than I
[0:14] thought. Um in particular they now have
[0:18] these AI
[0:19] agents which are more dangerous than AI
[0:22] that just answers questions because they
[0:24] can do things in the world. Um, so I
[0:27] think things have got, if anything
[0:29] scarier than they were before. Um, I
[0:31] don't know if we want to call it AGI
[0:33] super intelligence, whatever, very
[0:35] capable AI system. Do you have a a
[0:38] timeline in mind for when you think
[0:40] that's coming? So, a year ago, I thought
[0:44] it was there's a good chance it comes
[0:46] between five and 20 years from now. Um
[0:50] so I guess I should believe there's a
[0:51] good chance it comes between four and 19
[0:53] years from now. Um, I think that's still
[0:57] what I guess. Okay. Which is sooner than
[0:59] when we spoke because you were still
[1:01] thinking like 20 years. Yeah. Um, I
[1:03] think it may, you know, there's a good
[1:05] chance it'll be here in 10 years or less
[1:07] now. So, in 4 to 19 years, we reached
[1:10] this point. What does that look like?
[1:14] So, I don't really want to speculate on
[1:16] what it would look like if I decided to
[1:18] take over. There's so many ways it could
[1:20] do it. And I'm not even talking about
[1:21] taking over. We can talk about that. I'm
[1:23] sure we will talk about that. But
[1:25] putting aside that kind of takeover just
[1:28] like a super intelligent artificial
[1:31] intelligence like what what kind of
[1:33] things would is this capable of or would
[1:35] be doing? So the sort of good scenario
[1:39] is we would all be like the sort of dumb
[1:41] CEO of a big company who has an
[1:43] extremely intelligent assistant who
[1:46] actually makes everything work but does
[1:48] what the CEO wants. So the CEO thinks
[1:50] they're doing things, but actually it's
[1:52] all done by the assistant and the CEO
[1:54] feels just great because everything they
[1:55] sort of decide to do works out. That's
[1:58] the good scenario. And I've heard you
[2:00] point out a few areas where you think
[2:02] there's reason to be optimistic about
[2:06] what this future looks like. Yes. Yeah.
[2:08] So why don't we take each of them? So
[2:11] areas like healthcare um they will be
[2:15] much better at reading medical images
[2:17] for example. That's a minor thing. Um I
[2:20] made a prediction some years ago they'd
[2:21] be better by now and they're about
[2:23] comparable with the experts by now. Um
[2:26] they'll soon be considerably better
[2:27] because they'll have had a lot more
[2:29] experience. One of these things can look
[2:31] at millions of X-rays and learn from
[2:32] millions of them and a doctor can't. Um
[2:36] they'll be very good family doctors. So
[2:39] you can imagine a family doctor who's
[2:40] seen a 100 million people including half
[2:44] a dozen people with your very very rare
[2:47] condition. They'd just be a much better
[2:49] family doctor. A family doctor who can
[2:51] integrate information about your genome
[2:53] with the results of all the tests on you
[2:55] and all the tests on your relatives um
[2:57] the whole history and doesn't forget
[2:59] things. That would be much much better
[3:01] already.
[3:02] um AI combined with a doctor is much
[3:06] better at doing diagnosis in difficult
[3:08] cases than a doctor alone. So we're
[3:11] going to get much better healthcare from
[3:12] these things and they'll design better
[3:14] drugs too. Uh education is another
[3:17] field. Yes, in education we know that um
[3:21] if you have a private tutor you can
[3:23] learn stuff about twice as fast. Um
[3:26] these things eventually will be
[3:28] extremely good private tutors who know
[3:30] exactly what it is you misunderstand and
[3:33] exactly what example to give you to
[3:35] clarify it to you so you understand. So
[3:37] maybe you'll be able to learn things
[3:39] three or four times as fast with these
[3:40] things. Um, that's bad news for
[3:43] universities but good news for people.
[3:45] Yeah. Do you think the university system
[3:47] will survive this period? I think many
[3:50] aspects of it will. I think it's still
[3:52] the case that a graduate student in a
[3:55] good group in a good university is the
[3:57] sort of best source of truly original
[3:59] research and I think that'll probably
[4:01] survive. You need a kind of
[4:03] apprenticeship.
[4:04] Some people hope this will help solve
[4:06] the climate crisis. I think it will
[4:08] help. Um it'll make better materials.
[4:12] We'll be able to make better batteries
[4:13] for example. Um I'm sure AI will be
[4:15] involved in designing them. Um, people
[4:18] are using it for carbon capture from the
[4:21] atmosphere. I'm not convinced that's
[4:23] going to work just because of the energy
[4:24] considerations, but it might. In
[4:26] general, we're going to get much better
[4:28] materials. We might even get room
[4:30] temperature
[4:31] superconductivity, which would mean you
[4:33] can have lots of solar plants in the
[4:35] desert and we can be thousands of miles
[4:37] away. Uh, any other positives we should
[4:40] tick off? Well, more or less any
[4:42] industry it's going to make more
[4:43] efficient because almost every company
[4:45] wants to predict things from data and AI
[4:49] is very good at doing predictions. It's
[4:50] better than the methods we had
[4:52] previously almost always. Um so it's
[4:56] going to make it's going to cause huge
[4:58] increases in productivity. It's going to
[5:00] mean when you call up a call center
[5:02] when you call up um Microsoft to
[5:04] complain that something doesn't work and
[5:06] you get a call center, the person in the
[5:08] call center will be actually an AI who
[5:10] will be much better informed. Yeah. When
[5:12] I asked you a couple years ago about job
[5:14] displacements, you seem to think that
[5:16] wasn't a big concern. Is that still your
[5:18] thinking? No, I'm thinking it will be a
[5:20] big concern. AI's got so much better in
[5:22] the last few years that I mean, if I had
[5:25] a job in a call center, I'd be very
[5:27] worried. Yeah. or maybe a job as a
[5:30] lawyer or a job as a journalist or a job
[5:32] as an accountant. Yeah. Any doing
[5:35] anything routine I think investigatively
[5:37] journalists I think will last quite a
[5:38] long time because you need a lot of
[5:41] initiative plus some moral outrage and I
[5:44] think journalists will be in business
[5:46] for a bit but beyond call centers what
[5:48] are your concerns about jobs? Well any
[5:50] routine job so a sort of standard
[5:52] secretarial job something like a
[5:54] parallegal for example those jobs have
[5:57] had it. Have you thought about what how
[5:59] we move forward in a world where all
[6:01] these jobs go away? So it's like this.
[6:04] It ought to be that if you can increase
[6:07] productivity, everybody benefits. Um the
[6:11] people who are doing those jobs can work
[6:12] a few hours a week instead of 60 hours a
[6:14] week. Um they don't need two jobs
[6:16] anymore. They can get paid lots of money
[6:17] for doing one job because they're just
[6:19] as productive using AI assistance. But
[6:22] we know it's not going to be like that.
[6:24] We know what's going to happen is the
[6:26] extremely rich are going to get even
[6:28] more extremely rich and the not very
[6:31] welloff are going to have to work three
[6:33] jobs. Now I think no one likes this
[6:35] question but we like to ask it this idea
[6:38] of p doom how likely it is and I am
[6:42] curious if you see this as a a quite
[6:45] possible thing or it's just so bad that
[6:49] even though the likelihood isn't very
[6:50] high we should just be very concerned
[6:52] about it. where are you on that scale of
[6:54] probability?
[6:56] So I think um most of the experts in the
[6:59] field would
[7:01] agree that if you consider the
[7:04] possibility that these things will get
[7:05] much smarter than us and then just take
[7:07] control away from us just take over the
[7:10] probability of that happening is very
[7:13] likely more than 1% and very likely less
[7:16] than 99%. Yeah, I think all the pretty
[7:19] much all the experts can agree on that
[7:21] but that's not very helpful. No, but
[7:23] it's a good start. It it might happen
[7:26] and it might not happen and then
[7:28] different people disagree on what the
[7:30] numbers are. I'm in the unfortunate
[7:33] position of happening to agree with Elon
[7:34] Musk on this. Um, which is that it's
[7:37] sort of 10 to 20% chance that these
[7:39] things will take over. Um, but that's
[7:41] just a wild guess. Yeah. Um, I think
[7:45] reasonable people would say it's quite a
[7:47] lot more than 1% and quite a lot less
[7:48] than 99%. But we're dealing with
[7:51] something we've got no experience of.
[7:54] Um, we have no real good way of
[7:57] estimating what the probabilities are.
[7:59] It seems to me at this point it's
[8:00] inevitable that we're going to find out.
[8:03] We are going to find out. Yes, we
[8:05] because um it seems extremely likely
[8:08] that these things will get smarter than
[8:10] us already. They're much more
[8:12] knowledgeable than us. So, GPT4 knows
[8:15] thousands of times more than a normal
[8:16] person. It's a not very good expert at
[8:18] everything and eventually it successes
[8:21] will be a good expert at everything. Um
[8:23] they'll be able to see connections
[8:25] between different fields that nobody's
[8:26] seen before. Yeah. Yeah. I'm als I'm
[8:29] also interested in in understanding okay
[8:31] there's this terrible 10 to 20% chance
[8:34] but or more or or more or less or less
[8:37] but let's just take as a premise that
[8:39] there's a 80% chance that they don't
[8:41] take over and wipe us out. So that's the
[8:43] most likely scenario. Do you still think
[8:45] it would be net positive or net negative
[8:48] if it's not the worst outcome? Okay, if
[8:51] we can stop them taking over
[8:54] um that would be good. The only way
[8:56] that's going to happen is if we put
[8:57] serious effort into it. But I think once
[8:59] people understand that this is coming
[9:01] there will be a lot of pressure to put
[9:03] serious effort into it. If we just carry
[9:05] on like now just trying to make profits
[9:07] it's going to happen. They're going to
[9:08] take over. Um we we have to have the
[9:11] public put pressure on governments to do
[9:12] something serious about it. But even if
[9:15] the AIs don't take over, there's the
[9:17] issue of bad actors using AI for bad
[9:19] things. So mass surveillance, for
[9:22] example, which is already happening in
[9:24] China. If you look at what's happening
[9:25] in the west of China to the weaguers, um
[9:29] the AI is terrible for them. I I to
[9:32] board a plane to come to Toronto, I had
[9:34] to take a facial recognition photo from
[9:36] our US government. Right. When I come
[9:38] into Canada, you put your passport and
[9:41] it looks at you and it looks at your
[9:43] passport. Every time it fails to
[9:45] recognize me. Um everybody else, it
[9:48] recognizes people from all different
[9:50] nationalities. It recognizes me. It
[9:52] can't recognize. And I'm particularly
[9:54] indignant since I assume it's using
[9:56] neural nets.
[9:58] You didn't carve out an exception, did
[10:00] you? No. No. It just there's something
[10:02] about me that it doesn't like.
[10:05] Um, I have to find some place to work it
[10:08] in. So, this is as good a place as any.
[10:09] Let's talk a little bit about the Nobel.
[10:11] Can you paint the picture of the day you
[10:13] found out? So, I was sort of half
[10:17] asleep. I had my cell phone upside down
[10:20] on the bedside table with the sound
[10:21] turned off.
[10:23] But when a phone call comes, the screen
[10:26] lights up and I saw this little line of
[10:28] light because I happened to be lying on
[10:30] the pillow with my head on this side and
[10:32] the it was here facing the phone rather
[10:34] than facing away. Just happened to be
[10:36] facing the phone. I saw this little line
[10:37] of light and I was in California and it
[10:40] was 1:00 in the morning and most people
[10:42] who call me on the east coast or in
[10:44] Europe. Yeah. You don't use do not
[10:46] disturb. No. No. Okay. Um I just I turn
[10:49] off the sound. I turn off the sound. Got
[10:51] it. And I thought I was just curious
[10:54] about who on earth is calling me at four
[10:56] o'clock in the morning on the east
[10:57] coast. This is crazy. So I picked it up
[11:00] and there was this long phone number
[11:03] with a country code I didn't recognize.
[11:05] And then this Swedish voice comes on and
[11:07] asks if it's me and I say, "Yeah, it's
[11:09] me." And they say, "I won the Nobel
[11:11] Prize in physics." Well, I don't do
[11:12] physics, right? So I thought this might
[11:17] be a prank. In fact, I thought the most
[11:19] likely thing was that it was a prank. I
[11:21] was aware that the Nobel prizes were
[11:23] coming up. Okay. Because I was very
[11:25] interested in whether Demis would get
[11:26] the Nobel Prize for chemistry and I knew
[11:28] that was being announced the next day.
[11:30] Okay. Um but I sort of I don't do
[11:33] physics. I'm a psychologist hiding in
[11:36] computer science and I get the Nobel
[11:38] Prize in physics. Was it a mistake?
[11:41] Well, one thing that occurred to me is
[11:43] if it's a mistake, can they take it
[11:44] back? So, but for the next couple of
[11:47] days, I did the following reasoning. So
[11:50] what's the chance a psychologist will
[11:51] get the Nobel Prize in physics? Well
[11:54] maybe one in two million. Now, what's
[11:56] the chance if it's my dream I'll get the
[11:59] Nobel Prize in physics? Well, maybe one
[12:01] in two. So, if it's one in two in my
[12:04] dream and one in two million in reality
[12:06] that makes it a million times more
[12:08] likely that this is a dream than that
[12:10] it's reality. And for the next couple of
[12:12] days, I went around thinking, you know
[12:15] are you quite sure this isn't a dream?
[12:17] You've walked me into this very wacky
[12:18] territory, but it is part of this
[12:20] discussion. Some people think we're
[12:22] living in a simulation and that AGI is
[12:25] not evidence, but hints toward maybe
[12:28] that's the reality in which we live.
[12:30] Yeah, I don't really believe that. I
[12:32] think that's kind of wacky. Okay, so
[12:33] let's put But I don't think I don't
[12:34] think it's totally nonsense. I've seen
[12:36] the Matrix, too. Oh, okay. Okay. Wacky
[12:38] but not totally. Okay. I thought here's
[12:41] where I kind of wanted to head with the
[12:42] Nobel. Um, I think you've said something
[12:45] to the effect of you hope to use your
[12:49] credibility to convey a message to the
[12:52] world. Can you kind of explain what that
[12:54] is? Yes. That um AI is potentially very
[12:57] dangerous and there's two sets of
[12:59] dangers. There's bad actors using it for
[13:02] bad things and there's AI itself taking
[13:04] over and they're quite different kinds
[13:06] of threat. And we know bad actors are
[13:08] already using it for bad things. I mean
[13:10] it's it was used during Brexit to make
[13:15] British people vote to leave Europe in a
[13:17] crazy way. So, a company called
[13:19] Cambridge Analytica was getting
[13:21] information from Facebook and using AI.
[13:23] Um, and AI's developed a lot since then.
[13:26] It was probably used to get Trump
[13:28] elected. I
[13:29] mean, they had information from Facebook
[13:32] and it probably helped with that. We
[13:34] don't know for sure because it was never
[13:35] really investigated. Um, but now it's
[13:39] much more comp competent and so people
[13:42] can use it far more effectively for
[13:44] things like cyber attacks. Um, designing
[13:47] new
[13:48] viruses. Um, obviously fake videos for
[13:52] manipulating elections. Um, targeted
[13:55] fake videos by using information about
[13:58] people to give them just what will make
[14:00] them indignant. Yeah.
[14:03] um autonomous lethal weapons. They're
[14:07] all the big arms selling countries are
[14:10] busy trying to make autonomous lethal
[14:11] weapons. America and Russia and China
[14:13] and Britain and Israel. I think Canada's
[14:16] probably a bit too wimpy for that. The
[14:20] question then is what to do about it.
[14:23] What type of regulation do you think we
[14:25] should pursue? Okay, so we need to
[14:27] distinguish these two different kinds of
[14:28] threat. the bad actors using it for bad
[14:31] things and the AI itself taking over.
[14:34] I've talked mainly about that second
[14:36] threat, not because I think it's more
[14:38] important than the other threats, but
[14:39] because people thought it was science
[14:41] fiction. And I want to use my reputation
[14:43] to say no, it's not science fiction. We
[14:45] really need to worry about that. Um, and
[14:48] if you ask what should we do about it
[14:50] it's not like climate change. Climate
[14:52] change, just stop burning carbon and
[14:55] it'll all be okay in the long run. It'll
[14:57] be terrible for a while, but in the long
[14:58] run, it'll be okay if you don't burn
[14:59] carbon. Um, for AI taking over, we don't
[15:03] know what to do about it. We don't know.
[15:05] For example, the researchers don't know
[15:07] if there's any way to prevent that, but
[15:09] we should certainly try very hard, and
[15:12] the big companies aren't going to do
[15:13] that. If you look what the big companies
[15:14] are doing right now, they're lobbying to
[15:17] get less AI regulation. There's hardly
[15:19] any regulation as it is, but they want
[15:20] less um because they want short-term
[15:23] profits. We need people to put pressure
[15:26] on governments to insist that the big
[15:29] companies do serious safety research. So
[15:32] in California, they had very sensible
[15:34] bill, bill 1047, where they said that at
[15:38] least what big companies have to do is
[15:39] test things carefully and report the
[15:41] results of their tests. And they didn't
[15:43] even like that. So does that make you
[15:46] think regulation will not happen or how
[15:48] does it happen? It depends very much on
[15:50] what governments we get. Um I think
[15:52] under the current US government
[15:54] regulation is not going to happen. Um
[15:57] all of the big AI companies have got
[16:00] into bed with Trump and yeah it's just a
[16:04] bad situation. Elon Musk who is
[16:07] obviously so imshed in the Trump
[16:10] administration has been someone
[16:11] concerned about AI safety for a very
[16:13] long time. Yes, he's a funny mixture.
[16:16] Um, he has some crazy views like going
[16:20] to Mars, which I just think is
[16:21] completely crazy. However, because it
[16:24] won't happen or because it shouldn't be
[16:25] a priority. Because however bad you make
[16:28] the Earth, it's always going to be way
[16:30] more hospitable than Mars. Even if you
[16:32] had a global nuclear war, the Earth is
[16:35] going to be much more hospitable than
[16:36] Mars. Mars just isn't hospitable. Um
[16:39] obviously he's done some great things
[16:40] like electric cars and um helping
[16:44] Ukraine with communications with his
[16:46] Starlink. Um so he's done some good
[16:49] things, but right now he seems to be
[16:53] um fueled by powering ketamine and
[16:59] um he's doing a lot of crazy things. So
[17:02] he's got this funny mixture of views.
[17:03] So, so his history of being concerned
[17:06] about AI safety doesn't make you feel
[17:08] any better about the current
[17:09] administration. I don't think it's going
[17:10] to slow him down from doing unsafe
[17:13] things with AI. So, already they're
[17:15] releasing the weights for their AI large
[17:19] language models. Um, which is a crazy
[17:22] thing to do. Okay. These companies
[17:23] should not be releasing the weights.
[17:25] Meta releases the weights. Open AAI just
[17:26] announced they're about to release
[17:27] weights. Do you think that's I don't
[17:29] think they should be doing that because
[17:30] once you release the weights, you've got
[17:32] rid of the main barrier to using these
[17:35] things. So if you look at nuclear
[17:37] weapons, the reason only a few countries
[17:40] have nuclear weapons is because it's
[17:41] hard to get the file material. If you
[17:44] were to be able to buy file material on
[17:46] Amazon, many more companies would have
[17:48] nuclear many more countries would have
[17:50] nuclear weapons. Um, the equivalent of
[17:53] physile material for AI is the weights
[17:56] of a big model because it costs hundreds
[17:58] of millions of dollars to train a really
[18:00] big model. Not maybe the final training
[18:03] run, but all the research that goes into
[18:05] the things you do before the final
[18:06] training run. Hundreds of millions of
[18:08] dollars which a small cult or a bunch of
[18:11] cyber criminals can't afford. Um, once
[18:14] you release the weights, they can then
[18:16] start from there and fine-tune it for
[18:18] doing all sorts of things for just a few
[18:19] million dollars. So it's I think it's
[18:21] just crazy releasing weights and people
[18:22] talk about it like open source but it's
[18:25] very very different from open source. In
[18:27] open source software you release the
[18:29] code and then lots of people look at
[18:32] that code and say hey that might be a
[18:34] bug in that line and so they fix it.
[18:36] When you release the weights people
[18:37] don't look at the weights and say hey
[18:38] that weight might be a little bit wrong.
[18:40] No they just take this foundation model
[18:43] with the weights they've got now and
[18:44] they train it to do something bad. Yeah.
[18:46] The problem with the argument though, as
[18:48] articulated by your former colleague Yan
[18:50] Lakun among others, is the alternative
[18:52] is you have this tiny handful of
[18:54] companies that control this massively
[18:56] powerful technology. I think that's
[18:58] better than everybody controlling the
[18:59] massively powerful technology. I mean
[19:02] you could say the same for nuclear
[19:03] weapons. Would you like to have just a
[19:04] few countries controlling them or don't
[19:05] you think everybody should have them?
[19:07] One thing I'm taking from this is you
[19:09] have real concerns about it sounds like
[19:12] all of the major companies right now
[19:15] doing what's in society's best interest
[19:17] rather than what's in their profit
[19:19] motive. Is that the right way to hear
[19:20] you? I think the way companies work is
[19:24] they're legally required to try and
[19:26] maximize profits for their shareholders.
[19:28] They're not legally required. Well
[19:31] maybe public interest companies are, but
[19:32] most of them aren't legally required to
[19:35] do things that are good for society.
[19:37] Which, if any of them would you feel
[19:38] good about working for today?
[19:41] I used to feel good about working for
[19:43] Google because Google was very
[19:44] responsible. Um, it didn't release these
[19:47] big, it was the first to have these big
[19:48] chat bots and it didn't release them.
[19:50] Um, I'd feel less happy working for them
[19:53] today.
[19:56] Um, yeah, I wouldn't be happy working
[19:58] for any of them today. If if I worked
[19:59] for any of them, I'd be more happy with
[20:01] Google than most of the others. But were
[20:03] you disappointed when Google went back
[20:04] on its promise not to support uh
[20:06] military uses of AI? Very disappointed.
[20:09] I was very particularly since I knew
[20:11] Sergi Brin didn't didn't like military
[20:14] use of AI. But why do you think they did
[20:15] it?
[20:18] I can't really speculate with any inside
[20:21] information. I don't have any inside
[20:22] information about where they did it. I
[20:24] could speculate that they were worried
[20:27] about
[20:29] um being illreated by the current
[20:32] administration if they wouldn't um use
[20:35] their technology to make weapons for the
[20:36] US. Here's the toughest question I'll
[20:39] probably ask you today. Do you not still
[20:41] hold a lot of Google stock still? Um I
[20:44] hold some Google stock. Um most of my
[20:48] savings are not in Google stock anymore.
[20:51] Um, but yeah, I hold some Google stock
[20:54] and when Google goes up, I'm happy and
[20:55] when it goes down, I'm unhappy. So, I
[20:57] have a vested interest in Google. But I
[21:00] if they put in strong AI regulations
[21:03] that made Google less valuable, but um
[21:07] increase the chance of humanity
[21:09] surviving, I'd be very happy. Um, one of
[21:12] the most prominent labs has obviously
[21:14] been Open AI and they have lost so many
[21:17] of their top people. What have you made
[21:19] of that? Um, that open AI was set up
[21:24] explicitly to develop super intelligence
[21:27] safely and as the years went by, safety
[21:30] went more and more into the background.
[21:32] They were going to spend a certain
[21:34] fraction of their computation on safety
[21:36] and then they reaged on that. So, and
[21:39] now they're trying to go public. They're
[21:40] not now trying to be a for-profit
[21:42] company. um they're trying to get rid of
[21:44] all the um basically all the commitment
[21:47] to safety as far as I can see. So, and
[21:50] they've lost a lot of really good
[21:51] researchers in particular a former
[21:53] student of mine, Ilia Sutska, who's a
[21:55] really good researcher and was one of
[21:57] the people largely responsible for their
[21:59] development of GPT2 and then from there
[22:01] on to GPT4. Um did you talk to him
[22:05] before all that drama that led to his
[22:06] departure? No, he's very discreet. He
[22:09] doesn't talk he wouldn't talk to me
[22:11] about anything that was confidential to
[22:13] open AI. Um I was quite proud of him for
[22:17] firing Sam Arman even though it was very
[22:20] naive. So the problem was that OpenAI
[22:25] was about to have a new funding round
[22:28] and in that new funding round all the
[22:30] employees were going to be able to turn
[22:31] their paper money in OpenAI shares into
[22:34] real money. Yeah. Paper money meaning
[22:36] really hypothetical money. hypothetical
[22:38] money that would disappear if Open AI
[22:40] went bust. Tough time for an
[22:41] insurrection. So, a week or two before
[22:44] everybody's going to get maybe of the
[22:46] order of a million dollars each by
[22:47] cashing in their shares. Um maybe more.
[22:51] That's a bad time for an insurrection.
[22:53] So, the employees massively came out in
[22:54] favor of Sam Antman. But it wasn't
[22:56] because they um wanted Sam Antman. It's
[22:59] because they wanted to get that be able
[23:01] to turn their paper money into real
[23:02] money. Yeah. So, it was naive to do it
[23:06] then. Did it surprise you that he made
[23:08] that mistake or was this kind of the
[23:10] principled but maybe not fully
[23:12] calculated decision that you would
[23:13] expect?
[23:15] I don't know. Ilia is brilliant and has
[23:19] a strong moral compass. So, he's he's
[23:22] good on morality and he's very good
[23:24] technically, but in terms of
[23:26] manipulating people, he's maybe not so
[23:28] good. M I mean this is a little bit of a
[23:31] a wild card question but I do think it's
[23:32] interesting and relevant to the field
[23:34] and relevant to people discussing what's
[23:36] going on. You talked about Ilia being
[23:38] discreet. There does seem to be this
[23:40] culture of NDAs throughout the industry
[23:43] and so it's hard to even know what
[23:45] people think because people are
[23:46] unwilling or unable to even discuss
[23:48] what's going on. I'm not sure I can
[23:51] comment on that because when I left
[23:52] Google I I think I had to sign a whole
[23:54] bunch of NDAs. In fact, when I joined
[23:56] Google, I think I had to sign a whole
[23:58] bunch of NDAs that would apply when I
[23:59] left, and I have no idea what they said.
[24:01] I can't remember them anymore. Do you
[24:04] feel at all muzzled by them? No. Okay.
[24:06] Do you think it's a factor though that
[24:08] the public has a harder time
[24:10] understanding what's going on because
[24:11] people aren't allowed to tell us what's
[24:13] going on? I don't really know. I You'd
[24:15] have to know. You'd have to know which
[24:18] people weren't telling you. Okay. So
[24:20] you don't see this as a I don't see it
[24:21] as a big deal. It's a big deal. Got it.
[24:23] I think it was a big deal that Open AI
[24:26] appeared to have something um that said
[24:29] that if you'd already got shares, they
[24:32] could take the money away from you. Um
[24:35] yeah, that I think was a big deal and
[24:37] they they rapidly backed down on that
[24:38] when that became public. That was what
[24:40] their public statement said they did.
[24:42] They didn't present any contracts for
[24:44] the public to judge whether they had
[24:45] reversed that, but they said they had
[24:46] reversed it. Yes. Um there's a number of
[24:48] just important kind of hot buttony
[24:51] things. Hot button is actually not even
[24:53] a great word, but relevant issues I just
[24:54] like to get your your feedback on. One
[24:57] is the US and kind of the West's
[24:59] orientation to China in their efforts to
[25:02] pursue AI. Do you agree with this idea
[25:04] that we should be trying to restrain
[25:06] China? There's this idea of export
[25:08] controls, this idea that we should have
[25:09] democracies reach AGI first. What's your
[25:13] thinking on all that? First of all, you
[25:15] have to decide which countries are still
[25:17] democracies.
[25:19] Um and my thinking on that is in the
[25:23] long run it's not going to make much
[25:25] difference. It may slow things down by a
[25:26] few years but clearly um if you prevent
[25:30] China from getting the most advanced
[25:31] technology people know how this advanced
[25:33] technology works. So, China's just
[25:36] invested many many billions maybe
[25:39] hundreds of billions um of the order of
[25:41] 100 billion I think in making
[25:44] lithography machines or in in in getting
[25:47] their own homebased technology that does
[25:49] this stuff. Um so it'll slow them down a
[25:53] bit but it will actually force them to
[25:56] develop their own industry and in the
[25:57] long run um they're very competent and
[26:00] they will and so it'll just slow things
[26:02] down for a few years. But race is the
[26:04] right framework. We shouldn't be trying
[26:06] to cooperate with communist China. I
[26:09] wouldn't describe it as communist
[26:10] anymore. I used the loaded term
[26:13] specifically because why wouldn't you
[26:16] cooperate, right? The only rationale to
[26:18] not cooperate is if you think they're a
[26:21] malignant force. Well, there's areas in
[26:23] which we won't cooperate where we is, I
[26:26] guess, I'm not sure who we is anymore
[26:28] because I'm in Canada now and we used to
[26:30] be sort of Canada and the US, but it's
[26:31] not anymore. Yeah. Um obviously the
[26:35] countries are not going to cooperate on
[26:37] developing lethal autonomous weapons
[26:39] because the lethal autonomous weapons to
[26:41] be used against other countries. So but
[26:44] we've had treaties and other types of
[26:46] weapons as you've pointed out. We could
[26:47] have treaties not to develop them but
[26:49] cooperating in making them better.
[26:51] They're not going to do that. Sure.
[26:53] Sure. Sure. Now there is one area where
[26:55] they will cooperate which is on the
[26:57] existential threat. if they ever get
[27:00] serious about worrying about the
[27:02] existential threat and doing stuff about
[27:03] it, they will collaborate on that ways
[27:06] of stopping AI taking over because we're
[27:08] all in the same boat. So, at the height
[27:10] of the Cold War, the Soviet Union and
[27:12] the US collaborated on preventing a
[27:14] global nuclear war and even countries
[27:18] that are very hostile to each other will
[27:20] collaborate when their interests align
[27:22] and their interests will align when it's
[27:24] AI versus humanity.
[27:27] Um, there's this question of fair use
[27:30] whether it's okay to have the content of
[27:33] billions of humans created over many
[27:35] years kind of scooped up and repurposed
[27:39] into models that will replace some of
[27:42] those same people that created the
[27:44] training data. Where do you fall on
[27:45] that? I think I sort of fall all over
[27:49] the place on that in the sense that it's
[27:51] a very complicated issue. So initially
[27:55] it seems yeah they should have to pay
[27:57] pay for that. But suppose I have a
[28:00] musician who produces a song in a
[28:02] particular genre and ask well how did
[28:05] they produce the song in that genre?
[28:07] Where did where did their ability to
[28:08] produce songs in that genre came from?
[28:10] Well it came from listening to songs by
[28:11] other musicians in that genre. So they
[28:14] listen to these songs, they kind of
[28:16] internalize things about the structure
[28:17] of the songs and then they generated
[28:19] stuff in that genre and the stuff they
[28:21] generated is different. So it's not
[28:24] theft um and that's accepted. Well
[28:27] that's what the AI is doing. The AI is
[28:29] absorbing all this information and then
[28:31] producing new stuff. It's not just
[28:33] taking taking and patching it together.
[28:35] It's generating new stuff that has the
[28:37] same underlying themes. And so it's no
[28:41] more stealing than a person does when
[28:44] they do the same thing. But the point is
[28:46] it's doing it um at a massive scale. And
[28:51] no musician has ever put every other
[28:53] musician out of business. Exactly. So in
[28:56] Britain for example, the government
[28:59] doesn't seem to have any interest in
[29:03] protecting the creative artists. And if
[29:07] you look at the economy, the creative
[29:08] artists are work worth a lot to Britain.
[29:11] So I have a friend called BB Bankron
[29:14] saying we should protect creative
[29:15] artists. It's very important to the
[29:18] economy and just letting AI walk off
[29:20] with it all um seems unfair. UBI
[29:24] universal basic income, is this part of
[29:26] the solution to the displacements of AI?
[29:28] You think? I think it may be necessary
[29:31] to stop people starving. Um I don't
[29:34] think it totally solves the problem but
[29:37] even if you had quite high UBI
[29:40] um it doesn't solve the problem of human
[29:42] dignity for a lot of people um who they
[29:45] are is particularly for academics who
[29:48] they are is mixed up in their work.
[29:50] That's who they are. If they become
[29:52] unemployed just getting the same money
[29:54] doesn't totally compensate. They're not
[29:56] who they are anymore. Yeah. I tend to
[29:59] think that's true as well. I saw you
[30:00] give this quote at one point though
[30:01] where you said you might have been
[30:02] happier if you were a woodworker. Well
[30:05] yes, cuz I I really like being a
[30:07] carpenter. And isn't there an
[30:09] alternative where you're born a hundred
[30:10] years later where you don't have to
[30:12] waste all your time on these neural nets
[30:13] and you just get to enjoy woodworking
[30:16] while taking in a monthly income? Yeah
[30:18] but there's a difference between doing
[30:20] it as a hobby and doing it to make a
[30:22] living somehow. It's more real doing it
[30:23] to make a living. So you don't think a
[30:25] future where we get to pursue our
[30:27] hobbies and don't have to contribute to
[30:29] the economy? That might that might be
[30:30] fine. Yeah. Um if everybody was doing
[30:33] that, but if you're in some
[30:35] disadvantaged group who are getting
[30:37] universal basic income and you're
[30:39] getting less income than um other people
[30:43] because employers will want you to do
[30:44] that so they can get other people to
[30:46] work for them. Um that's going to be
[30:48] very different. I'm interested in this
[30:51] idea of robot rights. I don't know if
[30:54] there's a better term to describe it
[30:55] but at some point you're going to have
[30:57] these massively intelligent AIs. They're
[30:59] going to be agentic and doing all kinds
[31:01] of things in the world. Should they be
[31:03] able to own property? Should they be
[31:05] able to vote? Should they be able to
[31:07] marry humans in a loving relationship?
[31:10] Like what what or even if they if
[31:11] they're just smarter than us and if it's
[31:13] a better form of intelligence than what
[31:15] we've got, um should it be fine for them
[31:18] to just take over and humans be history?
[31:20] Yeah, let's go to that bigger idea
[31:23] second. Would I'm curious on the on the
[31:25] more narrow idea unless you think the
[31:27] narrow questions are irrelevant because
[31:28] the big question takes pre. No, I think
[31:30] the narrow questions irrelevant. Yeah.
[31:32] So, I used to be worried about this
[31:34] question. I used to think, well, if
[31:36] they're smarter than us, um, why
[31:38] shouldn't they have the same rights as
[31:39] us? Yeah. And now I think, well, we're
[31:43] people. What we care about is people.
[31:46] Um, I eat cows. I mean, I know lots of
[31:49] people don't, but I eat cows. And the
[31:51] reason I'm happy eating cows is because
[31:52] they're cows. Um, and I'm a person. Um
[31:56] and the same for these super intelligent
[31:58] AIs. They may be smarter than us, but
[32:00] what I care about is people. And so, I'm
[32:02] willing to be mean to them. I'm willing
[32:05] to deny them their rights because I want
[32:09] what's best for people. Yeah. Um, now
[32:12] they won't agree with that and they may
[32:15] win
[32:16] but that's my current position on
[32:19] whether AI should have rights, which is
[32:21] even if they're intelligent, even if
[32:22] they have sensations and emotions and
[32:24] feelings and all that stuff, um, they're
[32:26] not people, and people's what I care
[32:27] about, but they're going to seem so much
[32:29] like people. I feel like it's going to
[32:31] they're going to be able to fake it.
[32:32] Yes. They're going to be able to seem
[32:32] very like people. Yeah. Yeah. Do you
[32:35] suspect we'll end up giving them rights?
[32:38] I don't know. Okay. I tend to avoid this
[32:41] issue because there's more immediate
[32:44] problems like bad uses of AI or the
[32:47] issue of whether they will try and take
[32:48] over and how to prevent that. Yeah. And
[32:51] it sounds kind of flaky if you start
[32:52] talking about them having rights. Most
[32:54] people you've lost most people when you
[32:57] go there. Even just sticking with people
[32:59] there seems to be real soon, if it's not
[33:02] already here, this um ability to use AI
[33:06] to select what babies we have. Are you
[33:08] concerned at all about that line embryo
[33:11] selection? You mean selecting for the
[33:13] sex or selecting for the intelligence
[33:16] and the eye color and the likelihood to
[33:18] get pancreatic cancer and the you know
[33:21] the list goes down and down and down of
[33:22] all the things we might select. I think
[33:24] if you could select a baby that was less
[33:25] likely to get pancreatic cancer that
[33:27] would be a great thing. I'm willing to
[33:29] say that. Okay. So we this is a thing we
[33:32] should pursue. We should make healthier
[33:33] stronger, better babies.
[33:36] Um it's very difficult territory. Right.
[33:39] It is. That's why I'm asking about it.
[33:41] But some aspects of it um seem to make
[33:44] sense to me. Like if you're an a normal
[33:49] healthy couple and you have a fetus and
[33:53] you can predict that it's going to have
[33:55] very serious problems and maybe not live
[33:57] very long. Um it seems to me it makes
[34:01] sense to abort it and have a healthy
[34:04] baby. that just seems sensible to me.
[34:06] Now, I know a lot of religious people
[34:07] wouldn't agree with that at all. Um, but
[34:10] for me, if you could make those
[34:12] predictions reliably, um, that just
[34:15] seems to make sense to me. I've been a
[34:18] little bit holding us back from kind of
[34:20] the central thing that I think you want
[34:22] people to take away, which is this idea
[34:24] of of machines taking over and the
[34:26] impact of that. So, I'd like to just
[34:28] discuss that as fully as you'd like or
[34:31] that we can. like how do you want to
[34:33] frame this issue? How should people
[34:35] think about it? One thing to bear in
[34:38] mind is how many examples do you know of
[34:40] less intelligent things controlling much
[34:42] more intelligent things? So we know that
[34:45] things are more or less equal
[34:46] intelligence, the less intelligent one
[34:48] can control the more intelligent one. Um
[34:51] but with a big gap in intelligence
[34:53] there's very very few examples where the
[34:56] more intelligent one isn't in control.
[34:58] So that's something you should bear in
[34:59] mind. That's a big worry. I think the
[35:02] situation we're in right
[35:04] now, the best way to understand it
[35:07] emotionally is we're like somebody who
[35:10] has this really cute tiger cup. It's
[35:14] just such a cute tiger
[35:17] cup. Now, unless you can be very sure
[35:22] that it's not going to want to kill you
[35:23] when it's grown up, you should worry. M.
[35:28] And to extend the metaphor, you put it
[35:30] in a cage, you kill it. What do you do
[35:32] with the tiger cub? Well, the point
[35:34] about the tiger cub is it's just
[35:36] physically stronger than you. So, you
[35:38] can still control it because you're more
[35:39] intelligent. Yeah. Um, things that are
[35:42] more intelligent than you. We have no
[35:43] experience of that, right? People aren't
[35:45] used to thinking about it. People think
[35:48] somehow you constrain it. You don't
[35:50] allow it to press buttons or whatever.
[35:51] Um, things more intelligent than you
[35:54] they're going to be able to manipulate
[35:56] you. So another way of thinking about it
[35:57] is imagine that there's this
[36:00] kindergarten. There's these two and
[36:02] three year olds and the two and three
[36:03] year olds are in charge and you just
[36:05] work for them in the kindergarten and
[36:07] you're not that much more intelligent
[36:09] than a two or threey old. Not compared
[36:10] with super intelligence, but you are
[36:11] more intelligent. Um so how hard would
[36:14] it be for you to get control? Well, you
[36:17] just tell them all you're going to get
[36:18] free candy and if they just sort of sign
[36:21] this or just agree verbally to this um
[36:24] you get free candy for as long as you
[36:25] like. and you'll be in control. They
[36:28] won't they won't have any idea what's
[36:30] going on. And with super intelligences
[36:33] they're going to be so much smarter than
[36:34] us, we'll have no idea what they're up
[36:37] to. And so what do we do?
[36:41] Um, we worry
[36:43] about whether there's a way to build a
[36:47] super intelligence so that it doesn't
[36:49] want to take control. I don't think
[36:51] there's a way of stopping it take
[36:52] control if it wants to. So there's one
[36:54] possibility is never build a super
[36:56] intelligence. You think that's possible?
[36:59] I mean it's conceivable, but I don't
[37:01] think it's going to happen because
[37:02] there's too many too much competition
[37:04] between countries and between companies
[37:06] and they're all after the next shiny
[37:08] thing and it's developing very very
[37:10] fast. So I don't think we're going to be
[37:13] able to avoid building super
[37:14] intelligence. It's going to happen. The
[37:17] issue is can we design it in such a way
[37:20] that it never wants to take control that
[37:22] it's always benevolent. Um that's a very
[37:26] tricky issue. Just people say well we'll
[37:28] get it to align with human interests.
[37:31] But human interests don't align with
[37:33] each other. And if I say I've got two
[37:35] lines at right angle and I want you to
[37:37] show me a line parallel to both of them.
[37:39] That's kind of tricky, right? And if you
[37:42] look at the Middle East for example
[37:45] there's people with very strong views
[37:47] that don't align. So how are you going
[37:50] to get AI to align with human interests?
[37:53] Human interests don't align with each
[37:55] other. So that's one problem. It's going
[37:57] to be very hard to figure out how to get
[38:01] super intelligence that doesn't want to
[38:03] take over and doesn't want to ever hurt
[38:05] us. Um but we should certainly try. And
[38:09] trying is kind of just an iterative
[38:10] process. Month by month, year by year
[38:13] we try to Yeah. So obviously if you're
[38:15] going to develop something that might
[38:17] want to take
[38:19] over when it's just slightly less
[38:21] intelligent than you are, and we're very
[38:23] close to that now, um you should kind of
[38:26] look at what it'll do to try and take
[38:28] over. So if you look at the current AIS
[38:31] you can see they're already capable of
[38:33] deliberate deception. They're capable of
[38:35] pretending to be stupider than they are.
[38:38] um of lying to you so that they can kind
[38:41] of confuse you into not understanding
[38:43] what they're up to. Um we need to be
[38:46] very aware of all that and to study all
[38:48] that and study about whether there's a
[38:49] way to stop them doing that. When we
[38:51] spoke a couple years ago, I was
[38:53] surprised at you voicing concerns
[38:55] because you hadn't really done much of
[38:57] that before and now you're voicing them
[39:00] quite clearly and loudly. Was it mostly
[39:04] that you felt more liberated to say this
[39:06] stuff or was it really a really big sea
[39:09] change in how you saw it in these last
[39:10] few years? When we spoke a couple of
[39:12] years ago, I was still working at Google
[39:14] then. It was in March and I didn't
[39:16] resign till um the end of April. Um but
[39:20] I was thinking about leaving then. Um
[39:22] and I had had a kind of epiphany before
[39:25] we spoke where I realized that these
[39:27] things might be a better form of
[39:29] intelligence than us. And that got me
[39:31] very scared. And you didn't think that
[39:34] before just because you thought the time
[39:35] horizon was so different. No, it wasn't
[39:37] just that. It was because of the
[39:38] research I was doing at Google. Okay. I
[39:40] was trying to figure out whether you
[39:43] could design analog large language
[39:46] models that would use much less power.
[39:49] Mhm. Um, and I began to fully realize
[39:53] the advantage of being digital. So all
[39:55] the models we've got at present are
[39:56] digital. And if you're a digital model
[40:01] you can have exactly the same neural
[40:04] network with the same weights in it
[40:06] running on several different pieces of
[40:08] hardware, like thousands of different
[40:09] pieces of
[40:10] hardware. And then you can get one piece
[40:12] of hardware to look at one bit of the
[40:14] internet and another piece of hardware
[40:15] to look at another bit of the internet.
[40:17] And each piece of hardware can say, how
[40:19] would I like to change my internal
[40:21] parameters, my weights, so I can absorb
[40:24] the information I just saw.
[40:27] And each of these separate pieces of
[40:29] hardware can do that. And then they can
[40:31] just average all the changes to the
[40:33] weights because they're all using the
[40:35] same weights in exactly the same way.
[40:37] And so averaging makes sense. You and I
[40:39] can't do that. And if they've got a
[40:41] trillion weights, they're sharing
[40:43] information at like a trillions of bits
[40:46] every time they do this averaging. Now
[40:49] you and I when I want to get some
[40:51] knowledge from my head into your head, I
[40:53] can't just take the strength of the
[40:54] connections between neurons and average
[40:56] them with the strength of the
[40:57] connections between your neurons because
[40:58] our neurons are different. We we're
[41:00] analog and we're just very different
[41:02] brains. So the only way I have getting
[41:05] knowledge to you is I do some actions
[41:08] and if you trust me, you try and change
[41:10] the connection strengths in your brain
[41:13] so that you might do the same things.
[41:15] And if you ask, well, how efficient is
[41:17] that? Well, if I give you a sentence
[41:19] it's only a few hundred bits of
[41:21] information at most. So, it's very slow.
[41:23] We communicate just a few bits per
[41:26] second. These large language models
[41:29] running on digital systems can
[41:31] communicate trillions of bits a second.
[41:34] So, they're billions of times better
[41:35] than us at sharing information. That got
[41:38] me scared. Right. But what surprised you
[41:40] or what changed your thinking was you
[41:41] were thinking the analog was going to be
[41:43] the path previously. No, I was thinking
[41:46] if we want to use much less power, yeah
[41:49] we should think about whether it's
[41:50] possible to do this analog. Yeah. And
[41:52] because you can use much less power, you
[41:54] can also be much sloppier in the design
[41:56] of the system. Because what's going to
[41:58] happen is you don't have to manufacture
[42:00] a system that does precisely what you
[42:02] tell it to, which is what a computer is.
[42:04] You can manufacture a system with a lot
[42:06] of slop in it, and it will learn to use
[42:08] that sloppy system, which is what our
[42:10] brains are. Do you think the technology
[42:11] is no longer destined for that solution
[42:14] but is going to stick with the digital
[42:15] solution? I think it'll probably stick
[42:17] with a digital solution. Now, it's quite
[42:19] possible that we can get these digital
[42:21] computers to design better analog
[42:24] hardware better than us. Um, I think
[42:27] that may be the long-term future. You
[42:29] got into this field because you wanted
[42:30] to know how the brain works. Yes. Do you
[42:32] think we're getting closer to that
[42:34] through this? I think for a while we
[42:36] did. So I think we've learned a lot at a
[42:39] very general level about how the brain
[42:41] works.
[42:42] So 30 years ago or 50 years ago, if you
[42:46] ask people, well, could you have a big
[42:48] random neural network with random
[42:50] connection strengths and then could you
[42:52] show it data and have it learn to do
[42:55] difficult things like recognize what
[42:57] someone's saying or answer questions
[43:00] just by showing it lots of data? Almost
[43:02] everybody would have said, "That's
[43:03] crazy. There's no way you're going to do
[43:05] that. it has to have lots of pre-wired
[43:07] structure that comes from evolution.
[43:09] Well, it turns out they were wrong. It
[43:11] turns out you can have a big random
[43:13] neural network. Um, and it can learn
[43:15] just from data. Now, that doesn't mean
[43:17] we don't have a lot of pre-wired
[43:18] structure, but basically most of what we
[43:22] know comes from learning from data, not
[43:24] from all this pre-wired structure. So
[43:26] that's a huge advance in understanding
[43:28] the brain. Now, the issue is how do you
[43:31] get the information that tells you
[43:32] whether to increase or decrease the
[43:34] connection strength? If you can get that
[43:36] information, we know that we can then
[43:37] train a big system that starts with
[43:39] random weights to do wonderful things.
[43:41] The brain needs to get information like
[43:43] that and it probably gets it in a
[43:45] different way from the standard
[43:46] algorithm used in these big AI models
[43:49] which is called back propagation. The
[43:51] brain probably doesn't use back
[43:53] propagation. Nobody can figure out how
[43:55] it could be doing it. Um it's probably
[43:58] getting the gradient information that is
[44:00] how changing a weight will improve the
[44:02] performance in a different way. But we
[44:04] do know now that if it can get that
[44:06] great information, it can be really
[44:07] effective at learning. Do you know if
[44:10] any of the labs now are using their
[44:12] models to try to pursue new ideas in AI
[44:16] development? Almost certainly. Okay.
[44:19] Yeah. And in particular, Deep Mind is
[44:21] very interested in using AI for doing
[44:24] science. And one piece of science is AI.
[44:27] Sure. I mean, was that something you
[44:30] trying when you were there? like this
[44:31] bootstrapping idea of maybe the next
[44:33] innovation could be created by the AI
[44:36] itself. So there's elements of that. So
[44:39] for example, they were using AI to do
[44:42] layout on chips that were going to be
[44:43] used for AI. So Google's AI chips um
[44:47] their um tensor processing units. Um
[44:51] they used AI to develop those chips. So
[44:55] I'm curious if just in your normal
[44:57] day-to-day life you despair. You fear
[45:00] for the future and assume it won't be so
[45:03] good. I don't despair, but mainly
[45:07] because even I find it very hard to take
[45:10] it seriously. Ah, it's very hard to get
[45:13] your head around the fact that we're at
[45:14] this very very special point in history
[45:17] where in a ve relatively short time
[45:21] everything might totally change a change
[45:24] of a scale we've never seen before. Um
[45:26] it's hard to absorb that emotionally.
[45:29] It is. And I do notice even though
[45:31] people maybe are concerned, I've never
[45:34] seen a protest. There's no real
[45:36] political movement around this idea. The
[45:38] world is changing and no one really
[45:40] seems to care that much. Um, among the
[45:44] AI
[45:46] researchers, people are more aware of
[45:49] it. Um, so the people I know who are
[45:52] kind of most depressed about it are
[45:53] serious AI researchers.
[45:56] Um I have started doing practical things
[46:00] like because AI is going to be very good
[46:02] at designing
[46:04] um cyber
[46:06] attacks. Um I don't think the Canadian
[46:09] banks are safe anymore. So Canadian
[46:11] banks are about as safe as you can get.
[46:13] Okay? They're very well regulated
[46:15] compared with US banks.
[46:17] Um but over the next 10 years, I
[46:20] wouldn't be at all surprised if there
[46:22] was a cyber attack that took down a
[46:24] Canadian bank. What does take down mean?
[46:26] Suppose that the bank holds shares that
[46:29] I own, right? Suppose the cyber attack
[46:32] sells those
[46:33] shares. Now my money's gone. So I
[46:37] actually now spread my money between
[46:38] three banks. Okay. So now your mattress.
[46:41] That's the first practical thing I've
[46:42] done because I think if a cyber attack
[46:44] takes down one Canadian bank, the others
[46:46] will get a lot more serious. Okay.
[46:48] Anything else like that? What else?
[46:50] That's the main thing. That's where I
[46:51] noticed I actually did something
[46:52] practical.
[46:54] that flowed from my belief that um very
[46:58] scary times are coming. Okay. Uh when we
[47:01] spoke a couple years ago, you had said
[47:03] you know, AI is like an idiot savant
[47:05] but humans are still much better at
[47:06] reasoning, right? That's changed. Okay.
[47:09] Explain. Previously, what the large
[47:12] language models would do is they'd spit
[47:14] out one word at a time and that would be
[47:16] it. Now they spit out words and they're
[47:20] looking at the words they spit out. And
[47:23] they will spit out words that aren't the
[47:25] answer to the question yet. They'll spit
[47:27] out words. It's called chain of thought
[47:28] reasoning. And so now they can reflect
[47:31] on the words they spat out already. And
[47:33] that gives them room to do some thinking
[47:36] in and you can see what they're
[47:37] thinking. It's wonderful. Yeah. Well
[47:40] it's wonderful if you're a researcher.
[47:42] And a lot of people from old-fashioned
[47:44] AI said, "Well, you know, these things
[47:46] can't reason. They're not really
[47:48] intelligent because they can't reason.
[47:50] And you're going to need to use
[47:51] old-fashioned AI and turn things into
[47:54] logical forms in order to do proper
[47:56] reasoning. Well, they were just utterly
[47:58] wrong. Um, neural nets are going to do
[48:00] the reasoning. And the way they're going
[48:01] to do the reasoning is by this chain of
[48:03] thought, by spitting out stuff that they
[48:05] don't reflect upon. Yeah. You said at
[48:07] the beginning that the last two years
[48:09] the development has been faster than you
[48:11] expected. Are there other examples of
[48:12] that? Things you've seen that if you
[48:13] said, "Wow, it's half it's fast." That's
[48:15] the main example. It's got much better
[48:17] at generating images and things too, but
[48:20] the main thing is that it can now do
[48:22] reasoning quite well. Okay. And that you
[48:24] can see what it's thinking. Like why is
[48:26] that important or where does that lead
[48:27] that is meaningful? Um well, it's very
[48:31] good that you can see what they're
[48:32] thinking because there's these examples
[48:35] where you give it the goal. You give it
[48:39] a goal and you can see it doing
[48:41] reasoning to try and achieve this goal
[48:44] by deceiving people and you can see it
[48:46] doing that. It's like I could hear the
[48:48] voice in your head. Yeah. The other
[48:51] thing we we moved moved through, but
[48:53] maybe I don't know if you have anything
[48:54] more to say about is just it's
[48:56] remarkable that there are so many tech
[48:58] figures that are now have an important
[49:00] role in Washington DC at this very
[49:02] moment where what Washington DC does
[49:05] could be really important to the
[49:07] evolution, the regulation of this
[49:09] technology. Does that concern you? How
[49:12] do you see that?
[49:15] those tech
[49:17] figures are primarily concerned with
[49:20] their companies making
[49:22] profits. So that concerns me a lot.
[49:26] Yeah. I don't see how things really
[49:28] change unless either there's strong
[49:30] regulation or this moves away from this
[49:32] for-profit model. And I don't see how
[49:34] those things happen either. I think if
[49:37] the public realized what was happening
[49:40] they will put a lot of pressure on
[49:41] governments to insist that the AI
[49:43] companies develop this more safely.
[49:45] Okay, that's the best I can do. It's
[49:47] it's not very satisfactory, but it's the
[49:49] best I can think of. And more safely
[49:51] means more resources from those
[49:53] companies toward safety research. Yes.
[49:56] For example, the fraction of their
[49:57] computer time they spend on safety
[49:59] research should be a significant
[50:01] fraction like a third. Right now, it's
[50:04] much much less. There's one company
[50:06] Anthropic, that's more concerned with
[50:08] safety than the others. It was set up to
[50:10] be concerned with safety by people who
[50:12] left Open AI because Open AI wasn't
[50:14] enough concerned with safety. And
[50:16] Anthropic does spend more time on safety
[50:18] research, but still probably not enough.
[50:19] There is this view among many that Open
[50:22] AI has talked a good game about these
[50:24] issues, but is not living out those
[50:26] values. Is that your perspective? Yes.
[50:29] What evidence do you see of that? that
[50:32] all their best safety researchers left
[50:33] because they believed that too. Um that
[50:37] they were set up as a company um that
[50:41] was going to develop area safety and
[50:43] their main goal was not to make profits
[50:44] but to develop our safety and they're
[50:46] now busy lobbying the California
[50:49] Attorney General to allow them to change
[50:53] to a for-profit company. Um there's lots
[50:56] of evidence for that, right? Um and I
[50:58] should give you a chance to hold up
[50:59] anyone as a good actor here that people
[51:01] should feel better about. You mentioned
[51:02] Anthropic. Is that the name or do you
[51:05] see of the companies? Anthropic is the
[51:08] most concerned with safety and a lot of
[51:10] the safety researchers who left OpenAI
[51:13] went to Anthropic and so Anthropic has
[51:15] much more of a culture concerned with
[51:17] safety. Okay. But um they have
[51:20] investments from big companies. Yeah.
[51:22] you have to get money from somewhere and
[51:24] I'm worried that those investments will
[51:26] force them into um releasing things
[51:29] faster than they should. And when I
[51:31] asked you which you'd feel comfortable
[51:32] working for, you said none of them, I
[51:34] think, or just maybe Google. I should
[51:36] have said maybe Google Anthropic. Okay.
[51:39] Thank you so much for all this time and
[51:41] the rest of your time today. I really
[51:42] appreciate it. Okay. You haven't got the
[51:44] rest yet. I haven't got I'm counting on
[51:46] it.