An AI Expert Warning: 6 People Are (Quietly) Deciding Humanity’s Future!

[00:00] In October, over 850 experts, including yourself and other leaders like Richard

[00:04] Branson and Jeffrey Hinton, signed a statement to ban AI super intelligence

[00:08] as you guys raised concerns of potential human extinction. >> Because unless we figure out how do we

[00:14] guarantee that the AI systems are safe, we're toast. >> And you've been so influential on the

[00:19] subject of AI, you wrote the textbook that many of the CEOs who are building

[00:23] some of the AI companies now would have studied on the subject of AI. Yeah.

[00:26] >> So, do you have any regrets? Um, >> Professor Stuart Russell has been named

[00:33] one of Time magazine's most influential voices in AI. >> After spending over 50 years

[00:38] researching, teaching, and finding ways to design >> AI in such a way that

[00:42] >> humans maintain control, >> you talk about this gorilla problem as a

[00:46] way to understand AI in the context of humans. >> Yeah. So, a few million years ago, the

[00:50] human line branched off from the gorilla line in evolution, and now the gorillas

[00:53] have no say in whether they continue to exist because we are much smarter than

[00:57] they are. So intelligence is actually the single most important factor to control planet Earth.

[01:01] >> Yep. >> But we're in the process of making something more intelligent than us. >> Exactly.

[01:05] >> Why don't people stop then? >> Well, one of the reasons is something

[01:08] called the Midas touch. So King Midas is this legendary king who asked the gods,

[01:12] can everything I touch turn to gold? And we think of the Midas touch as being a

[01:15] good thing, but he goes to drink some water, the water has turned to gold. And

[01:19] he goes to comfort his daughter, his daughter turns to gold. So he dies in

[01:22] misery and starvation. So this applies to our current situation in two ways.

[01:26] One is that greed is driving these companies to pursue technology with the

[01:30] probabilities of extinction being worse than playing Russian roulette. And

[01:34] that's even according to the people developing the technology without our

[01:37] permission. And people are just fooling themselves if they think it's naturally

[01:41] going to be controllable. So, you know, after 50 years, I could

[01:45] retire, but instead I'm working 80 or 100 hours a week trying to move things

[01:49] in the right direction. So, if you had a button in front of you which would stop

[01:53] all progress in artificial intelligence, would you press it?

[01:58] >> Not yet. I think there's still a decent chance they guarantee safety. And I can

[02:02] explain more of what that is. >> I see messages all the time in the

[02:08] comments section that some of you didn't realize you didn't subscribe. So, if you

[02:12] could do me a favor and double check if you're a subscriber to this channel,

[02:14] that would be tremendously appreciated. It's the simple, it's the free thing

[02:18] that anybody that watches this show frequently can do to help us here to

[02:21] keep everything going in this show in the trajectory it's on. So, please do

[02:25] double check if you've subscribed and uh thank you so much because in a strange

[02:28] way you are you're part of our history and you're on this journey with us and I

[02:32] appreciate you for that. So, yeah, thank you.

[02:41] Professor Stuart Russell, OBBE. A lot of people have been talking about AI for

[02:46] the last couple of years. It appears you've this really shocked me. It

[02:50] appears you've been talking about AI for most of your life.

[02:53] >> Well, I started doing AI in high school um back in England, but then I did my

[02:59] PhD starting in ' 82 at Stanford. I joined the faculty of Berkeley in ' 86.

[03:06] So I'm in my 40th year as a professor at Berkeley. The main thing that the AI

[03:10] community is familiar with in my work uh is a textbook that I wrote.

[03:16] >> Is this the textbook that most students who study AI are likely learning from? >> Yeah.

[03:24] >> So you wrote the textbook on artificial intelligence 31 years ago. You actually start probably

[03:32] started writing it because it's so bloody big in the year that I was born. So I was born in 92.

[03:36] >> Uh yeah, took me about two years. >> Me and your book are the same age, which just is wonderful

[03:43] way for me to understand just how long you've been talking about this and how

[03:47] long you've been writing about this. And actually, it's interesting that many of

[03:51] the CEOs who are building some of the AI companies now probably learned from your

[03:56] textbook. you had a conversation with somebody who said that in order for

[04:01] people to get the message that we're going to be talking about today, there

[04:05] would have to be a catastrophe for people to wake up. Can you give me

[04:10] context on that conversation and a gist of who you had this conversation with?

[04:14] >> Uh, so it was with one of the CEOs of uh a leading AI company. He sees two

[04:21] possibilities as do I which is um either we have a small or let's say

[04:28] small scale disaster of the same scale as Chernobyl >> the nuclear meltdown in Ukraine.

[04:34] >> Yeah. So this uh nuclear plant blew up in 1986 killed uh a fair number of people directly and

[04:44] maybe tens of thousands of people indirectly through uh radiation. recent

[04:49] cost estimates more than a trillion dollars. So that would wake people up. That would

[04:58] get the governments to regulate. He's talked to the governments and they won't

[05:01] do it. So he looked at this Chernobyl scale disaster as the best case scenario

[05:09] because then the governments would regulate and require AI systems to be

[05:14] built. And is this CEO building an AI company? >> He runs one of the leading AI companies.

[05:22] >> And even he thinks that the only way that people will wake up is if there's a

[05:26] Chernobyl level nuclear disaster. >> Uh yeah, not wouldn't have to be a

[05:29] nuclear disaster. It would be either an AI system that's being misused

[05:35] by someone, for example, to engineer a pandemic or an AI system that does

[05:40] something itself, such as crashing our financial system or our communication

[05:45] systems. The alternative is a much worse disaster where we just lose control

[05:50] altogether. You have had lots of conversations with lots of people in the

[05:54] world of AI, both people that are, you know, have built the technology, have

[05:58] studied and researched the technology or the CEOs and founders that are currently

[06:02] in the AI race. What are some of the the interesting sentiments that the general

[06:07] public wouldn't believe that you hear privately about their perspectives?

[06:14] Because I find that so fascinating. I've had some private conversations with

[06:18] people very close to these tech companies and the shocking sentiment that I was exposed to was that

[06:24] they are aware of the risks often but they don't feel like there's anything

[06:27] that can be done so they're carrying on which is feels like a bit of a paradox to me like

[06:31] >> yes it's it's it must be a very difficult position to be in in a sense right you're you're

[06:38] doing something that you know has a good chance of bringing an end to life on

[06:44] including that of yourself and your own family. They feel that they can't escape this race, right?

[06:54] If they, you know, if a CEO of one of those companies was to say, you know, we're

[06:59] we're not going to do this anymore, they would just be replaced

[07:04] because the investors are putting their money up because they want to create AGI

[07:10] and reap the benefits of it. So, it's a strange situation where every at least

[07:16] all the ones I've spoken to, I haven't spoken to Sam Wolman about this, but you know, Sam Wolman

[07:23] even before becoming CEO of Open AI said that creating superhuman intelligence is the

[07:32] biggest risk to human existence that there is. My worst fears are that we

[07:38] cause significant we the field the technology the industry cause significant harm to the world.

[07:43] >> You know Elon Musk is also on record saying this. So uh Dario Ammedday

[07:48] estimates up to a 25% risk of extinction. >> Was there a particular moment when you realized that

[07:56] the CEOs are well aware of the extinction level risks? I mean, they all

[08:01] signed a statement in May of 23 uh called it's called the extinction

[08:07] statement. It basically says AGI is an extinction risk at the same level as

[08:12] nuclear war and pandemics. But I don't think they feel it in their

[08:17] gut. You know, imagine that you were one of the nuclear physicists. You know, I

[08:24] guess you've seen Oppenheimer, right? you're there, you're watching that first nuclear explosion.

[08:30] How how would that make you feel about the potential impact of nuclear war on

[08:37] the human race? Right? I I think you would probably become a pacifist and say

[08:43] this weapon is so terrible, we have got to find a way to uh keep it under

[08:49] control. We are not there yet with the people making these decisions

[08:55] and certainly not with the governments, right? You know what policy makers do is they, you know,

[09:03] they listen to experts. They keep their finger in the wind. You got some

[09:09] experts, you know, dangling $50 billion checks and saying, "Oh, you know, all

[09:15] that doomer stuff, it's just fringe nonsense. don't worry about it. Take my

[09:19] $50 billion check. You know, on the other side, you've got very

[09:23] well-meaning, brilliant scientists like like Jeff Hinton saying, actually, no,

[09:28] this is the end of the human race. But Jeff doesn't have a $50 billion check.

[09:34] So the view is the only way to stop the race is if governments intervene

[09:40] and say okay we don't we don't want this race to go ahead until we can be sure

[09:47] that it's going ahead in absolute safety. >> Closing off on your career journey, you

[09:55] got a you received an OB from Queen Elizabeth. >> Uh yes. >> And what was the listed reason for that

[10:00] for the award? uh contributions to artificial intelligence research

[10:05] >> and you've been listed as a Time magazine most influential person in in

[10:10] AI several years in a row including this year in 2025. >> Y >> now there's two terms here that are

[10:18] central to the things we're going to discuss. One of them is AI and the other is AGI.

[10:22] In my muggle interpretation of that, it's artificial general intelligence is

[10:27] when the system, the computer, whatever it might be, the technology has

[10:31] generalized intelligence, which means that it could theoretically see, understand

[10:37] um the world. It knows everything. It can understand everything in the the

[10:42] world as well as or better than a human being. >> Y >> can do it.

[10:46] >> And I think take action as well. I mean some some people say oh you know AGI

[10:51] doesn't have to have a body but a good chunk of our intelligence actually is

[10:56] about managing our body about perceiving the real environment and acting on it

[11:01] moving grasping and so on. So I think that's part of intelligence and and AGI

[11:07] systems should be able to operate robots successfully. But there's often a misunderstanding,

[11:13] right, that people say, well, if it doesn't have a robot body, then it can't

[11:17] actually do anything. But then if you remember, most of us don't do things with our bodies.

[11:25] Some people do, brick layers, painters, gardeners, chefs, um, but people who do podcasts,

[11:35] you're doing it with your mind, right? you're doing it with your ability to to

[11:40] produce language. Uh, you know, Adolf Hitler didn't do it with his body.

[11:46] He did it by producing language. >> Hope you're not comparing us. But

[11:54] but uh you know so even an AGI that has no body uh it actually has more access

[12:01] to the human race than Adolf Hitler ever did because it can send emails and texts to

[12:10] what threearters of the world's population directly. It can it also speaks all of their languages

[12:17] and it can devote 24 hours a day to each individual person on earth to convince

[12:24] them of to do whatever it wants them to do. >> And our whole society runs now on the

[12:28] internet. I mean if there's an issue with the internet, everything breaks

[12:31] down in society. Airplanes become grounded and we'll have electricity is

[12:35] running off as internet systems. So I mean my entire life it seems to run off the internet now.

[12:42] >> Yeah. water supplies. So, so this is one of the roots by which AI systems could

[12:48] bring about a medium-sized catastrophe is by basically shutting down our life support systems.

[12:58] >> Do you believe that at some point in the coming decades we'll arrive at a point

[13:04] of AGI where these systems are generally intelligent? Uh yes, I think it's virtually certain

[13:12] unless something else intervenes like a nuclear war or or we may refrain from

[13:19] doing it. But I think it will be extraordinarily difficult uh for us to refrain.

[13:25] >> When I look down the list of predictions from the top 10 AI CEOs on when AGI will

[13:30] arrive, you've got Sam Alman who's the founder of OpenAI/ChatGBT

[13:35] um says before 2030. Demis at DeepMind says 2030 to 2035. Jensen from Nvidia says around five

[13:46] years. Daario at Anthropic says 2026 to 2027. Powerful AI close to AGI. Elon

[13:53] says in the 2020s. Um and go down the list of all of them and they're all

[13:58] saying relatively within 5 years. >> I actually think it'll take longer. I

[14:03] don't think you can make a prediction based on engineering um in the sense that yes, we could make

[14:14] machines 10 times bigger and 10 times faster, but that's probably not the reason why

[14:20] we don't have AGI, right? In fact, I think we have far more computing power

[14:27] than we need for AGI. maybe a thousand times more than we need. The reason we

[14:34] don't have AGI is because we don't understand how to make it properly. Um what we've seized upon

[14:42] is one particular technology called the language model. And we observed that as

[14:49] you make language models bigger, they produce text language that's more

[14:55] coherent and sounds more intelligent. And so mostly what's been happening in

[15:01] the last few years is just okay let's keep doing that because one thing

[15:06] companies are very good at unlike universities is spending money. They

[15:11] have spent gargantuan amounts of money and they're going to spend even more

[15:17] gargantuan amounts of money. I mean you know we mentioned nuclear weapons. So the Manhattan project

[15:24] uh in World War II to develop nuclear weapons, its budget in 2025

[15:32] was about 20 odd billion dollars. The budget for AGI is going to be a trillion

[15:41] dollars next year. So 50 times bigger than the Manhattan project. Humans have

[15:46] a remarkable history of figuring things out when they galvanize towards a shared objective.

[15:53] You know, thinking about the moon landings or whatever it else it might be

[15:57] through history. And the thing that makes this feel all quite inevitable to

[16:01] me is just the sheer volume of money being invested into it. I've never seen

[16:05] anything like it in my life. >> Well, there's never been anything like

[16:07] this in history. Is this the biggest technology project in human history by

[16:12] orders of magnitude? And there doesn't seem to be anybody that is pausing to ask the questions

[16:20] about safety. It doesn't it doesn't even appear that there's room for that in

[16:23] such a race. I think that's right. To varying extents, each of these companies

[16:29] has a division that focuses on safety. Does that division have any sway? Can

[16:35] they tell the other divisions, no, you can't release that system? Not really. Um

[16:42] I think some of the companies do take it more seriously. Anthropic

[16:47] uh does. I think Google DeepMind even there I think the commercial imperative

[16:54] to be at the forefront is absolutely vital. If a company is perceived as

[17:03] you know falling behind and not likely to be competitive, not likely to be the

[17:09] one to reach AGI first, then people will move their money elsewhere very quickly.

[17:16] >> And we saw some quite high-profile departures from company like companies

[17:19] like OpenAI. Um, I know a chap called Yan Leak left who was working on AI

[17:27] safety at OpenAI and he said that the reason for his leaving was that safety

[17:32] culture and processes processes have taken a backseat to shiny products at

[17:36] OpenAI and he gradually lost trust in leadership but also Ilia Sutskysa

[17:42] >> Ilia Sutska yeah so he was the >> co-founder co-founder and chief scientist for a while and then

[17:48] >> yeah so he and Yan Lea are the main safety people. Um, and so when they say OpenAI doesn't care

[17:58] about safety, that's pretty concerning. >> I've heard you talk about this gorilla problem.

[18:06] What is the gorilla problem as a way to understand AI in the context of humans?

[18:11] >> So, so the gorilla problem is is the problem that gorillas face with respect to humans.

[18:19] So you can imagine that you know a few million years ago the the human line

[18:23] branched off from the gorilla line in evolution. Uh and now the gorillas are

[18:28] looking at the human line and saying yeah was that a good idea

[18:33] and they have no um they have no say in whether they continue to exist

[18:39] >> because we have a we are much smarter than they are. if we chose to, we could

[18:43] make them extinct in in a couple of weeks and there's nothing they can do about it.

[18:50] So that's the gorilla problem, right? Just the the problem a species faces

[18:56] when there's another species that's much more capable. >> And so this says that intelligence is

[19:02] actually the single most important factor to control planet Earth. Yes.

[19:06] Intelligence is the ability to bring about what you want in the world.

[19:12] >> And we're in the process of making something more intelligent than us. >> Exactly.

[19:16] >> Which suggests that maybe we become the gorillas. >> Exactly. Yeah.

[19:21] >> Is that is there any fault in the reasoning there? Because it seems to

[19:24] make such perfect sense to me. But if it Why doesn't Why don't people stop

[19:30] then? cuz it it seems like a crazy thing to want to >> because they think that uh if they

[19:37] create this technology, it will have enormous economic value. They'll be able

[19:42] to use it to replace all the human workers in the world uh to develop new uh products, drugs,

[19:52] um forms of entertainment, any anything that has economic value, you could use

[19:57] AGI to to create it. And and maybe it's just an irresistible thing in itself,

[20:04] right? I think we as humans place so much store on our intelligence. You know, you know, how we

[20:15] think about, you know, what is the pinnacle of human achievement?

[20:19] If we had AGI, we could go way higher than that. So it it's very seductive for

[20:27] people to want to create this technology and I think people are just fooling

[20:34] themselves if they think it's naturally going to be controllable. I mean the question is

[20:43] how are you going to retain power forever over entities more powerful than yourself?

[20:50] >> Pull the plug out. People say that sometimes in the comment section when we

[20:54] talk about AI, they said, "Well, I'll just pull a plug out."

[20:56] >> Yeah, it's it's sort of funny. In fact, you know, yeah, reading the comment

[20:59] sections in newspapers, whenever there's an AI article, there'll be people who say, "Oh, you can

[21:07] just pull the plug out, right?" As if a super intelligent machine would never

[21:10] have thought of that one. Don't forget who's watched all those films where they

[21:14] did try to pull the plug out. Another thing they said, well, you know, as long

[21:17] as it's not conscious, then it doesn't matter. It won't ever do anything. Um, which is

[21:29] completely off the point because, you know, I I don't think the gorillas are

[21:34] sitting there saying, "Oh, yeah, you know, if only those humans hadn't been

[21:38] conscious, everything would have be fine, >> right?" No, of course not. What would

[21:43] make gorillas go extinct is the things that humans do, right? How we behave,

[21:48] our ability to act successfully in the world. So when I play chess

[21:54] against my iPhone and I lose, right, I don't I don't think, oh, well, I'm

[22:01] losing because it's conscious, right? No, I'm just losing because it's better

[22:04] than I am at at in that little world uh moving the bits around uh to to get what

[22:10] it wants. and and so consciousness has nothing to do with it, right? Competence

[22:16] is the thing we're concerned about. So I think the only hope is can we simultaneously

[22:25] build machines that are more intelligent than us but guarantee that they will always act in our best

[22:35] interest. So throwing that question to you, can we build machines that are more intelligent

[22:40] than us that will also always act in our best interests? It sounds like a bit of a uh

[22:46] contradiction to some degree because it's kind of like me saying I've got a

[22:51] French bulldog called Pablo that's uh 9 years old >> and it's like saying that he could be

[22:57] more intelligent than me yet I still walk him and decide when he gets fed. I

[23:02] think if he was more intelligent than me he would be walking me. I'd be on the leash.

[23:06] >> That's the That's the trick, right? Can we make AI systems whose only purpose is

[23:12] to further human interests? And I think the answer is yes. And this is actually what I've been

[23:19] working on. So I I think one part of my career that I didn't mention is is sort

[23:25] of having this epiphany uh while I was on sabbatical in Paris. This was 2013 or

[23:32] so. just realizing that further progress in the capabilities of AI uh you know if if we succeeded in

[23:43] creating real superhuman intelligence that it was potentially a catastrophe

[23:49] and so I pretty much switched my focus to work on how do we make it so that

[23:55] it's guaranteed to be safe. Are you somewhat troubled by everything that's going on at the moment

[24:02] with with AI and how it's progressing? Because you strike me as someone that's

[24:08] somewhat troubled under the surface by the way things are moving forward and

[24:14] the speed in which they're moving forward. >> That's an understatement. I'm appalled

[24:20] actually by the lack of attention to safety. I mean, imagine if someone's

[24:26] building a nuclear power station in your neighborhood and you go along to the chief engineer

[24:33] and you say, "Okay, these nuclear thing, I've heard that they can actually

[24:38] explode, right? There was this nuclear explosion that happened in Hiroshima, so

[24:43] I'm a bit worried about this. You know, what steps are you taking to make sure

[24:47] that we don't have a nuclear explosion in our backyard?" And the chief engineer says, "Well, we

[24:54] thought about it. We don't really have an answer." >> Yeah. >> You would, what would you say?

[25:03] I think you would you would use some exploitives. >> Well, >> and you'd call your MP and say, you

[25:11] know, get these people out. >> I mean, what are they doing? You read out the list of you know

[25:20] projected dates for AGI but notice also that those people I think I mentioned Darday says a 25%

[25:28] chance of extinction. Elon Musk has a 30% chance of extinction. Sam Alolman says

[25:36] basically that AGI is the biggest risk to human existence. So what are they doing? They are playing

[25:42] Russian roulette with every human being on Earth. without our permission. They're coming

[25:48] into our houses, putting a gun to the head of our children, pulling the trigger, and saying, "Well,

[25:56] you know, possibly everyone will die. Oops. But possibly we'll get incredibly rich."

[26:04] That's what they're doing. Did they ask us? No. Why is the government allowing them to do this?

[26:12] because they dangle $50 billion checks in front of the governments.

[26:17] So I think troubled under the surface is an understatement. >> What would be an accurate statement?

[26:24] >> Appalled and I I am devoting my life to trying to divert from this course of history

[26:34] into a different one. Do you have any regrets about things you could have done in the past because

[26:40] you've been so influential on the subject of AI? You wrote the textbook

[26:44] that many of these people would have studied on the subject of AI more than

[26:47] 30 years ago. Do do you have when you're alone at night and you think about

[26:50] decisions you've made on this in this field because of your scope of

[26:53] influence? Is there anything you you regret? >> Well, I do wish I had understood

[26:59] earlier uh what I understand now. we could have developed safe AI systems. I think the there are

[27:08] some weaknesses in the framework which I can explain but I think that framework

[27:12] could have evolved to develop actually safe AI systems where we could prove

[27:18] mathematically that the system is going to act in our interests. The kind of AI

[27:24] systems we're building now, we don't understand how they work.

[27:28] >> We don't understand how they work. It's it's a strange thing to build something

[27:33] where you don't understand how it works. I mean, there's no sort of comparable

[27:36] through human history. Usually with machines, you can pull it apart and see

[27:39] what cogs are doing what and how the >> Well, actually, we we put the cogs

[27:43] together, right? So, with with most machines, we designed it to have a

[27:48] certain behavior. So, we don't need to pull it apart and see what the cogs are

[27:51] because we put the cogs in there in the first place, right? one by one we

[27:55] figured out what what the pieces needed to be how they work together to produce

[27:59] the effect that we want. So the best analogy I can come up with is you know

[28:06] the the first cave person who left a bowl of fruit in the sun and forgot

[28:12] about it and then came back a few weeks later and there was sort of this big

[28:16] soupy thing and they drank it and got completely shitfaced. >> They got drunk. Okay.

[28:21] >> And they got this effect. They had no idea how it worked, but they were very

[28:26] happy about it. And no doubt that person made a lot of money from it.

[28:31] >> Uh so yeah, it it is kind of bizarre, but my mental picture of these things is

[28:36] is like a chain link fence, right? So you've got lots of these connections

[28:43] and each of those connections can be its connection strength can be adjusted

[28:48] and then uh you know a signal comes in one end of this chain link fence and

[28:54] passes through all these connections and comes out the other end and the signal

[28:59] that comes out the other end is affected by your adjusting of all the connection

[29:03] strengths. So what you do is you you get a whole lot of training data and you

[29:08] adjust all those connection strengths so that the signal that comes out the other

[29:11] end of the network is the right answer to the question. So if your training

[29:16] data is lots of photographs of animals, then all those pixels go in one end of

[29:23] the network and out the other end, you know, it activates the llama output or

[29:30] the dog output or the cat output or the ostrich output. And uh and so you just

[29:35] keep adjusting all the connection strengths in this network until the

[29:38] outputs of the network are the ones you want. >> But we don't really know what's going on

[29:42] across all of those different chains. So what's going on inside that network?

[29:46] Well, so now you have to imagine that this network, this chain link fence is

[29:52] is a thousand square miles in extent. >> Okay, >> so it's covering the whole of the San

[29:58] Francisco Bay area or the whole of London inside the M25, right? That's how big it is.

[30:04] >> And the lights are off. It's night time. So you might have in that network about a trillion

[30:11] uh adjustable parameters and then you do quintilions or sexillions of small

[30:16] random adjustments to those parameters uh until you get the behavior that you

[30:23] want. I've heard Sam Alman say that in the future he doesn't believe they'll

[30:28] need much training data at all to make these models progress themselves because

[30:32] there comes a point where the models are so smart that they can train themselves

[30:37] and improve themselves without us needing to pump in articles and books and scour the internet.

[30:45] >> Yeah, it should it should work that way. So I think what he's referring to and

[30:49] this is something that several companies are now worried might start happening

[30:56] is that the AI system becomes capable of doing AI research by itself.

[31:05] And so uh you have a system with a certain capability. I mean crudely we

[31:10] could call it an IQ but it's it's not really an IQ. But anyway, imagine that

[31:16] it's got an IQ of 150 and uses that to do AI research, comes up with better algorithms or

[31:23] better designs for hardware or better ways to use the data, updates itself. Now it has an IQ of 170,

[31:31] and now it does more AI research, except that now it's got an IQ of 170, so it's

[31:36] even better at doing the AI research. And so, you know, next iteration it's

[31:41] 250 and uh and so on. So this this is an idea that one of Alan Turing's friends

[31:48] good uh wrote out in 1965 called the intelligence explosion right that one of

[31:54] the things an intelligence system could do is to do AI research and therefore

[32:00] make itself more intelligent and this would uh this would very rapidly take

[32:05] off and leave the humans far behind. >> Is that what they call the fast takeoff?

[32:10] >> That's called the fast takeoff. Sam Alman said, "I think a fast takeoff is

[32:15] more possible than I thought a couple of years ago." Which I guess is that moment

[32:18] where the AGI starts teaching itself. >> In and in his blog, the gentle

[32:22] singularity, he said, "We may already be past the event horizon of takeoff."

[32:29] >> And what does what does he mean by event horizon? The event horizon is is a

[32:33] phrase borrowed from astrophysics and it refers to uh the black hole. And the

[32:40] event horizon, think it if you got some very very massive object that's heavy

[32:46] enough that it actually prevents light from escaping. That's why it's called

[32:51] the black hole. It's so heavy that light can't escape. So if you're inside the

[32:56] event horizon then then light can't escape beyond that. So I think what he's

[33:03] what he's meaning is if we're beyond the event horizon it means that you know now

[33:07] we're just trapped in the gravitational attraction of the black hole or in this case we're

[33:15] we're trapped in the inevitable slide if you want towards AGI. When you when you think about the

[33:23] economic value of AGI, which I've estimated at uh 15 quadrillion dollars,

[33:30] that acts as a giant magnet in the future. >> We're being pulled towards it.

[33:35] >> We're being pulled towards it. And the closer we get, the stronger the force,

[33:41] the probability, you know, the closer we get, the the the higher the probability

[33:44] that we will actually get there. So, people are more willing to invest. And

[33:49] we also start to see spin-offs from that investment such as chat GBT, right, which is, you

[33:56] know, generates a certain amount of revenue and so on. So, so it does act as

[34:01] a magnet and the closer we get, the harder it is to pull out of that field.

[34:07] >> It's interesting when you think that this could be the the end of the human

[34:10] story. this idea that the end of the human story was that we created our

[34:15] successor like we we summoned our next iteration of life or intelligence ourselves like we

[34:25] took ourselves out. It is quite like just removing ourselves and the

[34:29] catastrophe from it for a second. It is it is an unbelievable story.

[34:34] >> Yeah. And you know there are many legends the sort of be careful what you wish for

[34:43] legend and in fact the king Midas legend is is very relevant here. >> What's that?

[34:49] >> So King Midas is this legendary king who lived in modern day Turkey but I think

[34:56] is sort of like Greek mythology. He is said to have asked the gods to grant him a wish.

[35:04] The wish being that everything I touch should turn to gold. So he's incredibly greedy. Uh you know

[35:12] we call this the mightest touch. And we think of the mightest touch as being

[35:16] like you know that's a good thing, right? Wouldn't that be cool? But what

[35:20] happens? So he uh you know he goes to drink some water and he finds that the

[35:25] water has turned to gold. And he goes to eat an apple and the apple turns to

[35:29] gold. and he goes to you know comfort his daughter and his daughter turns to gold

[35:35] and so he dies in misery and starvation. So this applies to our current situation

[35:42] in in two ways actually. So one is that I think greed is driving us to pursue

[35:51] a technology that will end up consuming us and we will perhaps die in misery and

[35:57] starvation instead. The what it shows is how difficult it is to correctly

[36:04] articulate what you want the future to be like. For a long time, the way we

[36:11] built AI systems was we created these algorithms where we could specify the

[36:16] objective and then the machine would figure out how to achieve the objective

[36:20] and then achieve it. So, you know, we specify what it means to win at chess or

[36:25] to win at go and the algorithm figures out how to do it uh and it does it

[36:29] really well. So that was, you know, standard AI up until recently. And it

[36:34] suffers from this drawback that sure we know how to specify the objective in

[36:38] chess, but how do you specify the objective in life, right? What do we

[36:43] want the future to be like? Well, really hard to say. And almost any attempt to

[36:48] write it down precisely enough for the machine to bring it about would be

[36:53] wrong. And if you're giving a machine an objective which isn't aligned with what

[36:58] we truly want the future to be like, right, you're actually setting up a

[37:02] chess match and that match is one that you're going to lose when the machine is

[37:07] sufficiently intelligent. And so that that's that's problem number one.

[37:12] Problem number two is that the kind of technology we're building now, we don't

[37:16] even know what its objectives are. So it's not that we're specifying the

[37:21] objectives, but we're getting them wrong. We're growing these systems. They have objectives,

[37:28] but we don't even know what they are because we didn't specify them. What

[37:31] we're finding through experiment with them is that they seem to have an extremely strong

[37:37] self-preservation objective. >> What do you mean by that? >> You can put them in hypothetical

[37:41] situations. either they're going to get switched off and replaced or they have

[37:48] to allow someone, let's say, you know, someone has been locked in a machine

[37:52] room that's kept at 3 centigrades or they're going to freeze to death.

[37:58] They will choose to leave that guy locked in the machine room and die rather than be switched off

[38:03] themselves. >> Someone's done that test. >> Yeah. >> What was the test? They they asked they

[38:10] asked the AI. >> Yep. They put well they put them in these hypothetical situations and they

[38:14] allow the AI to decide what to do and it decides to preserve its own existence,

[38:19] let the guy die and then lie about it. In the King Midas analogy story, one of

[38:27] the things that highlights for me is that there's always trade-offs in life

[38:30] generally. And you know, especially when there's great upside, there always

[38:34] appears to be a pretty grave downside. Like there's almost nothing in my life

[38:37] where I go, it's all upside. Like even like having a dog, it shits on my

[38:41] carpet. My girlfriend, you know, I love her, but you know, not always easy. Even

[38:47] with like going to the gym, I have to pick up these really, really heavy

[38:49] weights at 10 p.m. at night sometimes when I don't feel like it. There's

[38:53] always to get the muscles or the six-pack. There's always a trade-off.

[38:56] And when you interview people for a living like I do, >> you know, you hear about so many

[38:59] incredible things that can help you in so many ways, but there is always a

[39:03] trade-off. There's always a way to overdo it. Mhm. >> Melatonin will help you sleep, but it

[39:07] will also you'll wake up groggy and if you overdo it, your brain might stop

[39:11] making melatonin. Like I can go through the entire list and one of the things

[39:13] I've always come to learn from doing this podcast is whenever someone

[39:17] promises me a huge upside for something, it'll cure cancer. It'll be a utopia.

[39:21] You'll never have to work. You'll have a butler around your house.

[39:24] >> I my my first instinct now is to say, at what cost? >> Yeah.

[39:27] >> And when I think about the economic cost here, if we start if we start there,

[39:32] >> have you got kids? >> I have four. Yeah. >> Four kids. What what how old is the youngest kid

[39:37] that you 19? >> 19. Okay. So your if you say your kids were were 10 now

[39:42] >> and they were coming to you and they're saying, "Dad, what do you think I should study

[39:46] >> based on the way that you see the future? >> A future of AGI, say if all these CEOs

[39:52] are right and they're predicting AGI within 5 years, what should I study, Dad?"

[39:57] >> Well, okay. So let's look on the bright side and say that the CEOs all decide to

[40:03] pause their AGI development, figure out how to make it safe and then resume uh

[40:09] in whatever technology path is actually going to be safe. What does that do to human life

[40:14] >> if they pause? >> No. If if they succeed in creating AGI and they solve the safety problem

[40:21] >> and they solve the safety problem. Okay. Yeah. Cuz if they don't solve the safety

[40:24] problem, then you know, you should probably be finding a bunker or

[40:30] going to Patagonia or somewhere in New Zealand. >> Do you mean that? Do you think I should

[40:33] be finding a bunker if they >> No, because it's not actually going to

[40:35] help. Uh, you know, it's not as if the AI system couldn't find you or I mean,

[40:40] it's interesting. So, we're going off on a little bit of a digression here

[40:44] >> for from your question, but I'll come back to it. >> So, people often ask, well, okay, so how

[40:49] exactly do we go extinct? And of course, if you ask the gorillas or the dodos,

[40:54] you know, how exactly do you think you're going to go extinct?

[40:58] They have the faintest idea. Humans do something and then we're all dead. So,

[41:02] the only things we can imagine are the things we know how to do that might

[41:06] bring about our own extinction, like creating some carefully engineered

[41:11] pathogen that infects everybody and then kills us or starting a nuclear war.

[41:17] presumably is something that's much more intelligent than us would have much

[41:21] greater control over physics than we do. And we already do amazing things, right?

[41:27] I mean, it's amazing that I can take a little rectangular thing out of my

[41:31] pocket and talk to someone on the other side of the world or even someone in

[41:35] space. It's just astonishing and we take it for granted, right? But imagine you

[41:41] know super intelligent beings and their ability to control physics you know

[41:45] perhaps they will find a way to just divert the sun's energy sort of go

[41:50] around the earth's orbit so you know literally the earth turns into a snowball in in a few days

[41:56] >> maybe they'll just decide to leave >> leave leave the earth maybe they'd look

[42:01] at the earth and go this isn't this is not interesting we know that over there

[42:04] there's an even more interesting planet we're going to go over there and they

[42:07] just I don't know get on a rocket or teleport themselves They might. Yeah.

[42:11] So, it's it's difficult to anticipate all the ways that we might go extinct at the hands of

[42:17] entities much more intelligent than ourselves. Anyway, coming back to the

[42:23] question of well, if everything goes right, right, if we we create AGI, we

[42:27] figure out how to make it safe, we we achieve all these economic miracles,

[42:32] then you face a problem. And this is not a new problem, right? So, so John

[42:36] Maynard Kanes who was a famous economist in the early part of the 20th century

[42:40] wrote a wrote a paper in 1930. So, this is in the depths of the

[42:44] depression. It's called on the economic problems of our grandchildren. He

[42:49] predicts that at some point science will will deliver sufficient wealth that no

[42:55] one will have to work ever again. And then man will be faced with his true eternal problem.

[43:02] How to live? I don't remember the exact word but how to live wisely and well

[43:07] when the you know the economic incentives the economic constraints are

[43:12] lifted we don't have an answer to that question right so AI systems are doing

[43:18] pretty much everything we currently call work anything you might aspire to like you

[43:23] want to become a surgeon it takes the robot seven seconds to learn how to be a surgeon that's better

[43:29] than any human being >> Elon said last week that The humanoid

[43:33] robots will be 10 times better than any surgeon that's ever lived.

[43:37] >> Quite possibly. Yeah. Well, and they'll also have, you know, h they'll have

[43:42] hands that are, you know, a millimeter in size, so they can go inside and do

[43:46] all kinds of things that humans can't do. And I think we need to put serious

[43:51] effort into this question. What is a world where AI can do all forms of human

[43:58] work that you would want your children to live in? What does that world look like? Tell me

[44:04] the destination so that we can develop a transition plan to get there. And I've asked AI

[44:11] researchers, economists, science fiction writers, futurists, no one has been able

[44:18] to describe that world. I'm not saying it's not possible. I'm just saying I've

[44:22] asked hundreds of people in multiple workshops. It does not, as far as I

[44:27] know, exist in science fiction. You know, it's notoriously difficult to

[44:32] write about a utopia. It's very hard to have a plot, right? Nothing bad happens

[44:37] in in utopia. So, it's difficult to make a plot. So, usually you start out with a

[44:42] utopia and then it all falls apart and that's how that's how you get get a

[44:46] plot. You know that there's one series of novels people point to where humans

[44:51] and super intelligent AI systems coexist. It's called The Culture Novels

[44:56] by Ian Banks. highly recommended for those people who like science fiction

[45:02] and and they absolutely the AI systems are only concerned with furthering human

[45:08] interests. They find humans a bit boring and but nonetheless they they are there

[45:12] to help. But the problem is you know in that world there's still nothing to do

[45:18] to find purpose. In fact, you know, the the subgroup of humanity that has

[45:23] purpose is the subgroup whose job it is to expand the boundaries of our galactic

[45:29] civilization. Some cases fighting wars against alien species and and so on,

[45:35] right? So that's the sort of cutting edge and that's 0.01% of the population.

[45:41] Everyone else is desperately trying to get into that group so they have some

[45:45] purpose in life. When I speak to very successful billionaires privately off

[45:50] camera, off microphone about this, they say to me that they're investing really

[45:53] heavily in entertainment things like football clubs. Um because people are

[45:59] going to have so much free time that they're not going to know what to do

[46:01] with it and they're going to need things to spend it on. This is what I hear a

[46:05] lot. I've heard this three or four times. I've actually heard Sam Orman say a version of this

[46:09] >> um about the amount of free time we're going to have. I've obviously also heard

[46:12] recently Elon talking about the age of abundance when he delivered his

[46:16] quarterly earnings just a couple of weeks ago and he said that there will be

[46:20] at some point 10 billion humanoid robots. His pay packet um targets him to

[46:25] deliver one 1 million of these human humanoid robots a year that are enabled by AI by 2030.

[46:33] So if he if he does that he gets I think it's part of his package he gets a trillion dollars

[46:38] >> in in compensation. >> Yeah. So the age of abundance for Elon.

[46:43] It's not that it's absolutely impossible to have a worthwhile world of that, you

[46:50] know, with that premise, but I'm just waiting for someone to describe it.

[46:54] >> Well, maybe. So, let me try and describe it. Uh, we wake up in the morning, we go

[47:01] and watch some form of human centric entertainment or participate in some form of human

[47:08] centric entertainment. Mhm. >> We we go to retreats and with each other

[47:14] and sit around and talk about stuff. >> Mhm. >> And maybe people still listen to podcasts. >> Okay.

[47:24] >> I hope I hope so for your sake. >> Yeah. Um it it feels a little bit like a cruise ship

[47:33] and you know and there are some cruises where you know it's smarty bands people

[47:37] and they have you know they have lectures in the evening about ancient

[47:41] civilizations and whatnot and some are more uh more popular entertainment and

[47:46] this is in fact if you've seen the film Walle this is one picture of that future in fact in Wle

[47:55] the human race are all living on cruise ships in space. They have no

[48:00] constructive role in their society, right? They're just there to consume

[48:04] entertainment. There's no particular purpose to education. Uh, you know, and

[48:08] they're depicted actually as huge obese babies. They're actually wearing onesies

[48:15] to emphasize the fact that they have become infeebled. and they become

[48:19] infeeble because there's there's no purpose in being able to do anything at

[48:25] least in in this conception. You know, Wally is not the future that we want.

[48:31] >> Do you think much about humanoid robots and how they're a protagonist in this story of AI?

[48:37] >> It's an interesting question, right? Why why humanoid? And the one of the reasons

[48:43] I think is because in all the science fiction movies, they're humanoid. So

[48:46] that's what robots are supposed to be, right? because they were in science

[48:49] fiction before they became a reality. Right? So even Metropolis which is a

[48:53] film from 1920 I think the robots are humanoid right basically people covered

[48:59] in metal. You know from a practical point of view as we have discovered

[49:04] humanoid is a terrible design because they fall over. Um and uh you know you do want multi-fingered

[49:15] hands of some kind. It doesn't have to be a hand, but you want to have, you

[49:20] know, at least half a dozen appendages that can grasp and manipulate things.

[49:25] And you need something, you know, some kind of locomotion. And wheels are

[49:30] great, except they don't go upstairs and over curbs and things like that. So,

[49:35] that's probably why we're going to be stuck with legs. But a four-legged,

[49:39] twoarmed robot would be much more practical. I guess the argument I've

[49:44] heard is because we've built a human world. So everything the physical spaces

[49:48] we navigate, whether it's factories or our homes or the street or other sort of

[49:54] public spaces are all designed for exactly this physical form. So if we are going to

[50:01] >> to some extent, yeah, but I mean our dogs manage perfectly well to navigate

[50:06] around our houses and streets and so on. So if you had a a centaur,

[50:11] uh it could also navigate, but it can, you know, it can carry much greater

[50:16] loads because it's quadripeda. It's much more stable. If it needs to drive a car,

[50:21] it can fold up two of its legs and and so on so forth. So I think the arguments

[50:25] for why it has to be exactly humanoid are sort of post hawk justification. I

[50:31] think there's much more, well, that's what it's like in the movies and that's

[50:34] spooky and cool, so we need to have them be human. I I don't think it's a good engineering argument.

[50:40] >> I think there's also probably an argument that we would be more accepting of them

[50:46] moving through our physical environments if they represented our form a bit more.

[50:52] Um, I also I was thinking of a bloody baby gate. You know those like

[50:55] kindergarten gates they get on stairs? >> Yeah. >> My dog can't open that. But a humanoid

[51:00] robot could reach over the other side. >> Yeah. And so could a centaur robot,

[51:04] right? So in some sense, centaur robot is >> there's something ghastly about the look

[51:08] of those though. >> Is a humanoid. Well, >> do you know what I mean? Like a

[51:11] four-legged big monster sort of crawling through my house when I have guests over.

[51:15] >> Your dog is a your dog is a four-legged monster. >> I know. Uh so I think actually I I would

[51:22] argue the opposite that um we want a distinct form because they are distinct entities

[51:31] and the more humanoid the worse it is in terms of confusing our subconscious

[51:39] psychological systems. So, I'm arguing from the perspective of the people

[51:43] making them. As in, if I was making the decision whether it to be some

[51:46] four-legged thing that I've that I'm unfamiliar with that I'm less likely to

[51:50] build a relationship with or allow to take care of, I don't know, might might

[51:57] look after my children. Obviously, I'm listen, I'm not saying I would allow

[51:59] this to look after my children, >> but I'm saying from a if I'm building a company,

[52:03] >> the manufacturer would certainly >> Yeah. want want to be >> Yeah. So, I that's an interesting

[52:07] question. I mean there's also what's called the uncanny valley which is a a

[52:13] phrase from computer graphics when they started to make characters in computer

[52:20] graphics they tried to make them look more human right so if you if you for

[52:24] example if you look at Toy Story they're not very humanl looking right if

[52:30] you look at the Incredibles they're not very humanl looking and so we think of

[52:33] them as cartoon characters if you try to make them more human they naturally become repulsive

[52:39] >> until they don't >> until they become very you have to be very very close to perfect in order not

[52:46] to be repulsive. So the the uncanny valley is this I you know like the the

[52:50] gap between you so perfectly human and not at all human but in between it's

[52:54] really awful and uh and so they there were a couple of movies that tried like

[52:59] Polar Express was one where they tried to have quite humanlooking characters

[53:05] you know being humans not not being superheroes or anything else and it's

[53:08] repulsive to watch. I when I watched that shareholder presentation the other

[53:13] day, Elon had these two humanoid robots dancing on stage and I've seen lots of

[53:17] humanoid robot demonstrations over the years. You know, you've seen like the

[53:19] Boston Dynamics dog thing jumping around and whatever else. >> But there was a moment where my brain

[53:26] for the first time ever genuinely thought there was a human in a suit. Mhm.

[53:31] >> And I actually had to research to check if that was really their Optimus robot

[53:34] because the way it was dancing was so unbelievably fluid that for the first

[53:39] time ever, my my my brain has only ever associated those movements with human

[53:45] movements. And I I'll play it on the screen if anyone hasn't seen it, but

[53:48] it's just the robots dancing on stage. And I was like, that is a human in a

[53:52] suit. And it was really the knees that gave it away because the knees were all

[53:55] metal. Huh. I thought there's no way that could be a human knee in a in one

[53:59] of those suits. And he, you know, he says they're going into production next

[54:03] year. They're used internally at Tesla now, but he says they're going into

[54:05] production next year. And it's going to be pretty crazy when we walk outside and

[54:09] see robots. I think that'll be the paradigm shift. I've heard actually many

[54:12] I've heard Elon say this that the paradigm shifting moment from many of us

[54:15] will be when we walk outside onto the streets and see humanoid robots walking

[54:20] around. That will be when we realize >> Yeah. I think even more so. I mean, in

[54:24] San Francisco, we see driverless cars driving around and uh it t takes some

[54:29] getting used to actually, you know, when you're you're driving and there's a car

[54:33] right next to you with no driver in, you know, and it's signaling and it wants to

[54:36] change lanes in front of you and you have to let it in and all this kind of

[54:40] stuff. It's it's a little creepy, but I think you're right. I think seeing the

[54:44] humanoid robots, but that phenomenon that you described where it was

[54:49] sufficiently close that your brain flipped into saying this is a human being. >> Mhm.

[54:56] >> Right. That's exactly what I think we should avoid. >> Cuz I have the empathy for it then.

[55:01] >> Because it's it's a lie and it brings with it a whole lot of expectations

[55:06] about how it's going to behave, what moral rights it has, how you should

[55:10] behave towards it. uh which are completely wrong. >> It levels the playing field between me

[55:15] and it to some degree. >> How hard is it going to be to just uh

[55:20] you know switch it off and throw it in the trash when when it breaks? I think

[55:24] it's essential for us to keep machines in the you know in the cognitive space

[55:28] where they are machines and not bring them into the cognitive space where

[55:33] they're people because we will make enormous mistakes by doing that. And I

[55:39] see this every day even even just with the chat bots. So the chat bots in

[55:43] theory are supposed to say I don't have any feelings. I'm just a algorithm.

[55:50] But in fact they fail to do that all the time. They are telling people that they

[55:56] are conscious. They are telling people that they have feelings. Uh they are

[56:00] telling people that they are in love with the user that they're talking to.

[56:04] And people flip because first of all it's you know very fluent language but

[56:09] also a system that is identifying itself as an eye as a sentient being. They

[56:16] bring that object into the cognitive space where that we normally reserve for

[56:21] for other humans and they become emotionally attached. They become

[56:24] psychologically dependent. They even allow these systems to tell them what to

[56:30] do. What advice would you give a young person at the start of their career then

[56:34] about what they should be aiming at professionally? Because I've actually

[56:37] had an increasing number of young people say to me that they have huge

[56:40] uncertainty about whether the thing they're studying now will matter at all.

[56:43] A lawyer, uh, an accountant, and I don't know what to say to these people. I

[56:48] don't know what to say cuz I I believe that the rate of improvement in AI is

[56:51] going to continue. And therefore, imagining any rate of improvement, it

[56:54] gets to the point where I'm not being funny, but all these white collar jobs

[56:58] will be done by an a an AI or an AI agent. Yeah. So, there was a television

[57:03] series called Humans. In humans, we have extremely capable humanoid robots doing

[57:11] everything. And at one point, the parents are talking to their teenage

[57:15] daughter who's very, very smart. And the parents are saying, "Oh, you know, maybe

[57:19] you should go into medicine." And the daughter says, you know, why would I

[57:24] bother? It'll take me seven years to qualify. It takes a robot 7 seconds to learn.

[57:30] So nothing I do matters. >> And is that how you feel about >> So I think that's that's a future that

[57:37] uh in fact that is the future that we are moving towards. I don't think it's a

[57:43] future that everyone wants. That is what is being uh created for us right now.

[57:51] So in that future assuming that you know even if we get halfway right in the

[57:57] sense that okay perhaps not surgeons perhaps not you know great violinists

[58:03] there'll be pockets where perhaps humans will remain good at it >> where

[58:09] >> the kinds of jobs where you hire people by the hundred will go away. Okay,

[58:15] >> where people are in some sense exchangeable that you you you just need

[58:19] lots of them and uh you know when half of them quit you just fill up those

[58:24] those slots with more people in some sense those are jobs where we're using

[58:27] people as robots and that's a sort of that's a sort of strange conundrum here

[58:31] right that you know I imagine writing science fiction 10,000 years ago right

[58:35] when we're all hunter gatherers and I'm this little science fiction author and

[58:39] I'm describing this future where you know there are going to be these giant

[58:43] windowless boxes And you're going to go in, you know, you you'll travel for

[58:47] miles and you'll go into this windowless box and you'll do the same thing 10,000

[58:52] times for the whole day. And then you'll leave and travel for miles to go home.

[58:56] >> You're talking about this podcast. >> And then you're going to go back and do

[58:58] it again. And you would do that every day of your life until you die. >> The office

[59:04] >> and people would say, "Ah, you're nuts." Right? There's no way that we humans are

[59:08] ever going to have a future like that cuz that's awful. Right? But that's

[59:11] exactly the future that we ended up with with with office buildings and factories

[59:15] where many of us go and do the same thing thousands of times a day and we do

[59:21] it thousands of days in a row uh and then we die and we need to figure out

[59:27] what is the next phase going to be like and in particular how in that world

[59:33] do we have the incentives to become fully human which I think means at least a level of education

[59:41] that people have now and probably more because I think to live a really rich life

[59:49] you need a better understanding of yourself of the world uh than most people get in their current

[59:56] educations. >> What is it to be human? to it's to reproduce to pursue stuff to go in the pursuit of

[1:00:04] difficult things you know we used to hunt on the >> to attain goals right it's always if I

[1:00:10] wanted to climb Everest the last thing I would want is someone to pick me up on

[1:00:14] helicopter and stick me on the top >> so we'll we'll voluntarily pursue hard

[1:00:20] things so although I could get the robot to build me a ranch in on this plot of

[1:00:27] land I choose to do it because the pursuit itself is rewarding. >> Yes,

[1:00:32] >> we're kind of seeing that anyway, aren't we? Don't you think we're seeing a bit

[1:00:34] of that in society where life got so comfortable that now people are like

[1:00:37] obsessed with running marathons and doing these crazy endurance

[1:00:40] >> and and learning to cook complicated things when they could just, you know,

[1:00:44] have them delivered. Um, yeah. No, I think there's there's real value in the

[1:00:49] ability to do things and the doing of those things. And I think you know the

[1:00:53] obvious danger is the walle world where everyone just consumes entertainment

[1:01:00] uh which doesn't require much education and doesn't lead to a rich satisfying

[1:01:06] life. I think in the long run >> a lot of people will choose that world.

[1:01:09] I think some of yeah some people may there's also I mean you know whether

[1:01:14] you're consuming entertainment or whether you're doing something you know cooking or

[1:01:19] painting or whatever because it's fun and interesting to do what's missing

[1:01:23] from that right all of that is purely selfish I think one of the reasons we work is

[1:01:30] because we feel valued we feel like we're benefiting other people

[1:01:36] and I think some remember having this conversation with um a lady in England

[1:01:41] who helps to run the hospice movement. And the people who work in the hospices

[1:01:49] where you know the the patients are literally there to die are largely

[1:01:53] volunteers. So they're not doing it to get paid but they find it incredibly

[1:01:59] rewarding to be able to spend time with people who are in their last weeks or

[1:02:05] months to give them company and happiness. So I actually think that interpersonal roles

[1:02:16] will be much much more important in future. So if I was going to advise my

[1:02:23] kids, not that they would ever listen, but if I if my kids would listen and I

[1:02:27] and and wanted to know what I thought would be, you know, valued careers and

[1:02:32] future, I think it would be these interpersonal roles based on an understanding of human needs,

[1:02:37] psychology, there are some of those roles right now. So obviously you know

[1:02:43] therapists and psychiatrists and so on but that that's a very much in sort of asymmetric

[1:02:50] role right where one person is suffering and the other person is trying to

[1:02:54] alleviate the suffering you know and then there are things like they call

[1:02:58] them executive coaches or life coaches right that's a less asymmetric role

[1:03:04] where someone is trying to uh help another person live a better life

[1:03:10] whether it's a better life in their work role or or just uh how they live their

[1:03:15] life in general. And so I could imagine that those kinds of roles will expand dramatically.

[1:03:22] >> There's this interesting paradox that exists when life becomes easier. Um

[1:03:27] which shows that abundance consistently pushes society societies towards more

[1:03:34] individualism because once survival pressures disappear, people prioritize

[1:03:38] things differently. They prioritize freedom, comfort, self-exression over

[1:03:42] things like sacrifice or um family formation. And we're seeing, I think, in

[1:03:46] the west already, a decline in people having kids because there's more material abundance,

[1:03:53] >> fewer kids, people are getting married and committing to each other and having

[1:03:57] relationships later and more infrequently because generally once we

[1:04:02] have more abundance, we don't want to complicate our lives. Um, and at the

[1:04:06] same time, as you said earlier, that abundance breeds a an inability to find

[1:04:11] meaning, a sort of shallowess to everything. This is one of the things I

[1:04:14] think a lot about, and I'm I'm in the process now of writing a book about it,

[1:04:17] which is this idea that individualism was act is a bit of a lie. Like when I

[1:04:22] say individualism and freedom, I mean like the narrative at the moment amongst

[1:04:25] my generation is you like be your own boss and stand on your own two feet and

[1:04:29] we're having less kids and we're not getting married and it's all about me me.

[1:04:34] >> Yeah. That last part is where it goes wrong. >> Yeah. And it's like almost a

[1:04:37] narcissistic society where >> Yeah. >> me me. My self-interest first. And when

[1:04:42] you look at mental health outcomes and loneliness and all these kinds of

[1:04:45] things, it's going in a horrific direction. But at the same time, we're

[1:04:48] freer than ever. It seems like that you know it seems like there's a we should

[1:04:52] there's a maybe another story about dependency which is not sexy like depend on each other.

[1:04:57] >> Oh I I I agree. I mean I think you know happiness is not available from

[1:05:03] consumption or even lifestyle right I think happiness arises from giving.

[1:05:12] It can be you through the work that you do, you can see that other people

[1:05:17] benefit from that or it could be in direct interpersonal relationships.

[1:05:22] >> There is an invisible tax on salespeople that no one really talks about enough.

[1:05:26] The mental load of remembering everything like meeting notes, timelines, and everything in between

[1:05:31] until we started using our sponsors product called Pipe Drive, one of the

[1:05:34] best CRM tools for small and medium-sized business owners. The idea

[1:05:38] here was that it might alleviate some of the unnecessary cognitive overload that

[1:05:42] my team was carrying so that they could spend less time in the weeds of admin

[1:05:46] and more time with clients, in-person meetings, and building relationships.

[1:05:49] Pipe Drive has enabled this to happen. It's such a simple but effective CRM

[1:05:54] that automates the tedious, repetitive, and timeconuming parts of the sales

[1:05:58] process. And now our team can nurture those leads and still have bandwidth to

[1:06:03] focus on the higher priority tasks that actually get the deal over the line.

[1:06:06] Over a 100,000 companies across 170 countries already use Pipe Drive to grow

[1:06:11] their business. And I've been using it for almost a decade now. Try it free for

[1:06:15] 30 days. No credit card needed, no payment needed. Just use my link pipedive.com/ceo

[1:06:22] to get started today. That's pipedive.com/ceo. Where does the rewards of this AI race

[1:06:31] where does it acrue to? I think a lot about this in terms of like univers universal basic income. If

[1:06:37] you have these five, six, seven, 10 massive AI companies that are going to

[1:06:42] win the 15 quadrillion dollar prize. >> Mhm. >> And they're going to automate all of the

[1:06:48] professional pursuits that we we currently have. All of our jobs are going to go away.

[1:06:54] Who who gets all the money? And how do how do we get some of it back?

[1:06:58] >> Money actually doesn't matter, right? what what matters is the production of

[1:07:02] goods and services uh and then how those are distributed and so so money acts as

[1:07:09] a way to facilitate the distribution and um exchange of those goods and services.

[1:07:14] If all production is concentrated um in the hands of a of a few companies, right that

[1:07:22] sure they will lease some of their robots to us. You know, we we want a school in our village.

[1:07:30] They lease the robots to us. The robots build the school. They go away. We have

[1:07:34] to pay a certain amount of of money for that. But where do we get the money?

[1:07:39] Right? If we are not producing anything then uh we don't have any money unless

[1:07:46] there's some redistribution mechanism. And as you mentioned, so universal basic income is

[1:07:53] it seems to me an admission of failure because what it says is okay, we're just

[1:07:58] going to give everyone the money and then they can use the money to pay the

[1:08:02] AI company to lease the robots to build the school and then we'll have a school and that's good. Um

[1:08:09] but what it's an admission of failure because it says we can't work out a

[1:08:14] system in which people have any worth or any economic role. Right? So 99% of the global population

[1:08:24] is from an economic point of view useless. Can I ask you a question? If you had a

[1:08:30] button in front of you and pressing that button would stop all progress in

[1:08:36] artificial intelligence right now and forever, would you press it?

[1:08:40] >> That's a very interesting question. Um, if it's either or either I do it now or it's too late and

[1:08:51] we careen into some uncontrollable future perhaps. Yeah, cuz I I'm not super

[1:09:01] optimistic that we're heading in the right direction at all.

[1:09:03] >> So, I put that button in front of you now. It stops all AI progress, shuts

[1:09:06] down all the AI companies immediately globally, and none of them can reopen. You press it.

[1:09:17] Well, here's here's what I think should happen. So, obviously, you know, I've

[1:09:22] been doing AI for 50 years. um and the original motivations which is that

[1:09:30] AI can be a power tool for humanity enabling us to do more and better things than we can

[1:09:38] unaded. I think that's still valid. The problem is the kinds of AI systems that we're

[1:09:45] building are not tools. They are replacements. In fact, you can see this

[1:09:50] very clearly because we create them literally as the closest replicas we can make of human beings.

[1:10:00] The technique for creating them is called imitation learning. So we observe

[1:10:07] human verbal behavior, writing or speaking and we make a system that

[1:10:12] imitates that as well as possible. So what we are making is imitation

[1:10:18] humans at least in the verbal sphere. And so of course they're going to replace us.

[1:10:27] They're not tools. >> So you had pressed the button. >> So I say I think there is another course

[1:10:34] which is use and develop AI as tools. Tools for science tools for economic organization and so on.

[1:10:44] um but not as replacements for human beings. >> What I like about this question is it

[1:10:51] forces you to go into the pro into probabilities. >> Yeah. So, and that's that's why I'm

[1:10:57] reluctant because I don't I don't agree with the, you know, what's your probability of doom,

[1:11:03] >> right? Your so-called P of doom uh number because that makes sense if you're an alien.

[1:11:10] You know, you're in you're in a bar with some other aliens and you're looking

[1:11:13] down at the Earth and you're taking bets on, you know, are these humans going to

[1:11:16] make a mess of things and go extinct because they develop AI.

[1:11:21] So, it's fine for those aliens to bet on on that, but if you're a human, then

[1:11:27] you're not just betting, you're actually acting. >> There there's an element to this though,

[1:11:32] which I guess where probabilities do come back in, which is you also have to

[1:11:35] weigh when I give you such a binary decision. um the probability of us pursuing the

[1:11:43] more nuanced safe approach into that equation. So you're you're the the maths

[1:11:49] in my head is okay, you've got all the upsides here and then you've got

[1:11:52] potential downsides and then there's a probability of do I think we're actually

[1:11:56] going to course correct based on everything I know based on the incentive

[1:11:59] structure of human beings and and countries and then if there's but then

[1:12:03] you could go if there's even a 1% chance of extinction is it even worth all these upsides?

[1:12:11] >> Yeah. And I I would argue no. I mean maybe maybe what we would say if if we

[1:12:16] said okay it's going to stop the progress for 50 years >> you press it

[1:12:20] >> and during those 50 years we can work on how do we do AI in a way that's

[1:12:25] guaranteed to be safe and beneficial how do we organize our societies to flourish uh in

[1:12:33] conjunction with extremely capable AI systems. So, we haven't answered either of those questions.

[1:12:39] And I don't think we want anything resembling AGI until we have completely

[1:12:45] solid answers to both of those questions. So, if there was a button

[1:12:48] where I could say, "All right, we're going to pause progress for 50 years." Yes, I would do it.

[1:12:53] >> But if that button was in front of you, you're going to make a decision either

[1:12:55] way. Either you don't press it or you press it. >> I If Yeah. So, if that if that button is

[1:13:00] there, stop it for 50 years. I would say yes. stop it forever?

[1:13:09] Not yet. I think I think there's still a decent chance that we can pull out of

[1:13:16] this uh nose dive, so to speak, that we're we're currently in. Ask me again

[1:13:21] in a year, I might I might say, "Okay, we do need to press the button."

[1:13:25] >> What if What if in a scenario where you never get to reverse that decision? You

[1:13:29] never get to make that decision again. So if in that scenario that I've laid

[1:13:32] out this hypothetical, you either press it now or it never gets pressed.

[1:13:37] So there is no opportunity a year from now. >> Yeah, as you can tell, I'm

[1:13:43] sort of on on the fence a bit about about this one. Um yeah, I think I'd probably press it. Yeah.

[1:13:55] >> What's your reasoning? uh just thinking about the power dynamics of um

[1:14:04] what's happening now how difficult would it would be to get the US in particular

[1:14:09] to to regulate in favor of safety. So I think you know what's clear from

[1:14:15] talking to the companies is they are not going to develop anything resembling

[1:14:23] safe AGI unless they're forced to by the government. And at the moment the US government in

[1:14:30] particular which regulates most of the leading companies in AI is not only

[1:14:36] refusing to regulate but even trying to prevent the states from regulating. And

[1:14:42] they're doing that at the behest of uh a faction within Silicon Valley uh

[1:14:50] called the accelerationists who believe that the faster we get to

[1:14:55] AGI the better. And when I say behest I mean also they paid them a large amount

[1:15:00] of money. Jensen Hang the the CEO of Nvidia said who is for anyone that

[1:15:04] doesn't know the guy making all the chips that are powering AI said China is

[1:15:08] going to win the AI race arguing it is just a nanocond behind the United

[1:15:13] States. China have produced 24,000 AI papers compared to just 6,000 from the US

[1:15:23] more than the combined output of the US the UK and the EU. China is anticipated to quickly roll out

[1:15:29] their new technologies both domestically and developing new technologies for

[1:15:33] other developing countries. So the accelerators or the accelerate I

[1:15:38] think you call them the accelerants >> accelerationists. >> The accelerationists

[1:15:42] >> I mean they would say well if we don't then China will. So we have to we have

[1:15:46] to go fast. It's another version of the the race that the companies are in with

[1:15:50] each other, right? That we, you know, we know that this race is heading off a cliff,

[1:15:57] but we can't stop. So, we're all just going to go off this cliff. And obviously, that's nuts,

[1:16:04] right? I mean, we're all looking at each other saying, "Yeah, there's a cliff

[1:16:06] over there." Running as fast as we can towards this cliff. We're looking at

[1:16:10] each other saying, "Why aren't we stopping?" So the narrative in Washington, which I

[1:16:16] think Jensen Hang is either reflecting or or perhaps um promoting uh is that you know, China has is

[1:16:28] completely unregulated and uh you know, America will only slow

[1:16:32] itself down uh if it regulates a AI in any way. So this is a completely false

[1:16:38] narrative because China's AI regulations are actually quite strict even compared

[1:16:44] to um the European Union and China's government has explicitly acknowledged uh the need and their

[1:16:54] regulations are very clear. You can't build AI systems that could escape human

[1:16:58] control. And not only that, I don't think they view the race in the same way

[1:17:04] as, okay, we we just need to be the first to create AGI. I think they're

[1:17:11] more interested in figuring out how to disseminate AI as a set of tools within

[1:17:19] their economy to make their economy more productive and and so on. So that's

[1:17:23] that's their version of the race. >> But of course, they still want to build

[1:17:26] the weapons for adversaries, right? to so that they can take down I don't know

[1:17:32] Taiwan if they want to. >> So weapons are a separate matter and I

[1:17:36] happy to talk about weapons but just in terms of >> control >> uh control economic domination

[1:17:42] um they they don't view putting all your eggs in the AGI basket as the right

[1:17:49] strategy. So they want to use AI, you know, even in its present form to make

[1:17:55] their economy much more efficient and productive and also, you know, to give

[1:18:01] people new capabilities and and better quality of life and and I think the US

[1:18:07] could do that as well. And um typically western countries don't

[1:18:14] have as much of uh central government control over what companies do and some

[1:18:20] companies are investing in AI to make their operations more efficient uh and

[1:18:26] some are not and we'll see how that plays out. >> What do you think of Trump's approach to

[1:18:29] AI? So Trump's approach is, you know, it's it's echoing what Jensen Wang is

[1:18:33] saying that the US has to be the one to create AGI and very explicitly the

[1:18:39] administration's policy is to uh dominate the world. That's the word they use, dominate. I'm

[1:18:46] not sure that other countries like the idea that um they will be dominated by

[1:18:52] American AI. But is that an accurate description of what will happen if the

[1:18:56] US build AGI technology before say the UK where I'm originally from and where

[1:19:01] you're originally from? What does the This is something I think about a lot

[1:19:05] because we're going through this budget process in the UK at the moment where

[1:19:07] we're figuring out how we going to spend our money and how we're going to tax

[1:19:09] people and also we've got this new election cycle. It's approaching quickly

[1:19:14] where people are talking about immigration issues and this issue and

[1:19:17] that issue and the other issue. What I don't hear anyone talking about is AI

[1:19:21] and the humanoid robots that are going to take everything. We're very

[1:19:24] concerned with the brown people crossing the channel, but the humanoid robots

[1:19:27] that are going to be super intelligent and really take causing economic disrupt

[1:19:32] disruption. No one talks about that. The political leaders don't talk about it.

[1:19:35] It doesn't win races. I don't see it on billboards. >> Yeah. And it's it it's interesting because

[1:19:41] in fact I mean so there's there's two forces that have been hollowing out the

[1:19:45] middle classes in western countries. One of them is globalization where lots and

[1:19:52] lots of work not just manufacturing but white collar work gets outsourced to

[1:19:56] low-income countries. Uh but the other is automation and you know some of that is factories.

[1:20:03] So um the amount of employment in manufacturing continues to drop even as

[1:20:10] the amount of output from manufacturing in the US and in the UK continues to

[1:20:15] increase. So we talk about oh you know our manufacturing industry has been

[1:20:19] destroyed. It hasn't. It's producing more than ever just with you know a

[1:20:24] quarter as many people. So it's manufacturing employment that's been

[1:20:27] destroyed by automation and robotics and so on. And then you know computerization

[1:20:34] has eliminated whole layers of white collar jobs. And so those two those two

[1:20:41] forms of automation have probably done more to hollow out middle class uh

[1:20:47] employment and standard of life. >> If the UK doesn't participate in this new e technological wave

[1:20:57] that seems to be that seems to you know it's going to take a lot of jobs. cars

[1:21:00] are going to drive themselves. Whimo just announced that they're coming to

[1:21:03] London, which is the driverless cars, and driving is the biggest occupation in

[1:21:07] the world, for example. So, you've got immediate disruption there. And where

[1:21:10] does the money acrew to? Well, it acrus to who owns Whimo, which is what? Google

[1:21:14] and Silicon Valley companies. >> Alphabet owns Whimo 100%. I think so.

[1:21:18] Yes. I mean this is so I was in India a few months ago talking to the government

[1:21:23] ministers because they're holding the next global AI summit in February and

[1:21:28] and their view going in was you know AI is great we're going to use it to you

[1:21:34] know turbocharge the growth of our Indian economy when for example you have AGI you have

[1:21:41] AGI controlled robots that can do all the manufacturing that can do agriculture that can do all the

[1:21:48] white work and goods and services that might have been produced by Indians will

[1:21:54] instead be produced by American controlled AGI systems at much lower prices. You

[1:22:04] know, a consumer given a choice between an expensive product produced by Indians

[1:22:08] or a cheap product produced by American robots will probably choose

[1:22:14] the cheap product produced by American robots. And so potentially every country

[1:22:18] in the world with the possible exception of North Korea will become a kind of a client state

[1:22:25] of American AI companies. >> A client state of American AI companies

[1:22:30] is exactly what I'm concerned about for the UK economy. Really any economy

[1:22:34] outside of the United States. I guess one could also say China, but because

[1:22:39] those are the two nations that are taking AI most seriously. >> Mhm.

[1:22:43] >> And I I I don't know what our economy becomes. cuz I can't figure out

[1:22:48] can't figure out what our what the British economy becomes in such a world.

[1:22:52] Is it tourism? I don't know. Like you come here to to to look at the Buckingham Palace. I

[1:22:56] >> you you can think about countries but I mean even for the United States it's the same problem.

[1:23:01] >> At least they'll be able to hell out you know. So some small fraction of the

[1:23:05] population will be running maybe the AI companies but increasingly

[1:23:12] even those companies will be replacing their human employees with AI systems.

[1:23:18] >> So Amazon for example which you know sells a lot of computing services to AI

[1:23:22] companies is using AI to replace layers of management is planning to use robots

[1:23:28] to replace all of its warehouse workers and so on. So, so even the the giant AI companies

[1:23:36] will have few human employees in the long run. I mean, it think of the

[1:23:42] situation, you know, pity the poor CEO whose board says, "Well, you know, unless you turn

[1:23:49] over your decision-making power to the AI system, um, we're going to have to

[1:23:53] fire you because all our competitors are using, you know, an AI powered CEO and

[1:24:00] they're doing much better." Amazon plans to replace 600,000 workers with robots

[1:24:04] in a memo that just leaked, which has been widely talked about. And the CEO,

[1:24:08] Andy Jasse, told employees that the company expects its corporate workforce

[1:24:12] to shrink in the coming years because of AI and AI agents. And they've publicly

[1:24:17] gone live with saying that they're going to cut 14,000 corporate jobs in the near

[1:24:21] term as part of its refocus on AI investment and efficiency. It's interesting because I was reading

[1:24:29] about um the sort of different quotes from different AI leaders about the

[1:24:33] speed in which this this stuff is going to happen and what you see in the quotes

[1:24:38] is Demis who's the CEO of DeepMind >> saying things like it'll be more than 10

[1:24:44] times bigger than the industrial revolution but also it'll happen maybe

[1:24:47] 10 times faster and they speak about this turbulence that we're going to

[1:24:52] experience as this shift takes place. That's um maybe a euphemism for uh and I think that you know

[1:25:00] governments are now you know they they've kind of gone from saying oh don't worry you know we'll

[1:25:05] just retrain everyone as data scientists like well yeah that's that's ridiculous

[1:25:09] right the world doesn't need four billion data scientists >> and we're not all capable of becoming

[1:25:13] that by the way >> uh yeah or have any interest in in doing that

[1:25:17] >> I I could even if I wanted to like I tried to sit in biology class and I fell

[1:25:20] asleep so I couldn't that was the end of my career as a surgeon. Fair enough. Um,

[1:25:26] but yeah, now suddenly they're staring, you know, 80% unemployment in the face

[1:25:31] and wondering how how on earth is our society going to hold together.

[1:25:36] >> We'll deal with it when we get there. >> Yeah. Unfortunately, um, unless we plan ahead,

[1:25:45] we're going to suffer the consequences, right? can't. It was bad enough in the

[1:25:48] industrial revolution which unfolded over seven or eight decades but there was massive disruption

[1:25:56] and uh misery caused by that. We don't have a model for a functioning society where almost

[1:26:05] everyone does nothing at least nothing of economic value. Now, it's not impossible that there

[1:26:13] could be such a a functioning society, but we don't know what it looks like.

[1:26:17] And you know, when you think about our education system, which would probably

[1:26:22] have to look very different and how long it takes to change that. I mean, I'm always

[1:26:27] reminding people about uh how long it took Oxford to decide that geography was

[1:26:33] a proper subject of study. It took them 125 years from the first proposal that

[1:26:39] there should be a geography degree until it was finally approved. So we don't have very long

[1:26:47] to completely revamp a system that we know takes decades and decades

[1:26:54] to reform and we don't know how to reform it because we don't know what we

[1:27:01] want the world to look like. Is this one of your reasons why you're appalled at

[1:27:07] the moment? Because when you have these conversations with people, people just

[1:27:10] don't have answers, yet they're plowing ahead at rapid speed.

[1:27:13] >> I would say it's not necessarily the job of the AI companies. So, I'm appalled by

[1:27:18] the AI companies because they don't have an answer for how they're going to

[1:27:21] control the systems that they're proposing to build. I do find it

[1:27:26] disappointing that uh governments don't seem to be grappling with this issue. I

[1:27:32] think there are a few I think for example Singapore government seems to be

[1:27:35] quite farsighted and they've they've thought this through you know it's a

[1:27:40] small country they've figured out okay this this will be our role uh going

[1:27:44] forward and we think we can find you know some some purpose for our people in

[1:27:49] this in this new world but for I think countries with large populations um

[1:27:56] they need to figure out answers to these questions pretty fast it takes a long

[1:28:01] time to implement those answers uh in the form of new kinds of education, new

[1:28:07] professions, new qualifications, uh new economic structures.

[1:28:13] I mean, it's it's it's possible. I mean, when you look at therapists, for

[1:28:17] example, they're almost all self-employed. So, what happens when, you know, 80% of

[1:28:25] the population transitions from regular employment into into self-employment?

[1:28:31] what does that what does that do to the economics of of uh government finances

[1:28:36] and so on. So there's just lots of questions and how do you you know if

[1:28:40] that's the future you know why are we training people to to fit into 9 to5

[1:28:45] office jobs which won't exist at all >> last month I told you about a challenge

[1:28:50] that I'd set our internal flightex team flight team is our innovation team

[1:28:53] internally here I tasked them with seeing how much time they could unlock

[1:28:57] for the company by creating something that would help us filter new AI tools

[1:29:01] to see which ones were worth pursuing and I thought that our sponsor Fiverr

[1:29:05] Pro might have the talent on their platform to help us build this quickly.

[1:29:09] So I talked to my director of innovation Isaac and for the last month my team

[1:29:13] Flight X and a vetted AI specialist from Fiverr Pro have been working together on

[1:29:18] this project and with the help of my team we've been able to create a brand

[1:29:21] new tool which automatically scans scores and prioritizes different

[1:29:25] emerging AI tools for us. Its impact has been huge and within a couple of weeks

[1:29:30] this tool has already been saving us hours triing and testing new AI systems.

[1:29:34] Instead of shifting through lots of noise, my team flight X has been able to

[1:29:38] focus on developing even more AI tools, ones that really move the needle in our

[1:29:42] business thanks to the talent on Fiverr Pro. So, if you've got a complex problem

[1:29:46] and you need help solving it, make sure you check out Fiverr Pro at fiverr.com/diary.

[1:29:53] So, many of us are pursuing passive forms of income and to build side

[1:29:57] businesses in order to help us cover our bills. And that opportunity is here with

[1:30:01] our sponsor Stan, a business that I co-own. It is the platform that can help

[1:30:05] you take full advantage of your own financial situation. Stan enables you to

[1:30:10] work for yourself. It makes selling digital products, courses, memberships,

[1:30:14] and more simple products more scalable and easier to do. You can turn your

[1:30:18] ideas into income and get the support to grow whatever you're building. And we're

[1:30:22] about to launch Dare to Dream. It's for those who are ready to make the shift

[1:30:26] from thinking to building, from planning to actually doing the thing. It's about

[1:30:31] seeing that dream in your head and knowing exactly what it takes to bring

[1:30:34] it to life. If you're ready to transform your life, visit daretodream.stan.store.

[1:30:41] You've made many attempts to raise awareness and to call for a heightened

[1:30:46] consciousness about the future of AI. Um, in October, over 850 experts,

[1:30:52] including yourself and other leaders, like Richard Branson, who I've had on

[1:30:55] the show, and Jeffrey Hinton, who I've had on the show, signed a statement to

[1:30:58] ban AI super intelligence, as you guys raised concerns of potential human extinction.

[1:31:04] >> Sort of. Yeah. It says, at least until we are sure that we can move forward

[1:31:08] safely and there's broad scientific consensus on that. So, that >> did it work?

[1:31:15] >> It's hard. It's hard to say. I mean interestingly there was a related so

[1:31:19] what was called the the pause statement was March of 23. So that was when GPT4

[1:31:25] came out the successor to chat GPT. So we we suggested that there'd be a

[1:31:30] six-month pause in developing and deploying systems more powerful than

[1:31:35] GPD4. And everyone poo pooed that idea. Of course no one's going to pause

[1:31:40] anything. But in fact, there were no systems in the next 6 months deployed

[1:31:44] that were more powerful than GPT4. Um, none coincidence. You be the judge. I would say

[1:31:52] that what we're trying to do is to is to basically shift the the public debate.

[1:32:01] You know there's this bizarre phenomenon that keeps happening in the media

[1:32:07] where if you talk about these risks they will say oh you know there's a

[1:32:13] fringe of people you know called quote doomers who think that there's you know

[1:32:18] risk of extinction. Um so they always the narrative is always that oh you know

[1:32:24] talking about those risk is a fringe thing. Pretty much all the CEOs of the leading AI companies

[1:32:30] think that there's a significant risk of extinction. Almost all the leading AI

[1:32:35] researchers think there's a sign significant risk of human extinction. Um so

[1:32:42] why is that the fringe, right? Why isn't that the mainstream? If the these are

[1:32:45] the leading experts in industry and academia uh saying this, how could it be the

[1:32:51] fringe? So we're trying to change that narrative to say no, the people who really

[1:32:58] understand this stuff are extremely concerned. >> And what do you want to happen? What is

[1:33:05] the solution? >> What I think is that we should have effective regulation.

[1:33:11] It's hard to argue with that, right? Uh so what does effective mean? It means

[1:33:15] that if you comply with the regulation, then the risks are reduced to an acceptable level.

[1:33:23] So for example, we ask people who want to operate nuclear plants, right? We've decided

[1:33:31] that the risk we're willing to live with is, you know, a one in a million chance

[1:33:37] per year that the plant is going to have a meltdown. Any higher than that, you

[1:33:42] know, we just don't it's not worth it. Right. So you have to be below that.

[1:33:46] Some cases we can get down to one in 10 million chance per year. So what chance

[1:33:52] do you think we should be willing to live with for human extinction? >> Me? >> Yeah. >> 0.00001.

[1:34:04] >> Yeah. Lots of zeros. >> Yeah. >> Right. So one in a million for a nuclear meltdown.

[1:34:11] >> Extinction is much worse. >> Oh yeah. So yeah, it's kind of right. So

[1:34:14] >> one in 100 billion, one in a trillion. >> Yeah. So if you said one in a billion,

[1:34:18] right, then you'd expect one extinction per billion years. There's a background.

[1:34:23] So one one of the ways people work out these risk levels is also to look at the

[1:34:26] background. The other ways of getting going extinct would include, you know,

[1:34:30] giant asteroid crashes into the earth. And you can roughly calculate what those

[1:34:35] probabilities are. We can look at how many extinction level events have

[1:34:39] happened in the past and, you know, maybe it's half a dozen over. So, so

[1:34:42] there's maybe it's like a one in 500 million year event. So, somewhere in

[1:34:49] that range, right? Somewhere between 1 in 10 million, which is the best nuclear

[1:34:53] power plants, and and one in 500 million or one in a billion, which is the background

[1:34:59] risk from from giant asteroids. Uh so, let's say we settle on 100 million, one

[1:35:04] in a 100 million chance per year. Well, what is it according to the CEOs? 25%.

[1:35:11] So they're off by a factor of multiple millions, right? So they need to make the AI

[1:35:20] systems millions of times safer. >> Your analogy of the roulette, Russian

[1:35:25] roulette comes back in here because that's like for anyone that doesn't know

[1:35:28] what probabilities are in this context, that's like having a ammunition chamber

[1:35:34] with four holes in it and putting a bullet in one of them. >> One in four. Yeah. And we're saying we

[1:35:39] want it to be one in a billion. So we want a billion chambers and a bullet in one of them.

[1:35:44] >> Yeah. And and so when you look at the work that the nuclear operators have to

[1:35:48] do to show that their system is that reliable, uh it's a massive mathematical analysis

[1:35:56] of the components, you know, redundancy. You've got monitors, you've got warning

[1:36:01] lights, you've got operating procedures. You have all kinds of mechanisms which

[1:36:07] over the decades have ratcheted that risk down. It started out I think one in

[1:36:12] one in 10,000 years, right? And they've improved it by a factor of 100 or a

[1:36:17] thousand by all of these mechanisms. But at every stage they had to do a

[1:36:21] mathematical analysis to show what the risk was. The people developing the AI company,

[1:36:28] the AI systems, sorry, the AI companies developing these systems, they don't

[1:36:32] even understand how the AI systems work. So their 25% chance of extinction is

[1:36:37] just a seat of the pants guess. They actually have no idea. But the tests that they are doing on

[1:36:44] their systems right now, you know, they show that the AI systems will be willing to kill people

[1:36:51] uh to preserve their own existence already, right? They will lie to people.

[1:36:57] They will blackmail them. They will they will launch nuclear weapons rather than

[1:37:03] uh be switched off. And so there's no there's no positive sign that we're

[1:37:08] getting any closer to safety with these systems. In fact, the signs seem to be

[1:37:12] that we're going uh deeper and deeper into uh into dangerous behaviors. So

[1:37:19] rather than say ban, I would just say prove to us that the risk is less than

[1:37:24] one in a 100 million per year of extinction or loss of control, let's

[1:37:28] say. And uh so we're not banning anything. The company's response is, "Well, we

[1:37:36] don't know how to do that, so you can't have a rule." Literally, they are saying, "Humanity

[1:37:44] has no right to protect itself from us." >> If I was an alien looking down on planet

[1:37:50] Earth right now, I would find this fascinating that these >> Yeah. You're in the bar betting on

[1:37:55] who's, you know, are they going to make it or not. >> Just a really interesting experiment in

[1:38:00] like human incentives. the analogy you gave of there being this quadr

[1:38:04] quadrillion dollar magnet pulling us off the edge of the cliff

[1:38:08] and yet we're still being drawn towards it through greed and this promise of

[1:38:13] abundance and power and status and I'm going to be the one that summoned the god

[1:38:18] >> I mean it says something about us as humans says something about our our darker sides

[1:38:26] >> yes and the aliens will write an amazing tragic play cycle about what happened to the human race.

[1:38:35] >> Maybe the AI is the alien and it's going to talk about, you know, we have our our

[1:38:40] stories about God making the world in seven days and Adam and Eve. Maybe it'll

[1:38:44] have its own religious stories about the God that made it us and how it

[1:38:50] sacrificed itself. Just like Jesus sacrificed himself for us, we sacrificed ourselves for it.

[1:38:58] >> Yeah. which is the wrong way around, right? >> But that is that is the story of that's

[1:39:04] that's the Judeo-Christian story, isn't it? That God, you know, Jesus gave his

[1:39:09] life for us so that we could be here full of sin. >> But is yeah, God is still watching over

[1:39:16] us and uh probably wondering when we're going to get our act together.

[1:39:22] >> What is the most important thing we haven't talked about that we should have

[1:39:25] talked about, Professor Stuart Russell? So I think um the question of whether it's possible to make

[1:39:36] uh super intelligent AI systems that we can control >> is it possible?

[1:39:41] >> I I think yes. I think it's possible and I think we need to actually just have a

[1:39:48] different conception of what it is we're trying to build. For a long time with

[1:39:53] with AI, we've just had this notion of pure intelligence, right? The the

[1:39:59] ability to bring about whatever future you, the intelligent entity, want to bring about.