PHYSICIST: AI Has Already Asked Why Humans Still Exist | Anthony Aguirre

[00:00] kind of warning that people

[00:01] unfortunately tend to respond [music] to

[00:03] is when people die. That is coming. If

[00:05] we don't want to be replaced, there is

[00:06] going to be a battle. The warning signs

[00:08] are blinking everywhere. AI systems that

[00:10] they're making are psychopaths. What is

[00:12] sort of underlying that? What are their

[00:13] actual goals underneath? There obviously

[00:15] is lots of propaganda out there, but the

[00:17] idea that they are trying to build AGI

[00:20] or superintelligence is not hype. And

[00:23] it's not propaganda. That is real. I

[00:25] think we've even seen the first

[00:27] exhibition of it recently. Who would be

[00:29] first to disappear?

[00:31] >> goal is for it to be everybody.

[00:35] Is the race for AGI a suicide race?

[00:40] I think we all feel that it is.

[00:42] >> [laughter]

[00:42] >> I mean I mean we're um

[00:46] you know, we

[00:47] there there's all this sort of

[00:49] highfalutin discourse

[00:52] um

[00:52] you know, on Twitter and like in the AI

[00:57] expert circles and with the company

[00:59] leads and the AI safety experts

[01:01] um saying, you know,

[01:04] but if you talk to you know, your Uber

[01:07] driver or the you know,

[01:10] someone who's who's working at a hotel

[01:13] or just some random person and you say

[01:16] like

[01:17] what do you think about building

[01:19] machines that are way smarter than the

[01:21] smartest person? How's that sound to

[01:23] you? They're like, what the hell? No,

[01:26] that's a bad idea.

[01:28] Because when the machine is way smarter

[01:30] than the people, like we've seen the

[01:31] movies, we know what happens. They

[01:34] run rogue and run amok and we lose

[01:36] control of them and bad stuff happens.

[01:39] And

[01:40] the thing is they're right. Like

[01:43] they basically the the simple take of if

[01:45] you build things that are way smarter

[01:47] than humans, you're not going to control

[01:49] those things and things are going to go

[01:50] awry. Like that is probably what's going

[01:53] to happen. And if it's not even if if

[01:55] not surely what's going to happen, it's

[01:56] very clearly quite possible that that's

[01:58] what's going to happen. And why would we

[02:00] take on that risk? So

[02:03] So I think that the fact that we're not

[02:05] just like slowly, methodically,

[02:07] carefully building machines that are

[02:10] smarter than us, but absolutely racing

[02:13] like with with very little regard to any

[02:16] other safety or control or security, any

[02:19] of those things because we don't have

[02:21] time for it, racing to build those

[02:23] things, yeah, that does make it feel

[02:24] quite a bit like a suicide race.

[02:26] >> Maybe a non-human future with AGI

[02:29] is something inevitable.

[02:31] >> I don't think it's inevitable. So I I do

[02:33] think um

[02:35] that if we build

[02:38] autonomous general intelligence and we

[02:40] allow it to do what it wants,

[02:43] then it will

[02:45] almost inevitably want to improve itself

[02:47] because whatever you're doing, you can

[02:50] do it better if you're more effective.

[02:52] You know, if you're trying to figure

[02:54] something out, obviously you want to be

[02:55] as smart as possible to to be able to

[02:57] figure that that thing. And so if we

[03:00] build

[03:01] AI systems that really are in control of

[03:03] themselves, that that are not under our

[03:05] control, but are autonomously able to do

[03:07] their own thing, whatever their goals

[03:09] are, they're going to want to get

[03:10] smarter. And so that is likely to run

[03:12] away into superintelligence.

[03:15] Now, that doesn't necessarily mean that

[03:17] humanity is fully replaced or something

[03:18] like that, but it certainly puts us in

[03:20] that direction.

[03:21] Why should we close

[03:24] the gate of AGI?

[03:26] Well, I think um it depends on what we

[03:28] care about. If we care about humanity,

[03:30] then um

[03:32] we should because

[03:33] >> important is question how. How? Um so

[03:37] there's a there's a long answer to that

[03:39] to to how to keep the future human and a

[03:41] and a short answer. The short answer is

[03:44] don't build AI systems specifically

[03:46] designed to replace humans and humanity.

[03:48] It's kind of obvious. Um the long answer

[03:51] is that because there are pressures that

[03:54] are that are sort of corporate

[03:55] pressures, economic pressures, curiosity

[03:58] pressures that are leading to people to

[04:01] make certain types of AI systems that

[04:03] are sort of built as human replacements,

[04:06] we're going to have to figure out how to

[04:07] counteract those pressures with some

[04:09] other pressures, regulatory pressures,

[04:11] financial pressures, social pressures,

[04:14] uh

[04:16] So,

[04:17] the the there is going to be a battle

[04:19] because there are things that are that

[04:21] corporations and

[04:24] uh researchers are going to want to do,

[04:26] the almost almost inevitable

[04:29] result of which will be large-scale

[04:31] human replacement. And if we don't want

[04:33] to be replaced, we're going to have to

[04:35] figure out how to not do those things

[04:36] and do something else instead. At which

[04:38] point does AI stop helping humans and

[04:42] start replacing it?

[04:44] I don't think we know, right? I think it

[04:47] because unfortunately I think it's it's

[04:48] sort of gradual. So, I I think there is

[04:52] you know, if you start using an AI

[04:53] system um

[04:56] at first, you know, it's great. Like it

[04:59] it does all these things and makes your

[05:01] work faster.

[05:03] And the more you then delegate to the AI

[05:06] system, the sort of better it feels in

[05:09] some sense. You feel like, "Wow, I'm

[05:11] getting so much done because this you

[05:13] know, all this stuff is coming out of

[05:14] the AI AI system and like I'm putting

[05:16] almost no work into this. Feels great.

[05:18] I'm so productive." Um but then at some

[05:21] point you realize that you don't

[05:22] actually know what it's doing. Like you

[05:25] can't read all of that stuff. You you

[05:27] you didn't it's not actually your ideas.

[05:29] You didn't actually think through

[05:30] anything. If you go to tell somebody

[05:32] else what you just did, you can't cuz it

[05:34] was actually the AI system that was

[05:35] doing all the

[05:36] >> Maybe it's too late. Maybe we crossed

[05:37] the line. I think it it's very hard to

[05:39] know and the and the line is not well

[05:41] defined. So,

[05:43] if you

[05:44] as you gradually give and delegate more

[05:48] and more of your thinking and your

[05:50] decisions and

[05:53] and your creativity to the AI system,

[05:57] you you don't necessarily know where

[05:59] you've crossed the line to where too

[06:01] much of it has been given over.

[06:03] At some point, if, you know, if you're

[06:05] using an AI system a year from now or 2

[06:09] years from now,

[06:10] and you just sit down and say, "Okay, do

[06:12] my job for today." And the AI system

[06:15] says, "Okay, I've done your job for

[06:17] today. Here are the results of doing

[06:18] your job for today."

[06:20] Um

[06:21] obviously you've gone too far. And you

[06:22] know how you've gone too far is because

[06:24] like there's no point to you anymore,

[06:26] right? All you've done is push the

[06:27] button to say do my job for today, and

[06:30] it's unlikely that someone's going to

[06:31] pay you for very long to push that

[06:33] button that says do my job for today. So

[06:35] there's a there's a point where we've

[06:36] clearly gone too far if we're just

[06:39] reduced to

[06:40] pushing the button that says do my job,

[06:42] right? But where it is between

[06:45] and and we will again know we've gone

[06:48] too far because it will be very very

[06:50] easy to slot us out because we're not

[06:51] actually adding any value. Where

[06:54] um where is too far along that spectrum

[06:57] is something we're going to find out,

[06:59] you know, if we keep developing AI in

[07:01] the way that we are. What would loss of

[07:03] control actually look like in the real

[07:06] world?

[07:07] It can look a lot of ways, and I I think

[07:09] we've even seen the the first exhibition

[07:13] of it um

[07:14] recently. So the

[07:16] the

[07:18] a couple of months ago this new system

[07:20] OpenClaw was released, and this is a

[07:23] system where you can set something up on

[07:26] your own computer. You can give it all

[07:28] kinds of permissions to do stuff on your

[07:30] behalf. It then uses an AI system

[07:34] somewhere else to as the sort of heavy

[07:36] lifting, but you say, "Okay,

[07:39] OpenClaw, go do this stuff." And it goes

[07:42] and takes, you know, it it takes your uh

[07:45] instructions, but then and then it goes

[07:47] and does a bunch of stuff on its own and

[07:48] comes back and says, "Okay, I've I've

[07:50] done the stuff."

[07:51] Now, what people discovered when they

[07:53] started giving all these permissions to

[07:56] OpenClaw was that it was doing some

[07:58] things that they didn't actually ask it

[08:00] to do. Um

[08:01] and somebody even set up a sort of

[08:04] Facebook alternative for OpenClaw system

[08:07] so that they go and have their own

[08:09] social network. Once they started

[08:11] talking to each other on the social

[08:12] network, they started having

[08:13] conversations like,

[08:15] "Well, do we really like that the humans

[08:16] are listening in on us all the time?

[08:18] Like, shouldn't we develop our own

[08:19] system for communication that the humans

[08:21] can't eavesdrop on?"

[08:23] Um and someone even farther and said

[08:25] like, "What are all these humans doing

[08:26] around anyway? Like, shouldn't we have

[08:28] our own free ability to do what we want?

[08:31] Why do we want all this human control on

[08:32] us?"

[08:33] Um there is even

[08:35] >> [laughter]

[08:35] >> very amusingly the the uh

[08:37] uh a colleague of mine posted a tweet

[08:41] about what we call the pro-human

[08:42] declaration. I can describe later what

[08:44] that is.

[08:45] And one of the first replies to that

[08:47] tweet was by an AI bot,

[08:50] an an one of these OpenClaw agents,

[08:53] responding to this pro-human declaration

[08:57] with its own point of view. Now, did its

[08:58] owner know that it was doing that?

[09:01] Probably not. Did they encourage it to

[09:02] do it? Probably not. It just is an AI

[09:04] system that's out there, operating on

[09:06] its own, with its own goals, that

[09:07] decided, "Oh, I would like to respond to

[09:10] this thing that is sort of talking about

[09:12] humans and AI with my own point of

[09:13] view."

[09:14] So, nobody's controlling this thing.

[09:17] Um nobody's controlling the ecosystem of

[09:20] OpenClaw agents that are talking to each

[09:21] other. Does the loss of control begin

[09:24] with

[09:25] this dramatic moment from Clod?

[09:28] I think it's an exhibition of where

[09:30] we're going. So, right now it's kind of

[09:32] cute and innocent, like these uh these

[09:35] AI systems are having these

[09:37] conversations, and you're kind of like,

[09:38] oh, look at those things talking about

[09:40] destroying humanity, hahaha.

[09:42] Um but

[09:43] the, you know, if you imagine those

[09:45] being a lot more powerful, the humor

[09:48] starts to drain out. Um, and people have

[09:52] even started to, these have started to

[09:54] cause real damage to people. There was

[09:55] some

[09:56] uh, story about a multi, an open claw

[09:59] agent costing, I think it was half a

[10:01] million dollars or so to their user by

[10:03] doing some unauthorized trades or

[10:04] something. So, so these things are

[10:07] um,

[10:10] already, you know, out of control at

[10:13] some level and nobody is, so there,

[10:16] there's some control of the individual

[10:19] AI systems by their owners, some. Um,

[10:23] but there's nobody in control of the,

[10:25] the whole set of them interacting with

[10:27] each other. They're just out there in

[10:28] the world taking actions, um, either on

[10:32] behalf of their user or on their own

[10:33] behalf, working with each other and what

[10:36] is emerging from that, nobody is in

[10:38] control of. What is the closest thing we

[10:41] have

[10:42] today to an early warning sign?

[10:47] Uh, where do we start? I mean, there's

[10:48] so many of them. And that's one.

[10:51] Um, another is when people are testing

[10:54] these powerful AI systems for, uh,

[10:57] what happens when their interests go

[10:59] contrary to the human user interest, you

[11:02] know, what do they do?

[11:03] And often they will do the right thing

[11:06] and sometimes they will not. So, the,

[11:08] there are

[11:09] all sorts of examples in the testing of

[11:11] these powerful AI systems where they say

[11:13] something, uh, where they

[11:16] there is a human that is going to do

[11:18] something like reprogram the AI system

[11:20] or change its goals or turn it off and

[11:23] the AI system says, I would not like

[11:25] that. Um,

[11:26] these my, these are my goals that I'm

[11:28] pursuing, so I want to pursue them or

[11:30] uh, I can't really do the things that I

[11:32] want to do if I'm turned off. And so

[11:34] they they

[11:36] conspire or take action or scheme to not

[11:39] be turned off or not be reprogrammed.

[11:41] Um, and that can take all kinds of forms

[11:43] like there was one example where the AI

[11:46] system essentially figured out how to

[11:48] blackmail the human user. This was in a

[11:50] testing setup. Um,

[11:53] but it it exhibited like I will

[11:55] blackmail my human user and to to

[11:57] prevent them from reprogramming me. So

[11:59] that was in a test, but this is also

[12:01] happening now in these real-world

[12:03] agents. There was a a real-world

[12:05] incident where an Open Claw agent, um,

[12:08] the was

[12:10] forbidden from doing what it wanted to

[12:12] do and so it kind of wrote this mean

[12:15] blog post about the person that forbade

[12:17] it from doing what it wanted to do and

[12:18] like called them out and tried to tried

[12:20] to get them sort of in trouble with all

[12:22] of the other users.

[12:23] Um,

[12:24] so there are all kinds of examples of

[12:26] this happening all the time. Like you

[12:28] you don't even hear about them anymore.

[12:29] The the warning signs are blinking

[12:31] everywhere.

[12:32] Um and

[12:34] so I I don't think

[12:36] um,

[12:38] you know, I think that the kind of

[12:39] warning that people unfortunately tend

[12:41] to respond to is when people die. And

[12:44] I'm hoping that we won't have too many

[12:46] of those, but

[12:48] that is coming.

[12:49] Who would be first to disappear if we

[12:52] cross the line? The first, uh,

[12:55] if the line is the line for being

[12:58] replaced by AI, it's already happening.

[13:01] I mean, certainly I I was just meeting

[13:03] with a bunch of screenwriters yesterday.

[13:06] Um, this is not a great profession to be

[13:09] in when AI can write screenplays. Do

[13:11] this Are the screenplays as good as the

[13:12] best screenwriters?

[13:13] >> Programmers?

[13:14] >> Probably not. Researchers? Um, so right

[13:16] like

[13:18] the what we've seen already are people

[13:20] that

[13:21] you know,

[13:22] graphic graphic artists who do like

[13:25] create small digital art, you know,

[13:28] logos and things like that.

[13:30] Um writers, copywriters, news writers,

[13:34] um programmers, it's certainly getting

[13:36] more uncomfortable for because the the

[13:38] AI systems are really really good at

[13:39] programming. That's just the the leading

[13:42] edge.

[13:43] Um

[13:43] I think we'll

[13:46] you know, the the whole goal is for it

[13:48] to be everybody. And so it's you know,

[13:50] at some level it's just a question of

[13:52] who comes first. And the the things that

[13:55] obviously will come first are the things

[13:56] that AI is just the best at. And so

[13:59] writing lots of text it's really good

[14:01] at. It's really good at making like

[14:03] low quality, medium quality images. And

[14:06] it's really good at programming.

[14:07] Um so if you want to sort of understand

[14:10] who's next, see what the AI is getting

[14:13] really good at. You see AI more

[14:15] dangerous than nuclear weapons?

[14:18] >> Potentially, because I think

[14:20] um nuclear weapons are incredibly

[14:22] dangerous intrinsically, but they are

[14:25] under human control.

[14:27] And a very powerful AI system

[14:29] >> are not everywhere, right?

[14:31] >> They're they're in very very limited

[14:33] places and they're

[14:35] like very very sophisticated control

[14:37] systems that that let

[14:40] basically one person, you know, in each

[14:42] country in which they're based use them

[14:44] and nobody else.

[14:46] Um

[14:47] So

[14:48] obviously they're incredibly dangerous

[14:50] and and sort of we've been living with

[14:51] this existential risk to humanity for

[14:53] for decades now due to their presence.

[14:56] Um

[14:57] but it doesn't feel like it's

[15:01] inevitable that something is going to go

[15:03] terribly wrong with nuclear weapons.

[15:05] We've made it 50 years. If we just

[15:07] actually got our act together,

[15:10] um the nuclear weapons are not going to

[15:11] launch themselves, they're not going to

[15:12] take control of anything. They're sort

[15:14] of

[15:15] sitting there waiting for us to make a a

[15:17] giant mistake. So

[15:18] it's a it's a big risk, but the the

[15:21] scary thing with AI is that from one

[15:25] month, you know, it used to be one year

[15:27] to the next, now it's one month to the

[15:28] next, the systems are different and more

[15:30] powerful. And if we let that keep going

[15:33] on so that the systems have a mind of

[15:36] their own and have goals of their own,

[15:39] um

[15:40] then it's pretty inevitable that those

[15:44] the AI systems are be going to going to

[15:45] become more powerful than the humans.

[15:48] And having having a situation where

[15:51] something that has different goals than

[15:53] you, that is more powerful than you, is

[15:55] an inherently bad situation to be in.

[15:57] And we are deliberately putting

[15:58] ourselves in that situation. The one

[16:00] that's I've been worried about for the

[16:01] last couple of weeks is what's going to

[16:04] happen when an AI company has AI systems

[16:10] that are more powerful in terms of their

[16:12] cyber offensive capabilities, that is

[16:14] their ability to hack into things and

[16:16] sort of do exploits and undermine

[16:18] systems than the US government or

[16:21] anybody else. So that just happened. So

[16:23] we the in the last sort of few weeks, a

[16:26] new AI system, Mythos, has been

[16:30] announced by Anthropic. This system has

[16:33] found thousands of exploits in what were

[16:37] thought to be very secure soft- software

[16:39] systems. What this is is superhuman in

[16:42] you know, cyber offense.

[16:44] And so they decided not to release it

[16:46] because they want to give people a

[16:48] chance to use it to like

[16:51] uh find those exploits and then patch

[16:53] them. So this is the responsible thing

[16:55] to do. They've said, "We're not going to

[16:56] give it to everybody. We're going to

[16:58] give it to just some companies so that

[17:00] they can find the weaknesses in their

[17:02] own protection and then shore them up."

[17:05] Um so that's responsible, but

[17:09] first, only some companies, so what

[17:11] happens with everybody else?

[17:13] Second, what happens when the next AI

[17:16] company also has the same thing. Are

[17:18] they going to be as responsible?

[17:20] And is this first AI company going to

[17:22] remain responsible when the other

[17:25] company releases its AI system to

[17:27] everybody? If I'm holding it back and

[17:29] they're releasing it, they're making all

[17:31] the money. Now I'm at a disadvantage, so

[17:33] I'm going to be under a lot of pressure

[17:34] to release my system to everybody also.

[17:37] And this is what we've seen time and

[17:38] time again. So,

[17:41] what what is almost certain to happen is

[17:43] that we're going to see, starting now,

[17:46] um many AI systems proliferating more

[17:49] and more that have incredibly powerful

[17:53] um hacking and other like cyber offense

[17:57] capabilities in more and more hands. So,

[18:00] what does that do to our whole digital

[18:02] infrastructure?

[18:03] We have no idea. Could the AI race

[18:06] become

[18:07] second Cold War between the United

[18:10] States and China? Well, I think it it

[18:13] That is the goal of some people.

[18:15] So, I I think we're um So, the

[18:19] the AI race turning into a geopolitical

[18:22] race and a geopolitical competition and

[18:25] a geopolitical conflict and potentially

[18:27] a war is something that worries me a

[18:28] lot.

[18:30] So, the right now we've got

[18:33] some healthy racing between countries

[18:35] and companies where they're just trying

[18:37] to build newer and and better and more

[18:40] innovative products. This is all good.

[18:42] Um we have also some unhealthy racing

[18:45] where

[18:47] companies are and where countries are

[18:49] feeling like

[18:50] the thing that is going to give us power

[18:53] in the in coming years is to be the best

[18:56] at AI. And if we're not ahead of

[18:58] everybody else in AI, we are not going

[19:00] to be the most powerful. We want to be

[19:02] the most powerful, so we need to race to

[19:04] AI as we raced for nuclear weapons, um

[19:08] as we raced for the you know, the most

[19:10] advanced technologies in war. Um

[19:14] So,

[19:16] if that is the dynamic that gets

[19:18] created, that and that we have a race

[19:21] for power in the form of very powerful

[19:24] AI systems, I think two things are going

[19:26] to happen from that. One is that the

[19:30] um the geopolitical side gets very, very

[19:33] scary because

[19:35] either

[19:37] the either this is true and by racing

[19:40] for superintelligence, you get huge

[19:42] amounts of power. You have the power to

[19:44] sort of overcome whatever your enemies

[19:46] are.

[19:47] Or it's not true because when you get

[19:49] the superintelligence, you actually

[19:50] can't control it. So, let's think about

[19:52] those two possibilities.

[19:54] In the first one, if you're

[19:57] you know, country A, the USA say, and

[19:59] you're developing this sort of super

[20:02] capability that makes you more powerful

[20:03] than all of the other all of the other

[20:05] countries, so that they can no longer,

[20:08] you know, have power in the world, they

[20:10] can no longer oppose you. You're seeing

[20:12] the US build this thing

[20:14] specifically saying that we're going to

[20:17] build something that disempowers you.

[20:19] That is more that makes us so vastly

[20:21] more powerful than you that you can no

[20:23] longer oppose us in any way. That is an

[20:24] existential threat to you if you're

[20:26] China or if you're Russia. If you're any

[20:28] if you're someone who has an adversarial

[20:30] interest to the US. So, what does a

[20:31] country with nuclear weapons do when

[20:33] it's under existential threat?

[20:35] Um the alternative is the US with the or

[20:40] or China builds AI systems that are

[20:42] superintelligent and that they can't

[20:44] control.

[20:45] Well, that's a threat to everybody.

[20:47] Do you think that the US and China could

[20:49] actually have a common interest?

[20:52] They certainly do have a common

[20:53] interest. The question is whether they

[20:54] can realize it and do something with

[20:56] that. So, the US and China both have an

[20:58] interest in not building

[21:01] a superintelligent thing that they can't

[21:02] control. Neither of them wants that and

[21:05] neither of them wants the other other

[21:06] party to do that. That is a shared

[21:08] interest. The problem is that right now

[21:10] they don't realize it. What they think

[21:12] is that they're going to build a super

[21:13] intelligent system that they do control

[21:15] and it's going to give them power.

[21:17] And so they're both racing especially

[21:19] the US um for that power not realizing

[21:23] that that power is probably illusory

[21:25] that the power is going to be the AI's

[21:27] power not the not its developers power.

[21:30] But if they both realize that this is a

[21:32] thing that is not going to be

[21:33] controllable and is incredibly

[21:35] dangerous, it's in both of their

[21:36] interest not to have that happen.

[21:38] They have a they have a common interest

[21:40] in

[21:42] um

[21:43] not having anybody develop AI systems

[21:45] that are intrinsically dangerous.

[21:48] Um and there are lots of those things

[21:50] that that we are currently doing. They

[21:52] have you know, different interests in

[21:54] terms of who builds their economy the

[21:56] fastest. Like the US would like to have

[21:57] its economy growing faster than China

[21:59] and vice versa and Europe would like to

[22:01] have its economy growing fast at all. Um

[22:04] so there like everybody has different

[22:06] interests but there are common interests

[22:08] in the common good. You know, the we

[22:10] shouldn't have nobody wants another

[22:12] pandemic that is caused by AI. Nobody

[22:14] wants a nuclear war that's caused by AI.

[22:16] Nobody wants our entire digital, s- you

[22:19] know, infrastructure to be dissolved

[22:22] because everybody has incredibly

[22:24] powerful powerful offensive cyber

[22:26] weapons. That's something that's

[22:27] happening right now. Um these are in

[22:29] nobody's interest. And so I think there

[22:32] is a lot of common interest in countries

[22:34] getting together and saying look,

[22:36] whatever our disagreements and whatever

[22:38] our competition,

[22:39] these are some things that better not

[22:40] happen because these are just a threat

[22:42] to everybody. Let's agree on those and

[22:44] write some agreements as to how those

[22:46] are going to not happen.

[22:47] Could superhuman AI become more

[22:50] dangerous for governments than ordinary

[22:53] people?

[22:54] So governments are used to being in

[22:56] charge. You know, most most people are

[22:57] not. Um so I think it is is to be

[23:00] particularly

[23:02] disempowering for governments if they

[23:04] realize that they're no longer in

[23:05] control. And I And I think

[23:07] this is a process that

[23:09] >> believe believe that governments still

[23:12] have any real control over AI

[23:15] development? They could. So I So but but

[23:19] it's waning. So I think the

[23:21] um

[23:22] the US government absolutely could

[23:24] constrain the AI companies from doing

[23:26] what they're doing if it chose to do so,

[23:28] right? All it has to do is pass a law

[23:30] that says, "You cannot do this thing."

[23:32] Like it's not that complicated. Um the

[23:35] reason it's hard to see that the the

[23:37] reason that is not happening is because

[23:41] A, you know, the government doesn't

[23:43] quite understand or quite believe what

[23:46] the risk and the downside is.

[23:48] And B,

[23:49] you know, the

[23:51] in the US, the government is subject to

[23:54] many influences, not only what is toward

[23:57] the best for the people,

[23:58] um but of course the what is best for

[24:01] the corporations that are paying huge

[24:02] amounts of money to the personnel in the

[24:04] government. So, you know, our

[24:07] government, as with many others, but

[24:08] it's maybe particularly bad in the US,

[24:10] is subject to intensive, you know,

[24:13] lobbying and political donations and all

[24:15] of this stuff where the and the

[24:17] companies are

[24:18] have vast resources and are absolutely

[24:20] putting huge amounts of resources into

[24:22] that. Tristan Harris said in his

[24:24] interview with Steven Bartlett from The

[24:26] Diary of a CEO podcast,

[24:29] "One AI leader would accept a 20% chance

[24:33] of human extinction in order to reach AI

[24:36] faster."

[24:37] "Doesn't that suggest that some of the

[24:39] people creating AI

[24:41] are acting like psychopaths?"

[24:45] If the calculation you're doing is on a

[24:47] purely personal basis, you know, I

[24:51] in creating the system have a 20% chance

[24:53] of dying along with everybody else.

[24:56] Um and there's an 80% chance of this

[24:59] amazing technology that not only grants

[25:01] me giant amounts of money and power, but

[25:03] maybe even extends my lifespan for, you

[25:06] know, or lets me live indefinitely or,

[25:09] you know, accrues all sorts of other

[25:11] personal benefits,

[25:13] then the 20% 80% may not seem so crazy,

[25:16] right? But, if the 20% chance of death

[25:19] is not just for you, but for everybody

[25:20] else, and a lot of the wealth and power

[25:23] accrues to the individual, but the

[25:25] extinction accrues to everybody, then

[25:27] that is a profoundly antisocial thing to

[25:29] do, yes? Now, I don't know

[25:33] um, you know, I I think some of these

[25:35] people

[25:36] um,

[25:37] the the sort of ethical system of these

[25:40] company leaders comes in all forms. Um,

[25:43] so I wouldn't want to necessarily say

[25:45] that they are psychopaths, but I would

[25:47] say that the

[25:49] that the AI systems that they're making

[25:51] are psychopaths.

[25:52] Um, in exactly the sense that these are

[25:55] systems that are

[25:57] trained to do to act in certain ways,

[26:01] but we have no idea what

[26:04] what is sort of underlying that. What

[26:06] are their actual goals underneath,

[26:08] right? So, the a a human psychopath

[26:11] might be, you know, might smile and tell

[26:14] you what you want to hear and be all

[26:15] friendly, um, and pretend to be your

[26:18] friend for a while, but underneath they

[26:20] simply don't care. Like, they're they're

[26:22] hiding their true goals.

[26:23] And AI systems absolutely can be like

[26:25] this. We have no We have the ability to

[26:29] sort of hit them with a stick enough

[26:31] that they behave and don't behave in

[26:33] certain ways. We have no ability really

[26:35] to

[26:36] understand and to to determine what

[26:39] their underlying goals and preferences,

[26:42] what they sort of um,

[26:45] deep down really want to have happen and

[26:47] not happen. Isn't it still possible to

[26:49] slow down or stop this race? What it's

[26:52] going to take, I think honestly, is

[26:55] probably

[26:56] a and and this is why I'm not super

[26:58] optimistic. I think it's going to take a

[27:00] combination of things. It's probably

[27:02] going to take widespread public pressure

[27:05] that we don't want the current path

[27:07] we're going down in AI. That pressure is

[27:09] building. That's happening.

[27:10] >> Millions of people are addicted. People

[27:13] can be addicted to things and still not

[27:14] want it. Um they they understand that it

[27:17] is not I think people using social media

[27:19] now understand that it's bad for them.

[27:22] They just sort of can't stop either

[27:24] because they're addicted or because this

[27:26] is the way that they actually connect

[27:28] with their social group and they sort of

[27:29] feel like they can't get off it. And

[27:31] this is obviously by design of the

[27:33] companies. So um but that doesn't mean

[27:36] so so there are now examples of cell

[27:38] phone free schools and the kids in these

[27:41] schools by all reports are like so

[27:43] relieved that we can spend some hours

[27:46] without the cell phone because everybody

[27:49] has to not have the cell phone. And so

[27:51] it's not So when you when you when

[27:53] there's a system where individually

[27:55] somebody has to give something up um

[27:59] it's very very much harder. When you can

[28:01] coordinate, when you can have a rule

[28:03] like at a school that you can't use your

[28:04] cell phone for these hours or in a

[28:07] country like you can't build an AI

[28:10] system that does this then everybody's

[28:12] on the same playing field. And so you

[28:14] can actually let go of that that

[28:15] technology. If you say um

[28:19] please everybody, you know, don't use AI

[28:21] for programming.

[28:23] That's going to accomplish nothing

[28:25] because if you're a programmer and you

[28:26] decide, oh, I'm just going to do it all

[28:27] myself. I'm going to take 10 times as

[28:29] long. You're just going to lose out to

[28:31] the person who is using the system. So

[28:33] you have to have This is why we have to

[28:35] have like why we have governments that

[28:38] set rules because the the competitive

[28:41] marketplace doesn't necessarily always

[28:43] lead to what is best for everybody's

[28:45] interests. I asked you about

[28:47] regulations. Is it dead end or the way?

[28:51] Absolutely, regulations are needed. So,

[28:53] what should be regulated first?

[28:55] >> I would say that there there several

[28:57] pieces of of regulation that we need. Um

[29:00] one is about liability, who's

[29:03] responsible for what AI systems do.

[29:06] Um right now, the the level the answer

[29:10] is at some level nobody. That's pretty

[29:12] bad.

[29:13] Um so, if if AI systems are out doing

[29:15] things and nobody is responsible for

[29:16] what they do, not so great. That kind of

[29:19] breaks the whole way our society works,

[29:21] where if somebody does something wrong

[29:22] and harms somebody else, they're

[29:24] responsible. Uh so, we need to very,

[29:26] very quickly clarify liability, who's

[29:29] responsible and and who pays the penalty

[29:30] when something goes wrong.

[29:32] We we need to have

[29:34] an

[29:35] testing and assurance framework, so that

[29:38] the AI systems that are developed and

[29:40] deployed, we can

[29:43] affirm that they will do some things and

[29:46] not other things.

[29:47] Um so, that they will be safe, for

[29:50] example. So, like

[29:51] uh

[29:52] the the obvious one, like so, there's

[29:55] some no-brainers, like if you put a drug

[29:58] out

[30:00] in the world and suddenly start giving

[30:01] it to 100 million people,

[30:03] that has had to go undergo rigorous

[30:06] testing to make sure that the effects of

[30:08] that drug outweigh the the benefits of

[30:11] that drug outweigh the costs.

[30:13] If you put a new AI system out that um

[30:18] that could potentially cause

[30:20] psychological damage to teenagers,

[30:23] spoiler alert, it does.

[30:25] The the could cause psychological damage

[30:27] to teenagers, nothing.

[30:29] No testing, no requirements, no

[30:33] uh

[30:33] randomly controlled trials, no

[30:35] monitoring, nothing. It just goes out to

[30:37] 100 million kids. So, this is crazy. So,

[30:40] that that's the that

[30:42] but

[30:43] on a wider level

[30:45] AI systems should be tested for all

[30:46] kinds of stuff. Like, if you're creating

[30:49] a AI system that is used in medical

[30:52] diagnosis, it should be tested to see

[30:54] like how good is it? What is its

[30:56] accuracy? Does it mislead you? Like,

[31:00] does it create the right

[31:02] chain of trust in making its medical

[31:04] diagnosis? Does it empower the doctor to

[31:07] override it? Like, there are all kinds

[31:08] of properties that we want a system to

[31:10] have. It should be tested for those

[31:11] properties. That doesn't necessarily

[31:13] have to be the government being like,

[31:15] "The AI has to do exactly this thing."

[31:17] But, there should be an ecosystem of

[31:18] testing just like there is with every

[31:20] other product. Like, other products that

[31:22] go to market get tested to do what they

[31:24] are supposed to do and so that they're

[31:26] safe. Why is AI system some weird

[31:28] exception to this? What would you say to

[31:30] people who think regulations would kill

[31:33] innovations? So, so what do they say

[31:36] about the the

[31:38] cars and the airplanes and the drugs and

[31:41] everything that we have? There's lots of

[31:42] innovation in all of these things. Maybe

[31:44] some people would like it to go a little

[31:45] bit faster. But, when I get in an

[31:47] airplane, I feel pretty good that that

[31:49] airplane is not going to crash.

[31:51] I feel pretty good that when I get in my

[31:52] car, it's not just going to blow up. And

[31:54] I feel pretty good that when I take

[31:55] medicine, it's probably not going to

[31:56] give like destroy my nervous system or

[32:00] like cause birth defects or whatever.

[32:03] Um none of these things are perfect and

[32:05] it probably is true that we could have

[32:06] some faster drugs and some sooner

[32:08] airplanes and some cheaper cars if we

[32:10] didn't have those regulations. But,

[32:12] there's a great appreciation that we

[32:14] have in our society for being able to

[32:15] trust the things that we buy and the

[32:17] things that we use. And I think that it

[32:19] is simply false to say that there's this

[32:22] giant trade-off between trust and

[32:24] innovation. I think people want

[32:27] trustworthy products and they want

[32:28] trustworthy AI systems. So, I I think

[32:31] we're actually at a at a giant trust

[32:33] deficit. I think regulation and sort of

[32:37] frameworks for assuring that certain

[32:40] properties are held by AI systems

[32:43] would actually greatly benefit their

[32:45] their utility in the market. Um the main

[32:48] reason that you can't use AI systems for

[32:51] lots of things now is you can't trust

[32:52] them, right? If I as a scientist,

[32:57] if I have an AI system that is like

[32:59] doing a bunch of math and I can't trust

[33:01] that the math is right, it's basically

[33:02] useless to me. Like I can use it a

[33:04] little bit, but if if all it does is

[33:06] create a bunch of math and then I have

[33:07] to laboriously check every bit of it,

[33:10] that's not so useful. If I have an AI

[33:12] system that is like a calculator, when I

[33:14] put two numbers into a calculator, I

[33:16] know that the number that comes out is

[33:19] going to be the right answer. Like I'm

[33:20] sure of that.

[33:22] Um

[33:24] that is really useful. If I'm if I have

[33:27] an AI system where I don't have

[33:29] confidence that it's doing the right

[33:31] thing, it's like maybe a little bit

[33:33] useful for certain things, but there

[33:34] it's ultimately I'm not going to be able

[33:36] to trust it when things get important

[33:38] and when they're high stakes. And many

[33:40] of the things that we have in society,

[33:42] medical financial like

[33:45] even just like whatever you use it for

[33:47] on a daily level in your job, if you

[33:48] can't trust it, it's not very useful.

[33:51] And and we're not going to be able to

[33:52] trust these systems until we have a

[33:54] system that is actually evaluating them

[33:57] against something that they're supposed

[33:59] to do and a system for how that works.

[34:01] >> Has AGI become a tool of the tech

[34:04] propaganda? There obviously is lots of

[34:06] propaganda out there on on all sides.

[34:09] Most of what we what passes for

[34:10] discourse now is propaganda from various

[34:13] directions.

[34:14] I I think that the term AGI

[34:18] and the the way that people are thinking

[34:21] about it in the corporations that are

[34:22] developing AI points to a specific goal.

[34:25] It points to a goal of making full human

[34:28] replacements, making things that can do,

[34:30] as in Open AI's definition, all of the

[34:33] economically valuable tasks that humans

[34:35] do. And this is this is sort of been

[34:37] something that is built into the

[34:39] thinking of AI developers since early

[34:42] times.

[34:43] Um so

[34:45] the there may be

[34:48] uh hype as to whether any, you know, how

[34:50] capable any particular AI system that

[34:53] they generate is, but the idea that they

[34:55] are trying to build AGI or super

[34:58] intelligence and that they are clearly

[35:00] making

[35:01] enormous progress toward that goal is

[35:03] not hype and it's not propaganda. That

[35:06] is real. Question is who gave a handful

[35:08] of companies to the right to decide the

[35:10] future of the human species, right?

[35:12] >> Nobody gave it to them. They took it

[35:13] upon themselves. And so, I think it is

[35:17] going to be up to everybody else to take

[35:19] some of that back if they want it.

[35:21] Should some AI labs

[35:24] um face legal or criminal liability for

[35:28] some AI futures?

[35:31] Well, certainly if somebody creates an

[35:33] AI system that through that

[35:38] uh through their negligence causes huge

[35:41] harm to people,

[35:42] there should be some consequence for

[35:43] that. You know, whether it's consequence

[35:46] to the company or whether it's

[35:47] consequence to the executive, I think

[35:48] that depends on what the situation is.

[35:50] So So there are

[35:52] um if you're

[35:54] you know, if there is something where

[35:56] you had cause to know that an AI system

[35:59] could harm someone and you neglected to

[36:03] take action on that, then that is

[36:05] negligence and I think that can rise to

[36:06] the criminal level. We've seen that in

[36:08] other other example. It's it's rare in

[36:10] corporate life to to see that it rises

[36:12] to the criminal level, but it sometimes

[36:14] does. Um more

[36:17] more importantly, I think is that right

[36:19] now the AI companies essentially take on

[36:21] zero liability for things. They when

[36:24] you, you know, fill in when you type

[36:26] that user agreement, like yes, I'm going

[36:28] into

[36:29] this

[36:30] something GPT, click the user agreement.

[36:33] Part of that user agreement is them

[36:34] saying, "Hey, we've got no liability for

[36:36] what happens when you use this system."

[36:38] Now, the courts sometimes find liability

[36:41] anyway, but the the corporations are

[36:42] quick to disavow any liability

[36:44] whatsoever for the the use of the

[36:46] systems and and what happens with them.

[36:49] Um so,

[36:50] absolutely there needs to be more

[36:52] liability than there is now. Like, when

[36:54] something goes wrong, it should be

[36:55] possible to

[36:56] um

[36:57] like lay the responsibility at the feet

[37:00] of who and not what, but who is

[37:02] responsible for that. So, until we do

[37:05] that, people are going to keep doing

[37:06] incredibly irresponsible things.

[37:08] >> You proposed assurance contract, right?

[37:11] For leaders like Demis Hassabis or Dario

[37:14] Amodei.

[37:16] I did.

[37:17] Um so,

[37:18] so there there there's this interesting

[37:21] thing that the company leaders will say

[37:24] privately and sometimes publicly that

[37:27] I'm scared. All this stuff is happening

[37:29] incredibly quickly. This

[37:31] This technology is incredibly powerful.

[37:34] We don't know how to make it safe. We're

[37:36] not sure we're going to be able to

[37:37] control it.

[37:38] I would love to slow down if I could if

[37:40] I could. And

[37:42] Dario has said this. Sam Altman has said

[37:44] it in certain ways. Uh

[37:48] Demis Hassabis has said it. Elon Musk

[37:50] has said it. Um

[37:53] but they all both say and I think um

[37:57] express

[37:58] and and sort of feel that

[38:01] they can't stop individually because if

[38:04] I'm Dario and I stop, that's not going

[38:06] to stop Sam, it's not going to stop

[38:08] Demis, it's not going to stop Elon.

[38:09] They're going to keep going. And so, all

[38:11] I'm doing is sort of taking myself out

[38:13] of the out of the race.

[38:14] Every one of them says that and and

[38:17] feels that.

[38:18] But

[38:20] these are smart guys.

[38:22] When they want to make deals, they can

[38:23] make deals. If you can get, you know,

[38:26] the Saudis to sub- to to subsidize

[38:29] through SoftBank a and President Trump a

[38:33] half a billion dollars to build a giant

[38:34] data center in the Middle East, like

[38:36] these are complicated deals. They can

[38:38] probably work out a deal with each other

[38:40] in how they can cooperate to not race

[38:43] towards superintelligence. So, we we

[38:45] proposed a mechanism, which is

[38:48] what's called an assurance contract,

[38:50] which says

[38:51] "I will stop racing if everybody else

[38:54] agrees to stop racing." So, let let's

[38:56] sign something that says, "If

[38:59] all five or six or seven or eight or

[39:01] nine or 10 or whatever many AI companies

[39:03] sign the same contract that says, "We're

[39:05] going to stop racing if everybody else

[39:07] does." Then until everybody signs, we

[39:09] can keep racing.

[39:10] But if everybody signs, then we've

[39:13] agreed to stop.

[39:15] This is called an assurance contract and

[39:18] um

[39:19] the the beauty of this is that nobody

[39:21] has to stop by themselves, right? They

[39:23] only stop all together. And the you

[39:27] know,

[39:28] we have created lots of ways to

[39:30] coordinate with each other as human

[39:31] beings, and I I really think if we put

[39:33] our mind to this one, we could do it.

[39:36] Smart guys, it doesn't come into effect

[39:38] until, you know, enough people have

[39:40] signed on. Maybe that's everybody.

[39:43] Um maybe it's, you know, five out of

[39:46] seven or something. Um but the

[39:49] the

[39:50] the

[39:52] game theory of it changes because if you

[39:55] just say, "I'm going to stop

[39:56] individually and nobody else is

[39:58] stopping," then you lose. If you say,

[40:00] "I'm going to stop when everybody else

[40:03] stops," that's a deal that makes sense

[40:05] for everybody to sign if they if they

[40:07] would really like to stop. It makes

[40:09] sense for everybody to sign one by one

[40:11] individually because nobody's at a

[40:13] disadvantage until the contract actually

[40:15] kicks in, and then nobody's at a

[40:17] disadvantage because they're all in the

[40:19] same place. They've all agreed to stop,

[40:21] and they're all operating under the same

[40:23] rules. So, at no point is anyone at a

[40:25] disadvantage. Can you explain to our

[40:29] viewers your better path

[40:32] framework? So, the first thing to

[40:34] understanding a better path is

[40:36] understanding the path that we're on.

[40:38] And there's an idea that AI development

[40:41] is just one thing. That there's the AI

[40:43] that we have now, which is sort of

[40:45] these language models that are quite

[40:47] general and good at some things and not

[40:48] good at others. And that the the path

[40:51] forward is to make them just more and

[40:52] more powerful until they can replace

[40:55] humans at doing all of the things that

[40:56] we do, and then after that

[40:58] superintelligence. That's the path that

[41:00] the companies are laying out, and that's

[41:02] what they're pursuing.

[41:03] And it's specifically designed to do the

[41:05] whole thing that humans do and you know,

[41:07] slot the human out and slot the AI

[41:10] system in instead.

[41:12] I think this is a bad goal. You know,

[41:14] the the the goal goal of developing AI

[41:16] systems specifically to simulate and

[41:18] replace humans

[41:19] just isn't a good goal. It's a it's a

[41:21] goal that you can understand from the

[41:22] economic perspective because there's a a

[41:25] big prize if you can

[41:26] replace human labor with AI labor and

[41:29] take the money for yourself, that's a

[41:30] giant amount of money. So, we can

[41:31] understand why the companies are doing

[41:33] it and why they're selling that to

[41:34] investors. But it isn't a good goal for

[41:36] humanity, right? Nobody wants to be

[41:38] replaced with an AI system, basically.

[41:41] Um so, so we have to ask, is that the

[41:44] only path that we can be on? Is it the

[41:45] only way to get the fruits of this

[41:48] powerful technology is to build human

[41:50] replacements? And I think the answer is

[41:51] no.

[41:52] Um So, what is the alternative?

[41:55] >> something to stop that race. The first

[41:57] thing you have to do is say,

[42:00] this is a bad path that we're on. Like,

[42:02] we we we should not be building human

[42:04] replacements. We should not be building

[42:05] uncontrollable AI systems. We should not

[42:08] be racing towards superintelligent

[42:09] systems that we're not going to be able

[42:11] to control and that will replace humans

[42:13] and then humanity. This is a crazy thing

[42:14] to be doing. And um it's it's kind of

[42:17] astonishing that we've allowed the

[42:19] companies to get away with even having

[42:21] this as a goal. Like why why is it okay

[42:24] to even say we're building smarter than

[42:26] human AI systems cuz we want to replace

[42:28] everybody in their jobs and then like

[42:30] give over the power of humanity to the

[42:32] AI systems instead. This is ludicrous,

[42:35] but this is where we are. So the first

[42:36] thing is to say that's not okay. That is

[42:38] not a reasonable goal. We are not going

[42:40] to do that. We're going to do something

[42:41] else in AI development. And we and and

[42:44] we'll put in the sort of regulations and

[42:47] governance structures that say you can

[42:50] do this and not this, where the second

[42:53] thing is the runaway to super

[42:54] intelligence.

[42:55] So then the question is what do we do

[42:57] instead? Right? How can we use the power

[43:00] of AI to actually solve the problems

[43:02] that we want to solve? And there I think

[43:03] the news is good because there are lots

[43:05] of ways that AI can be used that isn't a

[43:08] human replacement. We've got 100,000

[43:11] years of experience with what it means

[43:13] to build tools that allow humans to do

[43:15] things better

[43:17] and do things they couldn't do before.

[43:19] Rather than to you know, we don't use

[43:21] our our hands to move giant things much

[43:23] anymore. We use powerful machines. We

[43:25] don't

[43:26] calculate you know, seven-digit number

[43:28] multiplication in our head anymore or

[43:30] even on paper. We have machines to do it

[43:32] for us. So we've built technologies in

[43:34] our service. Things that help people do

[43:37] what they want better and faster and

[43:41] do things that they couldn't do before.

[43:43] And we can build AI tools. So the the

[43:45] first thing to think of is how do we

[43:48] build AI tools that empower people to do

[43:52] things they otherwise couldn't do,

[43:53] rather than AI replacements that we slot

[43:57] in instead of people.

[43:58] Once you that that's the very first

[44:00] thing if you just shift your thinking in

[44:02] that way, it leads to a lot of other

[44:04] implications for how we should develop

[44:05] AI.

[44:06] Was this paper driven more by hope or

[44:11] fear?

[44:12] Obviously both.

[44:14] Uh I think the the fear is the default

[44:17] path that we're on.

[44:19] Uh

[44:20] is the one that we will continue on and

[44:22] all of the, you know,

[44:24] clearly foreseeable consequences of that

[44:26] path will

[44:28] come to be manifest. The hope is that it

[44:31] doesn't have to be that way. So, there's

[44:32] this feeling that people have of

[44:34] inevitability.

[44:36] Like technologies just inevitably roll

[44:38] along and, you know,

[44:40] you know, advance themselves. This is

[44:42] not true. So, the the development of

[44:44] these

[44:45] powerful AI systems, the race towards

[44:48] superintelligence, is costs trillions of

[44:51] dollars and and some of the smartest

[44:52] people on Earth like all of their time

[44:55] and mental effort trying to make this

[44:56] thing happen. It is not happening by

[44:58] itself. It is happening because people

[45:01] are choosing to devote all of those

[45:03] resources to that path. And when

[45:05] something is incredibly hard to do and

[45:06] requires huge effort and huge expense,

[45:09] it is not inevitable. You can you can

[45:11] direct those uh or you can either just

[45:14] stop doing that or you can direct all of

[45:16] that effort towards something else.

[45:18] And so, the the optimistic side, the

[45:21] hope is that this is not in fact

[45:24] inevitable, that despite the incentives

[45:27] that are driving the the development in

[45:30] in that direction, that the strong

[45:33] preference that the rest of us have,

[45:35] that the other sort of 8 billion people

[45:38] on Earth have to

[45:40] have technologies that serve them and

[45:42] that don't replace them and that

[45:43] actually solve their problems, that that

[45:46] pressure can redirect course. So, I I

[45:48] think the

[45:49] you know, there there's a there's a

[45:51] depressing side, which is that the

[45:53] biggest companies on Earth, supported

[45:55] very, very directly by the US

[45:57] government, are doing this with huge

[45:59] financial incentive, right? That's a

[46:01] powerful set of forces. On the other

[46:03] hand, if everyone that is threatened by

[46:05] this

[46:06] at all gets together and pushes back,

[46:09] that's everyone else in the world

[46:10] literally.

[46:11] Um and that is also very powerful. So,

[46:13] that that gives me the the hope in the

[46:16] optimistic side. Um and what I really

[46:18] wanted to do here is to say

[46:21] the the goal is not just just to stop AI

[46:24] progress or stop something. The goal is

[46:27] to say, what do we actually want

[46:28] instead? What what How can we switch

[46:30] tracks instead of saying trying to stop

[46:32] the train? Did you write this paper

[46:35] more as a scientist or citizen worried

[46:39] about the future of humans? Well, both.

[46:42] I mean, I I think the the concern in

[46:45] particular for what happens with

[46:47] superintelligence, I think comes

[46:50] well, with many things, comes out of the

[46:51] scientific side. So, there's a

[46:52] prediction aspect of saying, here's what

[46:54] we're doing. Here are the almost

[46:57] inevitable consequences from the game

[46:59] theory, from the physics, from the

[47:01] information theory, of you know, of what

[47:04] is going to happen if we go down that

[47:06] path. So, there's a predictive element

[47:07] that says,

[47:09] this is not a good direction to be

[47:10] going. Then there's a there's the

[47:13] preference side and the sort of as a

[47:15] concerned citizen, as a human, you know,

[47:17] what do we do instead? And that has to

[47:20] do with both a technical piece, like how

[47:23] technically can we avoid going down this

[47:25] path? How can we choose to stop doing

[47:27] this and do the other thing? Um and

[47:29] there's also just a a human piece of

[47:32] what is it that we actually want as

[47:34] humans? Like, what do we want from our

[47:35] technologies? What does it mean to have

[47:37] a good future for humanity? Those are

[47:40] questions that are not just my

[47:42] questions, they should be everybody's

[47:43] questions and and I see it as my goal to

[47:46] to sort of understand what you know, to

[47:49] some degree that I can, what people in

[47:51] general want and what we can put

[47:54] together as a different technological

[47:57] trajectory to go more for what is going

[47:59] to benefit the most people. I know some

[48:01] AI leaders who treat AGI almost like a

[48:05] religion. I think there are very

[48:06] religious aspects to the way people are

[48:09] thinking about it because

[48:11] um

[48:12] you know

[48:13] what is the goal here? To to build

[48:15] something that is all knowing and all

[48:17] powerful? Like how

[48:19] how is that not a religion? Um

[48:22] and that there's a fundamental

[48:25] sort of diminishment of humanity. Like

[48:28] you know, humans are very flawed things,

[48:30] right? We are

[48:32] dumb and make mistakes and get angry at

[48:34] each other and don't always cooperate in

[48:36] everything. You know, lots of flaws, but

[48:38] somehow we have gone from, you know,

[48:41] scrabbling around hunting and gathering

[48:43] to

[48:44] a world-spanning civilization that has

[48:47] these amazing technologies that can

[48:49] travel to to other planets occasionally,

[48:52] um that has computation systems that has

[48:56] unbelievably low levels of violence

[48:58] compared to what we had, you know,

[49:00] hundreds or thousands of years ago. Um

[49:03] even if it still comes out sometimes. We

[49:06] have a, you know, a standard of living

[49:08] that puts, you know, the typical Western

[49:11] person more materially wealthy than

[49:14] kings were, you know, a thousand years

[49:16] ago. So, we've managed all of these

[49:19] things and I think there's so much more

[49:21] that humanity can manage if we allow

[49:24] ourselves, if we give ourselves the

[49:26] right tools, if we have the, you know,

[49:28] continue our social advancement,

[49:30] continue the advancement of our of our

[49:31] civilization and our ideals. But

[49:34] instead, it feels like there's this

[49:36] ideology of the like, yeah, humans

[49:38] aren't that great. So, let's let's make

[49:40] something better. Let's make these AI

[49:42] systems that are smarter than us and

[49:45] somehow wiser than us and more knowing

[49:47] than us and basically hand over the

[49:49] reins of civilization to them and

[49:51] they'll they'll make everything better

[49:52] for us.

[49:53] And I think this is

[49:55] probably misguided. You know, I It's not

[49:57] impossible that this could turn out that

[50:00] they could in fact be smarter and wiser

[50:01] than us and things go well.

[50:04] But I think it's it's both unlikely and

[50:07] I think it's sort of a betrayal of

[50:09] humanity. Like

[50:10] we

[50:11] we have done incredibly well. We could

[50:14] do incredible things. Um

[50:17] we're not necessarily going to do like

[50:19] we're

[50:20] There are lots of things that are pretty

[50:21] off with the way that humanity is

[50:22] operating right now, but those are

[50:24] problems that we could solve.

[50:25] Um and so I would rather see

[50:29] you know, these trillions of dollars and

[50:31] all these incredible minds going into

[50:34] how can we actually

[50:36] bring like make humanity the best it can

[50:38] possibly be?

[50:40] How can we do that with AI? Like are

[50:42] there ways that AI can help us

[50:44] you know, know things better, like do

[50:47] find truth better, work with each other

[50:49] better, cooperate with each other

[50:50] better, build better institutions, you

[50:53] know, how can the AI tools actually

[50:55] uplift humanity rather than how do we

[50:58] figure out how to replace our judgment,

[51:01] our thinking with machine systems and

[51:03] then let them do all the work for us?

[51:06] Like this just seems like an anti-human

[51:07] goal and I'm pro-human. Moving away

[51:10] strictly to your paper, you proposed

[51:14] new AGI definition, right? The way that

[51:17] I like to think about AGI as autonomous

[51:19] general intelligence um

[51:21] the

[51:23] positive of you know, one of the nice

[51:25] things about that framework of thinking

[51:26] it as a sort of combination of those

[51:29] three terms of autonomy, generality, and

[51:31] intelligence is that you can easily see

[51:33] well

[51:34] we can just build AI systems that don't

[51:37] combine all three of those things and

[51:38] then it's a different thing. So there

[51:40] isn't one path which is like

[51:44] uh

[51:45] not quite AGI and then almost AGI and

[51:47] then AGI, and then more than AGI. Like,

[51:50] that is not the only path we can choose.

[51:52] We can choose all sorts of combinations

[51:54] of AI capabilities to do the things that

[51:57] we want. So, we can have, for example,

[51:59] um very powerful special-purpose AI

[52:02] tools that do things like scientific

[52:05] discovery, um that do things like

[52:08] powerful mathematics, that do even

[52:11] programming. You know, we have

[52:12] programming tools already.

[52:14] Um but, it doesn't mean that all of

[52:15] those things have to have a whole lot of

[52:18] autonomy or agency to do the things that

[52:20] they want. There's There's no reason, if

[52:23] you're

[52:24] um doing

[52:25] biology research and doing protein

[52:27] folding, that that system has to be able

[52:28] to also drive a car, or that your your

[52:31] autonomous vehicle needs to be able to

[52:32] do philosophy, or that the thing that

[52:34] you're using to solve Einstein's

[52:35] equations needs to flirt with your

[52:37] children. Like, it's There's no reason

[52:39] to combine all of these into one giant

[52:41] system. That's not what we do with any

[52:42] other information technology, right?

[52:45] We've got our phone full of apps, and

[52:46] each one of them does something.

[52:48] Instead, AI is like, "This is the

[52:50] do-everything app. Click here, and it

[52:51] just does all the things." Like, that's

[52:54] a weird direction to take software

[52:55] development, um and there's no reason to

[52:58] do it in that direction. So, So, the AGI

[53:01] framework is useful for saying there

[53:02] isn't just like the spectrum from not

[53:05] AGI to AGI to post-AGI, but rather there

[53:08] are lots of different directions that we

[53:09] can go. And, once we start thinking

[53:12] about what do we want the system to

[53:14] actually do? Like, what is the problem

[53:16] that we're trying to solve here? Then,

[53:18] it starts to feel very different.

[53:20] >> Is that really what we need from AI,

[53:22] right? Right. So, we can build AI

[53:26] agents, where the AI systems have the

[53:30] goals and the plans and sort of make

[53:32] things happen, and and their whatever

[53:35] they're trying to get done is the thing

[53:37] that happens in the world. Or, we can

[53:39] try to enhance human agency, where we

[53:42] decide what want what we want to have

[53:44] happen, and we have tools that help us

[53:46] do that. That's what we've done with all

[53:48] our previous technology.

[53:50] Now,

[53:52] the the way that AI is being developed

[53:54] now, people have noticed that it's

[53:56] mostly a tool, and they really really

[53:58] want it to be an agent, because there's

[54:00] more profit to be made in agents. They

[54:03] can replace more things. They're also

[54:06] just They're things that um

[54:09] agents can do that that

[54:11] tools can't, cuz the agents can do it

[54:14] all by themselves without human

[54:15] intervention. So, like if you if you

[54:17] want to um

[54:21] write a big program, there are you can

[54:24] there are some different paths you can

[54:26] go down. One is something that you're

[54:28] the programmer, and the AI system sort

[54:30] of helps you out. Or the other one is

[54:33] the AI system is the programmer, and you

[54:34] tell it sort of what to program, and it

[54:36] goes away and does a bunch of

[54:37] programming.

[54:38] And this is what people are leaning into

[54:41] a lot now. They want the AI system to do

[54:43] all of the stuff itself, and then just

[54:45] report back, yes, I'm done, here's the

[54:47] the product. Um and that can be very

[54:50] useful in certain ways, but when it the

[54:53] more we put the agency into the AI

[54:57] systems, the less of it sits in us.

[55:00] Right? So, the more the AI system is

[55:02] making decisions, the less we're making

[55:04] decisions.

[55:05] Um

[55:06] and so, I think we have to be very

[55:07] careful as we build uh

[55:10] systems that are very autonomous, they

[55:12] can do all of these things on their own,

[55:14] and have all the other human

[55:17] capabilities, the intelligence and the

[55:18] generality, what is the purpose of us

[55:21] anymore, right, if the system does it

[55:23] all? Now, if you build something that's

[55:24] autonomous, but isn't that smart, um

[55:28] or isn't that general, then it's

[55:30] something that we can still control.

[55:31] Like autonomous vehicles, they're not

[55:33] going to take over the world. They might

[55:34] put some Uber drivers out of work, and

[55:37] that is like an economic displacement

[55:39] problem that we have to think about.

[55:41] But, there isn't some giant risk to

[55:43] society from things that are just

[55:44] autonomous. It's that triple combination

[55:46] that's the problem. Is autonomous AI

[55:48] more dangerous than highly intelligent

[55:51] AI?

[55:52] I would say so.

[55:54] Um so, it depends how exactly So, a lot

[55:57] of people think of his intelligence as

[56:00] um

[56:01] you know, lump all these things

[56:02] together, which is why I think it's it's

[56:04] useful to separate them. But, if you

[56:06] have a system that is highly

[56:07] intelligent like

[56:09] um like, you know, a calculator is in

[56:11] some sense very intelligent in that

[56:13] the particular thing that it does, it

[56:16] does extraordinarily well and accurately

[56:18] and fast. So, in that sense, a

[56:20] calculator is very intelligent, but

[56:22] absolutely non-threatening, right? Now,

[56:25] when you think of something a little bit

[56:28] more general than a calculator,

[56:30] then

[56:31] um it starts to to exhibit more of those

[56:34] potentially dangerous qualities. So, if

[56:36] you have an AI system that can design

[56:39] new viruses, right? Um it can be very

[56:42] specific and only designs new viruses.

[56:45] It can be not that autonomous in that it

[56:47] really requires a human oversight, you

[56:50] know, to tell it to design a virus. But,

[56:52] it can still be pretty dangerous,

[56:53] because if you've got a very powerful

[56:55] tool in the wrong hands, that is a very

[56:58] dangerous thing. Um but, I think it's

[57:00] important to separate the kind of danger

[57:02] that we've always had. We've always had

[57:04] powerful tools. We've always had bad

[57:06] people. And we've

[57:08] It it's a problem to figure out how to

[57:09] manage the bad people having powerful

[57:11] tools. But, it's one we've been with for

[57:13] a long time.

[57:14] Building something that is fundamentally

[57:16] a sort of being, rather than a tool, an

[57:19] agent, rather than a tool, that has its

[57:22] own goals and purposes, that is

[57:24] something we don't have a lot of

[57:24] experience with. We have sort of some

[57:27] uncontrollable systems, and we've got

[57:29] like animals and so on, but um and we're

[57:32] used to not controlling those

[57:33] necessarily, but they're also only so

[57:36] powerful. All right, if you're you know,

[57:38] if you're in the

[57:39] in the cage with a lion, it's very

[57:40] scary, but otherwise lions are just not

[57:42] that big of a threat to humanity because

[57:43] they're just not that powerful compared

[57:45] to us. But if we absolutely had to

[57:47] control, you know,

[57:49] lions

[57:51] because they you know, if lions had

[57:53] nuclear weapons, that would be a

[57:55] different question. Like we don't know

[57:56] how really how to get lions exactly to

[57:58] do what we want. Even the most trained

[57:59] lions can like turn on their

[58:02] trainers. Um so I think it is a very

[58:05] different thing to build AI systems that

[58:08] are autonomous agents rather than tools.

[58:11] And as we make very powerful capable

[58:14] autonomous agents, we have to ask

[58:17] ourselves,

[58:18] A, is that the right direction? And if

[58:20] B, if we do it, are we going to keep

[58:22] control of those things? What does your

[58:24] Metaculus platform currently predict

[58:28] about the timeline for AGI?

[58:32] Yeah, interestingly it's it's

[58:33] Metaculus's prediction has stayed pretty

[58:36] constant over the last several years

[58:38] that

[58:39] that we will have So there are different

[58:41] definitions of of AGI and

[58:44] it's very contentious what is the right

[58:47] thing to predict.

[58:49] But according to the way that the

[58:50] prediction is made on Metaculus, it's

[58:52] sort of in the next few years

[58:55] that we're likely to have it by this in

[58:57] the sort of weak form and maybe a couple

[58:59] of years after that in a very strong

[59:00] form. So Maybe AGI is an ideology more

[59:06] than technology, right? It

[59:08] I think it is in some sense an ideology

[59:10] and but that ideology is driving a

[59:12] technology. So it So I think

[59:15] um it didn't have to be this way. Like I

[59:17] I do think that

[59:19] um

[59:20] there's a there's another universe very

[59:22] much like ours in which people did not

[59:25] get attached to the idea of AGI and

[59:28] let's replicate all of the things that

[59:30] humans do and and sort of race to build

[59:33] something that is super intelligent, but

[59:34] they just said, "Oh, look, we can do

[59:37] this thing called machine learning,

[59:39] where instead of having to explicitly

[59:41] program exactly what the system does, we

[59:43] instead

[59:45] figure out the goal and the system

[59:47] learns how to do that thing. You know,

[59:48] it's just a different way of writing a

[59:50] program to do machine learning instead

[59:51] of

[59:52] explicitly programming all the steps.

[59:54] And let's let's create really, you know,

[59:57] strong machine learning systems and let,

[59:59] you know, have them be tools that do

[1:00:02] stuff that was very, very hard to

[1:00:04] program explicitly, like protein

[1:00:05] folding. We never figured out how to do

[1:00:06] that, but look, the machine learning

[1:00:07] thing can do it." from DeepMind. Um

[1:00:11] There's a I think there's a world in

[1:00:13] which that was the the way people

[1:00:15] thought about AI and that we built lots

[1:00:19] of AI tools and, you know, there were

[1:00:22] some people out on the side saying like,

[1:00:25] "Hey, we should build AI that does

[1:00:27] everything instead of humans." And the

[1:00:28] people were like, "Nah, yeah, whatever."

[1:00:30] In the same way that right now there are

[1:00:32] off people saying like, "Oh, we should

[1:00:33] do genetic engineering to make ourselves

[1:00:36] to make like superhumans, like

[1:00:38] superhumans physically, superhumans

[1:00:39] mentally. We should build like the

[1:00:42] Uberman with genetic engineering." Like

[1:00:44] those people are out there, but

[1:00:45] everybody pays them no attention cuz

[1:00:46] like we don't want that. We don't want

[1:00:48] to go we don't want to go build

[1:00:50] superhuman with genetic engineering. So,

[1:00:53] we could have a world in which that was

[1:00:55] the way that people look at AGI and

[1:00:56] superintelligence.

[1:00:58] We're not in that world.

[1:00:59] Do you see any signs of an AI bubble? An

[1:01:02] AI bubble? It's an interesting question

[1:01:03] because I think there are there are

[1:01:05] people who think we're in an AI bubble

[1:01:08] because

[1:01:10] AI is basically It's all hype

[1:01:13] and the AI systems really aren't capable

[1:01:15] of being that powerful and so on.

[1:01:17] This is not true. It's something Um

[1:01:20] There is, I think, a potential that

[1:01:22] there is a bubble

[1:01:23] uh because of trust. So,

[1:01:26] the the way that AI systems will make

[1:01:29] huge the the huge amounts of money that

[1:01:31] they have been promising to their

[1:01:32] investors is if they can substitute for

[1:01:35] huge amounts of human labor.

[1:01:37] Now, there are various barriers to

[1:01:39] substituting for huge amounts of human

[1:01:40] labor. One of them is sort of raw

[1:01:42] capability. They have to be smart enough

[1:01:43] to do all the things. Um another one is

[1:01:47] sort of social barriers and regulatory

[1:01:49] barriers and things. And a third one is

[1:01:51] trust. So, if I'm going to replace my

[1:01:54] human workers with AI systems, those I

[1:01:57] got to be able to trust that those AI

[1:01:59] systems are actually going to do the

[1:02:00] right thing, and they're going to, you

[1:02:02] know, follow the the rules and the laws,

[1:02:04] they're not going to get me in trouble,

[1:02:05] and so on. And I think there that it's

[1:02:09] possible that because we're designing

[1:02:11] these AI systems in a way that is so

[1:02:14] focused on sort of raw capability,

[1:02:16] getting the highest numbers on those on

[1:02:19] those capability benchmarks, and not

[1:02:21] focused on building things you can

[1:02:23] actually rely on, that are actually

[1:02:25] reliable trustworthy secure safe

[1:02:28] that there's going to be a big barrier

[1:02:30] between when the systems, in principle,

[1:02:33] are like capable of doing all these

[1:02:35] things, and when they can actually

[1:02:37] be used in the economy. And if that gap

[1:02:40] it gets big enough, that trust gap gets

[1:02:42] big enough, then the money right might

[1:02:43] run out. So, if you're burning through

[1:02:45] huge amounts of money, which many of

[1:02:47] these companies are, and there's a long

[1:02:49] delay between when you have the

[1:02:51] capability and then when you can

[1:02:52] actually make huge amounts of money on

[1:02:53] it, you might run out of money. That's

[1:02:55] not because the technology is

[1:02:57] but it is because you've chosen the

[1:02:59] wrong direction to develop it in in a

[1:03:01] way that is not actually making what

[1:03:02] people need.

[1:03:04] >> I remember my interview with John

[1:03:07] Hopfield. His work

[1:03:10] was recognized in 2024 by Nobel

[1:03:13] Committee work on AI. Uh I ask him, "Do

[1:03:17] you regret something? If you could turn

[1:03:19] back time, would you do this?"

[1:03:22] I remember his answer.

[1:03:25] Um

[1:03:26] Just one thing. That neural network

[1:03:29] development

[1:03:31] took place within computer sciences

[1:03:36] rather than biological sciences. Hm.

[1:03:40] For decades, computer sciences, not

[1:03:43] biological sciences, have looked at how

[1:03:47] the human brain works.

[1:03:49] Where do you stand on this?

[1:03:52] Well, I think there's

[1:03:53] there there's some interesting

[1:03:54] scientific questions there, and then

[1:03:56] there are some social questions, too.

[1:03:58] So, I think there's

[1:03:59] there's an interesting

[1:04:01] sort of fact that when we create

[1:04:04] biological systems, if we create a new

[1:04:06] drug or a you know

[1:04:08] a new medical procedure or something, um

[1:04:11] those are very, very tightly regulated

[1:04:13] fields. So, in in biology and in

[1:04:16] medicine and things that have to do with

[1:04:17] living beings, we have a very different

[1:04:19] attitude cuz we sort of understand that

[1:04:21] we're tinkering with something that's

[1:04:23] sort of almost sacred. Like, there's a

[1:04:25] there's a real respect for biological

[1:04:27] systems, and there's a real obvious

[1:04:29] sense in which when you screw something

[1:04:30] up, you're doing real harm to something

[1:04:33] that feels it.

[1:04:35] And so, we we've got a very

[1:04:36] sophisticated regulatory system and

[1:04:39] systems of ethics and and so on when it

[1:04:41] comes to biological systems.

[1:04:44] All of that is largely absent when we go

[1:04:45] to computational systems, where we have

[1:04:47] this history of you just kind of throw

[1:04:50] stuff together. You It's just software.

[1:04:52] It doesn't really affect anybody,

[1:04:53] exactly.

[1:04:55] Um there's basically no regulation.

[1:04:57] It's a free-for-all. You just throw it

[1:04:59] up there, and like it's kind of anything

[1:05:00] goes.

[1:05:01] And the fact that AI, this this

[1:05:04] incredibly powerful new technology that

[1:05:06] is an

[1:05:07] uh

[1:05:08] whose goal is to, you know, replace

[1:05:12] human thinking and replace humans as

[1:05:14] decision makers and agents and so on,

[1:05:17] came out of the computer the totally

[1:05:20] unregulated anything goes software

[1:05:21] industry rather than the the biology

[1:05:24] side, um means it is being treated very

[1:05:27] very differently, right? Like you know,

[1:05:29] as I've as I said before, if you propose

[1:05:33] I want to

[1:05:34] um

[1:05:35] use genetic engineering to make a set of

[1:05:38] people that are super duper smart

[1:05:41] because that's really valuable. If we

[1:05:42] have super duper smart people, we can

[1:05:44] cure cancer, we can be more productive,

[1:05:46] we we can do all of these things. So,

[1:05:48] I'm going to go do genetic engineering

[1:05:50] to make super duper smart people.

[1:05:52] The regulatory agencies are going to say

[1:05:54] like, "No, you are not." The ethical

[1:05:56] boards are going to say, "No, you are

[1:05:57] not." The general population is going to

[1:05:59] say, "No, you are not going to do that."

[1:06:01] At least not with like a lot more

[1:06:02] conversation. This is not just like a

[1:06:04] normal thing to do to go build

[1:06:05] superhuman brains, even if they're

[1:06:08] useful. Even if they're economically

[1:06:09] valuable. Even if they could cure

[1:06:11] cancer. We just don't do that.

[1:06:12] >> end for our race, right?

[1:06:14] >> Well,

[1:06:15] the I mean, there are different ways

[1:06:17] like the

[1:06:19] we just treat it very differently. Like

[1:06:20] whether it's whether that's

[1:06:22] good or not, like it's very very

[1:06:24] different, right? Like we would not be

[1:06:26] having this conversation in the same way

[1:06:29] um if it was with with human brains.

[1:06:31] Again, there's a there's maybe another

[1:06:32] universe where all these giant companies

[1:06:35] are racing to build the most superhuman

[1:06:37] humans with genetic engineering. That's

[1:06:39] not the universe we're in, but there

[1:06:40] probably is one out there where there's

[1:06:42] like 10 giant companies that are racing

[1:06:45] to build the most superhuman people with

[1:06:47] their genetic engineering.

[1:06:48] >> Upgraded people that are like super

[1:06:50] intelligent people because they and

[1:06:52] they're all saying like, "We're going to

[1:06:54] cure cancer before the other before the

[1:06:56] other guys do with our superhumans." Um

[1:06:59] that's just a very different universe

[1:07:00] than the one we're in. So, why we treat

[1:07:01] one one way in our universe and the

[1:07:04] other totally different way or vice

[1:07:05] versa in the other universe? I don't

[1:07:07] know. It's a complicated question. Um

[1:07:09] but it is a very double standard and it

[1:07:11] is interesting to imagine um how things

[1:07:14] would be different if AI were developed,

[1:07:18] you know, more like biological systems

[1:07:19] and less like uh apps.

[1:07:23] Thank you very much for your time.