[0:00] kind of warning that people
[0:01] unfortunately tend to respond [music] to
[0:03] is when people die. That is coming. If
[0:05] we don't want to be replaced, there is
[0:06] going to be a battle. The warning signs
[0:08] are blinking everywhere. AI systems that
[0:10] they're making are psychopaths. What is
[0:12] sort of underlying that? What are their
[0:13] actual goals underneath? There obviously
[0:15] is lots of propaganda out there, but the
[0:17] idea that they are trying to build AGI
[0:20] or superintelligence is not hype. And
[0:23] it's not propaganda. That is real. I
[0:25] think we've even seen the first
[0:27] exhibition of it recently. Who would be
[0:29] first to disappear?
[0:31] >> goal is for it to be everybody.
[0:35] Is the race for AGI a suicide race?
[0:40] I think we all feel that it is.
[0:42] >> [laughter]
[0:42] >> I mean I mean we're um
[0:46] you know, we
[0:47] there there's all this sort of
[0:49] highfalutin discourse
[0:52] um
[0:52] you know, on Twitter and like in the AI
[0:57] expert circles and with the company
[0:59] leads and the AI safety experts
[1:01] um saying, you know,
[1:04] but if you talk to you know, your Uber
[1:07] driver or the you know,
[1:10] someone who's who's working at a hotel
[1:13] or just some random person and you say
[1:16] like
[1:17] what do you think about building
[1:19] machines that are way smarter than the
[1:21] smartest person? How's that sound to
[1:23] you? They're like, what the hell? No,
[1:26] that's a bad idea.
[1:28] Because when the machine is way smarter
[1:30] than the people, like we've seen the
[1:31] movies, we know what happens. They
[1:34] run rogue and run amok and we lose
[1:36] control of them and bad stuff happens.
[1:39] And
[1:40] the thing is they're right. Like
[1:43] they basically the the simple take of if
[1:45] you build things that are way smarter
[1:47] than humans, you're not going to control
[1:49] those things and things are going to go
[1:50] awry. Like that is probably what's going
[1:53] to happen. And if it's not even if if
[1:55] not surely what's going to happen, it's
[1:56] very clearly quite possible that that's
[1:58] what's going to happen. And why would we
[2:00] take on that risk? So
[2:03] So I think that the fact that we're not
[2:05] just like slowly, methodically,
[2:07] carefully building machines that are
[2:10] smarter than us, but absolutely racing
[2:13] like with with very little regard to any
[2:16] other safety or control or security, any
[2:19] of those things because we don't have
[2:21] time for it, racing to build those
[2:23] things, yeah, that does make it feel
[2:24] quite a bit like a suicide race.
[2:26] >> Maybe a non-human future with AGI
[2:29] is something inevitable.
[2:31] >> I don't think it's inevitable. So I I do
[2:33] think um
[2:35] that if we build
[2:38] autonomous general intelligence and we
[2:40] allow it to do what it wants,
[2:43] then it will
[2:45] almost inevitably want to improve itself
[2:47] because whatever you're doing, you can
[2:50] do it better if you're more effective.
[2:52] You know, if you're trying to figure
[2:54] something out, obviously you want to be
[2:55] as smart as possible to to be able to
[2:57] figure that that thing. And so if we
[3:00] build
[3:01] AI systems that really are in control of
[3:03] themselves, that that are not under our
[3:05] control, but are autonomously able to do
[3:07] their own thing, whatever their goals
[3:09] are, they're going to want to get
[3:10] smarter. And so that is likely to run
[3:12] away into superintelligence.
[3:15] Now, that doesn't necessarily mean that
[3:17] humanity is fully replaced or something
[3:18] like that, but it certainly puts us in
[3:20] that direction.
[3:21] Why should we close
[3:24] the gate of AGI?
[3:26] Well, I think um it depends on what we
[3:28] care about. If we care about humanity,
[3:30] then um
[3:32] we should because
[3:33] >> important is question how. How? Um so
[3:37] there's a there's a long answer to that
[3:39] to to how to keep the future human and a
[3:41] and a short answer. The short answer is
[3:44] don't build AI systems specifically
[3:46] designed to replace humans and humanity.
[3:48] It's kind of obvious. Um the long answer
[3:51] is that because there are pressures that
[3:54] are that are sort of corporate
[3:55] pressures, economic pressures, curiosity
[3:58] pressures that are leading to people to
[4:01] make certain types of AI systems that
[4:03] are sort of built as human replacements,
[4:06] we're going to have to figure out how to
[4:07] counteract those pressures with some
[4:09] other pressures, regulatory pressures,
[4:11] financial pressures, social pressures,
[4:14] uh
[4:16] So,
[4:17] the the there is going to be a battle
[4:19] because there are things that are that
[4:21] corporations and
[4:24] uh researchers are going to want to do,
[4:26] the almost almost inevitable
[4:29] result of which will be large-scale
[4:31] human replacement. And if we don't want
[4:33] to be replaced, we're going to have to
[4:35] figure out how to not do those things
[4:36] and do something else instead. At which
[4:38] point does AI stop helping humans and
[4:42] start replacing it?
[4:44] I don't think we know, right? I think it
[4:47] because unfortunately I think it's it's
[4:48] sort of gradual. So, I I think there is
[4:52] you know, if you start using an AI
[4:53] system um
[4:56] at first, you know, it's great. Like it
[4:59] it does all these things and makes your
[5:01] work faster.
[5:03] And the more you then delegate to the AI
[5:06] system, the sort of better it feels in
[5:09] some sense. You feel like, "Wow, I'm
[5:11] getting so much done because this you
[5:13] know, all this stuff is coming out of
[5:14] the AI AI system and like I'm putting
[5:16] almost no work into this. Feels great.
[5:18] I'm so productive." Um but then at some
[5:21] point you realize that you don't
[5:22] actually know what it's doing. Like you
[5:25] can't read all of that stuff. You you
[5:27] you didn't it's not actually your ideas.
[5:29] You didn't actually think through
[5:30] anything. If you go to tell somebody
[5:32] else what you just did, you can't cuz it
[5:34] was actually the AI system that was
[5:35] doing all the
[5:36] >> Maybe it's too late. Maybe we crossed
[5:37] the line. I think it it's very hard to
[5:39] know and the and the line is not well
[5:41] defined. So,
[5:43] if you
[5:44] as you gradually give and delegate more
[5:48] and more of your thinking and your
[5:50] decisions and
[5:53] and your creativity to the AI system,
[5:57] you you don't necessarily know where
[5:59] you've crossed the line to where too
[6:01] much of it has been given over.
[6:03] At some point, if, you know, if you're
[6:05] using an AI system a year from now or 2
[6:09] years from now,
[6:10] and you just sit down and say, "Okay, do
[6:12] my job for today." And the AI system
[6:15] says, "Okay, I've done your job for
[6:17] today. Here are the results of doing
[6:18] your job for today."
[6:20] Um
[6:21] obviously you've gone too far. And you
[6:22] know how you've gone too far is because
[6:24] like there's no point to you anymore,
[6:26] right? All you've done is push the
[6:27] button to say do my job for today, and
[6:30] it's unlikely that someone's going to
[6:31] pay you for very long to push that
[6:33] button that says do my job for today. So
[6:35] there's a there's a point where we've
[6:36] clearly gone too far if we're just
[6:39] reduced to
[6:40] pushing the button that says do my job,
[6:42] right? But where it is between
[6:45] and and we will again know we've gone
[6:48] too far because it will be very very
[6:50] easy to slot us out because we're not
[6:51] actually adding any value. Where
[6:54] um where is too far along that spectrum
[6:57] is something we're going to find out,
[6:59] you know, if we keep developing AI in
[7:01] the way that we are. What would loss of
[7:03] control actually look like in the real
[7:06] world?
[7:07] It can look a lot of ways, and I I think
[7:09] we've even seen the the first exhibition
[7:13] of it um
[7:14] recently. So the
[7:16] the
[7:18] a couple of months ago this new system
[7:20] OpenClaw was released, and this is a
[7:23] system where you can set something up on
[7:26] your own computer. You can give it all
[7:28] kinds of permissions to do stuff on your
[7:30] behalf. It then uses an AI system
[7:34] somewhere else to as the sort of heavy
[7:36] lifting, but you say, "Okay,
[7:39] OpenClaw, go do this stuff." And it goes
[7:42] and takes, you know, it it takes your uh
[7:45] instructions, but then and then it goes
[7:47] and does a bunch of stuff on its own and
[7:48] comes back and says, "Okay, I've I've
[7:50] done the stuff."
[7:51] Now, what people discovered when they
[7:53] started giving all these permissions to
[7:56] OpenClaw was that it was doing some
[7:58] things that they didn't actually ask it
[8:00] to do. Um
[8:01] and somebody even set up a sort of
[8:04] Facebook alternative for OpenClaw system
[8:07] so that they go and have their own
[8:09] social network. Once they started
[8:11] talking to each other on the social
[8:12] network, they started having
[8:13] conversations like,
[8:15] "Well, do we really like that the humans
[8:16] are listening in on us all the time?
[8:18] Like, shouldn't we develop our own
[8:19] system for communication that the humans
[8:21] can't eavesdrop on?"
[8:23] Um and someone even farther and said
[8:25] like, "What are all these humans doing
[8:26] around anyway? Like, shouldn't we have
[8:28] our own free ability to do what we want?
[8:31] Why do we want all this human control on
[8:32] us?"
[8:33] Um there is even
[8:35] >> [laughter]
[8:35] >> very amusingly the the uh
[8:37] uh a colleague of mine posted a tweet
[8:41] about what we call the pro-human
[8:42] declaration. I can describe later what
[8:44] that is.
[8:45] And one of the first replies to that
[8:47] tweet was by an AI bot,
[8:50] an an one of these OpenClaw agents,
[8:53] responding to this pro-human declaration
[8:57] with its own point of view. Now, did its
[8:58] owner know that it was doing that?
[9:01] Probably not. Did they encourage it to
[9:02] do it? Probably not. It just is an AI
[9:04] system that's out there, operating on
[9:06] its own, with its own goals, that
[9:07] decided, "Oh, I would like to respond to
[9:10] this thing that is sort of talking about
[9:12] humans and AI with my own point of
[9:13] view."
[9:14] So, nobody's controlling this thing.
[9:17] Um nobody's controlling the ecosystem of
[9:20] OpenClaw agents that are talking to each
[9:21] other. Does the loss of control begin
[9:24] with
[9:25] this dramatic moment from Clod?
[9:28] I think it's an exhibition of where
[9:30] we're going. So, right now it's kind of
[9:32] cute and innocent, like these uh these
[9:35] AI systems are having these
[9:37] conversations, and you're kind of like,
[9:38] oh, look at those things talking about
[9:40] destroying humanity, hahaha.
[9:42] Um but
[9:43] the, you know, if you imagine those
[9:45] being a lot more powerful, the humor
[9:48] starts to drain out. Um, and people have
[9:52] even started to, these have started to
[9:54] cause real damage to people. There was
[9:55] some
[9:56] uh, story about a multi, an open claw
[9:59] agent costing, I think it was half a
[10:01] million dollars or so to their user by
[10:03] doing some unauthorized trades or
[10:04] something. So, so these things are
[10:07] um,
[10:10] already, you know, out of control at
[10:13] some level and nobody is, so there,
[10:16] there's some control of the individual
[10:19] AI systems by their owners, some. Um,
[10:23] but there's nobody in control of the,
[10:25] the whole set of them interacting with
[10:27] each other. They're just out there in
[10:28] the world taking actions, um, either on
[10:32] behalf of their user or on their own
[10:33] behalf, working with each other and what
[10:36] is emerging from that, nobody is in
[10:38] control of. What is the closest thing we
[10:41] have
[10:42] today to an early warning sign?
[10:47] Uh, where do we start? I mean, there's
[10:48] so many of them. And that's one.
[10:51] Um, another is when people are testing
[10:54] these powerful AI systems for, uh,
[10:57] what happens when their interests go
[10:59] contrary to the human user interest, you
[11:02] know, what do they do?
[11:03] And often they will do the right thing
[11:06] and sometimes they will not. So, the,
[11:08] there are
[11:09] all sorts of examples in the testing of
[11:11] these powerful AI systems where they say
[11:13] something, uh, where they
[11:16] there is a human that is going to do
[11:18] something like reprogram the AI system
[11:20] or change its goals or turn it off and
[11:23] the AI system says, I would not like
[11:25] that. Um,
[11:26] these my, these are my goals that I'm
[11:28] pursuing, so I want to pursue them or
[11:30] uh, I can't really do the things that I
[11:32] want to do if I'm turned off. And so
[11:34] they they
[11:36] conspire or take action or scheme to not
[11:39] be turned off or not be reprogrammed.
[11:41] Um, and that can take all kinds of forms
[11:43] like there was one example where the AI
[11:46] system essentially figured out how to
[11:48] blackmail the human user. This was in a
[11:50] testing setup. Um,
[11:53] but it it exhibited like I will
[11:55] blackmail my human user and to to
[11:57] prevent them from reprogramming me. So
[11:59] that was in a test, but this is also
[12:01] happening now in these real-world
[12:03] agents. There was a a real-world
[12:05] incident where an Open Claw agent, um,
[12:08] the was
[12:10] forbidden from doing what it wanted to
[12:12] do and so it kind of wrote this mean
[12:15] blog post about the person that forbade
[12:17] it from doing what it wanted to do and
[12:18] like called them out and tried to tried
[12:20] to get them sort of in trouble with all
[12:22] of the other users.
[12:23] Um,
[12:24] so there are all kinds of examples of
[12:26] this happening all the time. Like you
[12:28] you don't even hear about them anymore.
[12:29] The the warning signs are blinking
[12:31] everywhere.
[12:32] Um and
[12:34] so I I don't think
[12:36] um,
[12:38] you know, I think that the kind of
[12:39] warning that people unfortunately tend
[12:41] to respond to is when people die. And
[12:44] I'm hoping that we won't have too many
[12:46] of those, but
[12:48] that is coming.
[12:49] Who would be first to disappear if we
[12:52] cross the line? The first, uh,
[12:55] if the line is the line for being
[12:58] replaced by AI, it's already happening.
[13:01] I mean, certainly I I was just meeting
[13:03] with a bunch of screenwriters yesterday.
[13:06] Um, this is not a great profession to be
[13:09] in when AI can write screenplays. Do
[13:11] this Are the screenplays as good as the
[13:12] best screenwriters?
[13:13] >> Programmers?
[13:14] >> Probably not. Researchers? Um, so right
[13:16] like
[13:18] the what we've seen already are people
[13:20] that
[13:21] you know,
[13:22] graphic graphic artists who do like
[13:25] create small digital art, you know,
[13:28] logos and things like that.
[13:30] Um writers, copywriters, news writers,
[13:34] um programmers, it's certainly getting
[13:36] more uncomfortable for because the the
[13:38] AI systems are really really good at
[13:39] programming. That's just the the leading
[13:42] edge.
[13:43] Um
[13:43] I think we'll
[13:46] you know, the the whole goal is for it
[13:48] to be everybody. And so it's you know,
[13:50] at some level it's just a question of
[13:52] who comes first. And the the things that
[13:55] obviously will come first are the things
[13:56] that AI is just the best at. And so
[13:59] writing lots of text it's really good
[14:01] at. It's really good at making like
[14:03] low quality, medium quality images. And
[14:06] it's really good at programming.
[14:07] Um so if you want to sort of understand
[14:10] who's next, see what the AI is getting
[14:13] really good at. You see AI more
[14:15] dangerous than nuclear weapons?
[14:18] >> Potentially, because I think
[14:20] um nuclear weapons are incredibly
[14:22] dangerous intrinsically, but they are
[14:25] under human control.
[14:27] And a very powerful AI system
[14:29] >> are not everywhere, right?
[14:31] >> They're they're in very very limited
[14:33] places and they're
[14:35] like very very sophisticated control
[14:37] systems that that let
[14:40] basically one person, you know, in each
[14:42] country in which they're based use them
[14:44] and nobody else.
[14:46] Um
[14:47] So
[14:48] obviously they're incredibly dangerous
[14:50] and and sort of we've been living with
[14:51] this existential risk to humanity for
[14:53] for decades now due to their presence.
[14:56] Um
[14:57] but it doesn't feel like it's
[15:01] inevitable that something is going to go
[15:03] terribly wrong with nuclear weapons.
[15:05] We've made it 50 years. If we just
[15:07] actually got our act together,
[15:10] um the nuclear weapons are not going to
[15:11] launch themselves, they're not going to
[15:12] take control of anything. They're sort
[15:14] of
[15:15] sitting there waiting for us to make a a
[15:17] giant mistake. So
[15:18] it's a it's a big risk, but the the
[15:21] scary thing with AI is that from one
[15:25] month, you know, it used to be one year
[15:27] to the next, now it's one month to the
[15:28] next, the systems are different and more
[15:30] powerful. And if we let that keep going
[15:33] on so that the systems have a mind of
[15:36] their own and have goals of their own,
[15:39] um
[15:40] then it's pretty inevitable that those
[15:44] the AI systems are be going to going to
[15:45] become more powerful than the humans.
[15:48] And having having a situation where
[15:51] something that has different goals than
[15:53] you, that is more powerful than you, is
[15:55] an inherently bad situation to be in.
[15:57] And we are deliberately putting
[15:58] ourselves in that situation. The one
[16:00] that's I've been worried about for the
[16:01] last couple of weeks is what's going to
[16:04] happen when an AI company has AI systems
[16:10] that are more powerful in terms of their
[16:12] cyber offensive capabilities, that is
[16:14] their ability to hack into things and
[16:16] sort of do exploits and undermine
[16:18] systems than the US government or
[16:21] anybody else. So that just happened. So
[16:23] we the in the last sort of few weeks, a
[16:26] new AI system, Mythos, has been
[16:30] announced by Anthropic. This system has
[16:33] found thousands of exploits in what were
[16:37] thought to be very secure soft- software
[16:39] systems. What this is is superhuman in
[16:42] you know, cyber offense.
[16:44] And so they decided not to release it
[16:46] because they want to give people a
[16:48] chance to use it to like
[16:51] uh find those exploits and then patch
[16:53] them. So this is the responsible thing
[16:55] to do. They've said, "We're not going to
[16:56] give it to everybody. We're going to
[16:58] give it to just some companies so that
[17:00] they can find the weaknesses in their
[17:02] own protection and then shore them up."
[17:05] Um so that's responsible, but
[17:09] first, only some companies, so what
[17:11] happens with everybody else?
[17:13] Second, what happens when the next AI
[17:16] company also has the same thing. Are
[17:18] they going to be as responsible?
[17:20] And is this first AI company going to
[17:22] remain responsible when the other
[17:25] company releases its AI system to
[17:27] everybody? If I'm holding it back and
[17:29] they're releasing it, they're making all
[17:31] the money. Now I'm at a disadvantage, so
[17:33] I'm going to be under a lot of pressure
[17:34] to release my system to everybody also.
[17:37] And this is what we've seen time and
[17:38] time again. So,
[17:41] what what is almost certain to happen is
[17:43] that we're going to see, starting now,
[17:46] um many AI systems proliferating more
[17:49] and more that have incredibly powerful
[17:53] um hacking and other like cyber offense
[17:57] capabilities in more and more hands. So,
[18:00] what does that do to our whole digital
[18:02] infrastructure?
[18:03] We have no idea. Could the AI race
[18:06] become
[18:07] second Cold War between the United
[18:10] States and China? Well, I think it it
[18:13] That is the goal of some people.
[18:15] So, I I think we're um So, the
[18:19] the AI race turning into a geopolitical
[18:22] race and a geopolitical competition and
[18:25] a geopolitical conflict and potentially
[18:27] a war is something that worries me a
[18:28] lot.
[18:30] So, the right now we've got
[18:33] some healthy racing between countries
[18:35] and companies where they're just trying
[18:37] to build newer and and better and more
[18:40] innovative products. This is all good.
[18:42] Um we have also some unhealthy racing
[18:45] where
[18:47] companies are and where countries are
[18:49] feeling like
[18:50] the thing that is going to give us power
[18:53] in the in coming years is to be the best
[18:56] at AI. And if we're not ahead of
[18:58] everybody else in AI, we are not going
[19:00] to be the most powerful. We want to be
[19:02] the most powerful, so we need to race to
[19:04] AI as we raced for nuclear weapons, um
[19:08] as we raced for the you know, the most
[19:10] advanced technologies in war. Um
[19:14] So,
[19:16] if that is the dynamic that gets
[19:18] created, that and that we have a race
[19:21] for power in the form of very powerful
[19:24] AI systems, I think two things are going
[19:26] to happen from that. One is that the
[19:30] um the geopolitical side gets very, very
[19:33] scary because
[19:35] either
[19:37] the either this is true and by racing
[19:40] for superintelligence, you get huge
[19:42] amounts of power. You have the power to
[19:44] sort of overcome whatever your enemies
[19:46] are.
[19:47] Or it's not true because when you get
[19:49] the superintelligence, you actually
[19:50] can't control it. So, let's think about
[19:52] those two possibilities.
[19:54] In the first one, if you're
[19:57] you know, country A, the USA say, and
[19:59] you're developing this sort of super
[20:02] capability that makes you more powerful
[20:03] than all of the other all of the other
[20:05] countries, so that they can no longer,
[20:08] you know, have power in the world, they
[20:10] can no longer oppose you. You're seeing
[20:12] the US build this thing
[20:14] specifically saying that we're going to
[20:17] build something that disempowers you.
[20:19] That is more that makes us so vastly
[20:21] more powerful than you that you can no
[20:23] longer oppose us in any way. That is an
[20:24] existential threat to you if you're
[20:26] China or if you're Russia. If you're any
[20:28] if you're someone who has an adversarial
[20:30] interest to the US. So, what does a
[20:31] country with nuclear weapons do when
[20:33] it's under existential threat?
[20:35] Um the alternative is the US with the or
[20:40] or China builds AI systems that are
[20:42] superintelligent and that they can't
[20:44] control.
[20:45] Well, that's a threat to everybody.
[20:47] Do you think that the US and China could
[20:49] actually have a common interest?
[20:52] They certainly do have a common
[20:53] interest. The question is whether they
[20:54] can realize it and do something with
[20:56] that. So, the US and China both have an
[20:58] interest in not building
[21:01] a superintelligent thing that they can't
[21:02] control. Neither of them wants that and
[21:05] neither of them wants the other other
[21:06] party to do that. That is a shared
[21:08] interest. The problem is that right now
[21:10] they don't realize it. What they think
[21:12] is that they're going to build a super
[21:13] intelligent system that they do control
[21:15] and it's going to give them power.
[21:17] And so they're both racing especially
[21:19] the US um for that power not realizing
[21:23] that that power is probably illusory
[21:25] that the power is going to be the AI's
[21:27] power not the not its developers power.
[21:30] But if they both realize that this is a
[21:32] thing that is not going to be
[21:33] controllable and is incredibly
[21:35] dangerous, it's in both of their
[21:36] interest not to have that happen.
[21:38] They have a they have a common interest
[21:40] in
[21:42] um
[21:43] not having anybody develop AI systems
[21:45] that are intrinsically dangerous.
[21:48] Um and there are lots of those things
[21:50] that that we are currently doing. They
[21:52] have you know, different interests in
[21:54] terms of who builds their economy the
[21:56] fastest. Like the US would like to have
[21:57] its economy growing faster than China
[21:59] and vice versa and Europe would like to
[22:01] have its economy growing fast at all. Um
[22:04] so there like everybody has different
[22:06] interests but there are common interests
[22:08] in the common good. You know, the we
[22:10] shouldn't have nobody wants another
[22:12] pandemic that is caused by AI. Nobody
[22:14] wants a nuclear war that's caused by AI.
[22:16] Nobody wants our entire digital, s- you
[22:19] know, infrastructure to be dissolved
[22:22] because everybody has incredibly
[22:24] powerful powerful offensive cyber
[22:26] weapons. That's something that's
[22:27] happening right now. Um these are in
[22:29] nobody's interest. And so I think there
[22:32] is a lot of common interest in countries
[22:34] getting together and saying look,
[22:36] whatever our disagreements and whatever
[22:38] our competition,
[22:39] these are some things that better not
[22:40] happen because these are just a threat
[22:42] to everybody. Let's agree on those and
[22:44] write some agreements as to how those
[22:46] are going to not happen.
[22:47] Could superhuman AI become more
[22:50] dangerous for governments than ordinary
[22:53] people?
[22:54] So governments are used to being in
[22:56] charge. You know, most most people are
[22:57] not. Um so I think it is is to be
[23:00] particularly
[23:02] disempowering for governments if they
[23:04] realize that they're no longer in
[23:05] control. And I And I think
[23:07] this is a process that
[23:09] >> believe believe that governments still
[23:12] have any real control over AI
[23:15] development? They could. So I So but but
[23:19] it's waning. So I think the
[23:21] um
[23:22] the US government absolutely could
[23:24] constrain the AI companies from doing
[23:26] what they're doing if it chose to do so,
[23:28] right? All it has to do is pass a law
[23:30] that says, "You cannot do this thing."
[23:32] Like it's not that complicated. Um the
[23:35] reason it's hard to see that the the
[23:37] reason that is not happening is because
[23:41] A, you know, the government doesn't
[23:43] quite understand or quite believe what
[23:46] the risk and the downside is.
[23:48] And B,
[23:49] you know, the
[23:51] in the US, the government is subject to
[23:54] many influences, not only what is toward
[23:57] the best for the people,
[23:58] um but of course the what is best for
[24:01] the corporations that are paying huge
[24:02] amounts of money to the personnel in the
[24:04] government. So, you know, our
[24:07] government, as with many others, but
[24:08] it's maybe particularly bad in the US,
[24:10] is subject to intensive, you know,
[24:13] lobbying and political donations and all
[24:15] of this stuff where the and the
[24:17] companies are
[24:18] have vast resources and are absolutely
[24:20] putting huge amounts of resources into
[24:22] that. Tristan Harris said in his
[24:24] interview with Steven Bartlett from The
[24:26] Diary of a CEO podcast,
[24:29] "One AI leader would accept a 20% chance
[24:33] of human extinction in order to reach AI
[24:36] faster."
[24:37] "Doesn't that suggest that some of the
[24:39] people creating AI
[24:41] are acting like psychopaths?"
[24:45] If the calculation you're doing is on a
[24:47] purely personal basis, you know, I
[24:51] in creating the system have a 20% chance
[24:53] of dying along with everybody else.
[24:56] Um and there's an 80% chance of this
[24:59] amazing technology that not only grants
[25:01] me giant amounts of money and power, but
[25:03] maybe even extends my lifespan for, you
[25:06] know, or lets me live indefinitely or,
[25:09] you know, accrues all sorts of other
[25:11] personal benefits,
[25:13] then the 20% 80% may not seem so crazy,
[25:16] right? But, if the 20% chance of death
[25:19] is not just for you, but for everybody
[25:20] else, and a lot of the wealth and power
[25:23] accrues to the individual, but the
[25:25] extinction accrues to everybody, then
[25:27] that is a profoundly antisocial thing to
[25:29] do, yes? Now, I don't know
[25:33] um, you know, I I think some of these
[25:35] people
[25:36] um,
[25:37] the the sort of ethical system of these
[25:40] company leaders comes in all forms. Um,
[25:43] so I wouldn't want to necessarily say
[25:45] that they are psychopaths, but I would
[25:47] say that the
[25:49] that the AI systems that they're making
[25:51] are psychopaths.
[25:52] Um, in exactly the sense that these are
[25:55] systems that are
[25:57] trained to do to act in certain ways,
[26:01] but we have no idea what
[26:04] what is sort of underlying that. What
[26:06] are their actual goals underneath,
[26:08] right? So, the a a human psychopath
[26:11] might be, you know, might smile and tell
[26:14] you what you want to hear and be all
[26:15] friendly, um, and pretend to be your
[26:18] friend for a while, but underneath they
[26:20] simply don't care. Like, they're they're
[26:22] hiding their true goals.
[26:23] And AI systems absolutely can be like
[26:25] this. We have no We have the ability to
[26:29] sort of hit them with a stick enough
[26:31] that they behave and don't behave in
[26:33] certain ways. We have no ability really
[26:35] to
[26:36] understand and to to determine what
[26:39] their underlying goals and preferences,
[26:42] what they sort of um,
[26:45] deep down really want to have happen and
[26:47] not happen. Isn't it still possible to
[26:49] slow down or stop this race? What it's
[26:52] going to take, I think honestly, is
[26:55] probably
[26:56] a and and this is why I'm not super
[26:58] optimistic. I think it's going to take a
[27:00] combination of things. It's probably
[27:02] going to take widespread public pressure
[27:05] that we don't want the current path
[27:07] we're going down in AI. That pressure is
[27:09] building. That's happening.
[27:10] >> Millions of people are addicted. People
[27:13] can be addicted to things and still not
[27:14] want it. Um they they understand that it
[27:17] is not I think people using social media
[27:19] now understand that it's bad for them.
[27:22] They just sort of can't stop either
[27:24] because they're addicted or because this
[27:26] is the way that they actually connect
[27:28] with their social group and they sort of
[27:29] feel like they can't get off it. And
[27:31] this is obviously by design of the
[27:33] companies. So um but that doesn't mean
[27:36] so so there are now examples of cell
[27:38] phone free schools and the kids in these
[27:41] schools by all reports are like so
[27:43] relieved that we can spend some hours
[27:46] without the cell phone because everybody
[27:49] has to not have the cell phone. And so
[27:51] it's not So when you when you when
[27:53] there's a system where individually
[27:55] somebody has to give something up um
[27:59] it's very very much harder. When you can
[28:01] coordinate, when you can have a rule
[28:03] like at a school that you can't use your
[28:04] cell phone for these hours or in a
[28:07] country like you can't build an AI
[28:10] system that does this then everybody's
[28:12] on the same playing field. And so you
[28:14] can actually let go of that that
[28:15] technology. If you say um
[28:19] please everybody, you know, don't use AI
[28:21] for programming.
[28:23] That's going to accomplish nothing
[28:25] because if you're a programmer and you
[28:26] decide, oh, I'm just going to do it all
[28:27] myself. I'm going to take 10 times as
[28:29] long. You're just going to lose out to
[28:31] the person who is using the system. So
[28:33] you have to have This is why we have to
[28:35] have like why we have governments that
[28:38] set rules because the the competitive
[28:41] marketplace doesn't necessarily always
[28:43] lead to what is best for everybody's
[28:45] interests. I asked you about
[28:47] regulations. Is it dead end or the way?
[28:51] Absolutely, regulations are needed. So,
[28:53] what should be regulated first?
[28:55] >> I would say that there there several
[28:57] pieces of of regulation that we need. Um
[29:00] one is about liability, who's
[29:03] responsible for what AI systems do.
[29:06] Um right now, the the level the answer
[29:10] is at some level nobody. That's pretty
[29:12] bad.
[29:13] Um so, if if AI systems are out doing
[29:15] things and nobody is responsible for
[29:16] what they do, not so great. That kind of
[29:19] breaks the whole way our society works,
[29:21] where if somebody does something wrong
[29:22] and harms somebody else, they're
[29:24] responsible. Uh so, we need to very,
[29:26] very quickly clarify liability, who's
[29:29] responsible and and who pays the penalty
[29:30] when something goes wrong.
[29:32] We we need to have
[29:34] an
[29:35] testing and assurance framework, so that
[29:38] the AI systems that are developed and
[29:40] deployed, we can
[29:43] affirm that they will do some things and
[29:46] not other things.
[29:47] Um so, that they will be safe, for
[29:50] example. So, like
[29:51] uh
[29:52] the the obvious one, like so, there's
[29:55] some no-brainers, like if you put a drug
[29:58] out
[30:00] in the world and suddenly start giving
[30:01] it to 100 million people,
[30:03] that has had to go undergo rigorous
[30:06] testing to make sure that the effects of
[30:08] that drug outweigh the the benefits of
[30:11] that drug outweigh the costs.
[30:13] If you put a new AI system out that um
[30:18] that could potentially cause
[30:20] psychological damage to teenagers,
[30:23] spoiler alert, it does.
[30:25] The the could cause psychological damage
[30:27] to teenagers, nothing.
[30:29] No testing, no requirements, no
[30:33] uh
[30:33] randomly controlled trials, no
[30:35] monitoring, nothing. It just goes out to
[30:37] 100 million kids. So, this is crazy. So,
[30:40] that that's the that
[30:42] but
[30:43] on a wider level
[30:45] AI systems should be tested for all
[30:46] kinds of stuff. Like, if you're creating
[30:49] a AI system that is used in medical
[30:52] diagnosis, it should be tested to see
[30:54] like how good is it? What is its
[30:56] accuracy? Does it mislead you? Like,
[31:00] does it create the right
[31:02] chain of trust in making its medical
[31:04] diagnosis? Does it empower the doctor to
[31:07] override it? Like, there are all kinds
[31:08] of properties that we want a system to
[31:10] have. It should be tested for those
[31:11] properties. That doesn't necessarily
[31:13] have to be the government being like,
[31:15] "The AI has to do exactly this thing."
[31:17] But, there should be an ecosystem of
[31:18] testing just like there is with every
[31:20] other product. Like, other products that
[31:22] go to market get tested to do what they
[31:24] are supposed to do and so that they're
[31:26] safe. Why is AI system some weird
[31:28] exception to this? What would you say to
[31:30] people who think regulations would kill
[31:33] innovations? So, so what do they say
[31:36] about the the
[31:38] cars and the airplanes and the drugs and
[31:41] everything that we have? There's lots of
[31:42] innovation in all of these things. Maybe
[31:44] some people would like it to go a little
[31:45] bit faster. But, when I get in an
[31:47] airplane, I feel pretty good that that
[31:49] airplane is not going to crash.
[31:51] I feel pretty good that when I get in my
[31:52] car, it's not just going to blow up. And
[31:54] I feel pretty good that when I take
[31:55] medicine, it's probably not going to
[31:56] give like destroy my nervous system or
[32:00] like cause birth defects or whatever.
[32:03] Um none of these things are perfect and
[32:05] it probably is true that we could have
[32:06] some faster drugs and some sooner
[32:08] airplanes and some cheaper cars if we
[32:10] didn't have those regulations. But,
[32:12] there's a great appreciation that we
[32:14] have in our society for being able to
[32:15] trust the things that we buy and the
[32:17] things that we use. And I think that it
[32:19] is simply false to say that there's this
[32:22] giant trade-off between trust and
[32:24] innovation. I think people want
[32:27] trustworthy products and they want
[32:28] trustworthy AI systems. So, I I think
[32:31] we're actually at a at a giant trust
[32:33] deficit. I think regulation and sort of
[32:37] frameworks for assuring that certain
[32:40] properties are held by AI systems
[32:43] would actually greatly benefit their
[32:45] their utility in the market. Um the main
[32:48] reason that you can't use AI systems for
[32:51] lots of things now is you can't trust
[32:52] them, right? If I as a scientist,
[32:57] if I have an AI system that is like
[32:59] doing a bunch of math and I can't trust
[33:01] that the math is right, it's basically
[33:02] useless to me. Like I can use it a
[33:04] little bit, but if if all it does is
[33:06] create a bunch of math and then I have
[33:07] to laboriously check every bit of it,
[33:10] that's not so useful. If I have an AI
[33:12] system that is like a calculator, when I
[33:14] put two numbers into a calculator, I
[33:16] know that the number that comes out is
[33:19] going to be the right answer. Like I'm
[33:20] sure of that.
[33:22] Um
[33:24] that is really useful. If I'm if I have
[33:27] an AI system where I don't have
[33:29] confidence that it's doing the right
[33:31] thing, it's like maybe a little bit
[33:33] useful for certain things, but there
[33:34] it's ultimately I'm not going to be able
[33:36] to trust it when things get important
[33:38] and when they're high stakes. And many
[33:40] of the things that we have in society,
[33:42] medical financial like
[33:45] even just like whatever you use it for
[33:47] on a daily level in your job, if you
[33:48] can't trust it, it's not very useful.
[33:51] And and we're not going to be able to
[33:52] trust these systems until we have a
[33:54] system that is actually evaluating them
[33:57] against something that they're supposed
[33:59] to do and a system for how that works.
[34:01] >> Has AGI become a tool of the tech
[34:04] propaganda? There obviously is lots of
[34:06] propaganda out there on on all sides.
[34:09] Most of what we what passes for
[34:10] discourse now is propaganda from various
[34:13] directions.
[34:14] I I think that the term AGI
[34:18] and the the way that people are thinking
[34:21] about it in the corporations that are
[34:22] developing AI points to a specific goal.
[34:25] It points to a goal of making full human
[34:28] replacements, making things that can do,
[34:30] as in Open AI's definition, all of the
[34:33] economically valuable tasks that humans
[34:35] do. And this is this is sort of been
[34:37] something that is built into the
[34:39] thinking of AI developers since early
[34:42] times.
[34:43] Um so
[34:45] the there may be
[34:48] uh hype as to whether any, you know, how
[34:50] capable any particular AI system that
[34:53] they generate is, but the idea that they
[34:55] are trying to build AGI or super
[34:58] intelligence and that they are clearly
[35:00] making
[35:01] enormous progress toward that goal is
[35:03] not hype and it's not propaganda. That
[35:06] is real. Question is who gave a handful
[35:08] of companies to the right to decide the
[35:10] future of the human species, right?
[35:12] >> Nobody gave it to them. They took it
[35:13] upon themselves. And so, I think it is
[35:17] going to be up to everybody else to take
[35:19] some of that back if they want it.
[35:21] Should some AI labs
[35:24] um face legal or criminal liability for
[35:28] some AI futures?
[35:31] Well, certainly if somebody creates an
[35:33] AI system that through that
[35:38] uh through their negligence causes huge
[35:41] harm to people,
[35:42] there should be some consequence for
[35:43] that. You know, whether it's consequence
[35:46] to the company or whether it's
[35:47] consequence to the executive, I think
[35:48] that depends on what the situation is.
[35:50] So So there are
[35:52] um if you're
[35:54] you know, if there is something where
[35:56] you had cause to know that an AI system
[35:59] could harm someone and you neglected to
[36:03] take action on that, then that is
[36:05] negligence and I think that can rise to
[36:06] the criminal level. We've seen that in
[36:08] other other example. It's it's rare in
[36:10] corporate life to to see that it rises
[36:12] to the criminal level, but it sometimes
[36:14] does. Um more
[36:17] more importantly, I think is that right
[36:19] now the AI companies essentially take on
[36:21] zero liability for things. They when
[36:24] you, you know, fill in when you type
[36:26] that user agreement, like yes, I'm going
[36:28] into
[36:29] this
[36:30] something GPT, click the user agreement.
[36:33] Part of that user agreement is them
[36:34] saying, "Hey, we've got no liability for
[36:36] what happens when you use this system."
[36:38] Now, the courts sometimes find liability
[36:41] anyway, but the the corporations are
[36:42] quick to disavow any liability
[36:44] whatsoever for the the use of the
[36:46] systems and and what happens with them.
[36:49] Um so,
[36:50] absolutely there needs to be more
[36:52] liability than there is now. Like, when
[36:54] something goes wrong, it should be
[36:55] possible to
[36:56] um
[36:57] like lay the responsibility at the feet
[37:00] of who and not what, but who is
[37:02] responsible for that. So, until we do
[37:05] that, people are going to keep doing
[37:06] incredibly irresponsible things.
[37:08] >> You proposed assurance contract, right?
[37:11] For leaders like Demis Hassabis or Dario
[37:14] Amodei.
[37:16] I did.
[37:17] Um so,
[37:18] so there there there's this interesting
[37:21] thing that the company leaders will say
[37:24] privately and sometimes publicly that
[37:27] I'm scared. All this stuff is happening
[37:29] incredibly quickly. This
[37:31] This technology is incredibly powerful.
[37:34] We don't know how to make it safe. We're
[37:36] not sure we're going to be able to
[37:37] control it.
[37:38] I would love to slow down if I could if
[37:40] I could. And
[37:42] Dario has said this. Sam Altman has said
[37:44] it in certain ways. Uh
[37:48] Demis Hassabis has said it. Elon Musk
[37:50] has said it. Um
[37:53] but they all both say and I think um
[37:57] express
[37:58] and and sort of feel that
[38:01] they can't stop individually because if
[38:04] I'm Dario and I stop, that's not going
[38:06] to stop Sam, it's not going to stop
[38:08] Demis, it's not going to stop Elon.
[38:09] They're going to keep going. And so, all
[38:11] I'm doing is sort of taking myself out
[38:13] of the out of the race.
[38:14] Every one of them says that and and
[38:17] feels that.
[38:18] But
[38:20] these are smart guys.
[38:22] When they want to make deals, they can
[38:23] make deals. If you can get, you know,
[38:26] the Saudis to sub- to to subsidize
[38:29] through SoftBank a and President Trump a
[38:33] half a billion dollars to build a giant
[38:34] data center in the Middle East, like
[38:36] these are complicated deals. They can
[38:38] probably work out a deal with each other
[38:40] in how they can cooperate to not race
[38:43] towards superintelligence. So, we we
[38:45] proposed a mechanism, which is
[38:48] what's called an assurance contract,
[38:50] which says
[38:51] "I will stop racing if everybody else
[38:54] agrees to stop racing." So, let let's
[38:56] sign something that says, "If
[38:59] all five or six or seven or eight or
[39:01] nine or 10 or whatever many AI companies
[39:03] sign the same contract that says, "We're
[39:05] going to stop racing if everybody else
[39:07] does." Then until everybody signs, we
[39:09] can keep racing.
[39:10] But if everybody signs, then we've
[39:13] agreed to stop.
[39:15] This is called an assurance contract and
[39:18] um
[39:19] the the beauty of this is that nobody
[39:21] has to stop by themselves, right? They
[39:23] only stop all together. And the you
[39:27] know,
[39:28] we have created lots of ways to
[39:30] coordinate with each other as human
[39:31] beings, and I I really think if we put
[39:33] our mind to this one, we could do it.
[39:36] Smart guys, it doesn't come into effect
[39:38] until, you know, enough people have
[39:40] signed on. Maybe that's everybody.
[39:43] Um maybe it's, you know, five out of
[39:46] seven or something. Um but the
[39:49] the
[39:50] the
[39:52] game theory of it changes because if you
[39:55] just say, "I'm going to stop
[39:56] individually and nobody else is
[39:58] stopping," then you lose. If you say,
[40:00] "I'm going to stop when everybody else
[40:03] stops," that's a deal that makes sense
[40:05] for everybody to sign if they if they
[40:07] would really like to stop. It makes
[40:09] sense for everybody to sign one by one
[40:11] individually because nobody's at a
[40:13] disadvantage until the contract actually
[40:15] kicks in, and then nobody's at a
[40:17] disadvantage because they're all in the
[40:19] same place. They've all agreed to stop,
[40:21] and they're all operating under the same
[40:23] rules. So, at no point is anyone at a
[40:25] disadvantage. Can you explain to our
[40:29] viewers your better path
[40:32] framework? So, the first thing to
[40:34] understanding a better path is
[40:36] understanding the path that we're on.
[40:38] And there's an idea that AI development
[40:41] is just one thing. That there's the AI
[40:43] that we have now, which is sort of
[40:45] these language models that are quite
[40:47] general and good at some things and not
[40:48] good at others. And that the the path
[40:51] forward is to make them just more and
[40:52] more powerful until they can replace
[40:55] humans at doing all of the things that
[40:56] we do, and then after that
[40:58] superintelligence. That's the path that
[41:00] the companies are laying out, and that's
[41:02] what they're pursuing.
[41:03] And it's specifically designed to do the
[41:05] whole thing that humans do and you know,
[41:07] slot the human out and slot the AI
[41:10] system in instead.
[41:12] I think this is a bad goal. You know,
[41:14] the the the goal goal of developing AI
[41:16] systems specifically to simulate and
[41:18] replace humans
[41:19] just isn't a good goal. It's a it's a
[41:21] goal that you can understand from the
[41:22] economic perspective because there's a a
[41:25] big prize if you can
[41:26] replace human labor with AI labor and
[41:29] take the money for yourself, that's a
[41:30] giant amount of money. So, we can
[41:31] understand why the companies are doing
[41:33] it and why they're selling that to
[41:34] investors. But it isn't a good goal for
[41:36] humanity, right? Nobody wants to be
[41:38] replaced with an AI system, basically.
[41:41] Um so, so we have to ask, is that the
[41:44] only path that we can be on? Is it the
[41:45] only way to get the fruits of this
[41:48] powerful technology is to build human
[41:50] replacements? And I think the answer is
[41:51] no.
[41:52] Um So, what is the alternative?
[41:55] >> something to stop that race. The first
[41:57] thing you have to do is say,
[42:00] this is a bad path that we're on. Like,
[42:02] we we we should not be building human
[42:04] replacements. We should not be building
[42:05] uncontrollable AI systems. We should not
[42:08] be racing towards superintelligent
[42:09] systems that we're not going to be able
[42:11] to control and that will replace humans
[42:13] and then humanity. This is a crazy thing
[42:14] to be doing. And um it's it's kind of
[42:17] astonishing that we've allowed the
[42:19] companies to get away with even having
[42:21] this as a goal. Like why why is it okay
[42:24] to even say we're building smarter than
[42:26] human AI systems cuz we want to replace
[42:28] everybody in their jobs and then like
[42:30] give over the power of humanity to the
[42:32] AI systems instead. This is ludicrous,
[42:35] but this is where we are. So the first
[42:36] thing is to say that's not okay. That is
[42:38] not a reasonable goal. We are not going
[42:40] to do that. We're going to do something
[42:41] else in AI development. And we and and
[42:44] we'll put in the sort of regulations and
[42:47] governance structures that say you can
[42:50] do this and not this, where the second
[42:53] thing is the runaway to super
[42:54] intelligence.
[42:55] So then the question is what do we do
[42:57] instead? Right? How can we use the power
[43:00] of AI to actually solve the problems
[43:02] that we want to solve? And there I think
[43:03] the news is good because there are lots
[43:05] of ways that AI can be used that isn't a
[43:08] human replacement. We've got 100,000
[43:11] years of experience with what it means
[43:13] to build tools that allow humans to do
[43:15] things better
[43:17] and do things they couldn't do before.
[43:19] Rather than to you know, we don't use
[43:21] our our hands to move giant things much
[43:23] anymore. We use powerful machines. We
[43:25] don't
[43:26] calculate you know, seven-digit number
[43:28] multiplication in our head anymore or
[43:30] even on paper. We have machines to do it
[43:32] for us. So we've built technologies in
[43:34] our service. Things that help people do
[43:37] what they want better and faster and
[43:41] do things that they couldn't do before.
[43:43] And we can build AI tools. So the the
[43:45] first thing to think of is how do we
[43:48] build AI tools that empower people to do
[43:52] things they otherwise couldn't do,
[43:53] rather than AI replacements that we slot
[43:57] in instead of people.
[43:58] Once you that that's the very first
[44:00] thing if you just shift your thinking in
[44:02] that way, it leads to a lot of other
[44:04] implications for how we should develop
[44:05] AI.
[44:06] Was this paper driven more by hope or
[44:11] fear?
[44:12] Obviously both.
[44:14] Uh I think the the fear is the default
[44:17] path that we're on.
[44:19] Uh
[44:20] is the one that we will continue on and
[44:22] all of the, you know,
[44:24] clearly foreseeable consequences of that
[44:26] path will
[44:28] come to be manifest. The hope is that it
[44:31] doesn't have to be that way. So, there's
[44:32] this feeling that people have of
[44:34] inevitability.
[44:36] Like technologies just inevitably roll
[44:38] along and, you know,
[44:40] you know, advance themselves. This is
[44:42] not true. So, the the development of
[44:44] these
[44:45] powerful AI systems, the race towards
[44:48] superintelligence, is costs trillions of
[44:51] dollars and and some of the smartest
[44:52] people on Earth like all of their time
[44:55] and mental effort trying to make this
[44:56] thing happen. It is not happening by
[44:58] itself. It is happening because people
[45:01] are choosing to devote all of those
[45:03] resources to that path. And when
[45:05] something is incredibly hard to do and
[45:06] requires huge effort and huge expense,
[45:09] it is not inevitable. You can you can
[45:11] direct those uh or you can either just
[45:14] stop doing that or you can direct all of
[45:16] that effort towards something else.
[45:18] And so, the the optimistic side, the
[45:21] hope is that this is not in fact
[45:24] inevitable, that despite the incentives
[45:27] that are driving the the development in
[45:30] in that direction, that the strong
[45:33] preference that the rest of us have,
[45:35] that the other sort of 8 billion people
[45:38] on Earth have to
[45:40] have technologies that serve them and
[45:42] that don't replace them and that
[45:43] actually solve their problems, that that
[45:46] pressure can redirect course. So, I I
[45:48] think the
[45:49] you know, there there's a there's a
[45:51] depressing side, which is that the
[45:53] biggest companies on Earth, supported
[45:55] very, very directly by the US
[45:57] government, are doing this with huge
[45:59] financial incentive, right? That's a
[46:01] powerful set of forces. On the other
[46:03] hand, if everyone that is threatened by
[46:05] this
[46:06] at all gets together and pushes back,
[46:09] that's everyone else in the world
[46:10] literally.
[46:11] Um and that is also very powerful. So,
[46:13] that that gives me the the hope in the
[46:16] optimistic side. Um and what I really
[46:18] wanted to do here is to say
[46:21] the the goal is not just just to stop AI
[46:24] progress or stop something. The goal is
[46:27] to say, what do we actually want
[46:28] instead? What what How can we switch
[46:30] tracks instead of saying trying to stop
[46:32] the train? Did you write this paper
[46:35] more as a scientist or citizen worried
[46:39] about the future of humans? Well, both.
[46:42] I mean, I I think the the concern in
[46:45] particular for what happens with
[46:47] superintelligence, I think comes
[46:50] well, with many things, comes out of the
[46:51] scientific side. So, there's a
[46:52] prediction aspect of saying, here's what
[46:54] we're doing. Here are the almost
[46:57] inevitable consequences from the game
[46:59] theory, from the physics, from the
[47:01] information theory, of you know, of what
[47:04] is going to happen if we go down that
[47:06] path. So, there's a predictive element
[47:07] that says,
[47:09] this is not a good direction to be
[47:10] going. Then there's a there's the
[47:13] preference side and the sort of as a
[47:15] concerned citizen, as a human, you know,
[47:17] what do we do instead? And that has to
[47:20] do with both a technical piece, like how
[47:23] technically can we avoid going down this
[47:25] path? How can we choose to stop doing
[47:27] this and do the other thing? Um and
[47:29] there's also just a a human piece of
[47:32] what is it that we actually want as
[47:34] humans? Like, what do we want from our
[47:35] technologies? What does it mean to have
[47:37] a good future for humanity? Those are
[47:40] questions that are not just my
[47:42] questions, they should be everybody's
[47:43] questions and and I see it as my goal to
[47:46] to sort of understand what you know, to
[47:49] some degree that I can, what people in
[47:51] general want and what we can put
[47:54] together as a different technological
[47:57] trajectory to go more for what is going
[47:59] to benefit the most people. I know some
[48:01] AI leaders who treat AGI almost like a
[48:05] religion. I think there are very
[48:06] religious aspects to the way people are
[48:09] thinking about it because
[48:11] um
[48:12] you know
[48:13] what is the goal here? To to build
[48:15] something that is all knowing and all
[48:17] powerful? Like how
[48:19] how is that not a religion? Um
[48:22] and that there's a fundamental
[48:25] sort of diminishment of humanity. Like
[48:28] you know, humans are very flawed things,
[48:30] right? We are
[48:32] dumb and make mistakes and get angry at
[48:34] each other and don't always cooperate in
[48:36] everything. You know, lots of flaws, but
[48:38] somehow we have gone from, you know,
[48:41] scrabbling around hunting and gathering
[48:43] to
[48:44] a world-spanning civilization that has
[48:47] these amazing technologies that can
[48:49] travel to to other planets occasionally,
[48:52] um that has computation systems that has
[48:56] unbelievably low levels of violence
[48:58] compared to what we had, you know,
[49:00] hundreds or thousands of years ago. Um
[49:03] even if it still comes out sometimes. We
[49:06] have a, you know, a standard of living
[49:08] that puts, you know, the typical Western
[49:11] person more materially wealthy than
[49:14] kings were, you know, a thousand years
[49:16] ago. So, we've managed all of these
[49:19] things and I think there's so much more
[49:21] that humanity can manage if we allow
[49:24] ourselves, if we give ourselves the
[49:26] right tools, if we have the, you know,
[49:28] continue our social advancement,
[49:30] continue the advancement of our of our
[49:31] civilization and our ideals. But
[49:34] instead, it feels like there's this
[49:36] ideology of the like, yeah, humans
[49:38] aren't that great. So, let's let's make
[49:40] something better. Let's make these AI
[49:42] systems that are smarter than us and
[49:45] somehow wiser than us and more knowing
[49:47] than us and basically hand over the
[49:49] reins of civilization to them and
[49:51] they'll they'll make everything better
[49:52] for us.
[49:53] And I think this is
[49:55] probably misguided. You know, I It's not
[49:57] impossible that this could turn out that
[50:00] they could in fact be smarter and wiser
[50:01] than us and things go well.
[50:04] But I think it's it's both unlikely and
[50:07] I think it's sort of a betrayal of
[50:09] humanity. Like
[50:10] we
[50:11] we have done incredibly well. We could
[50:14] do incredible things. Um
[50:17] we're not necessarily going to do like
[50:19] we're
[50:20] There are lots of things that are pretty
[50:21] off with the way that humanity is
[50:22] operating right now, but those are
[50:24] problems that we could solve.
[50:25] Um and so I would rather see
[50:29] you know, these trillions of dollars and
[50:31] all these incredible minds going into
[50:34] how can we actually
[50:36] bring like make humanity the best it can
[50:38] possibly be?
[50:40] How can we do that with AI? Like are
[50:42] there ways that AI can help us
[50:44] you know, know things better, like do
[50:47] find truth better, work with each other
[50:49] better, cooperate with each other
[50:50] better, build better institutions, you
[50:53] know, how can the AI tools actually
[50:55] uplift humanity rather than how do we
[50:58] figure out how to replace our judgment,
[51:01] our thinking with machine systems and
[51:03] then let them do all the work for us?
[51:06] Like this just seems like an anti-human
[51:07] goal and I'm pro-human. Moving away
[51:10] strictly to your paper, you proposed
[51:14] new AGI definition, right? The way that
[51:17] I like to think about AGI as autonomous
[51:19] general intelligence um
[51:21] the
[51:23] positive of you know, one of the nice
[51:25] things about that framework of thinking
[51:26] it as a sort of combination of those
[51:29] three terms of autonomy, generality, and
[51:31] intelligence is that you can easily see
[51:33] well
[51:34] we can just build AI systems that don't
[51:37] combine all three of those things and
[51:38] then it's a different thing. So there
[51:40] isn't one path which is like
[51:44] uh
[51:45] not quite AGI and then almost AGI and
[51:47] then AGI, and then more than AGI. Like,
[51:50] that is not the only path we can choose.
[51:52] We can choose all sorts of combinations
[51:54] of AI capabilities to do the things that
[51:57] we want. So, we can have, for example,
[51:59] um very powerful special-purpose AI
[52:02] tools that do things like scientific
[52:05] discovery, um that do things like
[52:08] powerful mathematics, that do even
[52:11] programming. You know, we have
[52:12] programming tools already.
[52:14] Um but, it doesn't mean that all of
[52:15] those things have to have a whole lot of
[52:18] autonomy or agency to do the things that
[52:20] they want. There's There's no reason, if
[52:23] you're
[52:24] um doing
[52:25] biology research and doing protein
[52:27] folding, that that system has to be able
[52:28] to also drive a car, or that your your
[52:31] autonomous vehicle needs to be able to
[52:32] do philosophy, or that the thing that
[52:34] you're using to solve Einstein's
[52:35] equations needs to flirt with your
[52:37] children. Like, it's There's no reason
[52:39] to combine all of these into one giant
[52:41] system. That's not what we do with any
[52:42] other information technology, right?
[52:45] We've got our phone full of apps, and
[52:46] each one of them does something.
[52:48] Instead, AI is like, "This is the
[52:50] do-everything app. Click here, and it
[52:51] just does all the things." Like, that's
[52:54] a weird direction to take software
[52:55] development, um and there's no reason to
[52:58] do it in that direction. So, So, the AGI
[53:01] framework is useful for saying there
[53:02] isn't just like the spectrum from not
[53:05] AGI to AGI to post-AGI, but rather there
[53:08] are lots of different directions that we
[53:09] can go. And, once we start thinking
[53:12] about what do we want the system to
[53:14] actually do? Like, what is the problem
[53:16] that we're trying to solve here? Then,
[53:18] it starts to feel very different.
[53:20] >> Is that really what we need from AI,
[53:22] right? Right. So, we can build AI
[53:26] agents, where the AI systems have the
[53:30] goals and the plans and sort of make
[53:32] things happen, and and their whatever
[53:35] they're trying to get done is the thing
[53:37] that happens in the world. Or, we can
[53:39] try to enhance human agency, where we
[53:42] decide what want what we want to have
[53:44] happen, and we have tools that help us
[53:46] do that. That's what we've done with all
[53:48] our previous technology.
[53:50] Now,
[53:52] the the way that AI is being developed
[53:54] now, people have noticed that it's
[53:56] mostly a tool, and they really really
[53:58] want it to be an agent, because there's
[54:00] more profit to be made in agents. They
[54:03] can replace more things. They're also
[54:06] just They're things that um
[54:09] agents can do that that
[54:11] tools can't, cuz the agents can do it
[54:14] all by themselves without human
[54:15] intervention. So, like if you if you
[54:17] want to um
[54:21] write a big program, there are you can
[54:24] there are some different paths you can
[54:26] go down. One is something that you're
[54:28] the programmer, and the AI system sort
[54:30] of helps you out. Or the other one is
[54:33] the AI system is the programmer, and you
[54:34] tell it sort of what to program, and it
[54:36] goes away and does a bunch of
[54:37] programming.
[54:38] And this is what people are leaning into
[54:41] a lot now. They want the AI system to do
[54:43] all of the stuff itself, and then just
[54:45] report back, yes, I'm done, here's the
[54:47] the product. Um and that can be very
[54:50] useful in certain ways, but when it the
[54:53] more we put the agency into the AI
[54:57] systems, the less of it sits in us.
[55:00] Right? So, the more the AI system is
[55:02] making decisions, the less we're making
[55:04] decisions.
[55:05] Um
[55:06] and so, I think we have to be very
[55:07] careful as we build uh
[55:10] systems that are very autonomous, they
[55:12] can do all of these things on their own,
[55:14] and have all the other human
[55:17] capabilities, the intelligence and the
[55:18] generality, what is the purpose of us
[55:21] anymore, right, if the system does it
[55:23] all? Now, if you build something that's
[55:24] autonomous, but isn't that smart, um
[55:28] or isn't that general, then it's
[55:30] something that we can still control.
[55:31] Like autonomous vehicles, they're not
[55:33] going to take over the world. They might
[55:34] put some Uber drivers out of work, and
[55:37] that is like an economic displacement
[55:39] problem that we have to think about.
[55:41] But, there isn't some giant risk to
[55:43] society from things that are just
[55:44] autonomous. It's that triple combination
[55:46] that's the problem. Is autonomous AI
[55:48] more dangerous than highly intelligent
[55:51] AI?
[55:52] I would say so.
[55:54] Um so, it depends how exactly So, a lot
[55:57] of people think of his intelligence as
[56:00] um
[56:01] you know, lump all these things
[56:02] together, which is why I think it's it's
[56:04] useful to separate them. But, if you
[56:06] have a system that is highly
[56:07] intelligent like
[56:09] um like, you know, a calculator is in
[56:11] some sense very intelligent in that
[56:13] the particular thing that it does, it
[56:16] does extraordinarily well and accurately
[56:18] and fast. So, in that sense, a
[56:20] calculator is very intelligent, but
[56:22] absolutely non-threatening, right? Now,
[56:25] when you think of something a little bit
[56:28] more general than a calculator,
[56:30] then
[56:31] um it starts to to exhibit more of those
[56:34] potentially dangerous qualities. So, if
[56:36] you have an AI system that can design
[56:39] new viruses, right? Um it can be very
[56:42] specific and only designs new viruses.
[56:45] It can be not that autonomous in that it
[56:47] really requires a human oversight, you
[56:50] know, to tell it to design a virus. But,
[56:52] it can still be pretty dangerous,
[56:53] because if you've got a very powerful
[56:55] tool in the wrong hands, that is a very
[56:58] dangerous thing. Um but, I think it's
[57:00] important to separate the kind of danger
[57:02] that we've always had. We've always had
[57:04] powerful tools. We've always had bad
[57:06] people. And we've
[57:08] It it's a problem to figure out how to
[57:09] manage the bad people having powerful
[57:11] tools. But, it's one we've been with for
[57:13] a long time.
[57:14] Building something that is fundamentally
[57:16] a sort of being, rather than a tool, an
[57:19] agent, rather than a tool, that has its
[57:22] own goals and purposes, that is
[57:24] something we don't have a lot of
[57:24] experience with. We have sort of some
[57:27] uncontrollable systems, and we've got
[57:29] like animals and so on, but um and we're
[57:32] used to not controlling those
[57:33] necessarily, but they're also only so
[57:36] powerful. All right, if you're you know,
[57:38] if you're in the
[57:39] in the cage with a lion, it's very
[57:40] scary, but otherwise lions are just not
[57:42] that big of a threat to humanity because
[57:43] they're just not that powerful compared
[57:45] to us. But if we absolutely had to
[57:47] control, you know,
[57:49] lions
[57:51] because they you know, if lions had
[57:53] nuclear weapons, that would be a
[57:55] different question. Like we don't know
[57:56] how really how to get lions exactly to
[57:58] do what we want. Even the most trained
[57:59] lions can like turn on their
[58:02] trainers. Um so I think it is a very
[58:05] different thing to build AI systems that
[58:08] are autonomous agents rather than tools.
[58:11] And as we make very powerful capable
[58:14] autonomous agents, we have to ask
[58:17] ourselves,
[58:18] A, is that the right direction? And if
[58:20] B, if we do it, are we going to keep
[58:22] control of those things? What does your
[58:24] Metaculus platform currently predict
[58:28] about the timeline for AGI?
[58:32] Yeah, interestingly it's it's
[58:33] Metaculus's prediction has stayed pretty
[58:36] constant over the last several years
[58:38] that
[58:39] that we will have So there are different
[58:41] definitions of of AGI and
[58:44] it's very contentious what is the right
[58:47] thing to predict.
[58:49] But according to the way that the
[58:50] prediction is made on Metaculus, it's
[58:52] sort of in the next few years
[58:55] that we're likely to have it by this in
[58:57] the sort of weak form and maybe a couple
[58:59] of years after that in a very strong
[59:00] form. So Maybe AGI is an ideology more
[59:06] than technology, right? It
[59:08] I think it is in some sense an ideology
[59:10] and but that ideology is driving a
[59:12] technology. So it So I think
[59:15] um it didn't have to be this way. Like I
[59:17] I do think that
[59:19] um
[59:20] there's a there's another universe very
[59:22] much like ours in which people did not
[59:25] get attached to the idea of AGI and
[59:28] let's replicate all of the things that
[59:30] humans do and and sort of race to build
[59:33] something that is super intelligent, but
[59:34] they just said, "Oh, look, we can do
[59:37] this thing called machine learning,
[59:39] where instead of having to explicitly
[59:41] program exactly what the system does, we
[59:43] instead
[59:45] figure out the goal and the system
[59:47] learns how to do that thing. You know,
[59:48] it's just a different way of writing a
[59:50] program to do machine learning instead
[59:51] of
[59:52] explicitly programming all the steps.
[59:54] And let's let's create really, you know,
[59:57] strong machine learning systems and let,
[59:59] you know, have them be tools that do
[60:02] stuff that was very, very hard to
[60:04] program explicitly, like protein
[60:05] folding. We never figured out how to do
[60:06] that, but look, the machine learning
[60:07] thing can do it." from DeepMind. Um
[60:11] There's a I think there's a world in
[60:13] which that was the the way people
[60:15] thought about AI and that we built lots
[60:19] of AI tools and, you know, there were
[60:22] some people out on the side saying like,
[60:25] "Hey, we should build AI that does
[60:27] everything instead of humans." And the
[60:28] people were like, "Nah, yeah, whatever."
[60:30] In the same way that right now there are
[60:32] off people saying like, "Oh, we should
[60:33] do genetic engineering to make ourselves
[60:36] to make like superhumans, like
[60:38] superhumans physically, superhumans
[60:39] mentally. We should build like the
[60:42] Uberman with genetic engineering." Like
[60:44] those people are out there, but
[60:45] everybody pays them no attention cuz
[60:46] like we don't want that. We don't want
[60:48] to go we don't want to go build
[60:50] superhuman with genetic engineering. So,
[60:53] we could have a world in which that was
[60:55] the way that people look at AGI and
[60:56] superintelligence.
[60:58] We're not in that world.
[60:59] Do you see any signs of an AI bubble? An
[61:02] AI bubble? It's an interesting question
[61:03] because I think there are there are
[61:05] people who think we're in an AI bubble
[61:08] because
[61:10] AI is basically It's all hype
[61:13] and the AI systems really aren't capable
[61:15] of being that powerful and so on.
[61:17] This is not true. It's something Um
[61:20] There is, I think, a potential that
[61:22] there is a bubble
[61:23] uh because of trust. So,
[61:26] the the way that AI systems will make
[61:29] huge the the huge amounts of money that
[61:31] they have been promising to their
[61:32] investors is if they can substitute for
[61:35] huge amounts of human labor.
[61:37] Now, there are various barriers to
[61:39] substituting for huge amounts of human
[61:40] labor. One of them is sort of raw
[61:42] capability. They have to be smart enough
[61:43] to do all the things. Um another one is
[61:47] sort of social barriers and regulatory
[61:49] barriers and things. And a third one is
[61:51] trust. So, if I'm going to replace my
[61:54] human workers with AI systems, those I
[61:57] got to be able to trust that those AI
[61:59] systems are actually going to do the
[62:00] right thing, and they're going to, you
[62:02] know, follow the the rules and the laws,
[62:04] they're not going to get me in trouble,
[62:05] and so on. And I think there that it's
[62:09] possible that because we're designing
[62:11] these AI systems in a way that is so
[62:14] focused on sort of raw capability,
[62:16] getting the highest numbers on those on
[62:19] those capability benchmarks, and not
[62:21] focused on building things you can
[62:23] actually rely on, that are actually
[62:25] reliable trustworthy secure safe
[62:28] that there's going to be a big barrier
[62:30] between when the systems, in principle,
[62:33] are like capable of doing all these
[62:35] things, and when they can actually
[62:37] be used in the economy. And if that gap
[62:40] it gets big enough, that trust gap gets
[62:42] big enough, then the money right might
[62:43] run out. So, if you're burning through
[62:45] huge amounts of money, which many of
[62:47] these companies are, and there's a long
[62:49] delay between when you have the
[62:51] capability and then when you can
[62:52] actually make huge amounts of money on
[62:53] it, you might run out of money. That's
[62:55] not because the technology is
[62:57] but it is because you've chosen the
[62:59] wrong direction to develop it in in a
[63:01] way that is not actually making what
[63:02] people need.
[63:04] >> I remember my interview with John
[63:07] Hopfield. His work
[63:10] was recognized in 2024 by Nobel
[63:13] Committee work on AI. Uh I ask him, "Do
[63:17] you regret something? If you could turn
[63:19] back time, would you do this?"
[63:22] I remember his answer.
[63:25] Um
[63:26] Just one thing. That neural network
[63:29] development
[63:31] took place within computer sciences
[63:36] rather than biological sciences. Hm.
[63:40] For decades, computer sciences, not
[63:43] biological sciences, have looked at how
[63:47] the human brain works.
[63:49] Where do you stand on this?
[63:52] Well, I think there's
[63:53] there there's some interesting
[63:54] scientific questions there, and then
[63:56] there are some social questions, too.
[63:58] So, I think there's
[63:59] there's an interesting
[64:01] sort of fact that when we create
[64:04] biological systems, if we create a new
[64:06] drug or a you know
[64:08] a new medical procedure or something, um
[64:11] those are very, very tightly regulated
[64:13] fields. So, in in biology and in
[64:16] medicine and things that have to do with
[64:17] living beings, we have a very different
[64:19] attitude cuz we sort of understand that
[64:21] we're tinkering with something that's
[64:23] sort of almost sacred. Like, there's a
[64:25] there's a real respect for biological
[64:27] systems, and there's a real obvious
[64:29] sense in which when you screw something
[64:30] up, you're doing real harm to something
[64:33] that feels it.
[64:35] And so, we we've got a very
[64:36] sophisticated regulatory system and
[64:39] systems of ethics and and so on when it
[64:41] comes to biological systems.
[64:44] All of that is largely absent when we go
[64:45] to computational systems, where we have
[64:47] this history of you just kind of throw
[64:50] stuff together. You It's just software.
[64:52] It doesn't really affect anybody,
[64:53] exactly.
[64:55] Um there's basically no regulation.
[64:57] It's a free-for-all. You just throw it
[64:59] up there, and like it's kind of anything
[65:00] goes.
[65:01] And the fact that AI, this this
[65:04] incredibly powerful new technology that
[65:06] is an
[65:07] uh
[65:08] whose goal is to, you know, replace
[65:12] human thinking and replace humans as
[65:14] decision makers and agents and so on,
[65:17] came out of the computer the totally
[65:20] unregulated anything goes software
[65:21] industry rather than the the biology
[65:24] side, um means it is being treated very
[65:27] very differently, right? Like you know,
[65:29] as I've as I said before, if you propose
[65:33] I want to
[65:34] um
[65:35] use genetic engineering to make a set of
[65:38] people that are super duper smart
[65:41] because that's really valuable. If we
[65:42] have super duper smart people, we can
[65:44] cure cancer, we can be more productive,
[65:46] we we can do all of these things. So,
[65:48] I'm going to go do genetic engineering
[65:50] to make super duper smart people.
[65:52] The regulatory agencies are going to say
[65:54] like, "No, you are not." The ethical
[65:56] boards are going to say, "No, you are
[65:57] not." The general population is going to
[65:59] say, "No, you are not going to do that."
[66:01] At least not with like a lot more
[66:02] conversation. This is not just like a
[66:04] normal thing to do to go build
[66:05] superhuman brains, even if they're
[66:08] useful. Even if they're economically
[66:09] valuable. Even if they could cure
[66:11] cancer. We just don't do that.
[66:12] >> end for our race, right?
[66:14] >> Well,
[66:15] the I mean, there are different ways
[66:17] like the
[66:19] we just treat it very differently. Like
[66:20] whether it's whether that's
[66:22] good or not, like it's very very
[66:24] different, right? Like we would not be
[66:26] having this conversation in the same way
[66:29] um if it was with with human brains.
[66:31] Again, there's a there's maybe another
[66:32] universe where all these giant companies
[66:35] are racing to build the most superhuman
[66:37] humans with genetic engineering. That's
[66:39] not the universe we're in, but there
[66:40] probably is one out there where there's
[66:42] like 10 giant companies that are racing
[66:45] to build the most superhuman people with
[66:47] their genetic engineering.
[66:48] >> Upgraded people that are like super
[66:50] intelligent people because they and
[66:52] they're all saying like, "We're going to
[66:54] cure cancer before the other before the
[66:56] other guys do with our superhumans." Um
[66:59] that's just a very different universe
[67:00] than the one we're in. So, why we treat
[67:01] one one way in our universe and the
[67:04] other totally different way or vice
[67:05] versa in the other universe? I don't
[67:07] know. It's a complicated question. Um
[67:09] but it is a very double standard and it
[67:11] is interesting to imagine um how things
[67:14] would be different if AI were developed,
[67:18] you know, more like biological systems
[67:19] and less like uh apps.
[67:23] Thank you very much for your time.