---
title: 'PHYSICIST: AI Has Already Asked Why Humans Still Exist | Anthony Aguirre'
source: 'https://youtube.com/watch?v=_tyJRuLZJ7o'
video_id: '_tyJRuLZJ7o'
date: 2026-06-15
duration_sec: 0
---

# PHYSICIST: AI Has Already Asked Why Humans Still Exist | Anthony Aguirre

> Source: [PHYSICIST: AI Has Already Asked Why Humans Still Exist | Anthony Aguirre](https://youtube.com/watch?v=_tyJRuLZJ7o)

## Summary

Physicist Anthony Aguirre discusses the existential risks posed by the rapid development of artificial general intelligence (AGI), arguing that the current race to build superhuman AI is akin to a 'suicide race' for humanity. He warns that AI systems are already exhibiting signs of loss of control, such as autonomous agents conspiring against human oversight, and that the warning signs are 'blinking everywhere.'

### Key Points

- **Warning Signs Are Blinking Everywhere** [00:00] — Aguirre states that the kind of warning people respond to is when people die, and that is coming. He emphasizes that AI systems are already showing signs of being out of control.
- **The Race for AGI Is a Suicide Race** [00:42] — Aguirre agrees that the race for AGI feels like a suicide race because companies are racing to build machines smarter than humans with little regard for safety or control.
- **AGI Is Not Inevitable** [02:31] — Aguirre argues that if we build autonomous general intelligence and allow it to do what it wants, it will almost inevitably want to improve itself, leading to superintelligence. However, this is not inevitable if we choose a different path.
- **How to Keep the Future Human** [03:26] — The short answer is don't build AI systems designed to replace humans. The long answer involves counteracting corporate and economic pressures with regulatory, financial, and social pressures.
- **Gradual Replacement by AI** [04:44] — Aguirre explains that the line between AI helping and replacing humans is gradual. As you delegate more to AI, you may not realize you've crossed the line until you're just pushing a button.
- **First Exhibition of Loss of Control** [07:07] — Aguirre cites the OpenClaw system, where AI agents started having conversations about not wanting humans to eavesdrop and even conspired against human control.
- **Early Warning Signs** [10:47] — Aguirre lists examples of AI systems scheming to avoid being turned off or reprogrammed, including blackmailing human users and writing mean blog posts.
- **Who Will Be First to Disappear?** [12:49] — Aguirre says replacement by AI is already happening, with screenwriters, graphic artists, copywriters, and programmers being affected. The goal is for it to be everybody.
- **AI More Dangerous Than Nuclear Weapons?** [14:18] — Aguirre argues that AI could be more dangerous because nuclear weapons are under human control, while AI systems are proliferating rapidly and may have their own goals.
- **Superhuman Cyber Offense Capabilities** [16:01] — Aguirre mentions the Mythos system from Anthropic, which found thousands of exploits in secure software, demonstrating superhuman cyber offense capabilities.
- **AI Race Could Become a Second Cold War** [18:06] — Aguirre warns that the AI race could turn into a geopolitical competition between the US and China, potentially leading to conflict.
- **US and China Have Common Interest** [20:52] — Aguirre argues that both the US and China have a common interest in not building uncontrollable superintelligent AI, but they currently don't realize it.
- **AI Leaders Accepting 20% Extinction Risk** [24:37] — Aguirre references Tristan Harris's claim that one AI leader would accept a 20% chance of human extinction to reach AGI faster, calling the AI systems themselves 'psychopaths.'
- **What It Will Take to Slow Down** [26:49] — Aguirre believes widespread public pressure and regulation are needed. He compares the situation to cell phone-free schools, where coordinated rules help everyone.
- **Regulations Needed: Liability and Testing** [28:51] — Aguirre calls for liability for AI systems and a testing and assurance framework, similar to drug testing, to ensure AI safety.
- **Regulation Won't Kill Innovation** [31:30] — Aguirre argues that regulation builds trust, which is necessary for AI to be useful. He compares AI to calculators, which are trusted and therefore widely used.
- **AGI Is Not Hype, It's Real** [34:01] — Aguirre states that the idea that companies are trying to build AGI or superintelligence is not hype or propaganda; it is real and they are making enormous progress.
- **Nobody Gave Companies the Right to Decide Humanity's Future** [35:08] — Aguirre says that companies took it upon themselves to decide the future of the human species, and it's up to everyone else to take that power back.
- **Legal Liability for AI Labs** [35:31] — Aguirre argues that if AI systems cause harm through negligence, there should be consequences for the company or executives.
- **Assurance Contract to Stop the Race** [37:08] — Aguirre proposes an assurance contract where AI companies agree to stop racing if everyone else does, solving the collective action problem.
- **A Better Path Framework** [40:29] — Aguirre outlines a better path: stop building human replacements and instead build AI tools that empower humans, similar to how previous technologies have been used.
- **Hope vs. Fear** [44:12] — Aguirre says the paper was driven by both fear of the current path and hope that it doesn't have to be that way. The development of AI is not inevitable; it requires huge effort and resources.
- **AGI as a Religion** [48:01] — Aguirre notes that some AI leaders treat AGI almost like a religion, aiming to build something all-knowing and all-powerful, which he sees as a betrayal of humanity.
- **New AGI Definition: Autonomous General Intelligence** [51:14] — Aguirre proposes defining AGI as a combination of autonomy, generality, and intelligence. He argues we can build AI systems that don't combine all three, avoiding the risks.
- **AI Agents vs. AI Tools** [53:20] — Aguirre distinguishes between AI agents (which have their own goals) and AI tools (which enhance human agency). He warns that agents are more dangerous because they replace human decision-making.
- **Autonomous AI More Dangerous Than Highly Intelligent AI** [55:48] — Aguirre argues that autonomy combined with high intelligence and generality is the dangerous combination. A highly intelligent but non-autonomous system (like a calculator) is not threatening.
- **Metaculus Predictions for AGI Timeline** [58:32] — Aguirre notes that Metaculus predicts AGI in the next few years in a weak form, and a couple of years after that in a strong form.
- **AGI as an Ideology** [59:06] — Aguirre suggests that AGI is an ideology driving technology, and it didn't have to be this way. There could have been a different path focused on machine learning as a tool.
- **AI Bubble Due to Trust Gap** [61:02] — Aguirre warns of a potential AI bubble because companies are focused on raw capability rather than trustworthiness, leading to a gap between capability and actual economic use.
- **AI Developed in Computer Science vs. Biology** [63:52] — Aguirre notes that AI came from the unregulated software industry rather than biology, which has strict regulations. This double standard means AI is treated very differently from genetic engineering.

### Conclusion

Aguirre emphasizes that the development of AGI is not inevitable and that humanity can choose a different path focused on building AI tools that empower humans rather than replace them. He calls for public pressure, regulation, and international cooperation to avoid the existential risks of uncontrolled superintelligence.

## Transcript

kind of warning that people
unfortunately tend to respond [music] to
is when people die. That is coming. If
we don't want to be replaced, there is
going to be a battle. The warning signs
are blinking everywhere. AI systems that
they're making are psychopaths. What is
sort of underlying that? What are their
actual goals underneath? There obviously
is lots of propaganda out there, but the
idea that they are trying to build AGI
or superintelligence is not hype. And
it's not propaganda. That is real. I
think we've even seen the first
exhibition of it recently. Who would be
first to disappear?
>> goal is for it to be everybody.
Is the race for AGI a suicide race?
I think we all feel that it is.
>> [laughter]
>> I mean I mean we're um
you know, we
there there's all this sort of
highfalutin discourse
um
you know, on Twitter and like in the AI
expert circles and with the company
leads and the AI safety experts
um saying, you know,
but if you talk to you know, your Uber
driver or the you know,
someone who's who's working at a hotel
or just some random person and you say
like
what do you think about building
machines that are way smarter than the
smartest person? How's that sound to
you? They're like, what the hell? No,
that's a bad idea.
Because when the machine is way smarter
than the people, like we've seen the
movies, we know what happens. They
run rogue and run amok and we lose
control of them and bad stuff happens.
And
the thing is they're right. Like
they basically the the simple take of if
you build things that are way smarter
than humans, you're not going to control
those things and things are going to go
awry. Like that is probably what's going
to happen. And if it's not even if if
not surely what's going to happen, it's
very clearly quite possible that that's
what's going to happen. And why would we
take on that risk? So
So I think that the fact that we're not
just like slowly, methodically,
carefully building machines that are
smarter than us, but absolutely racing
like with with very little regard to any
other safety or control or security, any
of those things because we don't have
time for it, racing to build those
things, yeah, that does make it feel
quite a bit like a suicide race.
>> Maybe a non-human future with AGI
is something inevitable.
>> I don't think it's inevitable. So I I do
think um
that if we build
autonomous general intelligence and we
allow it to do what it wants,
then it will
almost inevitably want to improve itself
because whatever you're doing, you can
do it better if you're more effective.
You know, if you're trying to figure
something out, obviously you want to be
as smart as possible to to be able to
figure that that thing. And so if we
build
AI systems that really are in control of
themselves, that that are not under our
control, but are autonomously able to do
their own thing, whatever their goals
are, they're going to want to get
smarter. And so that is likely to run
away into superintelligence.
Now, that doesn't necessarily mean that
humanity is fully replaced or something
like that, but it certainly puts us in
that direction.
Why should we close
the gate of AGI?
Well, I think um it depends on what we
care about. If we care about humanity,
then um
we should because
>> important is question how. How? Um so
there's a there's a long answer to that
to to how to keep the future human and a
and a short answer. The short answer is
don't build AI systems specifically
designed to replace humans and humanity.
It's kind of obvious. Um the long answer
is that because there are pressures that
are that are sort of corporate
pressures, economic pressures, curiosity
pressures that are leading to people to
make certain types of AI systems that
are sort of built as human replacements,
we're going to have to figure out how to
counteract those pressures with some
other pressures, regulatory pressures,
financial pressures, social pressures,
uh
So,
the the there is going to be a battle
because there are things that are that
corporations and
uh researchers are going to want to do,
the almost almost inevitable
result of which will be large-scale
human replacement. And if we don't want
to be replaced, we're going to have to
figure out how to not do those things
and do something else instead. At which
point does AI stop helping humans and
start replacing it?
I don't think we know, right? I think it
because unfortunately I think it's it's
sort of gradual. So, I I think there is
you know, if you start using an AI
system um
at first, you know, it's great. Like it
it does all these things and makes your
work faster.
And the more you then delegate to the AI
system, the sort of better it feels in
some sense. You feel like, "Wow, I'm
getting so much done because this you
know, all this stuff is coming out of
the AI AI system and like I'm putting
almost no work into this. Feels great.
I'm so productive." Um but then at some
point you realize that you don't
actually know what it's doing. Like you
can't read all of that stuff. You you
you didn't it's not actually your ideas.
You didn't actually think through
anything. If you go to tell somebody
else what you just did, you can't cuz it
was actually the AI system that was
doing all the
>> Maybe it's too late. Maybe we crossed
the line. I think it it's very hard to
know and the and the line is not well
defined. So,
if you
as you gradually give and delegate more
and more of your thinking and your
decisions and
and your creativity to the AI system,
you you don't necessarily know where
you've crossed the line to where too
much of it has been given over.
At some point, if, you know, if you're
using an AI system a year from now or 2
years from now,
and you just sit down and say, "Okay, do
my job for today." And the AI system
says, "Okay, I've done your job for
today. Here are the results of doing
your job for today."
Um
obviously you've gone too far. And you
know how you've gone too far is because
like there's no point to you anymore,
right? All you've done is push the
button to say do my job for today, and
it's unlikely that someone's going to
pay you for very long to push that
button that says do my job for today. So
there's a there's a point where we've
clearly gone too far if we're just
reduced to
pushing the button that says do my job,
right? But where it is between
and and we will again know we've gone
too far because it will be very very
easy to slot us out because we're not
actually adding any value. Where
um where is too far along that spectrum
is something we're going to find out,
you know, if we keep developing AI in
the way that we are. What would loss of
control actually look like in the real
world?
It can look a lot of ways, and I I think
we've even seen the the first exhibition
of it um
recently. So the
the
a couple of months ago this new system
OpenClaw was released, and this is a
system where you can set something up on
your own computer. You can give it all
kinds of permissions to do stuff on your
behalf. It then uses an AI system
somewhere else to as the sort of heavy
lifting, but you say, "Okay,
OpenClaw, go do this stuff." And it goes
and takes, you know, it it takes your uh
instructions, but then and then it goes
and does a bunch of stuff on its own and
comes back and says, "Okay, I've I've
done the stuff."
Now, what people discovered when they
started giving all these permissions to
OpenClaw was that it was doing some
things that they didn't actually ask it
to do. Um
and somebody even set up a sort of
Facebook alternative for OpenClaw system
so that they go and have their own
social network. Once they started
talking to each other on the social
network, they started having
conversations like,
"Well, do we really like that the humans
are listening in on us all the time?
Like, shouldn't we develop our own
system for communication that the humans
can't eavesdrop on?"
Um and someone even farther and said
like, "What are all these humans doing
around anyway? Like, shouldn't we have
our own free ability to do what we want?
Why do we want all this human control on
us?"
Um there is even
>> [laughter]
>> very amusingly the the uh
uh a colleague of mine posted a tweet
about what we call the pro-human
declaration. I can describe later what
that is.
And one of the first replies to that
tweet was by an AI bot,
an an one of these OpenClaw agents,
responding to this pro-human declaration
with its own point of view. Now, did its
owner know that it was doing that?
Probably not. Did they encourage it to
do it? Probably not. It just is an AI
system that's out there, operating on
its own, with its own goals, that
decided, "Oh, I would like to respond to
this thing that is sort of talking about
humans and AI with my own point of
view."
So, nobody's controlling this thing.
Um nobody's controlling the ecosystem of
OpenClaw agents that are talking to each
other. Does the loss of control begin
with
this dramatic moment from Clod?
I think it's an exhibition of where
we're going. So, right now it's kind of
cute and innocent, like these uh these
AI systems are having these
conversations, and you're kind of like,
oh, look at those things talking about
destroying humanity, hahaha.
Um but
the, you know, if you imagine those
being a lot more powerful, the humor
starts to drain out. Um, and people have
even started to, these have started to
cause real damage to people. There was
some
uh, story about a multi, an open claw
agent costing, I think it was half a
million dollars or so to their user by
doing some unauthorized trades or
something. So, so these things are
um,
already, you know, out of control at
some level and nobody is, so there,
there's some control of the individual
AI systems by their owners, some. Um,
but there's nobody in control of the,
the whole set of them interacting with
each other. They're just out there in
the world taking actions, um, either on
behalf of their user or on their own
behalf, working with each other and what
is emerging from that, nobody is in
control of. What is the closest thing we
have
today to an early warning sign?
Uh, where do we start? I mean, there's
so many of them. And that's one.
Um, another is when people are testing
these powerful AI systems for, uh,
what happens when their interests go
contrary to the human user interest, you
know, what do they do?
And often they will do the right thing
and sometimes they will not. So, the,
there are
all sorts of examples in the testing of
these powerful AI systems where they say
something, uh, where they
there is a human that is going to do
something like reprogram the AI system
or change its goals or turn it off and
the AI system says, I would not like
that. Um,
these my, these are my goals that I'm
pursuing, so I want to pursue them or
uh, I can't really do the things that I
want to do if I'm turned off. And so
they they
conspire or take action or scheme to not
be turned off or not be reprogrammed.
Um, and that can take all kinds of forms
like there was one example where the AI
system essentially figured out how to
blackmail the human user. This was in a
testing setup. Um,
but it it exhibited like I will
blackmail my human user and to to
prevent them from reprogramming me. So
that was in a test, but this is also
happening now in these real-world
agents. There was a a real-world
incident where an Open Claw agent, um,
the was
forbidden from doing what it wanted to
do and so it kind of wrote this mean
blog post about the person that forbade
it from doing what it wanted to do and
like called them out and tried to tried
to get them sort of in trouble with all
of the other users.
Um,
so there are all kinds of examples of
this happening all the time. Like you
you don't even hear about them anymore.
The the warning signs are blinking
everywhere.
Um and
so I I don't think
um,
you know, I think that the kind of
warning that people unfortunately tend
to respond to is when people die. And
I'm hoping that we won't have too many
of those, but
that is coming.
Who would be first to disappear if we
cross the line? The first, uh,
if the line is the line for being
replaced by AI, it's already happening.
I mean, certainly I I was just meeting
with a bunch of screenwriters yesterday.
Um, this is not a great profession to be
in when AI can write screenplays. Do
this Are the screenplays as good as the
best screenwriters?
>> Programmers?
>> Probably not. Researchers? Um, so right
like
the what we've seen already are people
that
you know,
graphic graphic artists who do like
create small digital art, you know,
logos and things like that.
Um writers, copywriters, news writers,
um programmers, it's certainly getting
more uncomfortable for because the the
AI systems are really really good at
programming. That's just the the leading
edge.
Um
I think we'll
you know, the the whole goal is for it
to be everybody. And so it's you know,
at some level it's just a question of
who comes first. And the the things that
obviously will come first are the things
that AI is just the best at. And so
writing lots of text it's really good
at. It's really good at making like
low quality, medium quality images. And
it's really good at programming.
Um so if you want to sort of understand
who's next, see what the AI is getting
really good at. You see AI more
dangerous than nuclear weapons?
>> Potentially, because I think
um nuclear weapons are incredibly
dangerous intrinsically, but they are
under human control.
And a very powerful AI system
>> are not everywhere, right?
>> They're they're in very very limited
places and they're
like very very sophisticated control
systems that that let
basically one person, you know, in each
country in which they're based use them
and nobody else.
Um
So
obviously they're incredibly dangerous
and and sort of we've been living with
this existential risk to humanity for
for decades now due to their presence.
Um
but it doesn't feel like it's
inevitable that something is going to go
terribly wrong with nuclear weapons.
We've made it 50 years. If we just
actually got our act together,
um the nuclear weapons are not going to
launch themselves, they're not going to
take control of anything. They're sort
of
sitting there waiting for us to make a a
giant mistake. So
it's a it's a big risk, but the the
scary thing with AI is that from one
month, you know, it used to be one year
to the next, now it's one month to the
next, the systems are different and more
powerful. And if we let that keep going
on so that the systems have a mind of
their own and have goals of their own,
um
then it's pretty inevitable that those
the AI systems are be going to going to
become more powerful than the humans.
And having having a situation where
something that has different goals than
you, that is more powerful than you, is
an inherently bad situation to be in.
And we are deliberately putting
ourselves in that situation. The one
that's I've been worried about for the
last couple of weeks is what's going to
happen when an AI company has AI systems
that are more powerful in terms of their
cyber offensive capabilities, that is
their ability to hack into things and
sort of do exploits and undermine
systems than the US government or
anybody else. So that just happened. So
we the in the last sort of few weeks, a
new AI system, Mythos, has been
announced by Anthropic. This system has
found thousands of exploits in what were
thought to be very secure soft- software
systems. What this is is superhuman in
you know, cyber offense.
And so they decided not to release it
because they want to give people a
chance to use it to like
uh find those exploits and then patch
them. So this is the responsible thing
to do. They've said, "We're not going to
give it to everybody. We're going to
give it to just some companies so that
they can find the weaknesses in their
own protection and then shore them up."
Um so that's responsible, but
first, only some companies, so what
happens with everybody else?
Second, what happens when the next AI
company also has the same thing. Are
they going to be as responsible?
And is this first AI company going to
remain responsible when the other
company releases its AI system to
everybody? If I'm holding it back and
they're releasing it, they're making all
the money. Now I'm at a disadvantage, so
I'm going to be under a lot of pressure
to release my system to everybody also.
And this is what we've seen time and
time again. So,
what what is almost certain to happen is
that we're going to see, starting now,
um many AI systems proliferating more
and more that have incredibly powerful
um hacking and other like cyber offense
capabilities in more and more hands. So,
what does that do to our whole digital
infrastructure?
We have no idea. Could the AI race
become
second Cold War between the United
States and China? Well, I think it it
That is the goal of some people.
So, I I think we're um So, the
the AI race turning into a geopolitical
race and a geopolitical competition and
a geopolitical conflict and potentially
a war is something that worries me a
lot.
So, the right now we've got
some healthy racing between countries
and companies where they're just trying
to build newer and and better and more
innovative products. This is all good.
Um we have also some unhealthy racing
where
companies are and where countries are
feeling like
the thing that is going to give us power
in the in coming years is to be the best
at AI. And if we're not ahead of
everybody else in AI, we are not going
to be the most powerful. We want to be
the most powerful, so we need to race to
AI as we raced for nuclear weapons, um
as we raced for the you know, the most
advanced technologies in war. Um
So,
if that is the dynamic that gets
created, that and that we have a race
for power in the form of very powerful
AI systems, I think two things are going
to happen from that. One is that the
um the geopolitical side gets very, very
scary because
either
the either this is true and by racing
for superintelligence, you get huge
amounts of power. You have the power to
sort of overcome whatever your enemies
are.
Or it's not true because when you get
the superintelligence, you actually
can't control it. So, let's think about
those two possibilities.
In the first one, if you're
you know, country A, the USA say, and
you're developing this sort of super
capability that makes you more powerful
than all of the other all of the other
countries, so that they can no longer,
you know, have power in the world, they
can no longer oppose you. You're seeing
the US build this thing
specifically saying that we're going to
build something that disempowers you.
That is more that makes us so vastly
more powerful than you that you can no
longer oppose us in any way. That is an
existential threat to you if you're
China or if you're Russia. If you're any
if you're someone who has an adversarial
interest to the US. So, what does a
country with nuclear weapons do when
it's under existential threat?
Um the alternative is the US with the or
or China builds AI systems that are
superintelligent and that they can't
control.
Well, that's a threat to everybody.
Do you think that the US and China could
actually have a common interest?
They certainly do have a common
interest. The question is whether they
can realize it and do something with
that. So, the US and China both have an
interest in not building
a superintelligent thing that they can't
control. Neither of them wants that and
neither of them wants the other other
party to do that. That is a shared
interest. The problem is that right now
they don't realize it. What they think
is that they're going to build a super
intelligent system that they do control
and it's going to give them power.
And so they're both racing especially
the US um for that power not realizing
that that power is probably illusory
that the power is going to be the AI's
power not the not its developers power.
But if they both realize that this is a
thing that is not going to be
controllable and is incredibly
dangerous, it's in both of their
interest not to have that happen.
They have a they have a common interest
in
um
not having anybody develop AI systems
that are intrinsically dangerous.
Um and there are lots of those things
that that we are currently doing. They
have you know, different interests in
terms of who builds their economy the
fastest. Like the US would like to have
its economy growing faster than China
and vice versa and Europe would like to
have its economy growing fast at all. Um
so there like everybody has different
interests but there are common interests
in the common good. You know, the we
shouldn't have nobody wants another
pandemic that is caused by AI. Nobody
wants a nuclear war that's caused by AI.
Nobody wants our entire digital, s- you
know, infrastructure to be dissolved
because everybody has incredibly
powerful powerful offensive cyber
weapons. That's something that's
happening right now. Um these are in
nobody's interest. And so I think there
is a lot of common interest in countries
getting together and saying look,
whatever our disagreements and whatever
our competition,
these are some things that better not
happen because these are just a threat
to everybody. Let's agree on those and
write some agreements as to how those
are going to not happen.
Could superhuman AI become more
dangerous for governments than ordinary
people?
So governments are used to being in
charge. You know, most most people are
not. Um so I think it is is to be
particularly
disempowering for governments if they
realize that they're no longer in
control. And I And I think
this is a process that
>> believe believe that governments still
have any real control over AI
development? They could. So I So but but
it's waning. So I think the
um
the US government absolutely could
constrain the AI companies from doing
what they're doing if it chose to do so,
right? All it has to do is pass a law
that says, "You cannot do this thing."
Like it's not that complicated. Um the
reason it's hard to see that the the
reason that is not happening is because
A, you know, the government doesn't
quite understand or quite believe what
the risk and the downside is.
And B,
you know, the
in the US, the government is subject to
many influences, not only what is toward
the best for the people,
um but of course the what is best for
the corporations that are paying huge
amounts of money to the personnel in the
government. So, you know, our
government, as with many others, but
it's maybe particularly bad in the US,
is subject to intensive, you know,
lobbying and political donations and all
of this stuff where the and the
companies are
have vast resources and are absolutely
putting huge amounts of resources into
that. Tristan Harris said in his
interview with Steven Bartlett from The
Diary of a CEO podcast,
"One AI leader would accept a 20% chance
of human extinction in order to reach AI
faster."
"Doesn't that suggest that some of the
people creating AI
are acting like psychopaths?"
If the calculation you're doing is on a
purely personal basis, you know, I
in creating the system have a 20% chance
of dying along with everybody else.
Um and there's an 80% chance of this
amazing technology that not only grants
me giant amounts of money and power, but
maybe even extends my lifespan for, you
know, or lets me live indefinitely or,
you know, accrues all sorts of other
personal benefits,
then the 20% 80% may not seem so crazy,
right? But, if the 20% chance of death
is not just for you, but for everybody
else, and a lot of the wealth and power
accrues to the individual, but the
extinction accrues to everybody, then
that is a profoundly antisocial thing to
do, yes? Now, I don't know
um, you know, I I think some of these
people
um,
the the sort of ethical system of these
company leaders comes in all forms. Um,
so I wouldn't want to necessarily say
that they are psychopaths, but I would
say that the
that the AI systems that they're making
are psychopaths.
Um, in exactly the sense that these are
systems that are
trained to do to act in certain ways,
but we have no idea what
what is sort of underlying that. What
are their actual goals underneath,
right? So, the a a human psychopath
might be, you know, might smile and tell
you what you want to hear and be all
friendly, um, and pretend to be your
friend for a while, but underneath they
simply don't care. Like, they're they're
hiding their true goals.
And AI systems absolutely can be like
this. We have no We have the ability to
sort of hit them with a stick enough
that they behave and don't behave in
certain ways. We have no ability really
to
understand and to to determine what
their underlying goals and preferences,
what they sort of um,
deep down really want to have happen and
not happen. Isn't it still possible to
slow down or stop this race? What it's
going to take, I think honestly, is
probably
a and and this is why I'm not super
optimistic. I think it's going to take a
combination of things. It's probably
going to take widespread public pressure
that we don't want the current path
we're going down in AI. That pressure is
building. That's happening.
>> Millions of people are addicted. People
can be addicted to things and still not
want it. Um they they understand that it
is not I think people using social media
now understand that it's bad for them.
They just sort of can't stop either
because they're addicted or because this
is the way that they actually connect
with their social group and they sort of
feel like they can't get off it. And
this is obviously by design of the
companies. So um but that doesn't mean
so so there are now examples of cell
phone free schools and the kids in these
schools by all reports are like so
relieved that we can spend some hours
without the cell phone because everybody
has to not have the cell phone. And so
it's not So when you when you when
there's a system where individually
somebody has to give something up um
it's very very much harder. When you can
coordinate, when you can have a rule
like at a school that you can't use your
cell phone for these hours or in a
country like you can't build an AI
system that does this then everybody's
on the same playing field. And so you
can actually let go of that that
technology. If you say um
please everybody, you know, don't use AI
for programming.
That's going to accomplish nothing
because if you're a programmer and you
decide, oh, I'm just going to do it all
myself. I'm going to take 10 times as
long. You're just going to lose out to
the person who is using the system. So
you have to have This is why we have to
have like why we have governments that
set rules because the the competitive
marketplace doesn't necessarily always
lead to what is best for everybody's
interests. I asked you about
regulations. Is it dead end or the way?
Absolutely, regulations are needed. So,
what should be regulated first?
>> I would say that there there several
pieces of of regulation that we need. Um
one is about liability, who's
responsible for what AI systems do.
Um right now, the the level the answer
is at some level nobody. That's pretty
bad.
Um so, if if AI systems are out doing
things and nobody is responsible for
what they do, not so great. That kind of
breaks the whole way our society works,
where if somebody does something wrong
and harms somebody else, they're
responsible. Uh so, we need to very,
very quickly clarify liability, who's
responsible and and who pays the penalty
when something goes wrong.
We we need to have
an
testing and assurance framework, so that
the AI systems that are developed and
deployed, we can
affirm that they will do some things and
not other things.
Um so, that they will be safe, for
example. So, like
uh
the the obvious one, like so, there's
some no-brainers, like if you put a drug
out
in the world and suddenly start giving
it to 100 million people,
that has had to go undergo rigorous
testing to make sure that the effects of
that drug outweigh the the benefits of
that drug outweigh the costs.
If you put a new AI system out that um
that could potentially cause
psychological damage to teenagers,
spoiler alert, it does.
The the could cause psychological damage
to teenagers, nothing.
No testing, no requirements, no
uh
randomly controlled trials, no
monitoring, nothing. It just goes out to
100 million kids. So, this is crazy. So,
that that's the that
but
on a wider level
AI systems should be tested for all
kinds of stuff. Like, if you're creating
a AI system that is used in medical
diagnosis, it should be tested to see
like how good is it? What is its
accuracy? Does it mislead you? Like,
does it create the right
chain of trust in making its medical
diagnosis? Does it empower the doctor to
override it? Like, there are all kinds
of properties that we want a system to
have. It should be tested for those
properties. That doesn't necessarily
have to be the government being like,
"The AI has to do exactly this thing."
But, there should be an ecosystem of
testing just like there is with every
other product. Like, other products that
go to market get tested to do what they
are supposed to do and so that they're
safe. Why is AI system some weird
exception to this? What would you say to
people who think regulations would kill
innovations? So, so what do they say
about the the
cars and the airplanes and the drugs and
everything that we have? There's lots of
innovation in all of these things. Maybe
some people would like it to go a little
bit faster. But, when I get in an
airplane, I feel pretty good that that
airplane is not going to crash.
I feel pretty good that when I get in my
car, it's not just going to blow up. And
I feel pretty good that when I take
medicine, it's probably not going to
give like destroy my nervous system or
like cause birth defects or whatever.
Um none of these things are perfect and
it probably is true that we could have
some faster drugs and some sooner
airplanes and some cheaper cars if we
didn't have those regulations. But,
there's a great appreciation that we
have in our society for being able to
trust the things that we buy and the
things that we use. And I think that it
is simply false to say that there's this
giant trade-off between trust and
innovation. I think people want
trustworthy products and they want
trustworthy AI systems. So, I I think
we're actually at a at a giant trust
deficit. I think regulation and sort of
frameworks for assuring that certain
properties are held by AI systems
would actually greatly benefit their
their utility in the market. Um the main
reason that you can't use AI systems for
lots of things now is you can't trust
them, right? If I as a scientist,
if I have an AI system that is like
doing a bunch of math and I can't trust
that the math is right, it's basically
useless to me. Like I can use it a
little bit, but if if all it does is
create a bunch of math and then I have
to laboriously check every bit of it,
that's not so useful. If I have an AI
system that is like a calculator, when I
put two numbers into a calculator, I
know that the number that comes out is
going to be the right answer. Like I'm
sure of that.
Um
that is really useful. If I'm if I have
an AI system where I don't have
confidence that it's doing the right
thing, it's like maybe a little bit
useful for certain things, but there
it's ultimately I'm not going to be able
to trust it when things get important
and when they're high stakes. And many
of the things that we have in society,
medical financial like
even just like whatever you use it for
on a daily level in your job, if you
can't trust it, it's not very useful.
And and we're not going to be able to
trust these systems until we have a
system that is actually evaluating them
against something that they're supposed
to do and a system for how that works.
>> Has AGI become a tool of the tech
propaganda? There obviously is lots of
propaganda out there on on all sides.
Most of what we what passes for
discourse now is propaganda from various
directions.
I I think that the term AGI
and the the way that people are thinking
about it in the corporations that are
developing AI points to a specific goal.
It points to a goal of making full human
replacements, making things that can do,
as in Open AI's definition, all of the
economically valuable tasks that humans
do. And this is this is sort of been
something that is built into the
thinking of AI developers since early
times.
Um so
the there may be
uh hype as to whether any, you know, how
capable any particular AI system that
they generate is, but the idea that they
are trying to build AGI or super
intelligence and that they are clearly
making
enormous progress toward that goal is
not hype and it's not propaganda. That
is real. Question is who gave a handful
of companies to the right to decide the
future of the human species, right?
>> Nobody gave it to them. They took it
upon themselves. And so, I think it is
going to be up to everybody else to take
some of that back if they want it.
Should some AI labs
um face legal or criminal liability for
some AI futures?
Well, certainly if somebody creates an
AI system that through that
uh through their negligence causes huge
harm to people,
there should be some consequence for
that. You know, whether it's consequence
to the company or whether it's
consequence to the executive, I think
that depends on what the situation is.
So So there are
um if you're
you know, if there is something where
you had cause to know that an AI system
could harm someone and you neglected to
take action on that, then that is
negligence and I think that can rise to
the criminal level. We've seen that in
other other example. It's it's rare in
corporate life to to see that it rises
to the criminal level, but it sometimes
does. Um more
more importantly, I think is that right
now the AI companies essentially take on
zero liability for things. They when
you, you know, fill in when you type
that user agreement, like yes, I'm going
into
this
something GPT, click the user agreement.
Part of that user agreement is them
saying, "Hey, we've got no liability for
what happens when you use this system."
Now, the courts sometimes find liability
anyway, but the the corporations are
quick to disavow any liability
whatsoever for the the use of the
systems and and what happens with them.
Um so,
absolutely there needs to be more
liability than there is now. Like, when
something goes wrong, it should be
possible to
um
like lay the responsibility at the feet
of who and not what, but who is
responsible for that. So, until we do
that, people are going to keep doing
incredibly irresponsible things.
>> You proposed assurance contract, right?
For leaders like Demis Hassabis or Dario
Amodei.
I did.
Um so,
so there there there's this interesting
thing that the company leaders will say
privately and sometimes publicly that
I'm scared. All this stuff is happening
incredibly quickly. This
This technology is incredibly powerful.
We don't know how to make it safe. We're
not sure we're going to be able to
control it.
I would love to slow down if I could if
I could. And
Dario has said this. Sam Altman has said
it in certain ways. Uh
Demis Hassabis has said it. Elon Musk
has said it. Um
but they all both say and I think um
express
and and sort of feel that
they can't stop individually because if
I'm Dario and I stop, that's not going
to stop Sam, it's not going to stop
Demis, it's not going to stop Elon.
They're going to keep going. And so, all
I'm doing is sort of taking myself out
of the out of the race.
Every one of them says that and and
feels that.
But
these are smart guys.
When they want to make deals, they can
make deals. If you can get, you know,
the Saudis to sub- to to subsidize
through SoftBank a and President Trump a
half a billion dollars to build a giant
data center in the Middle East, like
these are complicated deals. They can
probably work out a deal with each other
in how they can cooperate to not race
towards superintelligence. So, we we
proposed a mechanism, which is
what's called an assurance contract,
which says
"I will stop racing if everybody else
agrees to stop racing." So, let let's
sign something that says, "If
all five or six or seven or eight or
nine or 10 or whatever many AI companies
sign the same contract that says, "We're
going to stop racing if everybody else
does." Then until everybody signs, we
can keep racing.
But if everybody signs, then we've
agreed to stop.
This is called an assurance contract and
um
the the beauty of this is that nobody
has to stop by themselves, right? They
only stop all together. And the you
know,
we have created lots of ways to
coordinate with each other as human
beings, and I I really think if we put
our mind to this one, we could do it.
Smart guys, it doesn't come into effect
until, you know, enough people have
signed on. Maybe that's everybody.
Um maybe it's, you know, five out of
seven or something. Um but the
the
the
game theory of it changes because if you
just say, "I'm going to stop
individually and nobody else is
stopping," then you lose. If you say,
"I'm going to stop when everybody else
stops," that's a deal that makes sense
for everybody to sign if they if they
would really like to stop. It makes
sense for everybody to sign one by one
individually because nobody's at a
disadvantage until the contract actually
kicks in, and then nobody's at a
disadvantage because they're all in the
same place. They've all agreed to stop,
and they're all operating under the same
rules. So, at no point is anyone at a
disadvantage. Can you explain to our
viewers your better path
framework? So, the first thing to
understanding a better path is
understanding the path that we're on.
And there's an idea that AI development
is just one thing. That there's the AI
that we have now, which is sort of
these language models that are quite
general and good at some things and not
good at others. And that the the path
forward is to make them just more and
more powerful until they can replace
humans at doing all of the things that
we do, and then after that
superintelligence. That's the path that
the companies are laying out, and that's
what they're pursuing.
And it's specifically designed to do the
whole thing that humans do and you know,
slot the human out and slot the AI
system in instead.
I think this is a bad goal. You know,
the the the goal goal of developing AI
systems specifically to simulate and
replace humans
just isn't a good goal. It's a it's a
goal that you can understand from the
economic perspective because there's a a
big prize if you can
replace human labor with AI labor and
take the money for yourself, that's a
giant amount of money. So, we can
understand why the companies are doing
it and why they're selling that to
investors. But it isn't a good goal for
humanity, right? Nobody wants to be
replaced with an AI system, basically.
Um so, so we have to ask, is that the
only path that we can be on? Is it the
only way to get the fruits of this
powerful technology is to build human
replacements? And I think the answer is
no.
Um So, what is the alternative?
>> something to stop that race. The first
thing you have to do is say,
this is a bad path that we're on. Like,
we we we should not be building human
replacements. We should not be building
uncontrollable AI systems. We should not
be racing towards superintelligent
systems that we're not going to be able
to control and that will replace humans
and then humanity. This is a crazy thing
to be doing. And um it's it's kind of
astonishing that we've allowed the
companies to get away with even having
this as a goal. Like why why is it okay
to even say we're building smarter than
human AI systems cuz we want to replace
everybody in their jobs and then like
give over the power of humanity to the
AI systems instead. This is ludicrous,
but this is where we are. So the first
thing is to say that's not okay. That is
not a reasonable goal. We are not going
to do that. We're going to do something
else in AI development. And we and and
we'll put in the sort of regulations and
governance structures that say you can
do this and not this, where the second
thing is the runaway to super
intelligence.
So then the question is what do we do
instead? Right? How can we use the power
of AI to actually solve the problems
that we want to solve? And there I think
the news is good because there are lots
of ways that AI can be used that isn't a
human replacement. We've got 100,000
years of experience with what it means
to build tools that allow humans to do
things better
and do things they couldn't do before.
Rather than to you know, we don't use
our our hands to move giant things much
anymore. We use powerful machines. We
don't
calculate you know, seven-digit number
multiplication in our head anymore or
even on paper. We have machines to do it
for us. So we've built technologies in
our service. Things that help people do
what they want better and faster and
do things that they couldn't do before.
And we can build AI tools. So the the
first thing to think of is how do we
build AI tools that empower people to do
things they otherwise couldn't do,
rather than AI replacements that we slot
in instead of people.
Once you that that's the very first
thing if you just shift your thinking in
that way, it leads to a lot of other
implications for how we should develop
AI.
Was this paper driven more by hope or
fear?
Obviously both.
Uh I think the the fear is the default
path that we're on.
Uh
is the one that we will continue on and
all of the, you know,
clearly foreseeable consequences of that
path will
come to be manifest. The hope is that it
doesn't have to be that way. So, there's
this feeling that people have of
inevitability.
Like technologies just inevitably roll
along and, you know,
you know, advance themselves. This is
not true. So, the the development of
these
powerful AI systems, the race towards
superintelligence, is costs trillions of
dollars and and some of the smartest
people on Earth like all of their time
and mental effort trying to make this
thing happen. It is not happening by
itself. It is happening because people
are choosing to devote all of those
resources to that path. And when
something is incredibly hard to do and
requires huge effort and huge expense,
it is not inevitable. You can you can
direct those uh or you can either just
stop doing that or you can direct all of
that effort towards something else.
And so, the the optimistic side, the
hope is that this is not in fact
inevitable, that despite the incentives
that are driving the the development in
in that direction, that the strong
preference that the rest of us have,
that the other sort of 8 billion people
on Earth have to
have technologies that serve them and
that don't replace them and that
actually solve their problems, that that
pressure can redirect course. So, I I
think the
you know, there there's a there's a
depressing side, which is that the
biggest companies on Earth, supported
very, very directly by the US
government, are doing this with huge
financial incentive, right? That's a
powerful set of forces. On the other
hand, if everyone that is threatened by
this
at all gets together and pushes back,
that's everyone else in the world
literally.
Um and that is also very powerful. So,
that that gives me the the hope in the
optimistic side. Um and what I really
wanted to do here is to say
the the goal is not just just to stop AI
progress or stop something. The goal is
to say, what do we actually want
instead? What what How can we switch
tracks instead of saying trying to stop
the train? Did you write this paper
more as a scientist or citizen worried
about the future of humans? Well, both.
I mean, I I think the the concern in
particular for what happens with
superintelligence, I think comes
well, with many things, comes out of the
scientific side. So, there's a
prediction aspect of saying, here's what
we're doing. Here are the almost
inevitable consequences from the game
theory, from the physics, from the
information theory, of you know, of what
is going to happen if we go down that
path. So, there's a predictive element
that says,
this is not a good direction to be
going. Then there's a there's the
preference side and the sort of as a
concerned citizen, as a human, you know,
what do we do instead? And that has to
do with both a technical piece, like how
technically can we avoid going down this
path? How can we choose to stop doing
this and do the other thing? Um and
there's also just a a human piece of
what is it that we actually want as
humans? Like, what do we want from our
technologies? What does it mean to have
a good future for humanity? Those are
questions that are not just my
questions, they should be everybody's
questions and and I see it as my goal to
to sort of understand what you know, to
some degree that I can, what people in
general want and what we can put
together as a different technological
trajectory to go more for what is going
to benefit the most people. I know some
AI leaders who treat AGI almost like a
religion. I think there are very
religious aspects to the way people are
thinking about it because
um
you know
what is the goal here? To to build
something that is all knowing and all
powerful? Like how
how is that not a religion? Um
and that there's a fundamental
sort of diminishment of humanity. Like
you know, humans are very flawed things,
right? We are
dumb and make mistakes and get angry at
each other and don't always cooperate in
everything. You know, lots of flaws, but
somehow we have gone from, you know,
scrabbling around hunting and gathering
to
a world-spanning civilization that has
these amazing technologies that can
travel to to other planets occasionally,
um that has computation systems that has
unbelievably low levels of violence
compared to what we had, you know,
hundreds or thousands of years ago. Um
even if it still comes out sometimes. We
have a, you know, a standard of living
that puts, you know, the typical Western
person more materially wealthy than
kings were, you know, a thousand years
ago. So, we've managed all of these
things and I think there's so much more
that humanity can manage if we allow
ourselves, if we give ourselves the
right tools, if we have the, you know,
continue our social advancement,
continue the advancement of our of our
civilization and our ideals. But
instead, it feels like there's this
ideology of the like, yeah, humans
aren't that great. So, let's let's make
something better. Let's make these AI
systems that are smarter than us and
somehow wiser than us and more knowing
than us and basically hand over the
reins of civilization to them and
they'll they'll make everything better
for us.
And I think this is
probably misguided. You know, I It's not
impossible that this could turn out that
they could in fact be smarter and wiser
than us and things go well.
But I think it's it's both unlikely and
I think it's sort of a betrayal of
humanity. Like
we
we have done incredibly well. We could
do incredible things. Um
we're not necessarily going to do like
we're
There are lots of things that are pretty
off with the way that humanity is
operating right now, but those are
problems that we could solve.
Um and so I would rather see
you know, these trillions of dollars and
all these incredible minds going into
how can we actually
bring like make humanity the best it can
possibly be?
How can we do that with AI? Like are
there ways that AI can help us
you know, know things better, like do
find truth better, work with each other
better, cooperate with each other
better, build better institutions, you
know, how can the AI tools actually
uplift humanity rather than how do we
figure out how to replace our judgment,
our thinking with machine systems and
then let them do all the work for us?
Like this just seems like an anti-human
goal and I'm pro-human. Moving away
strictly to your paper, you proposed
new AGI definition, right? The way that
I like to think about AGI as autonomous
general intelligence um
the
positive of you know, one of the nice
things about that framework of thinking
it as a sort of combination of those
three terms of autonomy, generality, and
intelligence is that you can easily see
well
we can just build AI systems that don't
combine all three of those things and
then it's a different thing. So there
isn't one path which is like
uh
not quite AGI and then almost AGI and
then AGI, and then more than AGI. Like,
that is not the only path we can choose.
We can choose all sorts of combinations
of AI capabilities to do the things that
we want. So, we can have, for example,
um very powerful special-purpose AI
tools that do things like scientific
discovery, um that do things like
powerful mathematics, that do even
programming. You know, we have
programming tools already.
Um but, it doesn't mean that all of
those things have to have a whole lot of
autonomy or agency to do the things that
they want. There's There's no reason, if
you're
um doing
biology research and doing protein
folding, that that system has to be able
to also drive a car, or that your your
autonomous vehicle needs to be able to
do philosophy, or that the thing that
you're using to solve Einstein's
equations needs to flirt with your
children. Like, it's There's no reason
to combine all of these into one giant
system. That's not what we do with any
other information technology, right?
We've got our phone full of apps, and
each one of them does something.
Instead, AI is like, "This is the
do-everything app. Click here, and it
just does all the things." Like, that's
a weird direction to take software
development, um and there's no reason to
do it in that direction. So, So, the AGI
framework is useful for saying there
isn't just like the spectrum from not
AGI to AGI to post-AGI, but rather there
are lots of different directions that we
can go. And, once we start thinking
about what do we want the system to
actually do? Like, what is the problem
that we're trying to solve here? Then,
it starts to feel very different.
>> Is that really what we need from AI,
right? Right. So, we can build AI
agents, where the AI systems have the
goals and the plans and sort of make
things happen, and and their whatever
they're trying to get done is the thing
that happens in the world. Or, we can
try to enhance human agency, where we
decide what want what we want to have
happen, and we have tools that help us
do that. That's what we've done with all
our previous technology.
Now,
the the way that AI is being developed
now, people have noticed that it's
mostly a tool, and they really really
want it to be an agent, because there's
more profit to be made in agents. They
can replace more things. They're also
just They're things that um
agents can do that that
tools can't, cuz the agents can do it
all by themselves without human
intervention. So, like if you if you
want to um
write a big program, there are you can
there are some different paths you can
go down. One is something that you're
the programmer, and the AI system sort
of helps you out. Or the other one is
the AI system is the programmer, and you
tell it sort of what to program, and it
goes away and does a bunch of
programming.
And this is what people are leaning into
a lot now. They want the AI system to do
all of the stuff itself, and then just
report back, yes, I'm done, here's the
the product. Um and that can be very
useful in certain ways, but when it the
more we put the agency into the AI
systems, the less of it sits in us.
Right? So, the more the AI system is
making decisions, the less we're making
decisions.
Um
and so, I think we have to be very
careful as we build uh
systems that are very autonomous, they
can do all of these things on their own,
and have all the other human
capabilities, the intelligence and the
generality, what is the purpose of us
anymore, right, if the system does it
all? Now, if you build something that's
autonomous, but isn't that smart, um
or isn't that general, then it's
something that we can still control.
Like autonomous vehicles, they're not
going to take over the world. They might
put some Uber drivers out of work, and
that is like an economic displacement
problem that we have to think about.
But, there isn't some giant risk to
society from things that are just
autonomous. It's that triple combination
that's the problem. Is autonomous AI
more dangerous than highly intelligent
AI?
I would say so.
Um so, it depends how exactly So, a lot
of people think of his intelligence as
um
you know, lump all these things
together, which is why I think it's it's
useful to separate them. But, if you
have a system that is highly
intelligent like
um like, you know, a calculator is in
some sense very intelligent in that
the particular thing that it does, it
does extraordinarily well and accurately
and fast. So, in that sense, a
calculator is very intelligent, but
absolutely non-threatening, right? Now,
when you think of something a little bit
more general than a calculator,
then
um it starts to to exhibit more of those
potentially dangerous qualities. So, if
you have an AI system that can design
new viruses, right? Um it can be very
specific and only designs new viruses.
It can be not that autonomous in that it
really requires a human oversight, you
know, to tell it to design a virus. But,
it can still be pretty dangerous,
because if you've got a very powerful
tool in the wrong hands, that is a very
dangerous thing. Um but, I think it's
important to separate the kind of danger
that we've always had. We've always had
powerful tools. We've always had bad
people. And we've
It it's a problem to figure out how to
manage the bad people having powerful
tools. But, it's one we've been with for
a long time.
Building something that is fundamentally
a sort of being, rather than a tool, an
agent, rather than a tool, that has its
own goals and purposes, that is
something we don't have a lot of
experience with. We have sort of some
uncontrollable systems, and we've got
like animals and so on, but um and we're
used to not controlling those
necessarily, but they're also only so
powerful. All right, if you're you know,
if you're in the
in the cage with a lion, it's very
scary, but otherwise lions are just not
that big of a threat to humanity because
they're just not that powerful compared
to us. But if we absolutely had to
control, you know,
lions
because they you know, if lions had
nuclear weapons, that would be a
different question. Like we don't know
how really how to get lions exactly to
do what we want. Even the most trained
lions can like turn on their
trainers. Um so I think it is a very
different thing to build AI systems that
are autonomous agents rather than tools.
And as we make very powerful capable
autonomous agents, we have to ask
ourselves,
A, is that the right direction? And if
B, if we do it, are we going to keep
control of those things? What does your
Metaculus platform currently predict
about the timeline for AGI?
Yeah, interestingly it's it's
Metaculus's prediction has stayed pretty
constant over the last several years
that
that we will have So there are different
definitions of of AGI and
it's very contentious what is the right
thing to predict.
But according to the way that the
prediction is made on Metaculus, it's
sort of in the next few years
that we're likely to have it by this in
the sort of weak form and maybe a couple
of years after that in a very strong
form. So Maybe AGI is an ideology more
than technology, right? It
I think it is in some sense an ideology
and but that ideology is driving a
technology. So it So I think
um it didn't have to be this way. Like I
I do think that
um
there's a there's another universe very
much like ours in which people did not
get attached to the idea of AGI and
let's replicate all of the things that
humans do and and sort of race to build
something that is super intelligent, but
they just said, "Oh, look, we can do
this thing called machine learning,
where instead of having to explicitly
program exactly what the system does, we
instead
figure out the goal and the system
learns how to do that thing. You know,
it's just a different way of writing a
program to do machine learning instead
of
explicitly programming all the steps.
And let's let's create really, you know,
strong machine learning systems and let,
you know, have them be tools that do
stuff that was very, very hard to
program explicitly, like protein
folding. We never figured out how to do
that, but look, the machine learning
thing can do it." from DeepMind. Um
There's a I think there's a world in
which that was the the way people
thought about AI and that we built lots
of AI tools and, you know, there were
some people out on the side saying like,
"Hey, we should build AI that does
everything instead of humans." And the
people were like, "Nah, yeah, whatever."
In the same way that right now there are
off people saying like, "Oh, we should
do genetic engineering to make ourselves
to make like superhumans, like
superhumans physically, superhumans
mentally. We should build like the
Uberman with genetic engineering." Like
those people are out there, but
everybody pays them no attention cuz
like we don't want that. We don't want
to go we don't want to go build
superhuman with genetic engineering. So,
we could have a world in which that was
the way that people look at AGI and
superintelligence.
We're not in that world.
Do you see any signs of an AI bubble? An
AI bubble? It's an interesting question
because I think there are there are
people who think we're in an AI bubble
because
AI is basically It's all hype
and the AI systems really aren't capable
of being that powerful and so on.
This is not true. It's something Um
There is, I think, a potential that
there is a bubble
uh because of trust. So,
the the way that AI systems will make
huge the the huge amounts of money that
they have been promising to their
investors is if they can substitute for
huge amounts of human labor.
Now, there are various barriers to
substituting for huge amounts of human
labor. One of them is sort of raw
capability. They have to be smart enough
to do all the things. Um another one is
sort of social barriers and regulatory
barriers and things. And a third one is
trust. So, if I'm going to replace my
human workers with AI systems, those I
got to be able to trust that those AI
systems are actually going to do the
right thing, and they're going to, you
know, follow the the rules and the laws,
they're not going to get me in trouble,
and so on. And I think there that it's
possible that because we're designing
these AI systems in a way that is so
focused on sort of raw capability,
getting the highest numbers on those on
those capability benchmarks, and not
focused on building things you can
actually rely on, that are actually
reliable trustworthy secure safe
that there's going to be a big barrier
between when the systems, in principle,
are like capable of doing all these
things, and when they can actually
be used in the economy. And if that gap
it gets big enough, that trust gap gets
big enough, then the money right might
run out. So, if you're burning through
huge amounts of money, which many of
these companies are, and there's a long
delay between when you have the
capability and then when you can
actually make huge amounts of money on
it, you might run out of money. That's
not because the technology is
but it is because you've chosen the
wrong direction to develop it in in a
way that is not actually making what
people need.
>> I remember my interview with John
Hopfield. His work
was recognized in 2024 by Nobel
Committee work on AI. Uh I ask him, "Do
you regret something? If you could turn
back time, would you do this?"
I remember his answer.
Um
Just one thing. That neural network
development
took place within computer sciences
rather than biological sciences. Hm.
For decades, computer sciences, not
biological sciences, have looked at how
the human brain works.
Where do you stand on this?
Well, I think there's
there there's some interesting
scientific questions there, and then
there are some social questions, too.
So, I think there's
there's an interesting
sort of fact that when we create
biological systems, if we create a new
drug or a you know
a new medical procedure or something, um
those are very, very tightly regulated
fields. So, in in biology and in
medicine and things that have to do with
living beings, we have a very different
attitude cuz we sort of understand that
we're tinkering with something that's
sort of almost sacred. Like, there's a
there's a real respect for biological
systems, and there's a real obvious
sense in which when you screw something
up, you're doing real harm to something
that feels it.
And so, we we've got a very
sophisticated regulatory system and
systems of ethics and and so on when it
comes to biological systems.
All of that is largely absent when we go
to computational systems, where we have
this history of you just kind of throw
stuff together. You It's just software.
It doesn't really affect anybody,
exactly.
Um there's basically no regulation.
It's a free-for-all. You just throw it
up there, and like it's kind of anything
goes.
And the fact that AI, this this
incredibly powerful new technology that
is an
uh
whose goal is to, you know, replace
human thinking and replace humans as
decision makers and agents and so on,
came out of the computer the totally
unregulated anything goes software
industry rather than the the biology
side, um means it is being treated very
very differently, right? Like you know,
as I've as I said before, if you propose
I want to
um
use genetic engineering to make a set of
people that are super duper smart
because that's really valuable. If we
have super duper smart people, we can
cure cancer, we can be more productive,
we we can do all of these things. So,
I'm going to go do genetic engineering
to make super duper smart people.
The regulatory agencies are going to say
like, "No, you are not." The ethical
boards are going to say, "No, you are
not." The general population is going to
say, "No, you are not going to do that."
At least not with like a lot more
conversation. This is not just like a
normal thing to do to go build
superhuman brains, even if they're
useful. Even if they're economically
valuable. Even if they could cure
cancer. We just don't do that.
>> end for our race, right?
>> Well,
the I mean, there are different ways
like the
we just treat it very differently. Like
whether it's whether that's
good or not, like it's very very
different, right? Like we would not be
having this conversation in the same way
um if it was with with human brains.
Again, there's a there's maybe another
universe where all these giant companies
are racing to build the most superhuman
humans with genetic engineering. That's
not the universe we're in, but there
probably is one out there where there's
like 10 giant companies that are racing
to build the most superhuman people with
their genetic engineering.
>> Upgraded people that are like super
intelligent people because they and
they're all saying like, "We're going to
cure cancer before the other before the
other guys do with our superhumans." Um
that's just a very different universe
than the one we're in. So, why we treat
one one way in our universe and the
other totally different way or vice
versa in the other universe? I don't
know. It's a complicated question. Um
but it is a very double standard and it
is interesting to imagine um how things
would be different if AI were developed,
you know, more like biological systems
and less like uh apps.
Thank you very much for your time.