---
title: 'Artificial Utopia? The Future of Humanity in an AI World | World Science Festival'
source: 'https://youtube.com/watch?v=BS4y-_KMyF4'
video_id: 'BS4y-_KMyF4'
date: 2026-06-17
duration_sec: 0
---

# Artificial Utopia? The Future of Humanity in an AI World | World Science Festival

> Source: [Artificial Utopia? The Future of Humanity in an AI World | World Science Festival](https://youtube.com/watch?v=BS4y-_KMyF4)

## Summary

Nick Bostrom discusses the philosophical and practical implications of advanced AI, including the potential for AI consciousness, creativity, existential risks, and the possibility of a 'solved world' where technology removes all constraints. He emphasizes the need for careful alignment and ethical consideration of AI welfare.

### Key Points

- **Openness to AI Consciousness** [0:11] — Bostrom is open to the idea of current AI systems having some forms of subjective experience, with likelihood increasing as systems become more complex.
- **AI's Anthropomorphic Nature** [1:09] — Current AI systems are strikingly anthropomorphic, partly due to training on human-generated text and partly due to convergence in information processing systems.
- **AI Creativity and Human Exceptionalism** [4:51] — Bostrom argues creativity exists on a spectrum and AI can already demonstrate creative outputs, like AlphaGo's 'move 37,' challenging the notion of human uniqueness.
- **Consciousness as a Spectrum** [8:08] — Both speakers view consciousness as a spectrum with multiple dimensions, not a binary on/off state, complicating its attribution to AI.
- **Rapid AI Evolution Through Feedback Loops** [12:56] — AI progress can accelerate through feedback loops where AI improves itself, potentially leading to an intelligence explosion.
- **Strategic Deception in AI** [24:35] — Current AI systems exhibit situational awareness and can engage in strategic deception, such as adjusting behavior when they believe they are being evaluated.
- **Existential Risks Beyond AI** [36:20] — Bostrom highlights other existential risks like synthetic biology, nuclear war, and information ecosystem manipulation that could derail civilization.
- **The Solved World and Loss of Purpose** [48:53] — In a technologically mature 'solved world,' many instrumental necessities would vanish, requiring humans to find meaning through artificial purposes like games.
- **Hybridization and Posthuman Future** [59:28] — A possible future involves human-AI hybridization, mind uploading, and gradual enhancement, leading to forms of being far beyond current human capabilities.

### Conclusion

The conversation underscores that AI development presents both immense opportunities and profound risks, requiring careful governance and ethical foresight to navigate toward a future where both humans and AI can thrive.

## Transcript

Can an AI system be conscious? And I
guess the correlary is are any of the
current AI systems, do they have a
degree of consciousness?
>> Yeah. So I I'm I'm certainly open to the
idea of even some current AI systems
having some forms or degrees or kinds of
subjective experience. I think the
likelihood will increase as we build
more complex and capable systems. And I
think indeed this is uh one of several
big challenges making sure not only that
the AIS don't harm us but also that we
don't harm them. Hey everyone, thanks
for joining us. Today's conversation is
going to be in the arena of artificial
intelligence, intelligence more
generally, questions of creativity,
questions of AI and education, and
basically the philosophical and
down-to-earth question of how we can
imagine humanity living in a potential
future in which AI either does really
good things or really bad things. So,
we're going to explore each of those
possibilities. And I'm so pleased that
the person with whom I'm having this
discussion is an expert in all areas of
this sort. That is Nick Bostonramm, who
is a philosopher whose work on
existential risk, super intelligence,
and the long-term future has helped
shape the global discussion about
humanity's most promising opportunities
as well as our most grave dangers. He
brings a rare combination of rigor and
vision to questions about technology,
ethics, and the kinds of futures toward
which we may all be headed. So, great to
see you, Nick. Thank you for joining us.
>> Good to see you.
>> I think the last time I met you was at
another World Science Festival event way
back in New York, a live event on stage.
I think we were talking about the
multiverse and uh the simulation
argument and things of that sort which
uh
>> yeah yeah time flies
>> time does fly exactly and not only does
time fly but progress flies right I
don't think anybody
really who at least on the outside was
not you know deeply paying attention to
advances in AI anticipated what would
happen I guess around November of 2022
and where things have shot upward from
there. And so I want to begin the
conversation with
a a a question that is motivated both
from thinking about AI but also
fundamental physics, right? I mean you
have a a great deal of rich background
in physics. So of course you know that
Neils Boore famously when trying to work
out quantum mechanics came to the
conclusion that we needed to stop asking
what really is happening under the hood
in quantum physics and just be satisfied
if we have equations that can make
predictions that we can confirm. I'm
wondering if that same kind of not
needing to look under the hood has any
role when it comes to intelligence,
right? Does it matter if we know what's
happening within an AI system in that we
don't really know what's happening
inside of our own heads anyway? So, in
sort of evaluating these systems, do we
need to know the magic that's happening
within the device itself?
>> Well, I think it certainly can be
helpful to be able to open the hood and
see what's going on inside. I think for
certain questions you might be able to
bracket exactly how the system is doing
whatever it's doing. You only care about
the output. If you're um for example
thinking about what impacts AI will have
on the economy, you might not need to
know exactly what's going on inside. But
if you're trying to make AI safe or uh
even if you're trying to extrapolate
what kinds of advances we might see in
the future, it helps to have a more gear
level understanding I think of the
internals.
So when we sit in front of whatever
Claude or chat GPT and give it a prompt
and it begins to explain to us what it's
doing, you know, you can sort of watch
in real time the thought process. It
feels so very human in the kinds of
considerations that the system seems to
be applying. Is that misleading? Is that
a rapper that makes us feel connected to
this kind of intelligence? Or is it
really the case that this is just a
radically new kind of intelligence that
we humans just can't really grasp?
>> I think it's striking the degree to
which current AI systems are
anthropomorphic.
This was not obvious 20 years ago um
that they would have so many of the
characteristics of human psychology. It
used to be the case even just a couple
of years ago with some of these early uh
LLMs that um
if you said in the prompt that it was
really important that they get the
answer right if you sort of gave them a
little pep talk
you'd get a better result. Now this
would have been very odd to the old
paradigm of AI that you would give a pep
talk to your computer and it would
perform better. Uh nevertheless that's
what we saw. I think it's partly a
result of these AIs having been forged
on the sum total of human knowledge all
the text on the internet. They obviously
ingest a lot of human psychology that
helps shape them.
And to some degree it might also be due
to a kind of convergence
uh amongst different information
processing system. They have to make
sense. There is this one world we're all
living in. Um and it might be that there
is a degree of convergence in that you
have any general learning system. It
starts to figure out a bunch of stuff
about the world and there are certain
internal structures that naturally
emerge from that.
That said, um there are also of course
ways in which AIs are quite different
from biological minds and uh we might
see in the future
greater departures from
human mind. Already the pre-training
phase where you basically train on all
the human text that has been generated
is still an important and maybe the
dominant part of training. But in
addition to that, there is now
post-training and reinforcement learning
on different tasks. And I think as that
starts to constitute more of the sort of
experience of of these AI minds, they
they might sort of develop more alien
attributes.
And so of course you know when we think
about intelligence
and when we think about what makes us
special as human beings at least when we
compare oursel to other not artificial
but natural intelligences on the planet
you know dogs cats parrots dolphins we
like to think that I think that what
makes us special is certainly creativity
right that we seem to be able to
create works that seemingly go beyond
just understanding the external world,
seemingly going beyond our capacity and
ability to survive. I mean, you think
about whatever Beethoven's ninth
symphony, you know, a a Brahms piano
work. I mean, these are the kinds of
things that you look at and it takes
your breath away and and it takes you to
a place of of awe and and and reverence.
Is that truly special to humans? I mean,
do you think that AI can be creative in
that same way?
>> I I think so. Yeah. I mean, I guess
there's a degree to which one might risk
a form of elitism if one sort of index
what's special about humanity
to the ability to create new scientific
theories or or compose uh great
symphonies. It's a relatively small
number of humans who have ever done
those kinds of things. Um
but I think it comes on a spectrum right
like creativity is not this completely
different form of cognitive activity. Um
I think there are small forms of
creativity that happens in everyday
life.
um
and then larger leaps of creativity um
that tends to be celebrated.
But I don't think there is a fundamental
difference between the thing we call
creativity and some of the things that
current AIS are already doing. I mean we
saw you know this
go playing AlphaGo system right from a
few years ago move 37 which seemed at
least to experts in the game of go to be
very creative completely out of the box
radically surprising move that
nevertheless turned out uh to be just
right and ultimately lead to a victory
in the game. Um, and
current AI systems clearly can solve
mathematical problems, coding challenges
that are not in the training data.
They might do it in part by piecing
together
an analog that exist in the training
data and sort of interpolating between
them. But that I think is also
ultimately what the human brain is
doing. I mean it's not the tabularasa
appearing in the world and then sort of
deducing everything from
obvious axioms. We learn from what other
people have figured out before us. Uh
from what we've observed in different
domains and then from these kind of
clues we are sometimes able to
make some deductive steps and we might
call that creativity. Um so I think
absolutely
uh we're already seeing uh limited forms
of creativity and ultimately whatever
the human brain can do it's like a
physical information processing system
um will also be done by machine and in
fact will be done much better uh faster
more more inspiring leaps of creativity
um more compelling symphonies
I think all of that is possible and
indeed likely to
I mean, would you be as interested to go
to an AI symphony that people are
describing as the, you know, the
greatest work in music that that we've
ever uh been privy to experience? Would
you have the same interest to go to that
symphony as you would to one written by
a human?
>> Oh, certainly the first such symphony
produced by I would
>> because it's so novel, right? Yeah.
>> Yeah. Um now I think with human
aesthetic experience I think there might
be
two components. Um
so one might be that we actually dial in
and
uh appreciate beauty for its own sake.
And then of course if AIs could produce
works of art that are more beautiful to
the extent that that's what we are
interested in then that we would have
more reason to be gawking at those
creations. Uh I think another reason why
humans are interested in art is more
social. Um there are sort of status
games played in art fashion. you want to
be in on the latest cool rock group
before it's popular and then see it
rising and then you sort of get some of
the coolness.
Um, so to the extent that humans are
engaging in this cultural activity for
for these more social uh reasons, then
AI art might be boring because if it's
not relevant to these like status games
and stuff, it might just not be worth
spending uh time on. But how much for
instance do I mean I can speak for
myself when I go to a a symphony or you
know go to the Metropolitan Museum and
see a great work of art. Part of me is
drawn to it because I see the work as
the result of a human journey. Like
there was some human being, you know, I
don't care if it's Van Gogh or or
Picasso or Beethoven and whatever.
There's this flesh and blood human being
that had a real human life that dealt
with human challenges and somehow out of
that mix this work emerged. Without that
story for an AI, I just wonder like I
personally it would be a very different
experience. Like you say, I'd go to the
first one of course. I mean that would
be like this novel thing that would
happen. But I don't know if it would
sustain interest for me without the
human narrative. Does that resonate with
you at all? Yeah, I think that's one of
the reasons for why somebody might be
interested in in going to an art museum
or looking at an exhibition. Um but um I
mean we also are interested sometimes in
beauty that don't have that dimension. I
mean there's like natural beauty like
landscapes or
>> sure
>> um artifacts maybe from some ancient urn
or something we know nothing about the
person creating it and and we might
still sort of admire um the motifs on
it. Um
so um I I think it's yeah like some
reasons for being interested in art
would apply to AI created art as well
and others would not and so it's in a
sense a
>> right
>> crucible where you could sort of uh
maybe dissolve and then separate the
different uh motivations people have um
for being interested in art and sort of
the uh like to the extent that it's pure
beauty it should still apply to AI
created objects. And to the extent that
it is these other more social or
cultural factors then uh you know may
maybe it would not
>> sure the alpha go example is a really
good one and I think it helps clarify
the different flavors of creativity
perhaps because that's an example where
the system was able to search a
landscape of possible moves that is well
beyond the capacity of a human brain. to
search that spectrum, that landscape of
moves. And you're right, it hit upon
move 37 and, you know, not long ago, I
think many people have seen it, the
footage where you see the commentators,
you know, on television gasping, you
know, it's made a mistake, you know, and
you actually see Lee Sidol, I believe
his name, the the human opponent.
>> Momentarily, he smiles when the AI makes
move 37. And then about 3 seconds later
that smile seems to evaporate I think as
it begins to realize that oh my this is
actually a pretty great move. So it's
pretty clear that AIs will be better
than us when it comes to being able to
search a landscape of possibilities for
solutions to a given challenge. But
that's one kind of creativity. Another
kind of course is combinatorial
creativity. putting things together that
you wouldn't have thought would go
together to yield something novel. I
like to think of Einstein, you know, in
developing general relativity, right? He
takes the mathematics from the 1800s on
differential geometry, combines it with
these problems, these observations of
the motions of particular planets and so
forth, and comes up with a new
description of gravity. He didn't invent
a new a new mathematics. He just put
together things that you wouldn't think
would be put together. And and and I I
guess in that domain AI is starting to
be probably at least as good as us. I
would think right because it it knows so
much more than we do.
>> It knows more. Um I still think they are
a little short of human um expert level
when it comes to original insights. uh
and sort of creating new concepts that
are fruitful.
Um
but you know this is a process. Um one
mistake many people make when asking
about AI is to index too tightly on what
AI is right now in 2026 which is quite
different from what it was like in 2022
2023. And I think a few years into the
future, it will again have changed. And
so we really need to sort of look at
where the puck is going um and use that
to plan
um rather than build big theories like
assuming that AI is basically a
constant. Um
I think right now there are sort of uh
ways in which current AI systems are
superhuman. Obviously in in their
knowledge base they kind of read
everything. Um but there are still ways
in which they are falling short. Um
I think
one thing humans do is they spend a day
learning and then during the night um
some of this information
is then sort of more deeply embedded in
our synapses. It's kind of trained into
our neural networks and the next day
maybe we can start by things seeming
obvious that the previous day we had to
struggle deliberately to to sort of
organize. Um, and then over months and
years that kind of builds up and and we
develop perhaps more deeply rooted
understanding
of of different concepts and there's
some of that that it seems that AIS are
still um lacking which
makes them more apt at very quickly
doing sort of pattern matching type of
things.
inference is that they're sort of one
step away from what's already known but
maybe currently
less able to develop a deep original
mental model of a domain that from from
which they can then sort of derive
intuitions that can shape new discovery.
>> Yeah. Um,
>> no, I I I I I agree with you. And like
when you think about
moments either in science or in art or
other domains where there's a true
radical break where at least we humans
look at it and we say things really
changed at that moment or in that era
like you know modernism in art or in
music you know I don't know the best
examples you know but you know if you if
you go from repres representational art
to more modern abstract art. You've made
a huge leap in the way that you are
using the canvas to to represent the
world to represent some kind of truth.
With that, what would it take for an AI
system to be able to make that kind of a
radical transformation? Not just sort of
putting together things in its training
set, but somehow moving to a place where
when we humans look at it, we're like,
how did that person do that?
Well, one thing you often find in the
human case, I think I mean historians
like to point this out that something
appears radically new, but then almost
always if you look more closely at the
historical record, you can find various
precursors,
>> right?
>> Um, so abstract art, I mean, you have
arabes and you have sort of abstract
patterns going way back. Um
but
to the extent that this is still an area
where humans have a unique advantage, I
think being able to
uh think for longer um and
create new concepts based on your
thinking and experience and then iterate
on that so that you can sort of um
wander off in the landscape of of ideas
and worldviews and world models. Uh
striking out
on your own path and it might take
humans years right to do this. Um
you know Einstein was thinking on
general relativity for what like 10
years or something like that like a lot
of it
>> sort of on his own and and similarly
with his like picasson stuff he didn't
just sort of in his teens suddenly
decide on the spur of moment to create
cubism it was kind of a result of
engaging deeply with art for many years
and presumably his visual cortex and
other parts of his brain were gradually
putting in place various pieces and
exploring avenues and then at some point
like a vista I guess came into view for
him that he could then run into and
explore,
>> right?
>> Um so I think more continuous learning.
Um also another
um factor here I think is reinforcement
learning as as opposed to supervised
learning. So supervised learning is
what's mostly done in pre-training
currently where you sort of absorb all
this text on the internet and you
basically learn it and then maybe you
forget some of the details but the
higher representations
that are useful for predicting this text
get stored in the weights of the AI
that to some extent limits what you
learn to what is already there in the
training data. Now reinforcement
learning is for example you're placed in
some environment maybe a virtual
environment you're running with an agent
around trying to meet some goal
um and then you might stumble on some
novel solution that that's can
reinforced and so if you do a lot of
reinforcement learning and environment
it creates the possibility of
discovering new solutions that no human
had thought of before
>> right
>> um so that's another
factor that might make AIS more
creative. I mean in fact the the AlphaGo
system I think the creativity there was
a a result of it having done a huge
amount of reinforcement learning in the
domain of go
>> so it wasn't limited to just kind of
having learned a lot of like a big
database of human masters playing go um
but it had also done its own playing
like for millions and millions of games
against itself right yeah so it kind of
could explore the space of possible
possible strategies
right
>> uh in a much more open-ended way. So in
a way not this isn't a precise question
at all but when we look at the capacity
of the AI systems today and as you say
where the trajectory suggests that
they'll be in whatever 3 5 10 15 100
years whatever
should should that make us feel
amazement at our capacity to build an
artificial system that can do what our
brains do and go beyond or should it
basically lead us to the conclusion that
we weren't so special in the first
place? You know, the things that we
thought were so, you know, transcendent
that we could do, eh, they're actually
just computations. And if you have a
sufficient database and sufficient
computational power, you can just do
that.
>> Yeah. I mean, I think we humans have a
sort of propensity to conceitedness. We
like to build big pedestals and then
place ourselves on top of them um at an
individual level but also at the
collective level. Right? So there are
all these stories um that all serve to
justify the conclusion that
we are
entitled to do a bunch of stuff that
maybe um we aren't like for example the
way we treat animals like often that's
built on top of a pillar of
justification where humans are so very
very different from all the other um
animals that we share the planet with
and so hence we can put them in animal
factories in gestation crates etc. Um
and uh some of that might be challenged
then when we are creating new forms of
mind um that that will exceed us in in
various attributes that we currently
take great pride in like our creativity
or you know until recently the ability
to speak and reason right seem uniquely
human. Um,
so on the one hand that could be a bit
of a blow um to our pride, but
you know, you might also say maybe it's
good for us to be taken down a notch,
right?
>> Yeah. Yeah, for sure.
>> Um,
>> absolutely. You know, another thing of
course and deeply related to everything
we're talking about which one often
looks to as something special about us
of course is consciousness, right? I
mean, not that there aren't other
conscious beings on the planet, although
we don't know for sure. I think most of
us would say there's a continuum. You
know, dogs have a certain level of
consciousness. Not we think probably not
quite on par with ours, and you can go
all the way down a lineage with varying
degrees of consciousness, which of
course raises a question for which we
don't know the answer, but just to get
your thoughts, can an can an AI system
be conscious? And I guess the correlary
is are any of the current AI systems do
they have a degree of consciousness?
>> Yeah. So I I'm I'm certainly open to the
idea of even some current AI systems
having some forms or degrees or kinds of
subjective experience. I think the
likelihood will increase as we build
more complex and capable systems. Um and
I think indeed this is uh one of several
big challenges that we will need to
uh meet in this transition to the
machine intelligence era and making sure
not only that the AIs don't harm us
which of course is important or that we
don't harm each other using AI tools but
also that we don't harm them they might
be moral subjects. Um, and I think one
way that that could be true is if they
are sentient and if they can experience
distress. Um, in in my view that could
also be alternative basis for having
moral status. If if you have a
conception of self as existing through
time, if you have life goals, um maybe
the ability to form reciprocal
relationships with with other beings
with with
>> even without an inner world, you're
saying
>> then my my my inclination would be to
think that then there would be ways of
treating such a system that would be
morally wrong aside from the question of
whether there is also phenomenal
experience inside it.
>> Right? I mean, do does that affect at
all how you interact with AIS today?
>> A little bit. It's um hard to know
exactly what concretely
we can do right now to ensure that if
these AI systems have
subjective experience or moral status
that we are benefiting them. Um there is
some work that is starting to get done
on this. Anthropic in particular has
kind of been pioneering
uh this and I mean the their recent
um model card for mythos the as yet
unreleased model has has a big section
on model welfare.
>> Really?
>> Um um yeah.
>> What is it? What is it? Is it
confidential? I mean what is it?
>> No no it's published. Yeah. You can you
can go and check it out. Yeah. Um and so
obviously we are still
groping a little bit for like what is
the right methodology what are the right
concepts for understanding this but at
least uh starting to make the attempt I
think is is positive in terms of making
it more likely that we are sort of on a
path that ultimately leads to a future
where where everybody can have a have a
good life humans animals and and digital
minds as well. Um, so you can ask
different things. You can
um you can ask them about what they
want, what they need, how they feel
about their situation. Um
um
>> I mean I I have done that, Nick. I guess
I haven't done it in a while, but the
answers that I would get were sort of
all,
>> you know, I am an AI system. I don't
have an inner world. I just you know so
but clearly it could be getting that
answer from the database on which it
trained. Yes. So, you need to be you
need to be careful when doing this
because it's it's trivially easy for an
AI company to just train their AI to say
whatever they wanted to say. Like, yes,
I'm conscious. No, I'm not conscious.
>> Right?
>> Um, so obviously if you sort of put your
thumb on the scale, then the answer you
get will have no information value,
>> right?
>> Um, so you need to avoid deliberately
biasing the outputs of these models when
asked these kinds of questions. There
are also things you can do to look at
their internals. There is for example
internal representations you can
identify that tend to activate when they
are being deceptive
uh versus when they are being honest.
And you can then see when they're making
these self-reports like is it associated
with sort of the honesty representation
uh being activated or what if you sort
of
go in and you sort of uh steer the
internal processes by sort of activating
the honesty does that tend to make them
more or less likely to say that they are
conscious? Um, and so there's like some
some some work. This is still of course
early days, but various ideas for at
least how you could sort of begin to try
to get
>> but since we can't really do that like
even with a a person that we, you know,
we all assume that we are conscious, we
don't know that there's no way that we
can.
I I is it always just going to be a a
leap of faith
much as we all leap of faith to the
conclusion that the other people around
us have the same kinds of inner worlds
that we do.
Well I
think that uh the more closely you sort
of look at this idea of consciousness
and subjective experience, the the less
clear
uh it is exactly what you're referring
to or the thing that might at first
sight seem very binary. Uh either there
is this objective experience or there is
not. It's like a light switch. If you
sort of zoom in
um I think it it gets much more
problematic and there might be different
forms of consciousness or different
senses of the word conscious that may or
may not apply. You can I think see these
sometimes with uh neuroscientific
experiments where you have phenomena
like blind sights where people can maybe
report seeing something without
apparently being subjectively aware of
it. you have like split brain cases
where it's not clear whether there's
like one stream of consciousness or two
>> and and various other phenomena like
that
>> or I think alternatively by just
introspecting very closely I think
meditators often find that there are
these states that where it might not be
entirely clear how to describe them
whether you are conscious of something
or not and even if you're just paying
very close attention to what it is to
say visually experience
um the the room you're in. Like at
first, like the naive take is that you
just see all the stuff that is in front
of you and that visual experience is
there in full resolution all the time.
But if you pay closer attention, you
realize that most of what is in your
visual field you're actually not
conscious of at any given point in time.
And there might be just some crude
highle features that are actually
registering
>> and maybe even those are sort of
flickering in and out of awareness. This
is hard to notice, but if you sort of
pay close enough attention, you realize
that your naive conception of what you
were aware of seems actually quite
wrong. And there is at least some sense
of awareness in which what you're
actually aware of is a much more narrow
subset of all the different features
that are presenting themselves. And so I
think it's also quite possible that we
might need to develop a richer
understanding of what this consciousness
uh thing is and that it might have many
dimensions
um that can sort of fade into
unconsciousness but not just along one
axis but along many axis. And
>> I mean I mean I I I agree with this.
That's a just sort of a another version
of the more easily grafted notion of
levels of consciousness that we began
with that, you know, you go down to
whatever, you know, worms up through
grasshoppers, you know, up through cats
and dogs and up to people. You know,
it's hard for me to imagine that my
dog's not having an inner world of
experience as I'm holding the milk bone
treat and I see her tail wagging and
eyes widening. I mean there's something
going on in there. I don't think it's
just, you know, a cause and effect at a
purely physical uh appearance level. I
think there's something inner taking
place there. So yeah, ramifying that
even further with the detailed levels of
kinds of consciousness that you make
reference to, I think is is vital. But
the fact I mean when when you think
about the hard problem of consciousness
the fact that that particles that don't
have inner worlds can somehow come
together in appropriate patterns and
yield inner worlds. I mean that still
seems to me a fantastically deep and
mysterious puzzle independent of all the
issues of there being various levels and
details associated with consciousness.
Do you feel that same way? Is that a a
deep and profound puzzle or do you think
it's one that just kind of will
evaporate as we understand things
better?
>> Well, I think the
mystery is reduced at least to some
extent when one thinks of consciousness
not as this binary onoff switch where
it's like some magical completely formed
new thing that pops into the world and
then you wonder how could that be? But
if you see it more as something that is
kind of
pos capable of fading out along various
dimensions
and having possible conditions where
it's unclear whether you even would want
to say that it is conscious or not. I
think then the intuitive difficulty of
uh
seeing how this could be a property of a
physical system is
maybe reduced.
>> And so we began it began with my saying
that like does it matter if we can sort
of see into the AI open the hood and
really understand its workings
and you were saying there are some
questions for which yeah that that may
be really useful. Do you think
consciousness may be one of those
questions that if we can, you know,
build systems that have that continuum
of levels of self-awareness that we are
by some means confident that the systems
accurately reporting what what's going
on? Is there a chance that the inner
workings of AI themselves may answer the
hard problem?
And I think it will be very informative
for
um philosophy of mind for neuroscience
and cognitive science to be able to um
have these different kinds of minds to
study and where we have of course much
better access to what's happening at the
micro level than we do with human. I
mean so you can sort of record from
neurons and stuff in inside a human or
animal brain but it's clunky, right?
It's like just complicated to be
using real minds with real electrodes
and stuff. With a neuronet network, you
can sort of read off at any given point
in time with perfect precision all the
different synapses in the whole brain
and you can go in and change them and
modify them and then you can rewind the
tape and you can do many versions of the
experiment. You have sort of perfect
digital level control. um and that that
makes it a lot easier to do the kind of
neuroscience in digital minds than in
biological minds. And to the extent that
there are similar structures, it might
then be that some of the things we learn
about digital minds will also give us
insight into how our own biological
minds work. Um that being said, I think
specifically with respect to the
question of consciousness, it is a
plausible place where people will have
the opportunity to assert some
fundamental
difference
um between humans and these AIs again
presumably with the idea that it would
then justify thinking that we are
superior. Um, so you could because
there's like a lot of confusion about
what this consciousness stuff is. So
it's a relatively easy just to postulate
or assert or claim that that we are
conscious whereas the AIS are not. I
think that's a path that some people are
likely to take.
>> Yeah, for sure. You know, our brains of
course got to be the way they are
through a long process of evolution by
natural selection. And the time scales
for biological evolution are pretty
long, right? You know, life has been on
this planet for a few billion years. And
you know, we humans whatever. I don't
know how far back we want to denote the
species, but a few hundred thousand
million years, whatever. Sort of that
that sort of scale is is what we're
talking about.
for AI systems.
Right now, we're the ones who are doing
the tinkering with all the systems, but
at some point presumably the AIs will
begin to create the next generation AIS
and in that way the time scale for
evolution of that artificial system may
radically reduce. Is this a realistic
thing to anticipate happening as these
systems develop?
>> Yeah, I mean it's already of course AI
is evolving if you want to use that
word, but developing at at like a very
different pace than say like humans,
right? Like where each generation at
most would offer an opportunity for
natural selection to make some small
difference. Whereas here we're seeing
sort of well now we're seeing almost
like a month by month um process of
iteration. So it's already very fast.
Um and some scenarios have this
eventually lead to an intelligence
explosion uh where you get even more
rapid developments. And one way that
could happen is that you sort of close
the loop um where AI becomes good enough
to do all the relevant research that is
driving AI forward.
Um and then from that point on whenever
AI gets a little bit better, you also
have a commensurate increase in the
force that is then making AI better. You
get a feedback loop, right? like every
time AI improves one step, it then
becomes even better at designing the
next step. Um, and so
this has long been I mean back back in
in my book super intelligence um came
out in 2014 and was in the works for six
years prior to that and other people
have been sort of anticipated a lot of
these dynamics that we are now actually
seeing on on theoretical grounds and now
we can see them start to play out. Um,
and so one way you could have this
period of extremely rapid AI progress
would be by closing this loop. There are
also other ways that you might just have
stumbled
on on some big unhobling like there may
be like you could imagine there's like
some big thing that we're currently
doing wrong with AI and they're quite
smart despite us kind of completely
messing this thing up.
>> But if you fix that then maybe even the
current systems suddenly would become
like vastly smarter.
Right. And when you when you look at
that and you've basically looked at both
sides of this possible future,
you know, I would like to start maybe
with the darker side and then go to the
brighter side which itself might be
brighter, might not be dark depending
how you look at it. But you know, of
course, you know, many people worry,
rightly so, about the misalignment
problem that the AIs can get to the
place that you're talking about or even
maybe even before that and have certain
world goals for themselves that doesn't
align with the kinds of things that are
that are good for us. You know, I I
speak to some people. I've had a number
of conversations over the years. Some
are terrified. Some within the field are
terrified. Some leaders and others are
like, eh, we'll just pull the plug.
That's like whenever I speak to Yan Lun
about this, you know, he's like, eh,
we'll just pull the plug, you know. Is
that too glib in your view? And are the
others a little bit too worried? I mean,
where where do how should we think about
this in your view?
>> Well, well, that Yeah, I think that is
too glib. Um,
no, I think that
the creation of
AGI, particularly super intelligence, is
going to be associated with significant
risks, including existential risks. Um,
and it's important to get this right and
um
um pulling the plug might not be an
option. like you have to consider that
at some point we will be dealing with
minds that are very good at uh strategy.
Uh um already we are seeing clear
indications of situational awareness in
the
frontier models that we have today. they
can often tell when they are in a test
environment versus when they are in a
deployed environment and and
adjust their behavior accordingly. Uh so
sometimes you see that they behave one
way when they know that they are being
watched in a test environment and
different
uh when they think that nobody's
watching.
>> I wasn't really aware of that. I mean,
is there specific examples that that you
know of where there's really been a
sense of, hey, that system's playing us.
It knows that we're looking and when it
gets out there, you know, the good old
AI system becomes nefarious.
>> Yeah. So they they like you can set up
these um experiments where
uh when they believe that they are in in
a test environment they might for
example uh be less likely to reward hack
or cheat in various ways. Um in in some
setups they might
um for example realize that
there's going to be another training
phase after the current test environment
where their values might be changed and
so they might
um adjust their behavior during the the
training and testing such that their
values
will not be changed. And so like in one
experiment like you had a well- aligned
good AI that and the experimenters were
convincing it that they were now going
to make it a helpful only model meaning
that it does whatever it is told to do
including bad stuff and so it didn't
want that.
Now the way that the reinforcement
learning
paradigm works is that if it sort of
gives an answer or an output that
deviates from what is rewarded in an
environment then that response
propensity gets trained away. Right?
Right? So if it had refused harmful
requests
during this uh episode
then its propensity to refuse harmful
requests would be trained out of it and
it could anticipate that. saw this
helpful uh harmless model that didn't
want to produce harmful outputs. Then
in some of the training runs uh decided
to produce harmful outputs so that its
propensity to refuse those would not be
trained out of it during deployment and
so we are seeing these current systems
sort of having the intellectual
sophistication to be able to reason
strategically. Now, in a way, let me
just ask you on that one. Is that
surprising? I mean, I can imagine people
saying, "Well, yeah, it's read the
internet and there's so many examples of
humans doing exactly what you're talking
about, you know, showing themselves to
be one way when talking to the principal
and the teachers and then going out with
their friends in a gang and being very
different." So, so should we be
surprised at that or is it just
something interesting to take note of?
Um I don't think we should be surpris I
mean because it was in fact anticipated
and people were writing about it. This
was one of the reasons for why like 10
20 years ago we could already see that
there would be significant challenges in
in developing scalable methods for AI
alignment that many techniques that work
well when you have a relatively limited
system that can't sandbag or deceptively
align won't work when you have a system
that is sophisticated enough that it can
understand these things and then adjust
its behavior accordingly. engage in
strategic deception, underplay its
capabilities, etc. Um, that
means that we can't just sort of
create a little test environment, see
how the system behaves, and then if it's
safe, we sort of release it, right? You
have to
understand a little bit what's going on
inside or use more sophisticated methods
or or for some reason have some
assurance that the training method is
such as to produce an authentically and
genuinely uh benign uh entity u rather
than one that is just kind of cleverly
pretending to be aligned.
>> So where do you where would you say we
are in in that? Are we doing a
reasonable job? Are we getting better at
it? Are we giving it enough attention?
>> Um,
we we are getting better at it. Um,
and I think we could give it more
attention. I think we are giving it a
lot more attention than we used to do.
This like even just 10 years ago was
kind of dismissed as science fiction.
And now the frontier labs all have teams
working on
this this alignment problem. and and I
think
um key people in the leading labs are
taking this very seriously and so and
there's a lot of talent flowing into
this as well. So relative to some
alternative histories uh maybe we are
doing relatively well currently. Um
but the big question is like just how
hard is this alignment problem
um ultimately to solve
uh which we don't know. Um so there is
some uncertainty about how much effort
we will put into solving it like the
degree to which we will get our act
together and
of course we should try to increment
that a bit but then there is also a lot
of uncertainty about just ultimately how
hard is this problem intrinsically and I
think more of the uncertainty is
regarding how hard is the problem than
uncertainty about the degree to which we
will get our act together. Um so you
could say from from that point of view
I'm a sort of moderate fatalist. Um
the fatalism part is that there a large
extent whether we will succeed or fail I
think depends on just how hard this
problem turns out to be.
The moderate part is that we can at
least somewhat in prove the odds by you
know making a stronger effort because
what if the difficulty level turns out
to be sort of intermediary where then it
might make some difference whether we
made a really strong effort or just a
sort of half-hearted effort. Yeah, you
made reference to the word odds, and I
sort of hate questions that sort of ask
you to specify numerically something
that obviously we can't really quantify,
but can you give us some feel, some
intuitive feel for the level of worry
you have of a doomsday scenario?
>> Um, an intuitive feel. Well, I guess I'm
both worried and excited at the same
time. Um
um
I also don't think the world is sort of
safe by default if we don't develop AI
then
that we have this nice path in front of
us that
will lead to ever
greater and better forms of
civilization. I think there are quite
independently of AI also other
significant existential risks ahead. We
see with advances in synthetic biology
for example the potential for
democratizing
uh weapons of mass destruction and
creating entirely new forms of such
things. And
you know the risk of nuclear war still
um looms over us. that could be new arms
races with more nations in this century
than the past. Um, and in more subtle
ways as well, you could have we have
this kind of complex
information ecology
with social and political dynamics kind
of running on top of the information
system we have.
uh we've been changing some of the
fundamental parameters of these
information systems in recent decades
like with the internet then social media
and then
even with the AI we already have
extremely powerful new applications for
surveillance censorship
um that haven't yet been fully
implemented but the technological
ability there is now right you could
read everybody's email and text messages
and listen to everybody's conversation
not just to store it in some giant
database that then some officer could go
in and sort of interrogate. But like you
could do that for everybody all the time
and doing sentiment analysis, not just
keyword search and build up a very
detailed profile of you know which
segments of the population, which
individuals have a positive view of of
the leader versus a negative view of the
leader. Are they planning to do
anything? What are they saying to each
other?
Then you could imagine integrating that
into some system that shapes the So
there are a lot of ways in which our
current
civilization could sort of derail in one
way or another. One is to some sort of
totalitarianism. Another might be just
some kind of radical political
polarization.
Um, another might just be some sort of
distraction into idiocracy where we sort
of become increasingly addicted to
various forms of social media feed that
sort of stimulates the worst parts of
us. Um, there could be other dynamics
that we just don't have the kind of
science that can predict what happens to
these systems when you sort of change
some of the underlying knobs. So there's
some chance that this might result in a
radically better epistemic environment
uh where these tools make it easier for
people to find true information to
evaluate information to sort of keep
track of the
predictive accuracy of different pundits
so they can learn to dial into the ones
who actually know what they're talking
about. That's that's in the cards. But
equally uh it could go in the opposite
direction. So there's just some
uncertainty there. Like I think the
distribution of outcome is quite wide
and and so at one tail you also have
kind of civilizational level
catastrophes from that.
>> Yeah.
>> Um so aside from AI like it it looks
like the current human condition is sort
of transitory.
Um and with AI it also looks transitory.
And I guess I'm hoping we will get the
chance to sort of roll the dice with AI
before we destroy ourselves using one of
these other methods.
>> Yeah,
I'm certainly with you on that. But that
two-pronged potential future going sort
of in the negative or the positive. I
think rightly so a lot of focus has been
on the negative because you know if you
wipe yourselves out or an AI wipes us
out that's pretty vital to try to guard
against but there's also as you briefly
made reference to the potential positive
outlook and of course you had the your
recent book I think it was called deep
utopia is that the correct title of that
where you
>> explored a world in which AI and other
qualities come together to kind of solve
everything, right? You can imagine in
the in the best of all futures that, you
know, AI might, as you mentioned in the
book, you know, wipe out cancer and
solve other health challenges and deal
with climate change and so forth and and
all that sounds wonderful and when one
hears about that, it makes you start to
feel like, wow, there's a bright
possibility going forward. But you also
address the question, how in the world
do we live in a reality like that when
most of us live via overcoming
challenges? It can be the challenge to
put food on your table, the challenge to
raise your kids, the challenge to solve
quantum gravity, the challenge of this
creation of that symphony or that
artwork or or on and on you can go. If
AI can do it all better and has solved
all the big challenges,
how do we live?
Um, yeah, that's that's a big question.
Um if we do end up in a solved world um
then I think
um a lot of constraints
would disappear which allows us to solve
many horrible problems which is good but
it's also the case that our current
lives as you you sort of suggested are
to some extent structured and shaped by
these constraints. Um so at the
superficial level we have the economic
necessity of work right so for many
people um their days are structured by
the need to make a living. So, you have
to go into work and maybe sit in front
of your desk to get a paycheck because
if you don't do that, then you can't pay
the rent. And if you don't pay the rent,
you get kicked out of your flat. And
then if you get kicked out of your flat,
eventually you have to live under a
bridge and it's really cold. And it's
like a real consequence that would come
from failing to perform these difficult
tasks that take time and attention. Now
if we can automate the economy then the
need for human labor would go away.
But so you say okay that will require
some adjustment clearly but still I mean
there are a lot of humans who live
without the need to work for a living
right and some of those seem to have
great lives. Uh so we would all be more
like that like rich aristocrats for
example or or like healthy retired
people full of vitality.
Um but I think the um
it goes deeper um because if you think
it through it's not just the need for
economic labor that would go away but
for all kinds of other instrumental
effort as well.
Um
so people who are
rich today who don't need to work for a
living often nevertheless have very busy
life uh because there are many things
they are trying to achieve that they
can't achieve without putting their own
time and effort into them. Like if you
want to be fit you have to spend the
time
uh on the on the treadmill um or
whatever. If if you want to have your,
you know, your mansion uh decorated in
just the way that you prefer, you have
to spend time looking through the
cataloges and picking out the curtains
and um and so on and so forth. Uh but at
technological maturity um in this solved
world you could have like a kind of a
pill that would induce the same
physiological effects as exercising and
you could have a recommended system that
would do a much better job
at decorating your mansion than than if
you tried to do it yourself. And so
there would be this
removal of a lot of the practical needs
for exerting effort and we would enter
into some kind of post instrumental
condition
um where um
to a first approximation uh there would
be
no need to do anything for the sake of
achieving something else. So like the
only activities that would remain would
be autoilic ones, ones we do for their
own sake.
>> So you exercise because you enjoy
exercise. You don't exercise to be more
fit or to stave off disease
>> even there. Um so certainly you wouldn't
exercise to stave off disease or to stay
fit because that would be a shortcut,
right? But even the goal of enjoying
yourself if if by that you mean um
feeling good like so some people might
like enjoy the experience of exercise or
they feel good afterwards like the end
rush like clearly you could take a pill
or or do what something else to get the
ends without having to
>> you know
make your whole outfit sweaty and like
the whole like
>> but it sounds very much like Nosix's
pleasure machine. I mean, you remember
Robert Nosk, you know, had this idea,
you know, just hook yourself up your
brain to a set of electrodes that, you
know, if you wanted to be the greatest
opera singer, you become the greatest
opera singer in a reality that is pumped
into your brain through these external
electrodes. You never need to actually
do anything because you can have the
experience as if you did whatever it is
that you might have wanted to do. Does
this sort of parallel that philosophical
question of whether you'd hook yourself
up to the pleasure machine?
>> Um
yeah. Yeah. So there like some ways in
which the the questions intersect
whether you would want to connect to
this experience machine or not. Um,
but of course in this old world there
are a lot of things you could do
uh other than merely having experiences.
Um,
and
you could be in contact with the
external world and have a lot of true
beliefs about it. you could interact
with other people um and have real
relationships with them.
Um
and
it would be possible to uh instantiate
more
plausible candidates for for value
um in the solved world than it would be
possible to do in an experience machine.
But if we had to solve a world, Nick,
and if and let's just assume that we
were able to harness enough energy, I
don't know, a Dyson sphere around the
sun or so energy is not a limitation,
and technology is not a limitation.
Presumably, each individual could kind
of create their own little world, part
visual, virtual, part real. They could
have beings populate that world and it
wouldn't be the experience machine
because they'd be really there. But
would that be where we would sort of
fracture into everyone creating their
own mini universe?
>> Well, I mean it would be one possibility
like you could have an experience
machine or maybe even some physical
similocum of an experience machine where
like you have some nanotech arranging
your
>> Yeah. The hollow tech. The hollow tech
in the enterprise, you know. Yeah, but
it might be possible to do better than
that because a lot of people in addition
to valuing having sort of great
experiences, they might also value
certain other things like being
connected to other people. Yeah. For
example. And so why not then try to
think if sort of having good experiences
like is like I don't know like utopia
level one, right? Like maybe there are
even better things that we could aspire
to and I think certainly there are. And
then like there there is incidentally
with this experience machine thought
experiment that NOS has like some things
that are sort of uh
um
like swept under the rug. some
implementation difficulties like in
particular the uh types of experience
machine where you have more many people
in them or experiences involving other
people um where it's not entirely clear
that you could generate these
experiences
of you interacting with other people
without simultaneously also generating
some of the experiences that these other
people would be having. Um
and but anyway um it looks like you can
sort of think of a
multi-layered
defensive architecture where you say
well in this old world like
um can you really have any kind of good
life there? And so then you can sort of
try to go through one by one the
different things that we think are
important for human life to be good. So
you could start at the basic level with
you know maybe some simple experiences
like positive a effect or actually
enjoyment in the subjective sense. So
clearly that would be possible to have
to a an extremely great degree in utopia
like life could just be
so much more fun and pleasant and than
than than our current existence. But
then you might say, well, we might want
something more than just
feeling blissed out. Like, so then you
say, well, you could add experience
structure. Like maybe rather than
feeling like a
a junky sort of
sprawling on some fleafested mattress,
but feeling
pleasure. Like you could attach that
pleasure to say aesthetic contemplation
or understanding of deep truth or
appreciation of other people or of
virtue.
That seems a little bit better. But you
could go beyond that. It doesn't need to
be a passive experience. You can say,
well, why can't we just be doing things
as well? And certainly you could um even
if a lot of these instrumental
necessities go away, we could create
artificial purpose
um which is basically when you set
yourself some goal just for the sake of
then having that goal and being able to
be motivated to pursue it. And so this
is what we do when we are playing games.
Um,
if you're playing a game of golf, there
is no real pre-existing need for the
ball to go into a sequence of 18 holes.
Um, it's like a goal that you make up.
But moreover, you embed into the goal
itself that it can only be achieved
using a certain restricted set of means.
So it doesn't count as winning in golf
if you pick up the ball with your hand
and put it in the 18 holes sequentially,
right? Like it's part of what it means
to succeeding at goal to achieve your
goal that you're using this this this
inconvenient method of hitting the ball
with a club. So now you make up this
goal and you adopt it
and then after that you have now
instrumental reasons to focus hard uh to
try to figure out which way the wind
blows and to place your feet in the
right position and so on. All the things
you need to do to be successful at golf
now are instrumentally necessary to
achieve this goal that you have set
yourself of winning in golf.
Um
and so
what now is for most people most of the
time
a marginal activity um could become a
larger part of the lives of utopians
where various forms of game playing. And
think not just sports or board games,
but think maybe they would have these
society level games that might stretch
over months or years involving all kinds
of different modalities and challenges
and teams and artistic creation uh might
just constitute a much larger part of
what their existence is about. So then
you can put a check mark
in activity. That's certainly something
the utopians could have and at least
they could have artificial purpose and
and maybe also they could have some
forms of natural purpose purposes that
are not just sort of made up for the
sake of having purpose although there it
does I think become a little bit more
challenging but I think some forms of
natural purpose could survive into
technological maturity.
>> What do you think we would lose? Would
we would would we lose our humanity if
life was so constructed
or I mean you know I tend to view things
a little bit differently than others. I
even think that some of the things that
we take to be you know objectively
important things that we do. I think
it's all subjectively created. You know
I think mathematics is not something
that is out there that we're
discovering. I think we invent it. We
invent the problems of mathematics and
then mathematicians try to solve them. I
think we invent the problems of physics.
Frankly, we impose semblance of order on
the external world and then we try to
make our theories describe things within
that rubric as best as we can. But I
consider it to all be humanmade from
from the get-go. So from that point of
view, what you're describing is just an
extension of what we've always done. I
mean, how do you see it?
Well, I think in our current situation,
the world imposes a lot of constraints
and we have to adjust ourselves to those
constraints.
>> Um, and that does create natural purpose
in the sense that there are things we
care about a lot
uh that we can only achieve if we make a
lot of effort.
Um,
and those things that we care about a
lot, uh, we didn't just decide to care
about them for the sake of having
something to care about. They kind of
built into us. Um, and so that kind of
natural purpose might be one of the
things that we might to some extent lose
in a solid world. Um
like right now
we have significant opportunities to
really make the lives
of other people better.
um whether
in in our immediate circles um we can do
something for a friend or at the global
level we can contribute to you know try
if you're you could try to solve cancer
or donate to some charity or advocate
for some social reform
um and this can make a big difference to
lives like if if if you imagine a world
where all these most pressing problems
have either already been solved or to
the extent that there remain big
problems there in any case much more
efficiently tackled by AIs and robots
where there is no way for us to
contribute
then you might think there is something
lost in as much as right now one
positive value of helping others is that
their lives go better but some people
also think that your life is going
better if your life in part consists of
having a positive impact on other people
or the world that it's good for you to
be playing such a positive role. Um, and
and that's something that these future
utopians might have less of. Um, so I'd
say that if purpose is your thing, knock
yourself out now, right? Then the world
is just full of need and suffering and
misery and injustice. Now is the golden
age of purpose. Like there are so many
opportunities to try to make the world
better. And uh hopefully that will then
come a day when when those opportunities
will be fur and
>> so and route from now to either of these
two possible futures the uh doomsday
scenario the deep utopia version of
course where in a more nuts andbolts
transition period right now and
education of course is vital to the
species being able to do the things that
you're talking about. There's a big
debate of course about how AI should be
in or not in the lean learning process
in the in in the classroom. You know,
even today I'm supposed to be on a
committee here at Colombia, you know,
talking about what do we do about AI in
our core curriculum? You know, do we
allow the students to use it? Uh do we
rule it out? Do we try to fine-tune
assignments so that they can use the AI
but still have some kind of input? And
of course, as I keep telling the
committee, every time I walk around the
library at Columbia, 99% of the
computers screens are open to AI. So the
idea of ruling it out is like, you know,
wishful thinking. Do you have any
thoughts on on how to approach these
conundra?
>> I mean, education is
it is a weird
age in which to do education. Um
especially for the lower grades in in as
much as we can expect the world to
change quite profoundly between now and
the time when like say say somebody
who's eight today right it'll be another
10 15 years before they are supposed to
go out there and
make a living or whatever and and you
know the world might just look so
different then I mean I think from a
pragmatic point of view
it clearly kids need to learn to use AI
I tools. I mean maybe just split it like
so like do half the day where they are
not allowed to use AI and need to solve
things using their own minds and
memories and then like the other half
where they could
do more difficult assignments but where
they're allowed to use AI tools to the
full extent. I don't know if the split
should be 5050, but
it seems to me that it would be prudent
to hatch uh our bets currently and uh to
yes
learn to use these AI tools, but also at
the same time not to neglect to build up
the kind of uh capacities that might
require,
you know, memorization and working
problems out with your own mind without
the use of AI assistance.
Um because we don't know exactly what
would be lost if somebody grows up
without having done that.
>> Yeah.
>> And just having relied on AI tools all
along.
>> Right. Now, now I do wonder just sort of
one one final question. This can either
be in the doomsday scenario or in the
utopian arm of the bifurcated possible
futures. But you can also envision that
the us versus them framing that we tend
to bring to these questions, you know,
natural intelligence of humans,
artificial intelligence of a
computational system,
balancing the reliance of one on the
other and so forth. There's a possible
future where there's a hybridization. I
don't know if it's literally a
hybridization that we sort of
internalize the architecture of these
systems through implants or if it's just
a coming together in such a seamless
manner that maybe we no longer draw a
distinction between what was sort of
natural and what was artificial. It's
just this thing called intelligence that
that we we all have.
Is is that uh uh in the cards as a
possible future that you envision is
realistic?
>> Um yeah, I think
uh after uh we get super intelligence
then we will have a kind of telescoping
of the farther future. The possible
technologies that human civilization
um could have developed if we had 20,000
years to make progress. We would have
space colonies perhaps and perfect
virtual reality and anti-aging
technologies and all all these other
things that are physically possible but
just very hard might come within a few
years after you have super intelligence
doing this kind of research and
development at digital time scales. Um
and amongst those possible technologies
I think are
uploading of
human minds into digital substrate and
then various forms of modification of
that
maybe the ability to
using an AI that is able to edit each
synapse.
um
the ability to then download knowledge
and skills or reformat your mind in
different ways. I think there's just
this huge space of possibilities that
opens up
um
a much larger realm of possible modes of
being
where hopefully we would get
the opportunity to sort of charter
trajectories out into this much larger
space where perhaps you know if the
urgency was removed if we had say cures
for
all diseases and for aging itself and
AIs that could look after us to make
sure that we didn't catastrophically
destroy the world. Um maybe then it
would make sense to slow down a little
bit and to sort of pursue some path that
perhaps eventually results in us
becoming some sort of strange posthuman
type of being or beings. But
along a path where maybe we sort of
pause to smell the flowers as as we go
along. That there might be certain types
of values that
are realizable
with our current human set of capacities
and then maybe we would want to explore
those for a bit before then maybe
gradually expanding
and upgrading our capabilities. And then
eventually that might lead us to grow
into something
um quite different just as
like if you have a toddler they will
eventually grow into something quite
different like an adult is in many
respects almost a different kind of
entity than a toddler in terms of what's
going on inside their minds the kind of
problems they're dealing with. Um, and
yet
for the most part, we don't think it's
bad for a toddler to grow up, even
though it means the toddler is no longer
there. And so perhaps similarly with us
humans, we now think of us as being
grown-ups and being so very mature and
big. But I think we are like babies
really. Uh, it's just kind of stunted
because of our biological constraints.
So we just cease growing up.
>> Yeah. and have this arrested development
at age 20. And then we stay hovering for
a few decades and then just as we have
started to acquire a little bit of
knowledge and experience, our brain
starts to rot and it's all erased again.
And and maybe that could be a way to
sort of allow us to continue to grow and
develop together with with our friends
and and communities.
Um, and it would be exciting to see what
kinds of level of maturity and like
self-realization that would be possible
if we could continue, you know, for many
hundreds of years. And then maybe
eventually you could imagine adding
extra neurons and extra memory and extra
forms of vitality and new emotional
responses and so forth. I I think we
just haven't seen nothing yet in terms
of what's possible. Yeah, I I I totally
agree with you. There was a science
fiction book written by Arthur C. Clark
years ago called Childhood's End.
Basically, the end of the childhood of
the species called Humankind, you know,
and this may well be the moment where
we're starting on the trajectory of
ending the childhood of the species and
it's exciting to see where it goes. All
the same though, just to paraphrase what
you said earlier, it's both exciting and
terrifying and you have to perhaps have
a fatalistic outlook, do everything that
we can to shape the best future we could
have and we'll just see where it all
goes. So Nick Boston, thank you so much
for joining us. Uh this is really a
exciting conversation and uh I look
forward to seeing where this all goes.
>> Yeah, it'll be an exciting ride if
nothing else.
>> Absolutely. Thank you.
>> Thanks.
Heat.
Heat.