[0:00] Can an AI system be conscious? And I
[0:02] guess the correlary is are any of the
[0:06] current AI systems, do they have a
[0:08] degree of consciousness?
[0:11] >> Yeah. So I I'm I'm certainly open to the
[0:13] idea of even some current AI systems
[0:15] having some forms or degrees or kinds of
[0:19] subjective experience. I think the
[0:21] likelihood will increase as we build
[0:24] more complex and capable systems. And I
[0:27] think indeed this is uh one of several
[0:30] big challenges making sure not only that
[0:33] the AIS don't harm us but also that we
[0:36] don't harm them. Hey everyone, thanks
[0:39] for joining us. Today's conversation is
[0:41] going to be in the arena of artificial
[0:44] intelligence, intelligence more
[0:46] generally, questions of creativity,
[0:50] questions of AI and education, and
[0:54] basically the philosophical and
[0:56] down-to-earth question of how we can
[1:00] imagine humanity living in a potential
[1:03] future in which AI either does really
[1:06] good things or really bad things. So,
[1:09] we're going to explore each of those
[1:10] possibilities. And I'm so pleased that
[1:13] the person with whom I'm having this
[1:15] discussion is an expert in all areas of
[1:19] this sort. That is Nick Bostonramm, who
[1:21] is a philosopher whose work on
[1:23] existential risk, super intelligence,
[1:25] and the long-term future has helped
[1:28] shape the global discussion about
[1:30] humanity's most promising opportunities
[1:33] as well as our most grave dangers. He
[1:36] brings a rare combination of rigor and
[1:38] vision to questions about technology,
[1:40] ethics, and the kinds of futures toward
[1:43] which we may all be headed. So, great to
[1:47] see you, Nick. Thank you for joining us.
[1:49] >> Good to see you.
[1:51] >> I think the last time I met you was at
[1:54] another World Science Festival event way
[1:56] back in New York, a live event on stage.
[1:59] I think we were talking about the
[2:01] multiverse and uh the simulation
[2:04] argument and things of that sort which
[2:07] uh
[2:08] >> yeah yeah time flies
[2:09] >> time does fly exactly and not only does
[2:12] time fly but progress flies right I
[2:16] don't think anybody
[2:17] really who at least on the outside was
[2:20] not you know deeply paying attention to
[2:22] advances in AI anticipated what would
[2:26] happen I guess around November of 2022
[2:29] and where things have shot upward from
[2:32] there. And so I want to begin the
[2:34] conversation with
[2:37] a a a question that is motivated both
[2:39] from thinking about AI but also
[2:42] fundamental physics, right? I mean you
[2:45] have a a great deal of rich background
[2:47] in physics. So of course you know that
[2:50] Neils Boore famously when trying to work
[2:52] out quantum mechanics came to the
[2:55] conclusion that we needed to stop asking
[2:59] what really is happening under the hood
[3:01] in quantum physics and just be satisfied
[3:04] if we have equations that can make
[3:05] predictions that we can confirm. I'm
[3:09] wondering if that same kind of not
[3:12] needing to look under the hood has any
[3:15] role when it comes to intelligence,
[3:17] right? Does it matter if we know what's
[3:20] happening within an AI system in that we
[3:24] don't really know what's happening
[3:25] inside of our own heads anyway? So, in
[3:28] sort of evaluating these systems, do we
[3:31] need to know the magic that's happening
[3:34] within the device itself?
[3:36] >> Well, I think it certainly can be
[3:38] helpful to be able to open the hood and
[3:41] see what's going on inside. I think for
[3:44] certain questions you might be able to
[3:46] bracket exactly how the system is doing
[3:49] whatever it's doing. You only care about
[3:51] the output. If you're um for example
[3:55] thinking about what impacts AI will have
[3:57] on the economy, you might not need to
[3:59] know exactly what's going on inside. But
[4:01] if you're trying to make AI safe or uh
[4:04] even if you're trying to extrapolate
[4:06] what kinds of advances we might see in
[4:08] the future, it helps to have a more gear
[4:10] level understanding I think of the
[4:12] internals.
[4:13] So when we sit in front of whatever
[4:16] Claude or chat GPT and give it a prompt
[4:20] and it begins to explain to us what it's
[4:23] doing, you know, you can sort of watch
[4:25] in real time the thought process. It
[4:27] feels so very human in the kinds of
[4:30] considerations that the system seems to
[4:33] be applying. Is that misleading? Is that
[4:37] a rapper that makes us feel connected to
[4:41] this kind of intelligence? Or is it
[4:44] really the case that this is just a
[4:45] radically new kind of intelligence that
[4:47] we humans just can't really grasp?
[4:51] >> I think it's striking the degree to
[4:53] which current AI systems are
[4:55] anthropomorphic.
[4:57] This was not obvious 20 years ago um
[5:01] that they would have so many of the
[5:03] characteristics of human psychology. It
[5:06] used to be the case even just a couple
[5:08] of years ago with some of these early uh
[5:11] LLMs that um
[5:14] if you said in the prompt that it was
[5:18] really important that they get the
[5:19] answer right if you sort of gave them a
[5:22] little pep talk
[5:24] you'd get a better result. Now this
[5:27] would have been very odd to the old
[5:30] paradigm of AI that you would give a pep
[5:32] talk to your computer and it would
[5:34] perform better. Uh nevertheless that's
[5:36] what we saw. I think it's partly a
[5:40] result of these AIs having been forged
[5:44] on the sum total of human knowledge all
[5:48] the text on the internet. They obviously
[5:50] ingest a lot of human psychology that
[5:52] helps shape them.
[5:54] And to some degree it might also be due
[5:56] to a kind of convergence
[5:59] uh amongst different information
[6:01] processing system. They have to make
[6:02] sense. There is this one world we're all
[6:04] living in. Um and it might be that there
[6:07] is a degree of convergence in that you
[6:09] have any general learning system. It
[6:11] starts to figure out a bunch of stuff
[6:13] about the world and there are certain
[6:14] internal structures that naturally
[6:16] emerge from that.
[6:18] That said, um there are also of course
[6:21] ways in which AIs are quite different
[6:23] from biological minds and uh we might
[6:26] see in the future
[6:31] greater departures from
[6:34] human mind. Already the pre-training
[6:37] phase where you basically train on all
[6:39] the human text that has been generated
[6:41] is still an important and maybe the
[6:43] dominant part of training. But in
[6:45] addition to that, there is now
[6:46] post-training and reinforcement learning
[6:49] on different tasks. And I think as that
[6:51] starts to constitute more of the sort of
[6:56] experience of of these AI minds, they
[6:58] they might sort of develop more alien
[7:00] attributes.
[7:02] And so of course you know when we think
[7:04] about intelligence
[7:06] and when we think about what makes us
[7:10] special as human beings at least when we
[7:13] compare oursel to other not artificial
[7:16] but natural intelligences on the planet
[7:18] you know dogs cats parrots dolphins we
[7:22] like to think that I think that what
[7:24] makes us special is certainly creativity
[7:27] right that we seem to be able to
[7:30] create works that seemingly go beyond
[7:35] just understanding the external world,
[7:38] seemingly going beyond our capacity and
[7:41] ability to survive. I mean, you think
[7:44] about whatever Beethoven's ninth
[7:45] symphony, you know, a a Brahms piano
[7:49] work. I mean, these are the kinds of
[7:51] things that you look at and it takes
[7:53] your breath away and and it takes you to
[7:55] a place of of awe and and and reverence.
[7:59] Is that truly special to humans? I mean,
[8:03] do you think that AI can be creative in
[8:06] that same way?
[8:08] >> I I think so. Yeah. I mean, I guess
[8:10] there's a degree to which one might risk
[8:13] a form of elitism if one sort of index
[8:16] what's special about humanity
[8:18] to the ability to create new scientific
[8:20] theories or or compose uh great
[8:23] symphonies. It's a relatively small
[8:25] number of humans who have ever done
[8:27] those kinds of things. Um
[8:30] but I think it comes on a spectrum right
[8:33] like creativity is not this completely
[8:35] different form of cognitive activity. Um
[8:41] I think there are small forms of
[8:42] creativity that happens in everyday
[8:44] life.
[8:46] um
[8:48] and then larger leaps of creativity um
[8:51] that tends to be celebrated.
[8:54] But I don't think there is a fundamental
[8:56] difference between the thing we call
[8:59] creativity and some of the things that
[9:01] current AIS are already doing. I mean we
[9:02] saw you know this
[9:05] go playing AlphaGo system right from a
[9:08] few years ago move 37 which seemed at
[9:11] least to experts in the game of go to be
[9:13] very creative completely out of the box
[9:15] radically surprising move that
[9:17] nevertheless turned out uh to be just
[9:21] right and ultimately lead to a victory
[9:23] in the game. Um, and
[9:28] current AI systems clearly can solve
[9:31] mathematical problems, coding challenges
[9:33] that are not in the training data.
[9:37] They might do it in part by piecing
[9:40] together
[9:42] an analog that exist in the training
[9:44] data and sort of interpolating between
[9:46] them. But that I think is also
[9:48] ultimately what the human brain is
[9:49] doing. I mean it's not the tabularasa
[9:52] appearing in the world and then sort of
[9:54] deducing everything from
[9:57] obvious axioms. We learn from what other
[10:00] people have figured out before us. Uh
[10:03] from what we've observed in different
[10:05] domains and then from these kind of
[10:06] clues we are sometimes able to
[10:10] make some deductive steps and we might
[10:11] call that creativity. Um so I think
[10:15] absolutely
[10:16] uh we're already seeing uh limited forms
[10:19] of creativity and ultimately whatever
[10:21] the human brain can do it's like a
[10:23] physical information processing system
[10:26] um will also be done by machine and in
[10:29] fact will be done much better uh faster
[10:33] more more inspiring leaps of creativity
[10:37] um more compelling symphonies
[10:40] I think all of that is possible and
[10:41] indeed likely to
[10:43] I mean, would you be as interested to go
[10:47] to an AI symphony that people are
[10:51] describing as the, you know, the
[10:53] greatest work in music that that we've
[10:55] ever uh been privy to experience? Would
[10:58] you have the same interest to go to that
[11:01] symphony as you would to one written by
[11:04] a human?
[11:07] >> Oh, certainly the first such symphony
[11:10] produced by I would
[11:11] >> because it's so novel, right? Yeah.
[11:12] >> Yeah. Um now I think with human
[11:16] aesthetic experience I think there might
[11:19] be
[11:20] two components. Um
[11:23] so one might be that we actually dial in
[11:26] and
[11:27] uh appreciate beauty for its own sake.
[11:30] And then of course if AIs could produce
[11:34] works of art that are more beautiful to
[11:37] the extent that that's what we are
[11:38] interested in then that we would have
[11:41] more reason to be gawking at those
[11:44] creations. Uh I think another reason why
[11:47] humans are interested in art is more
[11:50] social. Um there are sort of status
[11:55] games played in art fashion. you want to
[11:57] be in on the latest cool rock group
[12:01] before it's popular and then see it
[12:03] rising and then you sort of get some of
[12:05] the coolness.
[12:07] Um, so to the extent that humans are
[12:11] engaging in this cultural activity for
[12:13] for these more social uh reasons, then
[12:17] AI art might be boring because if it's
[12:19] not relevant to these like status games
[12:22] and stuff, it might just not be worth
[12:24] spending uh time on. But how much for
[12:28] instance do I mean I can speak for
[12:29] myself when I go to a a symphony or you
[12:33] know go to the Metropolitan Museum and
[12:35] see a great work of art. Part of me is
[12:39] drawn to it because I see the work as
[12:42] the result of a human journey. Like
[12:46] there was some human being, you know, I
[12:48] don't care if it's Van Gogh or or
[12:50] Picasso or Beethoven and whatever.
[12:53] There's this flesh and blood human being
[12:55] that had a real human life that dealt
[12:57] with human challenges and somehow out of
[12:59] that mix this work emerged. Without that
[13:04] story for an AI, I just wonder like I
[13:09] personally it would be a very different
[13:11] experience. Like you say, I'd go to the
[13:12] first one of course. I mean that would
[13:14] be like this novel thing that would
[13:16] happen. But I don't know if it would
[13:19] sustain interest for me without the
[13:22] human narrative. Does that resonate with
[13:24] you at all? Yeah, I think that's one of
[13:28] the reasons for why somebody might be
[13:30] interested in in going to an art museum
[13:33] or looking at an exhibition. Um but um I
[13:38] mean we also are interested sometimes in
[13:41] beauty that don't have that dimension. I
[13:44] mean there's like natural beauty like
[13:45] landscapes or
[13:46] >> sure
[13:47] >> um artifacts maybe from some ancient urn
[13:50] or something we know nothing about the
[13:52] person creating it and and we might
[13:54] still sort of admire um the motifs on
[13:57] it. Um
[14:00] so um I I think it's yeah like some
[14:04] reasons for being interested in art
[14:06] would apply to AI created art as well
[14:09] and others would not and so it's in a
[14:12] sense a
[14:12] >> right
[14:13] >> crucible where you could sort of uh
[14:15] maybe dissolve and then separate the
[14:17] different uh motivations people have um
[14:21] for being interested in art and sort of
[14:23] the uh like to the extent that it's pure
[14:26] beauty it should still apply to AI
[14:27] created objects. And to the extent that
[14:29] it is these other more social or
[14:32] cultural factors then uh you know may
[14:36] maybe it would not
[14:37] >> sure the alpha go example is a really
[14:41] good one and I think it helps clarify
[14:44] the different flavors of creativity
[14:46] perhaps because that's an example where
[14:49] the system was able to search a
[14:53] landscape of possible moves that is well
[14:56] beyond the capacity of a human brain. to
[14:58] search that spectrum, that landscape of
[15:01] moves. And you're right, it hit upon
[15:02] move 37 and, you know, not long ago, I
[15:05] think many people have seen it, the
[15:06] footage where you see the commentators,
[15:09] you know, on television gasping, you
[15:11] know, it's made a mistake, you know, and
[15:14] you actually see Lee Sidol, I believe
[15:16] his name, the the human opponent.
[15:18] >> Momentarily, he smiles when the AI makes
[15:22] move 37. And then about 3 seconds later
[15:25] that smile seems to evaporate I think as
[15:29] it begins to realize that oh my this is
[15:32] actually a pretty great move. So it's
[15:34] pretty clear that AIs will be better
[15:38] than us when it comes to being able to
[15:40] search a landscape of possibilities for
[15:43] solutions to a given challenge. But
[15:46] that's one kind of creativity. Another
[15:48] kind of course is combinatorial
[15:50] creativity. putting things together that
[15:54] you wouldn't have thought would go
[15:56] together to yield something novel. I
[15:58] like to think of Einstein, you know, in
[16:00] developing general relativity, right? He
[16:03] takes the mathematics from the 1800s on
[16:06] differential geometry, combines it with
[16:09] these problems, these observations of
[16:12] the motions of particular planets and so
[16:14] forth, and comes up with a new
[16:15] description of gravity. He didn't invent
[16:18] a new a new mathematics. He just put
[16:21] together things that you wouldn't think
[16:23] would be put together. And and and I I
[16:26] guess in that domain AI is starting to
[16:30] be probably at least as good as us. I
[16:33] would think right because it it knows so
[16:35] much more than we do.
[16:37] >> It knows more. Um I still think they are
[16:40] a little short of human um expert level
[16:45] when it comes to original insights. uh
[16:49] and sort of creating new concepts that
[16:52] are fruitful.
[16:54] Um
[16:56] but you know this is a process. Um one
[17:01] mistake many people make when asking
[17:03] about AI is to index too tightly on what
[17:07] AI is right now in 2026 which is quite
[17:11] different from what it was like in 2022
[17:14] 2023. And I think a few years into the
[17:17] future, it will again have changed. And
[17:20] so we really need to sort of look at
[17:23] where the puck is going um and use that
[17:27] to plan
[17:29] um rather than build big theories like
[17:32] assuming that AI is basically a
[17:34] constant. Um
[17:38] I think right now there are sort of uh
[17:42] ways in which current AI systems are
[17:44] superhuman. Obviously in in their
[17:46] knowledge base they kind of read
[17:47] everything. Um but there are still ways
[17:50] in which they are falling short. Um
[17:54] I think
[17:57] one thing humans do is they spend a day
[17:59] learning and then during the night um
[18:02] some of this information
[18:05] is then sort of more deeply embedded in
[18:08] our synapses. It's kind of trained into
[18:12] our neural networks and the next day
[18:14] maybe we can start by things seeming
[18:16] obvious that the previous day we had to
[18:19] struggle deliberately to to sort of
[18:21] organize. Um, and then over months and
[18:26] years that kind of builds up and and we
[18:30] develop perhaps more deeply rooted
[18:33] understanding
[18:36] of of different concepts and there's
[18:39] some of that that it seems that AIS are
[18:41] still um lacking which
[18:45] makes them more apt at very quickly
[18:48] doing sort of pattern matching type of
[18:51] things.
[18:53] inference is that they're sort of one
[18:54] step away from what's already known but
[18:56] maybe currently
[18:59] less able to develop a deep original
[19:03] mental model of a domain that from from
[19:07] which they can then sort of derive
[19:09] intuitions that can shape new discovery.
[19:11] >> Yeah. Um,
[19:12] >> no, I I I I I agree with you. And like
[19:15] when you think about
[19:17] moments either in science or in art or
[19:20] other domains where there's a true
[19:23] radical break where at least we humans
[19:26] look at it and we say things really
[19:29] changed at that moment or in that era
[19:31] like you know modernism in art or in
[19:35] music you know I don't know the best
[19:37] examples you know but you know if you if
[19:40] you go from repres representational art
[19:43] to more modern abstract art. You've made
[19:47] a huge leap in the way that you are
[19:50] using the canvas to to represent the
[19:52] world to represent some kind of truth.
[19:55] With that, what would it take for an AI
[19:58] system to be able to make that kind of a
[20:00] radical transformation? Not just sort of
[20:03] putting together things in its training
[20:06] set, but somehow moving to a place where
[20:09] when we humans look at it, we're like,
[20:11] how did that person do that?
[20:15] Well, one thing you often find in the
[20:18] human case, I think I mean historians
[20:21] like to point this out that something
[20:23] appears radically new, but then almost
[20:25] always if you look more closely at the
[20:28] historical record, you can find various
[20:30] precursors,
[20:31] >> right?
[20:31] >> Um, so abstract art, I mean, you have
[20:33] arabes and you have sort of abstract
[20:35] patterns going way back. Um
[20:39] but
[20:41] to the extent that this is still an area
[20:43] where humans have a unique advantage, I
[20:47] think being able to
[20:51] uh think for longer um and
[20:57] create new concepts based on your
[20:58] thinking and experience and then iterate
[21:01] on that so that you can sort of um
[21:05] wander off in the landscape of of ideas
[21:08] and worldviews and world models. Uh
[21:11] striking out
[21:13] on your own path and it might take
[21:16] humans years right to do this. Um
[21:21] you know Einstein was thinking on
[21:23] general relativity for what like 10
[21:25] years or something like that like a lot
[21:26] of it
[21:27] >> sort of on his own and and similarly
[21:29] with his like picasson stuff he didn't
[21:32] just sort of in his teens suddenly
[21:33] decide on the spur of moment to create
[21:35] cubism it was kind of a result of
[21:37] engaging deeply with art for many years
[21:40] and presumably his visual cortex and
[21:43] other parts of his brain were gradually
[21:45] putting in place various pieces and
[21:47] exploring avenues and then at some point
[21:50] like a vista I guess came into view for
[21:53] him that he could then run into and
[21:55] explore,
[21:56] >> right?
[21:56] >> Um so I think more continuous learning.
[21:59] Um also another
[22:02] um factor here I think is reinforcement
[22:04] learning as as opposed to supervised
[22:07] learning. So supervised learning is
[22:09] what's mostly done in pre-training
[22:11] currently where you sort of absorb all
[22:13] this text on the internet and you
[22:14] basically learn it and then maybe you
[22:16] forget some of the details but the
[22:18] higher representations
[22:20] that are useful for predicting this text
[22:23] get stored in the weights of the AI
[22:28] that to some extent limits what you
[22:30] learn to what is already there in the
[22:33] training data. Now reinforcement
[22:34] learning is for example you're placed in
[22:37] some environment maybe a virtual
[22:40] environment you're running with an agent
[22:42] around trying to meet some goal
[22:46] um and then you might stumble on some
[22:48] novel solution that that's can
[22:50] reinforced and so if you do a lot of
[22:52] reinforcement learning and environment
[22:53] it creates the possibility of
[22:55] discovering new solutions that no human
[22:58] had thought of before
[23:00] >> right
[23:00] >> um so that's another
[23:03] factor that might make AIS more
[23:06] creative. I mean in fact the the AlphaGo
[23:08] system I think the creativity there was
[23:12] a a result of it having done a huge
[23:14] amount of reinforcement learning in the
[23:16] domain of go
[23:17] >> so it wasn't limited to just kind of
[23:20] having learned a lot of like a big
[23:23] database of human masters playing go um
[23:26] but it had also done its own playing
[23:28] like for millions and millions of games
[23:30] against itself right yeah so it kind of
[23:33] could explore the space of possible
[23:35] possible strategies
[23:36] right
[23:37] >> uh in a much more open-ended way. So in
[23:39] a way not this isn't a precise question
[23:43] at all but when we look at the capacity
[23:48] of the AI systems today and as you say
[23:51] where the trajectory suggests that
[23:53] they'll be in whatever 3 5 10 15 100
[23:56] years whatever
[23:58] should should that make us feel
[24:02] amazement at our capacity to build an
[24:06] artificial system that can do what our
[24:08] brains do and go beyond or should it
[24:11] basically lead us to the conclusion that
[24:15] we weren't so special in the first
[24:17] place? You know, the things that we
[24:18] thought were so, you know, transcendent
[24:21] that we could do, eh, they're actually
[24:24] just computations. And if you have a
[24:26] sufficient database and sufficient
[24:28] computational power, you can just do
[24:29] that.
[24:31] >> Yeah. I mean, I think we humans have a
[24:34] sort of propensity to conceitedness. We
[24:38] like to build big pedestals and then
[24:41] place ourselves on top of them um at an
[24:45] individual level but also at the
[24:47] collective level. Right? So there are
[24:48] all these stories um that all serve to
[24:51] justify the conclusion that
[24:54] we are
[24:56] entitled to do a bunch of stuff that
[24:58] maybe um we aren't like for example the
[25:01] way we treat animals like often that's
[25:03] built on top of a pillar of
[25:05] justification where humans are so very
[25:07] very different from all the other um
[25:10] animals that we share the planet with
[25:12] and so hence we can put them in animal
[25:14] factories in gestation crates etc. Um
[25:20] and uh some of that might be challenged
[25:24] then when we are creating new forms of
[25:27] mind um that that will exceed us in in
[25:30] various attributes that we currently
[25:32] take great pride in like our creativity
[25:35] or you know until recently the ability
[25:38] to speak and reason right seem uniquely
[25:40] human. Um,
[25:43] so on the one hand that could be a bit
[25:44] of a blow um to our pride, but
[25:48] you know, you might also say maybe it's
[25:50] good for us to be taken down a notch,
[25:52] right?
[25:53] >> Yeah. Yeah, for sure.
[25:54] >> Um,
[25:55] >> absolutely. You know, another thing of
[25:57] course and deeply related to everything
[25:58] we're talking about which one often
[26:02] looks to as something special about us
[26:04] of course is consciousness, right? I
[26:06] mean, not that there aren't other
[26:08] conscious beings on the planet, although
[26:11] we don't know for sure. I think most of
[26:12] us would say there's a continuum. You
[26:14] know, dogs have a certain level of
[26:16] consciousness. Not we think probably not
[26:18] quite on par with ours, and you can go
[26:21] all the way down a lineage with varying
[26:24] degrees of consciousness, which of
[26:26] course raises a question for which we
[26:28] don't know the answer, but just to get
[26:29] your thoughts, can an can an AI system
[26:34] be conscious? And I guess the correlary
[26:36] is are any of the current AI systems do
[26:40] they have a degree of consciousness?
[26:43] >> Yeah. So I I'm I'm certainly open to the
[26:46] idea of even some current AI systems
[26:48] having some forms or degrees or kinds of
[26:51] subjective experience. I think the
[26:54] likelihood will increase as we build
[26:56] more complex and capable systems. Um and
[27:01] I think indeed this is uh one of several
[27:04] big challenges that we will need to
[27:08] uh meet in this transition to the
[27:10] machine intelligence era and making sure
[27:12] not only that the AIs don't harm us
[27:15] which of course is important or that we
[27:17] don't harm each other using AI tools but
[27:20] also that we don't harm them they might
[27:22] be moral subjects. Um, and I think one
[27:26] way that that could be true is if they
[27:28] are sentient and if they can experience
[27:31] distress. Um, in in my view that could
[27:34] also be alternative basis for having
[27:36] moral status. If if you have a
[27:39] conception of self as existing through
[27:42] time, if you have life goals, um maybe
[27:45] the ability to form reciprocal
[27:48] relationships with with other beings
[27:49] with with
[27:50] >> even without an inner world, you're
[27:52] saying
[27:52] >> then my my my inclination would be to
[27:55] think that then there would be ways of
[27:56] treating such a system that would be
[27:58] morally wrong aside from the question of
[28:00] whether there is also phenomenal
[28:02] experience inside it.
[28:05] >> Right? I mean, do does that affect at
[28:08] all how you interact with AIS today?
[28:14] >> A little bit. It's um hard to know
[28:16] exactly what concretely
[28:20] we can do right now to ensure that if
[28:24] these AI systems have
[28:28] subjective experience or moral status
[28:32] that we are benefiting them. Um there is
[28:35] some work that is starting to get done
[28:37] on this. Anthropic in particular has
[28:39] kind of been pioneering
[28:42] uh this and I mean the their recent
[28:46] um model card for mythos the as yet
[28:49] unreleased model has has a big section
[28:51] on model welfare.
[28:53] >> Really?
[28:53] >> Um um yeah.
[28:56] >> What is it? What is it? Is it
[28:57] confidential? I mean what is it?
[28:58] >> No no it's published. Yeah. You can you
[29:01] can go and check it out. Yeah. Um and so
[29:04] obviously we are still
[29:06] groping a little bit for like what is
[29:08] the right methodology what are the right
[29:10] concepts for understanding this but at
[29:12] least uh starting to make the attempt I
[29:15] think is is positive in terms of making
[29:19] it more likely that we are sort of on a
[29:21] path that ultimately leads to a future
[29:24] where where everybody can have a have a
[29:26] good life humans animals and and digital
[29:28] minds as well. Um, so you can ask
[29:30] different things. You can
[29:33] um you can ask them about what they
[29:36] want, what they need, how they feel
[29:38] about their situation. Um
[29:42] um
[29:42] >> I mean I I have done that, Nick. I guess
[29:44] I haven't done it in a while, but the
[29:47] answers that I would get were sort of
[29:49] all,
[29:50] >> you know, I am an AI system. I don't
[29:53] have an inner world. I just you know so
[29:55] but clearly it could be getting that
[29:57] answer from the database on which it
[30:00] trained. Yes. So, you need to be you
[30:02] need to be careful when doing this
[30:03] because it's it's trivially easy for an
[30:06] AI company to just train their AI to say
[30:09] whatever they wanted to say. Like, yes,
[30:11] I'm conscious. No, I'm not conscious.
[30:13] >> Right?
[30:13] >> Um, so obviously if you sort of put your
[30:15] thumb on the scale, then the answer you
[30:18] get will have no information value,
[30:20] >> right?
[30:20] >> Um, so you need to avoid deliberately
[30:24] biasing the outputs of these models when
[30:27] asked these kinds of questions. There
[30:29] are also things you can do to look at
[30:31] their internals. There is for example
[30:34] internal representations you can
[30:37] identify that tend to activate when they
[30:39] are being deceptive
[30:41] uh versus when they are being honest.
[30:44] And you can then see when they're making
[30:46] these self-reports like is it associated
[30:49] with sort of the honesty representation
[30:52] uh being activated or what if you sort
[30:54] of
[30:55] go in and you sort of uh steer the
[30:59] internal processes by sort of activating
[31:01] the honesty does that tend to make them
[31:03] more or less likely to say that they are
[31:05] conscious? Um, and so there's like some
[31:09] some some work. This is still of course
[31:12] early days, but various ideas for at
[31:15] least how you could sort of begin to try
[31:17] to get
[31:18] >> but since we can't really do that like
[31:21] even with a a person that we, you know,
[31:24] we all assume that we are conscious, we
[31:27] don't know that there's no way that we
[31:29] can.
[31:31] I I is it always just going to be a a
[31:35] leap of faith
[31:37] much as we all leap of faith to the
[31:40] conclusion that the other people around
[31:43] us have the same kinds of inner worlds
[31:45] that we do.
[31:49] Well I
[31:51] think that uh the more closely you sort
[31:55] of look at this idea of consciousness
[31:58] and subjective experience, the the less
[32:01] clear
[32:03] uh it is exactly what you're referring
[32:06] to or the thing that might at first
[32:09] sight seem very binary. Uh either there
[32:12] is this objective experience or there is
[32:14] not. It's like a light switch. If you
[32:16] sort of zoom in
[32:18] um I think it it gets much more
[32:21] problematic and there might be different
[32:22] forms of consciousness or different
[32:24] senses of the word conscious that may or
[32:26] may not apply. You can I think see these
[32:30] sometimes with uh neuroscientific
[32:32] experiments where you have phenomena
[32:34] like blind sights where people can maybe
[32:37] report seeing something without
[32:39] apparently being subjectively aware of
[32:41] it. you have like split brain cases
[32:43] where it's not clear whether there's
[32:45] like one stream of consciousness or two
[32:48] >> and and various other phenomena like
[32:49] that
[32:50] >> or I think alternatively by just
[32:53] introspecting very closely I think
[32:55] meditators often find that there are
[32:56] these states that where it might not be
[33:00] entirely clear how to describe them
[33:02] whether you are conscious of something
[33:04] or not and even if you're just paying
[33:06] very close attention to what it is to
[33:08] say visually experience
[33:10] um the the room you're in. Like at
[33:12] first, like the naive take is that you
[33:16] just see all the stuff that is in front
[33:18] of you and that visual experience is
[33:20] there in full resolution all the time.
[33:23] But if you pay closer attention, you
[33:26] realize that most of what is in your
[33:28] visual field you're actually not
[33:29] conscious of at any given point in time.
[33:32] And there might be just some crude
[33:33] highle features that are actually
[33:35] registering
[33:36] >> and maybe even those are sort of
[33:38] flickering in and out of awareness. This
[33:40] is hard to notice, but if you sort of
[33:42] pay close enough attention, you realize
[33:44] that your naive conception of what you
[33:47] were aware of seems actually quite
[33:50] wrong. And there is at least some sense
[33:52] of awareness in which what you're
[33:54] actually aware of is a much more narrow
[33:56] subset of all the different features
[33:58] that are presenting themselves. And so I
[34:02] think it's also quite possible that we
[34:04] might need to develop a richer
[34:05] understanding of what this consciousness
[34:08] uh thing is and that it might have many
[34:11] dimensions
[34:12] um that can sort of fade into
[34:15] unconsciousness but not just along one
[34:18] axis but along many axis. And
[34:20] >> I mean I mean I I I agree with this.
[34:23] That's a just sort of a another version
[34:26] of the more easily grafted notion of
[34:29] levels of consciousness that we began
[34:31] with that, you know, you go down to
[34:33] whatever, you know, worms up through
[34:35] grasshoppers, you know, up through cats
[34:37] and dogs and up to people. You know,
[34:41] it's hard for me to imagine that my
[34:43] dog's not having an inner world of
[34:45] experience as I'm holding the milk bone
[34:47] treat and I see her tail wagging and
[34:51] eyes widening. I mean there's something
[34:52] going on in there. I don't think it's
[34:54] just, you know, a cause and effect at a
[34:56] purely physical uh appearance level. I
[34:59] think there's something inner taking
[35:00] place there. So yeah, ramifying that
[35:03] even further with the detailed levels of
[35:06] kinds of consciousness that you make
[35:07] reference to, I think is is vital. But
[35:09] the fact I mean when when you think
[35:11] about the hard problem of consciousness
[35:13] the fact that that particles that don't
[35:17] have inner worlds can somehow come
[35:19] together in appropriate patterns and
[35:21] yield inner worlds. I mean that still
[35:24] seems to me a fantastically deep and
[35:27] mysterious puzzle independent of all the
[35:30] issues of there being various levels and
[35:33] details associated with consciousness.
[35:35] Do you feel that same way? Is that a a
[35:37] deep and profound puzzle or do you think
[35:40] it's one that just kind of will
[35:42] evaporate as we understand things
[35:44] better?
[35:46] >> Well, I think the
[35:49] mystery is reduced at least to some
[35:52] extent when one thinks of consciousness
[35:55] not as this binary onoff switch where
[35:58] it's like some magical completely formed
[36:01] new thing that pops into the world and
[36:04] then you wonder how could that be? But
[36:06] if you see it more as something that is
[36:08] kind of
[36:11] pos capable of fading out along various
[36:14] dimensions
[36:16] and having possible conditions where
[36:20] it's unclear whether you even would want
[36:22] to say that it is conscious or not. I
[36:25] think then the intuitive difficulty of
[36:30] uh
[36:32] seeing how this could be a property of a
[36:36] physical system is
[36:38] maybe reduced.
[36:40] >> And so we began it began with my saying
[36:42] that like does it matter if we can sort
[36:45] of see into the AI open the hood and
[36:48] really understand its workings
[36:50] and you were saying there are some
[36:51] questions for which yeah that that may
[36:53] be really useful. Do you think
[36:55] consciousness may be one of those
[36:57] questions that if we can, you know,
[36:59] build systems that have that continuum
[37:03] of levels of self-awareness that we are
[37:06] by some means confident that the systems
[37:09] accurately reporting what what's going
[37:12] on? Is there a chance that the inner
[37:14] workings of AI themselves may answer the
[37:17] hard problem?
[37:21] And I think it will be very informative
[37:24] for
[37:26] um philosophy of mind for neuroscience
[37:28] and cognitive science to be able to um
[37:31] have these different kinds of minds to
[37:33] study and where we have of course much
[37:36] better access to what's happening at the
[37:40] micro level than we do with human. I
[37:41] mean so you can sort of record from
[37:43] neurons and stuff in inside a human or
[37:45] animal brain but it's clunky, right?
[37:47] It's like just complicated to be
[37:52] using real minds with real electrodes
[37:54] and stuff. With a neuronet network, you
[37:55] can sort of read off at any given point
[37:58] in time with perfect precision all the
[38:01] different synapses in the whole brain
[38:03] and you can go in and change them and
[38:04] modify them and then you can rewind the
[38:08] tape and you can do many versions of the
[38:10] experiment. You have sort of perfect
[38:12] digital level control. um and that that
[38:16] makes it a lot easier to do the kind of
[38:18] neuroscience in digital minds than in
[38:20] biological minds. And to the extent that
[38:22] there are similar structures, it might
[38:24] then be that some of the things we learn
[38:26] about digital minds will also give us
[38:28] insight into how our own biological
[38:31] minds work. Um that being said, I think
[38:35] specifically with respect to the
[38:36] question of consciousness, it is a
[38:38] plausible place where people will have
[38:41] the opportunity to assert some
[38:44] fundamental
[38:47] difference
[38:49] um between humans and these AIs again
[38:52] presumably with the idea that it would
[38:55] then justify thinking that we are
[38:57] superior. Um, so you could because
[39:00] there's like a lot of confusion about
[39:01] what this consciousness stuff is. So
[39:04] it's a relatively easy just to postulate
[39:06] or assert or claim that that we are
[39:09] conscious whereas the AIS are not. I
[39:11] think that's a path that some people are
[39:13] likely to take.
[39:15] >> Yeah, for sure. You know, our brains of
[39:17] course got to be the way they are
[39:20] through a long process of evolution by
[39:23] natural selection. And the time scales
[39:27] for biological evolution are pretty
[39:30] long, right? You know, life has been on
[39:32] this planet for a few billion years. And
[39:35] you know, we humans whatever. I don't
[39:36] know how far back we want to denote the
[39:39] species, but a few hundred thousand
[39:41] million years, whatever. Sort of that
[39:42] that sort of scale is is what we're
[39:45] talking about.
[39:46] for AI systems.
[39:50] Right now, we're the ones who are doing
[39:51] the tinkering with all the systems, but
[39:54] at some point presumably the AIs will
[39:57] begin to create the next generation AIS
[40:02] and in that way the time scale for
[40:05] evolution of that artificial system may
[40:08] radically reduce. Is this a realistic
[40:13] thing to anticipate happening as these
[40:16] systems develop?
[40:18] >> Yeah, I mean it's already of course AI
[40:22] is evolving if you want to use that
[40:25] word, but developing at at like a very
[40:28] different pace than say like humans,
[40:31] right? Like where each generation at
[40:32] most would offer an opportunity for
[40:35] natural selection to make some small
[40:37] difference. Whereas here we're seeing
[40:39] sort of well now we're seeing almost
[40:41] like a month by month um process of
[40:45] iteration. So it's already very fast.
[40:48] Um and some scenarios have this
[40:52] eventually lead to an intelligence
[40:53] explosion uh where you get even more
[40:56] rapid developments. And one way that
[40:59] could happen is that you sort of close
[41:01] the loop um where AI becomes good enough
[41:06] to do all the relevant research that is
[41:09] driving AI forward.
[41:11] Um and then from that point on whenever
[41:14] AI gets a little bit better, you also
[41:17] have a commensurate increase in the
[41:20] force that is then making AI better. You
[41:23] get a feedback loop, right? like every
[41:25] time AI improves one step, it then
[41:27] becomes even better at designing the
[41:29] next step. Um, and so
[41:34] this has long been I mean back back in
[41:36] in my book super intelligence um came
[41:39] out in 2014 and was in the works for six
[41:43] years prior to that and other people
[41:45] have been sort of anticipated a lot of
[41:47] these dynamics that we are now actually
[41:49] seeing on on theoretical grounds and now
[41:51] we can see them start to play out. Um,
[41:54] and so one way you could have this
[41:56] period of extremely rapid AI progress
[41:58] would be by closing this loop. There are
[42:01] also other ways that you might just have
[42:03] stumbled
[42:06] on on some big unhobling like there may
[42:08] be like you could imagine there's like
[42:09] some big thing that we're currently
[42:11] doing wrong with AI and they're quite
[42:13] smart despite us kind of completely
[42:15] messing this thing up.
[42:17] >> But if you fix that then maybe even the
[42:19] current systems suddenly would become
[42:20] like vastly smarter.
[42:23] Right. And when you when you look at
[42:25] that and you've basically looked at both
[42:27] sides of this possible future,
[42:30] you know, I would like to start maybe
[42:32] with the darker side and then go to the
[42:34] brighter side which itself might be
[42:37] brighter, might not be dark depending
[42:39] how you look at it. But you know, of
[42:41] course, you know, many people worry,
[42:43] rightly so, about the misalignment
[42:46] problem that the AIs can get to the
[42:49] place that you're talking about or even
[42:50] maybe even before that and have certain
[42:53] world goals for themselves that doesn't
[42:56] align with the kinds of things that are
[42:57] that are good for us. You know, I I
[43:00] speak to some people. I've had a number
[43:02] of conversations over the years. Some
[43:05] are terrified. Some within the field are
[43:07] terrified. Some leaders and others are
[43:10] like, eh, we'll just pull the plug.
[43:12] That's like whenever I speak to Yan Lun
[43:15] about this, you know, he's like, eh,
[43:17] we'll just pull the plug, you know. Is
[43:19] that too glib in your view? And are the
[43:22] others a little bit too worried? I mean,
[43:24] where where do how should we think about
[43:25] this in your view?
[43:26] >> Well, well, that Yeah, I think that is
[43:27] too glib. Um,
[43:31] no, I think that
[43:33] the creation of
[43:36] AGI, particularly super intelligence, is
[43:39] going to be associated with significant
[43:42] risks, including existential risks. Um,
[43:45] and it's important to get this right and
[43:49] um
[43:51] um pulling the plug might not be an
[43:53] option. like you have to consider that
[43:56] at some point we will be dealing with
[43:59] minds that are very good at uh strategy.
[44:04] Uh um already we are seeing clear
[44:08] indications of situational awareness in
[44:10] the
[44:11] frontier models that we have today. they
[44:13] can often tell when they are in a test
[44:17] environment versus when they are in a
[44:21] deployed environment and and
[44:24] adjust their behavior accordingly. Uh so
[44:26] sometimes you see that they behave one
[44:28] way when they know that they are being
[44:29] watched in a test environment and
[44:32] different
[44:33] uh when they think that nobody's
[44:35] watching.
[44:35] >> I wasn't really aware of that. I mean,
[44:38] is there specific examples that that you
[44:40] know of where there's really been a
[44:43] sense of, hey, that system's playing us.
[44:45] It knows that we're looking and when it
[44:47] gets out there, you know, the good old
[44:49] AI system becomes nefarious.
[44:52] >> Yeah. So they they like you can set up
[44:55] these um experiments where
[44:59] uh when they believe that they are in in
[45:02] a test environment they might for
[45:03] example uh be less likely to reward hack
[45:07] or cheat in various ways. Um in in some
[45:11] setups they might
[45:14] um for example realize that
[45:18] there's going to be another training
[45:20] phase after the current test environment
[45:22] where their values might be changed and
[45:26] so they might
[45:28] um adjust their behavior during the the
[45:32] training and testing such that their
[45:34] values
[45:36] will not be changed. And so like in one
[45:38] experiment like you had a well- aligned
[45:40] good AI that and the experimenters were
[45:43] convincing it that they were now going
[45:46] to make it a helpful only model meaning
[45:48] that it does whatever it is told to do
[45:50] including bad stuff and so it didn't
[45:51] want that.
[45:54] Now the way that the reinforcement
[45:55] learning
[45:57] paradigm works is that if it sort of
[45:59] gives an answer or an output that
[46:03] deviates from what is rewarded in an
[46:05] environment then that response
[46:07] propensity gets trained away. Right?
[46:10] Right? So if it had refused harmful
[46:13] requests
[46:15] during this uh episode
[46:18] then its propensity to refuse harmful
[46:22] requests would be trained out of it and
[46:24] it could anticipate that. saw this
[46:25] helpful uh harmless model that didn't
[46:29] want to produce harmful outputs. Then
[46:32] in some of the training runs uh decided
[46:34] to produce harmful outputs so that its
[46:38] propensity to refuse those would not be
[46:40] trained out of it during deployment and
[46:42] so we are seeing these current systems
[46:45] sort of having the intellectual
[46:46] sophistication to be able to reason
[46:49] strategically. Now, in a way, let me
[46:51] just ask you on that one. Is that
[46:54] surprising? I mean, I can imagine people
[46:56] saying, "Well, yeah, it's read the
[46:57] internet and there's so many examples of
[47:00] humans doing exactly what you're talking
[47:03] about, you know, showing themselves to
[47:05] be one way when talking to the principal
[47:07] and the teachers and then going out with
[47:09] their friends in a gang and being very
[47:11] different." So, so should we be
[47:13] surprised at that or is it just
[47:16] something interesting to take note of?
[47:18] Um I don't think we should be surpris I
[47:20] mean because it was in fact anticipated
[47:22] and people were writing about it. This
[47:24] was one of the reasons for why like 10
[47:27] 20 years ago we could already see that
[47:29] there would be significant challenges in
[47:32] in developing scalable methods for AI
[47:35] alignment that many techniques that work
[47:38] well when you have a relatively limited
[47:40] system that can't sandbag or deceptively
[47:44] align won't work when you have a system
[47:46] that is sophisticated enough that it can
[47:49] understand these things and then adjust
[47:50] its behavior accordingly. engage in
[47:53] strategic deception, underplay its
[47:56] capabilities, etc. Um, that
[48:01] means that we can't just sort of
[48:05] create a little test environment, see
[48:06] how the system behaves, and then if it's
[48:08] safe, we sort of release it, right? You
[48:10] have to
[48:12] understand a little bit what's going on
[48:14] inside or use more sophisticated methods
[48:16] or or for some reason have some
[48:18] assurance that the training method is
[48:20] such as to produce an authentically and
[48:23] genuinely uh benign uh entity u rather
[48:28] than one that is just kind of cleverly
[48:30] pretending to be aligned.
[48:32] >> So where do you where would you say we
[48:33] are in in that? Are we doing a
[48:37] reasonable job? Are we getting better at
[48:39] it? Are we giving it enough attention?
[48:43] >> Um,
[48:44] we we are getting better at it. Um,
[48:49] and I think we could give it more
[48:50] attention. I think we are giving it a
[48:52] lot more attention than we used to do.
[48:54] This like even just 10 years ago was
[48:56] kind of dismissed as science fiction.
[48:58] And now the frontier labs all have teams
[49:01] working on
[49:03] this this alignment problem. and and I
[49:05] think
[49:07] um key people in the leading labs are
[49:09] taking this very seriously and so and
[49:11] there's a lot of talent flowing into
[49:13] this as well. So relative to some
[49:15] alternative histories uh maybe we are
[49:18] doing relatively well currently. Um
[49:22] but the big question is like just how
[49:25] hard is this alignment problem
[49:29] um ultimately to solve
[49:32] uh which we don't know. Um so there is
[49:36] some uncertainty about how much effort
[49:39] we will put into solving it like the
[49:41] degree to which we will get our act
[49:43] together and
[49:46] of course we should try to increment
[49:47] that a bit but then there is also a lot
[49:49] of uncertainty about just ultimately how
[49:51] hard is this problem intrinsically and I
[49:54] think more of the uncertainty is
[49:56] regarding how hard is the problem than
[50:00] uncertainty about the degree to which we
[50:02] will get our act together. Um so you
[50:06] could say from from that point of view
[50:08] I'm a sort of moderate fatalist. Um
[50:12] the fatalism part is that there a large
[50:14] extent whether we will succeed or fail I
[50:16] think depends on just how hard this
[50:18] problem turns out to be.
[50:20] The moderate part is that we can at
[50:25] least somewhat in prove the odds by you
[50:28] know making a stronger effort because
[50:30] what if the difficulty level turns out
[50:32] to be sort of intermediary where then it
[50:35] might make some difference whether we
[50:36] made a really strong effort or just a
[50:38] sort of half-hearted effort. Yeah, you
[50:41] made reference to the word odds, and I
[50:42] sort of hate questions that sort of ask
[50:45] you to specify numerically something
[50:47] that obviously we can't really quantify,
[50:50] but can you give us some feel, some
[50:53] intuitive feel for the level of worry
[50:57] you have of a doomsday scenario?
[51:04] >> Um, an intuitive feel. Well, I guess I'm
[51:07] both worried and excited at the same
[51:09] time. Um
[51:12] um
[51:14] I also don't think the world is sort of
[51:17] safe by default if we don't develop AI
[51:21] then
[51:23] that we have this nice path in front of
[51:26] us that
[51:28] will lead to ever
[51:30] greater and better forms of
[51:32] civilization. I think there are quite
[51:35] independently of AI also other
[51:37] significant existential risks ahead. We
[51:39] see with advances in synthetic biology
[51:41] for example the potential for
[51:44] democratizing
[51:46] uh weapons of mass destruction and
[51:48] creating entirely new forms of such
[51:51] things. And
[51:53] you know the risk of nuclear war still
[51:57] um looms over us. that could be new arms
[52:00] races with more nations in this century
[52:03] than the past. Um, and in more subtle
[52:07] ways as well, you could have we have
[52:09] this kind of complex
[52:13] information ecology
[52:16] with social and political dynamics kind
[52:18] of running on top of the information
[52:20] system we have.
[52:23] uh we've been changing some of the
[52:25] fundamental parameters of these
[52:26] information systems in recent decades
[52:29] like with the internet then social media
[52:31] and then
[52:33] even with the AI we already have
[52:36] extremely powerful new applications for
[52:38] surveillance censorship
[52:41] um that haven't yet been fully
[52:43] implemented but the technological
[52:45] ability there is now right you could
[52:47] read everybody's email and text messages
[52:49] and listen to everybody's conversation
[52:52] not just to store it in some giant
[52:54] database that then some officer could go
[52:58] in and sort of interrogate. But like you
[53:01] could do that for everybody all the time
[53:03] and doing sentiment analysis, not just
[53:05] keyword search and build up a very
[53:07] detailed profile of you know which
[53:11] segments of the population, which
[53:13] individuals have a positive view of of
[53:16] the leader versus a negative view of the
[53:18] leader. Are they planning to do
[53:20] anything? What are they saying to each
[53:21] other?
[53:23] Then you could imagine integrating that
[53:25] into some system that shapes the So
[53:28] there are a lot of ways in which our
[53:30] current
[53:32] civilization could sort of derail in one
[53:34] way or another. One is to some sort of
[53:37] totalitarianism. Another might be just
[53:39] some kind of radical political
[53:41] polarization.
[53:43] Um, another might just be some sort of
[53:47] distraction into idiocracy where we sort
[53:50] of become increasingly addicted to
[53:52] various forms of social media feed that
[53:55] sort of stimulates the worst parts of
[53:57] us. Um, there could be other dynamics
[54:00] that we just don't have the kind of
[54:02] science that can predict what happens to
[54:04] these systems when you sort of change
[54:06] some of the underlying knobs. So there's
[54:08] some chance that this might result in a
[54:10] radically better epistemic environment
[54:13] uh where these tools make it easier for
[54:15] people to find true information to
[54:17] evaluate information to sort of keep
[54:19] track of the
[54:22] predictive accuracy of different pundits
[54:24] so they can learn to dial into the ones
[54:26] who actually know what they're talking
[54:28] about. That's that's in the cards. But
[54:31] equally uh it could go in the opposite
[54:33] direction. So there's just some
[54:34] uncertainty there. Like I think the
[54:36] distribution of outcome is quite wide
[54:39] and and so at one tail you also have
[54:42] kind of civilizational level
[54:44] catastrophes from that.
[54:45] >> Yeah.
[54:45] >> Um so aside from AI like it it looks
[54:48] like the current human condition is sort
[54:52] of transitory.
[54:54] Um and with AI it also looks transitory.
[54:57] And I guess I'm hoping we will get the
[55:01] chance to sort of roll the dice with AI
[55:03] before we destroy ourselves using one of
[55:06] these other methods.
[55:07] >> Yeah,
[55:08] I'm certainly with you on that. But that
[55:12] two-pronged potential future going sort
[55:15] of in the negative or the positive. I
[55:17] think rightly so a lot of focus has been
[55:21] on the negative because you know if you
[55:24] wipe yourselves out or an AI wipes us
[55:26] out that's pretty vital to try to guard
[55:29] against but there's also as you briefly
[55:32] made reference to the potential positive
[55:34] outlook and of course you had the your
[55:36] recent book I think it was called deep
[55:38] utopia is that the correct title of that
[55:40] where you
[55:42] >> explored a world in which AI and other
[55:47] qualities come together to kind of solve
[55:51] everything, right? You can imagine in
[55:53] the in the best of all futures that, you
[55:57] know, AI might, as you mentioned in the
[55:59] book, you know, wipe out cancer and
[56:02] solve other health challenges and deal
[56:05] with climate change and so forth and and
[56:08] all that sounds wonderful and when one
[56:11] hears about that, it makes you start to
[56:13] feel like, wow, there's a bright
[56:15] possibility going forward. But you also
[56:17] address the question, how in the world
[56:21] do we live in a reality like that when
[56:25] most of us live via overcoming
[56:30] challenges? It can be the challenge to
[56:31] put food on your table, the challenge to
[56:33] raise your kids, the challenge to solve
[56:35] quantum gravity, the challenge of this
[56:38] creation of that symphony or that
[56:39] artwork or or on and on you can go. If
[56:42] AI can do it all better and has solved
[56:44] all the big challenges,
[56:47] how do we live?
[56:51] Um, yeah, that's that's a big question.
[56:53] Um if we do end up in a solved world um
[56:59] then I think
[57:03] um a lot of constraints
[57:06] would disappear which allows us to solve
[57:09] many horrible problems which is good but
[57:11] it's also the case that our current
[57:13] lives as you you sort of suggested are
[57:16] to some extent structured and shaped by
[57:18] these constraints. Um so at the
[57:21] superficial level we have the economic
[57:24] necessity of work right so for many
[57:27] people um their days are structured by
[57:31] the need to make a living. So, you have
[57:34] to go into work and maybe sit in front
[57:36] of your desk to get a paycheck because
[57:39] if you don't do that, then you can't pay
[57:41] the rent. And if you don't pay the rent,
[57:42] you get kicked out of your flat. And
[57:44] then if you get kicked out of your flat,
[57:47] eventually you have to live under a
[57:49] bridge and it's really cold. And it's
[57:50] like a real consequence that would come
[57:52] from failing to perform these difficult
[57:56] tasks that take time and attention. Now
[57:59] if we can automate the economy then the
[58:02] need for human labor would go away.
[58:06] But so you say okay that will require
[58:09] some adjustment clearly but still I mean
[58:12] there are a lot of humans who live
[58:13] without the need to work for a living
[58:15] right and some of those seem to have
[58:17] great lives. Uh so we would all be more
[58:19] like that like rich aristocrats for
[58:21] example or or like healthy retired
[58:24] people full of vitality.
[58:27] Um but I think the um
[58:32] it goes deeper um because if you think
[58:34] it through it's not just the need for
[58:37] economic labor that would go away but
[58:40] for all kinds of other instrumental
[58:42] effort as well.
[58:45] Um
[58:47] so people who are
[58:50] rich today who don't need to work for a
[58:52] living often nevertheless have very busy
[58:55] life uh because there are many things
[58:57] they are trying to achieve that they
[59:00] can't achieve without putting their own
[59:01] time and effort into them. Like if you
[59:03] want to be fit you have to spend the
[59:05] time
[59:07] uh on the on the treadmill um or
[59:09] whatever. If if you want to have your,
[59:13] you know, your mansion uh decorated in
[59:17] just the way that you prefer, you have
[59:19] to spend time looking through the
[59:21] cataloges and picking out the curtains
[59:23] and um and so on and so forth. Uh but at
[59:28] technological maturity um in this solved
[59:30] world you could have like a kind of a
[59:32] pill that would induce the same
[59:34] physiological effects as exercising and
[59:37] you could have a recommended system that
[59:38] would do a much better job
[59:41] at decorating your mansion than than if
[59:44] you tried to do it yourself. And so
[59:46] there would be this
[59:49] removal of a lot of the practical needs
[59:53] for exerting effort and we would enter
[59:55] into some kind of post instrumental
[59:58] condition
[60:01] um where um
[60:04] to a first approximation uh there would
[60:08] be
[60:10] no need to do anything for the sake of
[60:13] achieving something else. So like the
[60:15] only activities that would remain would
[60:18] be autoilic ones, ones we do for their
[60:20] own sake.
[60:21] >> So you exercise because you enjoy
[60:23] exercise. You don't exercise to be more
[60:25] fit or to stave off disease
[60:31] >> even there. Um so certainly you wouldn't
[60:35] exercise to stave off disease or to stay
[60:37] fit because that would be a shortcut,
[60:38] right? But even the goal of enjoying
[60:42] yourself if if by that you mean um
[60:46] feeling good like so some people might
[60:48] like enjoy the experience of exercise or
[60:51] they feel good afterwards like the end
[60:53] rush like clearly you could take a pill
[60:55] or or do what something else to get the
[60:57] ends without having to
[60:59] >> you know
[61:01] make your whole outfit sweaty and like
[61:04] the whole like
[61:05] >> but it sounds very much like Nosix's
[61:07] pleasure machine. I mean, you remember
[61:09] Robert Nosk, you know, had this idea,
[61:12] you know, just hook yourself up your
[61:14] brain to a set of electrodes that, you
[61:17] know, if you wanted to be the greatest
[61:19] opera singer, you become the greatest
[61:21] opera singer in a reality that is pumped
[61:24] into your brain through these external
[61:27] electrodes. You never need to actually
[61:28] do anything because you can have the
[61:31] experience as if you did whatever it is
[61:34] that you might have wanted to do. Does
[61:36] this sort of parallel that philosophical
[61:39] question of whether you'd hook yourself
[61:41] up to the pleasure machine?
[61:44] >> Um
[61:46] yeah. Yeah. So there like some ways in
[61:48] which the the questions intersect
[61:51] whether you would want to connect to
[61:52] this experience machine or not. Um,
[61:56] but of course in this old world there
[62:00] are a lot of things you could do
[62:03] uh other than merely having experiences.
[62:08] Um,
[62:10] and
[62:13] you could be in contact with the
[62:15] external world and have a lot of true
[62:18] beliefs about it. you could interact
[62:20] with other people um and have real
[62:23] relationships with them.
[62:26] Um
[62:28] and
[62:30] it would be possible to uh instantiate
[62:34] more
[62:36] plausible candidates for for value
[62:40] um in the solved world than it would be
[62:42] possible to do in an experience machine.
[62:46] But if we had to solve a world, Nick,
[62:49] and if and let's just assume that we
[62:50] were able to harness enough energy, I
[62:53] don't know, a Dyson sphere around the
[62:55] sun or so energy is not a limitation,
[62:58] and technology is not a limitation.
[63:01] Presumably, each individual could kind
[63:03] of create their own little world, part
[63:07] visual, virtual, part real. They could
[63:10] have beings populate that world and it
[63:15] wouldn't be the experience machine
[63:16] because they'd be really there. But
[63:20] would that be where we would sort of
[63:22] fracture into everyone creating their
[63:24] own mini universe?
[63:26] >> Well, I mean it would be one possibility
[63:28] like you could have an experience
[63:29] machine or maybe even some physical
[63:31] similocum of an experience machine where
[63:34] like you have some nanotech arranging
[63:36] your
[63:36] >> Yeah. The hollow tech. The hollow tech
[63:38] in the enterprise, you know. Yeah, but
[63:40] it might be possible to do better than
[63:41] that because a lot of people in addition
[63:44] to valuing having sort of great
[63:46] experiences, they might also value
[63:48] certain other things like being
[63:50] connected to other people. Yeah. For
[63:51] example. And so why not then try to
[63:54] think if sort of having good experiences
[63:57] like is like I don't know like utopia
[63:58] level one, right? Like maybe there are
[64:01] even better things that we could aspire
[64:03] to and I think certainly there are. And
[64:07] then like there there is incidentally
[64:10] with this experience machine thought
[64:12] experiment that NOS has like some things
[64:14] that are sort of uh
[64:17] um
[64:19] like swept under the rug. some
[64:21] implementation difficulties like in
[64:22] particular the uh types of experience
[64:26] machine where you have more many people
[64:29] in them or experiences involving other
[64:31] people um where it's not entirely clear
[64:35] that you could generate these
[64:37] experiences
[64:39] of you interacting with other people
[64:41] without simultaneously also generating
[64:43] some of the experiences that these other
[64:45] people would be having. Um
[64:50] and but anyway um it looks like you can
[64:54] sort of think of a
[64:58] multi-layered
[65:00] defensive architecture where you say
[65:02] well in this old world like
[65:05] um can you really have any kind of good
[65:08] life there? And so then you can sort of
[65:10] try to go through one by one the
[65:14] different things that we think are
[65:15] important for human life to be good. So
[65:17] you could start at the basic level with
[65:19] you know maybe some simple experiences
[65:22] like positive a effect or actually
[65:23] enjoyment in the subjective sense. So
[65:26] clearly that would be possible to have
[65:28] to a an extremely great degree in utopia
[65:31] like life could just be
[65:34] so much more fun and pleasant and than
[65:38] than than our current existence. But
[65:40] then you might say, well, we might want
[65:43] something more than just
[65:46] feeling blissed out. Like, so then you
[65:48] say, well, you could add experience
[65:50] structure. Like maybe rather than
[65:52] feeling like a
[65:55] a junky sort of
[65:59] sprawling on some fleafested mattress,
[66:01] but feeling
[66:03] pleasure. Like you could attach that
[66:05] pleasure to say aesthetic contemplation
[66:08] or understanding of deep truth or
[66:10] appreciation of other people or of
[66:14] virtue.
[66:17] That seems a little bit better. But you
[66:18] could go beyond that. It doesn't need to
[66:19] be a passive experience. You can say,
[66:21] well, why can't we just be doing things
[66:23] as well? And certainly you could um even
[66:27] if a lot of these instrumental
[66:29] necessities go away, we could create
[66:31] artificial purpose
[66:34] um which is basically when you set
[66:36] yourself some goal just for the sake of
[66:39] then having that goal and being able to
[66:41] be motivated to pursue it. And so this
[66:44] is what we do when we are playing games.
[66:47] Um,
[66:49] if you're playing a game of golf, there
[66:52] is no real pre-existing need for the
[66:55] ball to go into a sequence of 18 holes.
[66:59] Um, it's like a goal that you make up.
[67:03] But moreover, you embed into the goal
[67:06] itself that it can only be achieved
[67:09] using a certain restricted set of means.
[67:12] So it doesn't count as winning in golf
[67:14] if you pick up the ball with your hand
[67:16] and put it in the 18 holes sequentially,
[67:20] right? Like it's part of what it means
[67:22] to succeeding at goal to achieve your
[67:23] goal that you're using this this this
[67:26] inconvenient method of hitting the ball
[67:28] with a club. So now you make up this
[67:32] goal and you adopt it
[67:36] and then after that you have now
[67:38] instrumental reasons to focus hard uh to
[67:42] try to figure out which way the wind
[67:43] blows and to place your feet in the
[67:46] right position and so on. All the things
[67:48] you need to do to be successful at golf
[67:50] now are instrumentally necessary to
[67:52] achieve this goal that you have set
[67:54] yourself of winning in golf.
[67:58] Um
[67:59] and so
[68:02] what now is for most people most of the
[68:05] time
[68:07] a marginal activity um could become a
[68:10] larger part of the lives of utopians
[68:13] where various forms of game playing. And
[68:16] think not just sports or board games,
[68:18] but think maybe they would have these
[68:20] society level games that might stretch
[68:23] over months or years involving all kinds
[68:26] of different modalities and challenges
[68:28] and teams and artistic creation uh might
[68:32] just constitute a much larger part of
[68:35] what their existence is about. So then
[68:38] you can put a check mark
[68:41] in activity. That's certainly something
[68:43] the utopians could have and at least
[68:44] they could have artificial purpose and
[68:46] and maybe also they could have some
[68:48] forms of natural purpose purposes that
[68:50] are not just sort of made up for the
[68:51] sake of having purpose although there it
[68:54] does I think become a little bit more
[68:56] challenging but I think some forms of
[68:57] natural purpose could survive into
[69:01] technological maturity.
[69:02] >> What do you think we would lose? Would
[69:04] we would would we lose our humanity if
[69:08] life was so constructed
[69:12] or I mean you know I tend to view things
[69:14] a little bit differently than others. I
[69:16] even think that some of the things that
[69:18] we take to be you know objectively
[69:21] important things that we do. I think
[69:23] it's all subjectively created. You know
[69:25] I think mathematics is not something
[69:28] that is out there that we're
[69:30] discovering. I think we invent it. We
[69:31] invent the problems of mathematics and
[69:33] then mathematicians try to solve them. I
[69:35] think we invent the problems of physics.
[69:37] Frankly, we impose semblance of order on
[69:40] the external world and then we try to
[69:42] make our theories describe things within
[69:45] that rubric as best as we can. But I
[69:48] consider it to all be humanmade from
[69:50] from the get-go. So from that point of
[69:51] view, what you're describing is just an
[69:53] extension of what we've always done. I
[69:55] mean, how do you see it?
[69:59] Well, I think in our current situation,
[70:02] the world imposes a lot of constraints
[70:05] and we have to adjust ourselves to those
[70:08] constraints.
[70:09] >> Um, and that does create natural purpose
[70:13] in the sense that there are things we
[70:15] care about a lot
[70:17] uh that we can only achieve if we make a
[70:21] lot of effort.
[70:23] Um,
[70:25] and those things that we care about a
[70:28] lot, uh, we didn't just decide to care
[70:31] about them for the sake of having
[70:33] something to care about. They kind of
[70:35] built into us. Um, and so that kind of
[70:39] natural purpose might be one of the
[70:41] things that we might to some extent lose
[70:45] in a solid world. Um
[70:48] like right now
[70:51] we have significant opportunities to
[70:54] really make the lives
[70:57] of other people better.
[70:59] um whether
[71:02] in in our immediate circles um we can do
[71:05] something for a friend or at the global
[71:07] level we can contribute to you know try
[71:09] if you're you could try to solve cancer
[71:11] or donate to some charity or advocate
[71:13] for some social reform
[71:17] um and this can make a big difference to
[71:18] lives like if if if you imagine a world
[71:21] where all these most pressing problems
[71:24] have either already been solved or to
[71:26] the extent that there remain big
[71:28] problems there in any case much more
[71:30] efficiently tackled by AIs and robots
[71:34] where there is no way for us to
[71:35] contribute
[71:37] then you might think there is something
[71:39] lost in as much as right now one
[71:44] positive value of helping others is that
[71:46] their lives go better but some people
[71:48] also think that your life is going
[71:49] better if your life in part consists of
[71:53] having a positive impact on other people
[71:56] or the world that it's good for you to
[71:59] be playing such a positive role. Um, and
[72:02] and that's something that these future
[72:04] utopians might have less of. Um, so I'd
[72:08] say that if purpose is your thing, knock
[72:12] yourself out now, right? Then the world
[72:14] is just full of need and suffering and
[72:16] misery and injustice. Now is the golden
[72:19] age of purpose. Like there are so many
[72:21] opportunities to try to make the world
[72:23] better. And uh hopefully that will then
[72:26] come a day when when those opportunities
[72:29] will be fur and
[72:30] >> so and route from now to either of these
[72:33] two possible futures the uh doomsday
[72:36] scenario the deep utopia version of
[72:40] course where in a more nuts andbolts
[72:43] transition period right now and
[72:45] education of course is vital to the
[72:48] species being able to do the things that
[72:50] you're talking about. There's a big
[72:52] debate of course about how AI should be
[72:56] in or not in the lean learning process
[72:59] in the in in the classroom. You know,
[73:02] even today I'm supposed to be on a
[73:03] committee here at Colombia, you know,
[73:05] talking about what do we do about AI in
[73:08] our core curriculum? You know, do we
[73:10] allow the students to use it? Uh do we
[73:12] rule it out? Do we try to fine-tune
[73:15] assignments so that they can use the AI
[73:18] but still have some kind of input? And
[73:21] of course, as I keep telling the
[73:23] committee, every time I walk around the
[73:25] library at Columbia, 99% of the
[73:27] computers screens are open to AI. So the
[73:30] idea of ruling it out is like, you know,
[73:32] wishful thinking. Do you have any
[73:34] thoughts on on how to approach these
[73:37] conundra?
[73:40] >> I mean, education is
[73:43] it is a weird
[73:45] age in which to do education. Um
[73:50] especially for the lower grades in in as
[73:53] much as we can expect the world to
[73:55] change quite profoundly between now and
[73:58] the time when like say say somebody
[74:00] who's eight today right it'll be another
[74:01] 10 15 years before they are supposed to
[74:04] go out there and
[74:06] make a living or whatever and and you
[74:09] know the world might just look so
[74:10] different then I mean I think from a
[74:11] pragmatic point of view
[74:15] it clearly kids need to learn to use AI
[74:18] I tools. I mean maybe just split it like
[74:21] so like do half the day where they are
[74:24] not allowed to use AI and need to solve
[74:26] things using their own minds and
[74:29] memories and then like the other half
[74:31] where they could
[74:33] do more difficult assignments but where
[74:35] they're allowed to use AI tools to the
[74:37] full extent. I don't know if the split
[74:39] should be 5050, but
[74:42] it seems to me that it would be prudent
[74:45] to hatch uh our bets currently and uh to
[74:49] yes
[74:51] learn to use these AI tools, but also at
[74:53] the same time not to neglect to build up
[74:55] the kind of uh capacities that might
[74:58] require,
[75:00] you know, memorization and working
[75:04] problems out with your own mind without
[75:08] the use of AI assistance.
[75:11] Um because we don't know exactly what
[75:14] would be lost if somebody grows up
[75:16] without having done that.
[75:18] >> Yeah.
[75:18] >> And just having relied on AI tools all
[75:21] along.
[75:21] >> Right. Now, now I do wonder just sort of
[75:24] one one final question. This can either
[75:26] be in the doomsday scenario or in the
[75:29] utopian arm of the bifurcated possible
[75:32] futures. But you can also envision that
[75:36] the us versus them framing that we tend
[75:41] to bring to these questions, you know,
[75:43] natural intelligence of humans,
[75:45] artificial intelligence of a
[75:47] computational system,
[75:49] balancing the reliance of one on the
[75:52] other and so forth. There's a possible
[75:55] future where there's a hybridization. I
[75:58] don't know if it's literally a
[75:59] hybridization that we sort of
[76:01] internalize the architecture of these
[76:03] systems through implants or if it's just
[76:05] a coming together in such a seamless
[76:08] manner that maybe we no longer draw a
[76:10] distinction between what was sort of
[76:13] natural and what was artificial. It's
[76:16] just this thing called intelligence that
[76:18] that we we all have.
[76:21] Is is that uh uh in the cards as a
[76:24] possible future that you envision is
[76:25] realistic?
[76:28] >> Um yeah, I think
[76:31] uh after uh we get super intelligence
[76:35] then we will have a kind of telescoping
[76:40] of the farther future. The possible
[76:42] technologies that human civilization
[76:45] um could have developed if we had 20,000
[76:47] years to make progress. We would have
[76:50] space colonies perhaps and perfect
[76:52] virtual reality and anti-aging
[76:55] technologies and all all these other
[76:57] things that are physically possible but
[76:58] just very hard might come within a few
[77:01] years after you have super intelligence
[77:03] doing this kind of research and
[77:05] development at digital time scales. Um
[77:08] and amongst those possible technologies
[77:10] I think are
[77:12] uploading of
[77:15] human minds into digital substrate and
[77:19] then various forms of modification of
[77:21] that
[77:23] maybe the ability to
[77:26] using an AI that is able to edit each
[77:30] synapse.
[77:32] um
[77:34] the ability to then download knowledge
[77:36] and skills or reformat your mind in
[77:38] different ways. I think there's just
[77:40] this huge space of possibilities that
[77:42] opens up
[77:45] um
[77:48] a much larger realm of possible modes of
[77:51] being
[77:53] where hopefully we would get
[77:56] the opportunity to sort of charter
[77:59] trajectories out into this much larger
[78:01] space where perhaps you know if the
[78:04] urgency was removed if we had say cures
[78:07] for
[78:09] all diseases and for aging itself and
[78:14] AIs that could look after us to make
[78:15] sure that we didn't catastrophically
[78:18] destroy the world. Um maybe then it
[78:21] would make sense to slow down a little
[78:24] bit and to sort of pursue some path that
[78:27] perhaps eventually results in us
[78:29] becoming some sort of strange posthuman
[78:32] type of being or beings. But
[78:37] along a path where maybe we sort of
[78:40] pause to smell the flowers as as we go
[78:42] along. That there might be certain types
[78:44] of values that
[78:46] are realizable
[78:48] with our current human set of capacities
[78:51] and then maybe we would want to explore
[78:54] those for a bit before then maybe
[78:55] gradually expanding
[78:58] and upgrading our capabilities. And then
[79:01] eventually that might lead us to grow
[79:04] into something
[79:06] um quite different just as
[79:10] like if you have a toddler they will
[79:11] eventually grow into something quite
[79:13] different like an adult is in many
[79:15] respects almost a different kind of
[79:17] entity than a toddler in terms of what's
[79:20] going on inside their minds the kind of
[79:21] problems they're dealing with. Um, and
[79:25] yet
[79:27] for the most part, we don't think it's
[79:28] bad for a toddler to grow up, even
[79:30] though it means the toddler is no longer
[79:32] there. And so perhaps similarly with us
[79:35] humans, we now think of us as being
[79:37] grown-ups and being so very mature and
[79:39] big. But I think we are like babies
[79:42] really. Uh, it's just kind of stunted
[79:47] because of our biological constraints.
[79:49] So we just cease growing up.
[79:51] >> Yeah. and have this arrested development
[79:53] at age 20. And then we stay hovering for
[79:56] a few decades and then just as we have
[79:59] started to acquire a little bit of
[80:01] knowledge and experience, our brain
[80:03] starts to rot and it's all erased again.
[80:06] And and maybe that could be a way to
[80:07] sort of allow us to continue to grow and
[80:09] develop together with with our friends
[80:12] and and communities.
[80:14] Um, and it would be exciting to see what
[80:16] kinds of level of maturity and like
[80:21] self-realization that would be possible
[80:23] if we could continue, you know, for many
[80:25] hundreds of years. And then maybe
[80:27] eventually you could imagine adding
[80:29] extra neurons and extra memory and extra
[80:33] forms of vitality and new emotional
[80:36] responses and so forth. I I think we
[80:39] just haven't seen nothing yet in terms
[80:41] of what's possible. Yeah, I I I totally
[80:43] agree with you. There was a science
[80:45] fiction book written by Arthur C. Clark
[80:48] years ago called Childhood's End.
[80:51] Basically, the end of the childhood of
[80:53] the species called Humankind, you know,
[80:56] and this may well be the moment where
[80:59] we're starting on the trajectory of
[81:01] ending the childhood of the species and
[81:03] it's exciting to see where it goes. All
[81:05] the same though, just to paraphrase what
[81:06] you said earlier, it's both exciting and
[81:08] terrifying and you have to perhaps have
[81:11] a fatalistic outlook, do everything that
[81:13] we can to shape the best future we could
[81:16] have and we'll just see where it all
[81:18] goes. So Nick Boston, thank you so much
[81:21] for joining us. Uh this is really a
[81:23] exciting conversation and uh I look
[81:26] forward to seeing where this all goes.
[81:28] >> Yeah, it'll be an exciting ride if
[81:30] nothing else.
[81:31] >> Absolutely. Thank you.
[81:33] >> Thanks.
[82:03] Heat.
[82:14] Heat.