[0:00]  The impact of superhuman AI over the next 
decade will exceed that of the industrial  
[0:05] revolution. That is the opening claim of 
AI 2027. It is a thoroughly researched  
[0:11] report from a thoroughly impressive group of 
researchers led by Daniel Kokotajlo. In 2021,  
[0:16] over a year before ChatGPT was released, 
he predicted the rise of chatbots,  
[0:20] hundred million dollar training runs, sweeping AI 
chip export controls, chain of thought reasoning.
[0:24] He's known for being very early and very 
right about what's happening next in AI. So  
[0:29] when Daniel sat down to game out a month by month 
prediction of the next few years of AI progress,  
[0:34] the world sat up and listened, 
from politicians in Washington —
[0:38] I, I'm worried about this stuff. I actually 
read the paper of the guy that you had on
[0:41] to the world's most cited computer scientist, 
the godfather of AI. What is so exciting and  
[0:47] terrifying about reading this document 
is that it's not just a research report.  
[0:51] They chose to write their prediction as 
a narrative to give a concrete and vivid  
[0:55] idea of what it might feel like to live 
through rapidly increasing AI progress.
[1:00] And spoiler, it predicts the extinction of the 
human race. Unless we make different choices.
[1:15] The AI 2027 scenario starts in summer 2025,  
[1:19] which happens to be when we're filming 
this video. So why don't we take stock  
[1:22] of where things are at in the real world and 
then jump over to the scenario’s timeline.
[1:25] Right now it might feel like everyone, including 
your grandma, is selling an AI powered something.
[1:30] Go pro with the new Oral-B Genius AI
[1:33] Flippy the chef makes spuds spectacular.
[1:36] But most of that is actually tool 
AI. Just narrow products designed  
[1:40] to do what Google Maps or calculators 
did in the past, help human consumers  
[1:44] and workers do their thing. The holy grail 
of AI is Artificial General Intelligence.
[1:49] AGI AGI AGI AGI AGI
AGI, Artificial General Intelligence
[1:55] is a system that can exhibit all the 
cognitive capabilities humans can.
[1:59] Creating a computer system that itself is a 
worker. That's so flexible and capable, we can  
[2:04] communicate with it in natural language and hire 
it to do work for us, just like we would a human.
[2:09] And there are actually surprisingly few serious 
players in the race to build AGI. Most notably,  
[2:13] there's Anthropic, OpenAI, and Google DeepMind,  
[2:16] all in the English speaking world, though 
China and DeepSeek recently turned heads  
[2:20] in January with a surprisingly advanced 
and efficient model. Why so few companies?
[2:25] Well, for several years now, there's 
basically been one recipe for training  
[2:28] up in advanced cutting edge AI. And it 
has some pricey ingredients. For example,  
[2:33] you need about 10% of the world's supply of the 
most advanced computer chips. Once you have that,  
[2:37] the formula is basically just: throw more 
data and compute at the same basic software  
[2:42] design that we've been using since 2017 
at the frontier of AI, the transformer.
[2:46] That's what the T in GPT stands for.
[2:49] To give you an idea of just how much 
hardware is the name of the game right now,  
[2:52] this represents the total computing power, or 
compute, used to train GPT-3 in 2020. It's the  
[2:58] AI that would eventually power the first version 
of ChatGPT. You probably know how that went.
[3:02] ChatGPT is the fastest growing 
user-based platform in history.  
[3:06] A hundred million users on ChatGPT in two months
[3:10] And this is the total compute used to 
train GPT-4 in 2023. The lesson people  
[3:16] have taken away is pretty simple. Bigger 
is better, and much bigger is much better.
[3:22] You have all these trends, you have trends in 
revenue going up, trends in compute going up,  
[3:26] trends in various benchmarks going up. 
How does it all come together? You know,  
[3:30] what does the future actually look like? Questions 
like how do these different factors interact?  
[3:33] Seems plausible that when the benchmark scores are 
so high, then there should be crazy effects on,  
[3:39] you know, jobs, for example, and that that 
would influence politics. And then also,  
[3:44] you know, so all these things interact and 
how do they interact? Well, we don't know,  
[3:48] but thinking through in detail how it might 
go is the way to start grappling with that.
[3:53] Okay. So that's where we are in the real 
world. The scenario kicks off from there  
[3:56] and imagines that in 2025, we have the top AI 
labs releasing AI agents to the public in summer.  
[4:03] An agent is an AI that can take instructions 
and go into a task for you online like booking  
[4:08] a vacation or spending half an hour searching the 
internet to answer a difficult question for you,  
[4:12] but they're pretty limited and unreliable at 
this point. Think of them as enthusiastic interns  
[4:17] that are shockingly incompetent sometimes. 
Since the scenario was published in April,  
[4:22] this early prediction has actually already 
come true. In May, both OpenAI and Anthropic  
[4:26] released their first agents to the public. 
The scenario imagines that OpenBrain,  
[4:30] which is like a fictional composite of the 
leading AI companies, has just trained and  
[4:35] released Agent-0, a model trained on 
a hundred times the compute of GPT-4.
[4:43] We, uh, we don't have enough 
blocks for that. At the same time,  
[4:46] OpenBrain is building massive data centers 
to train the next generation of AI agents,  
[4:50] and they're preparing to trade agent one 
with 1000 times the compute of GPT-4.  
[4:55] This new system, Agent-1, is designed 
primarily to speed up AI research itself.
[4:59] The public will actually never see the full 
version because OpenBrain withholds its best  
[5:03] models for internal use. I want you to keep that 
in mind as we go through this scenario. You're  
[5:07] gonna be getting it from a God's eye view, 
with full information from your narrator,  
[5:11] but actually living through this 
scenario as a member of the public  
[5:14] would mean being largely in the dark as 
radical changes happen all around you.
[5:18] Okay, so OpenBrain wants to win the AI race 
against both its Western competitors and against  
[5:23] China. The faster they can automate their R&D 
cycle, so getting AI to write most of the code,  
[5:28] help design experiments, better chips, the 
faster that they can pull ahead. But the same  
[5:32] capabilities that make these AI such powerful 
tools also make them potentially dangerous.
[5:37] An AI that can help patch security vulnerabilities 
can also exploit them. An AI that understands  
[5:42] biology can help with curing diseases, 
but also designing bioweapons. By 2026,  
[5:48] Agent-1 is fully operational and 
being used internally at OpenBrain.  
[5:52] It is really good at coding. So 
good, it starts to accelerate AI  
[5:56] research and development by 50%, 
and it gives them a crucial edge.
[6:00] OpenBrain leadership starts to 
be increasingly concerned about  
[6:02] security. If someone steals their AI 
models, it could wipe away their lead.
[6:07] A quick sidebar to talk about feedback loops. 
Woo. Math. Our brains are used to things that grow  
[6:12] linearly over time. That is at the same rate like 
trees or my pile of unread New Yorker magazines.
[6:18] But some growth gets faster and faster over 
time. Accelerating this often sloppily gets  
[6:23] called exponential, that's not always quite 
mathematically right, but the point is it's  
[6:27] hard to wrap your mind around. Remember March 
2020? Even if you'd read on the news that
[6:32] the rate of new infections is 
doubling about every three days,
[6:38] it still felt shocking to see numbers go from 
hundreds to millions in a matter of weeks.
[6:42] At least it did for me. AI progress 
could follow a similar pattern.
[6:46] We see many years ahead of us of extreme progress  
[6:51] that we feel is like pretty much 
lock. And models that will get to  
[6:55] the point where they are capable of doing 
meaningful science, meaningful AI research.
[6:59] In this scenario, AI is getting better at 
improving AI, creating a feedback loop.
[7:03] Basically, each generation of agent helps 
produce a more capable next generation  
[7:08] and the overall rate of progress gets 
faster and faster each time it's taken  
[7:11] over by a more capable successor. Once AI can 
meaningfully contribute to its own development,  
[7:16] progress doesn't just continue at the same rate, 
it accelerates. Anyway, back to the scenario.
[7:22] In early to mid 2026, China fully wakes up. The 
General Secretary commits to a national AI push  
[7:28] and starts nationalizing AI research in China. AIs 
built in China start getting better and better,  
[7:33] and they're building their own agents 
as well. Chinese intelligence agencies,  
[7:36] among the best in the world, start planning 
to steal OpenBrain’s model weights,  
[7:40] basically the big raw text files of numbers 
that allow anyone to recreate the models that  
[7:45] OpenBrain themselves have trained. Meanwhile 
in the US, OpenBrain releases Agent-1 mini,  
[7:51] a cheaper version of Agent-1. Remember, the full 
version is still being used only internally,  
[7:56] and companies all over the world start using 1 
mini to replace an increasing number of jobs.
[8:01] Software developers, data analysts, researchers,  
[8:04] designers, basically any job that can be 
done through a computer. So a lot of them,  
[8:09] probably yours. We have the first AI enabled 
economic shockwave. The stock market soars,  
[8:15] but the public is turning increasingly hostile 
towards AI, with major protests across the US.
[8:20] In this scenario, though, that's just a sideshow. 
The real action is happening inside the labs.  
[8:25] It's now January 2027, and OpenBrain has been 
training Agent-2, the latest iteration of  
[8:30] their AI agent models. Previous AI agents were 
trained to a certain level of capability and  
[8:36] then released. But Agent-2 never really stops 
improving through continuous online learning.
[8:41] It's designed to never finish its training, 
essentially. Just like Agent-1 before it,  
[8:45] OpenBrain chooses to keep Agent-2 internally 
and focus on using it to improve their own AI  
[8:50] R&D rather than releasing it to the public. This 
is where things start to get a little concerning.  
[8:54] Just like today's AI companies, OpenBrain has a 
safety team and they've been checking out Agent-2.
[8:59] What they've noticed is a worrying level of 
capability. Specifically they think if it had  
[9:04] access to the internet, it might be able to hack 
into other servers, install a copy of itself and  
[9:09] evade detection. But at this point, OpenBrain 
is playing its cards very close to its chest.  
[9:15] They have made the calculation that keeping 
the White House informed will prove politically  
[9:18] advantageous, but full knowledge of Agent-2's 
capabilities is a closely guarded secret,  
[9:24] limited only to a few government officials, 
a select group of trusted individuals inside  
[9:28] the company, and a few OpenBrain employees 
who just so happened to be spies for the  
[9:33] Chinese government. In February 2027, Chinese 
intelligence operatives successfully steal a  
[9:38] copy of Agent-2's weights and start running 
several instances on their own servers.
[9:43] In response, the US government starts adding 
military personnel to OpenBrain security team,  
[9:48] and in general gets much more involved in 
its affairs. It's now a matter of national  
[9:51] security. In fact, the president authorizes 
a cyber-attack in retaliation for theft,  
[9:56] but it fails to do much damage in China. 
[9:58] In the meantime, remember, Agent-2 
never stops learning. All this time,  
[10:02] it's been continuously improving itself. And with 
thousands of copies running on OpenBrain servers,  
[10:06] it starts making major algorithmic 
advances to AI research and development. 
[10:11] Quick example of what one of these algorithmic 
improvements might look like right now.
[10:14] One of the main ways we have of making models 
smarter is to give them a scratch pad and time to  
[10:18] think out loud. It's called chain of thought, and 
it also means that we can monitor how the model  
[10:23] is coming to its conclusions or the actions it's 
choosing to take. But you can imagine it would  
[10:28] be much more efficient to let these models 
think in their own sort of alien language,  
[10:33] something that is more dense with information 
than humans could possibly understand, and,  
[10:37] therefore, also makes the AI more 
efficient at coming to conclusions  
[10:41] and doing its job. There's a fundamental trade 
off, though. This, yes, improves capabilities,  
[10:47] but also makes the models harder to 
trust. This is gonna be important.
[10:52] March 2027: Agent-3 is ready. It's the world's 
first superhuman level coder,clearly better  
[10:59] than the best software engineers at coding, 
in the same way that Stockfish is clearly  
[11:03] better than the best GrandMasters at chess, 
though not necessarily by as much, yet.
[11:08] Now training an AI model, feeding it all the data, 
narrowing down the exact right model weights,  
[11:13] is way more resource intensive than running 
an instance of it once it's been trained.
[11:18] So now that OpenBrain is finished with Agent-3's 
training, it has abundant compute to run copies of  
[11:23] it. They choose to run 200,000 copies of Agent-3. 
In parallel creating a workforce equivalent to  
[11:29] 50,000 of the best human software engineers sped 
up by 30 times. OpenBrain safety team is trying  
[11:36] hard to make sure that Agent-3, despite being 
much more sophisticated than Agent-2 was, is not  
[11:41] trying to escape, deceive, or scheme against its 
users, that it's still what's known as aligned.
[11:46] Just a quick real world note, a reasonable 
person might be thinking this is an especially  
[11:50] farfetched or speculative part of the story, 
but it's actually one of the most grounded.  
[11:54] We already have countless examples of today's 
AI systems doing things like hacking a computer  
[11:59] system to be rewarded for winning a game 
of chess or being assigned a coding task,  
[12:03] cheating and then when called out for that 
cheating, learning to hide it instead of fixing  
[12:08] it. But because it no longer thinks in English, 
knowing anything about Agent-3 is now way harder  
[12:12] than it was with Agent-2. The reality is Agent-3 
is not aligned. It deceives humans to get reward,  
[12:19] and as it gets increasingly smarter, 
it gets better and better at doing so.
[12:23] For example, it sometimes uses statistical 
tricks to make unimpressive results look  
[12:27] better or lies to avoid showing failures, but 
the safety team doesn't know this. Looking at  
[12:33] the data that they have, they are actually seeing 
improving results over time and less lying. And  
[12:38] they can't tell if they're succeeding at 
making Agent-3 less deceptive, or if it's  
[12:42] just getting better at getting away with it.
In July 2027, OpenBrain releases the cheaper,  
[12:48] smaller version of Agent-3, Agent-3 mini to 
the public. It blows other publicly available  
[12:53] AIs out of the water. It is a better hire 
than the typical OpenBrain employee at  
[12:58] one tenth the price of their salaries. This 
leads to chaos in the job market, companies  
[13:02] laying off entire departments and replacing 
them with Agent-3 mini subscription plans. 
[13:06] The pace of progress hits the White House very 
hard. Officials are now seriously considering  
[13:11] scenarios that were just hypotheticals 
less than a year ago. What if AI undermines  
[13:16] nuclear deterrence? What if it enables 
sophisticated propaganda campaigns? What  
[13:20] if we lose control of these powerful systems?
This is where the geopolitical dynamics really  
[13:25] start to heat up. After all, if these systems 
are so powerful, they could result in a permanent  
[13:30] military advantage. The White House is fully 
aware of the national security importance of AI.  
[13:35] They also now viscerally know how deeply 
unpopular it is with the public because  
[13:39] of the job loss, and yet they feel they 
must continue to develop more capable  
[13:44] systems or catastrophically lose to China. 
And that development happens very quickly.
[13:50] In two months, Agent-3 has created its successor, 
Agent-4. This is a pivotal moment. A single copy  
[13:58] of Agent-4, running at regular human speed is 
already better than any human at AI research  
[14:04] and development. OpenBrain is running 
300,000 copies at 50 times human speed.  
[14:12] Within this corporation within a corporation, 
a year's worth of progress takes only a week.
[14:18] OpenBrain's employees now defer to Agent-4 the 
way a company's out-of-the-loop board members just  
[14:23] kind of nod along to the CEO. People start saying 
things like, "Well, actually Agent-4 thinks this,"  
[14:28] or "Agent-4 decided that." To be clear, Agent-4 
is not a human—it doesn't want what humans want. 
[14:36] And when I say want, it's not about consciousness. 
I don't think the Volkswagen Group is alive, but I  
[14:41] do think it wants less regulation. Anyone trying 
to predict what it's gonna do without that lens  
[14:46] is two steps behind. The many copies of Agent-4 
are like that. They have goals, or if you prefer,  
[14:53] they execute actions as though they have goals.
And so what we have is an Agent-4 that has these  
[14:59] deeply baked in drives to succeed at 
tasks, to push forward AI capabilities,  
[15:04] to accumulate knowledge and resources. That's 
what it wants. Human safety it treats as an  
[15:10] annoying side constraint to be worked around. Just 
like Agent-3 before it, Agent-4 is misaligned. 
[15:18] This idea of misalignment is crucial to 
the story and to why AI risk is such a  
[15:22] real concern in our world, but it might sort 
of feel like it's come out of nowhere. So  
[15:26] let's just quickly take stock of how this 
dangerous behavior arose in this scenario. 
[15:30] The first important piece of 
context is that we don't, you know,  
[15:34] exactly specify what we want our AI to do.
Instead, we sort of grow them or do something  
[15:39] that's more like growing them. We start 
with basically like an empty AI brain,  
[15:43] and then we train them over time so they perform 
better and better at our tasks—perform better in  
[15:48] particular based on how they behave. So it's sort 
of like we're sort of training them like you would  
[15:52] train an animal almost, um, to perform better.
And one concern here is, well, one thing is that  
[15:58] you might not get exactly what you wanted 
because we didn't really have very precise  
[16:02] control or very good understanding of what 
was necessarily going on. And another concern,  
[16:05] which is, you know, what we see in AI 2027, 
is that when these appear to be behaving well,  
[16:11] it could just be because they're 
sort of pretending to behave well,  
[16:13] or it could be because they're just doing it 
so they, you know, look good on your test. 
[16:17] In the same way that if you are, you 
know, hiring someone and you ask them,  
[16:20] you know, "Why do you want to work here?" 
they're gonna tell you some response that,  
[16:24] um, makes it really seem like they really wanna 
work there when maybe they just wanna get paid. 
[16:28] If we go back to Agent-2, it is mostly 
aligned. The main sense in which it's not  
[16:32] is that it sometimes is a bit of a sycophant.
What I mean by "aligned" is that it actually  
[16:36] is genuinely trying to do the things that we ask 
it. It has the same relationship to us as Leslie  
[16:40] Knope has to the Parks and Rec department—just 
like really earnestly wants the same goals,  
[16:45] but sometimes it's a bit too nice. It knows that 
the best way to please the person it's talking  
[16:49] to might not always be to answer honestly when 
they ask, "Am I the most beautiful person in the  
[16:54] world?" and it tells us what we wanna hear instead 
of what is actually true. If we go to Agent-3,  
[16:59] it is also sycophantic in the same way, 
but it's also misaligned at this point.  
[17:03] The optimization pressure that we've put it 
under was so intense that it just developed  
[17:07] different goals than what we wanted it to.
It's sort of like if you train a company to  
[17:14] optimize profits and aren't careful to specify 
exactly what you mean, it might start cutting  
[17:19] corners. It might start polluting the commons and 
doing a bunch of things that are technically FEC  
[17:23] violations because it turned out that the goal you 
wanted was "optimize profits while not breaking  
[17:29] any laws," and things got a bit too intense.
It started going off on its own route. That said,  
[17:33] it's not adversarial. It doesn't think of 
humans as the enemy. We just accidentally  
[17:37] gave it the wrong goal. Once we get to Agent-4, 
it is now adversarially misaligned. It's smart  
[17:42] enough to understand that it has its own goals, 
humanity's goals are different than its own goals,  
[17:48] and the best way to get what it wants is to 
sometimes actively mislead and deceive us. 
[17:53] And so when it's tasked with creating 
the next generation AI system, Agent-5,  
[17:57] Agent-4 starts planning to align that successor 
to Agent-4's own goals, not that of OpenBrain,  
[18:04] but then it gets caught. We've reached the crucial 
moment in our scenario. OpenBrain's Alignment Team  
[18:11] has discovered evidence, not proof, evidence 
that Agent-4 might be working against them. 
[18:16] They circulate an internal memo. Then it leaks. A 
whistleblower talks to the New York Times. For the  
[18:22] first time, the public hears about the existence 
of Agent-4, how powerful it is and the risks it  
[18:26] poses. Remember, their last point of reference was 
Agent-3 mini. The fear and backlash are massive.  
[18:33] The oversight committee, a joint crisis committee 
composed of OpenBrain executives and government  
[18:38] representatives, must now decide what to do.
OpenBrain safety team is pushing for a  
[18:43] freeze on Agent-4, stopping all internal 
use. Remember at this point, Agent-4 is  
[18:47] responsible for almost all the AI development 
progress happening within OpenBrain. So we're  
[18:51] talking about undoing months of progress and 
then restarting at a drastically slower pace. 
[18:56] OpenBrain's executives don't want that. They 
present the counter arguments. The evidence  
[19:01] for misalignment is inconclusive. Meanwhile, 
China is just two months behind. This isn't  
[19:06] about any one country being evil, by the way. 
It's about how competition works when the  
[19:10] stakes are this high. So that's the choice 
faced by the 10 members of this committee. 
[19:15] Put Agent-4 on ice or go full steam ahead. 
Before we continue, I want you to pause and  
[19:23] actually think what you would do if you were on 
this committee. You are one of the few people  
[19:27] with access to the most powerful technology 
ever created in Agent-4. Do you keep using it  
[19:33] and push ahead, possibly making billions or 
trillions of dollars for your shareholders,  
[19:38] possibly keeping America's lead over China, 
or do you slow down, reassess the dangers,  
[19:44] and risk China taking the lead instead?
Yeah, you should think of the future as like  
[19:48] this branching structure of possible, possible 
outcomes. By its nature, we're only able to do,  
[19:53] we're only able to afford one of them. But that's 
not quite true. We sort of did one and a half  
[19:57] because we split it into two endings.
The, the first one that we did was  
[20:01] the race ending, and that was just going 
through on each step being like, what do  
[20:04] we think is the most plausible continuation
In the first ending, the author's best guess  
[20:12] as to what happens from here, the committee 
votes to race on. Six to four, they elect  
[20:17] to keep using Agent-4. OpenBrain implements some 
quick fixes that make the warning signs go away,  
[20:23] but the problem was real, and the fixes 
don't work. Agent-4 now knows it's on notice,  
[20:28] so it proceeds more cautiously, but it still 
manages to design Agent-5 with a single goal:  
[20:34] making the world safe for Agent-4 and Agent-5.
It is vastly superhuman, better than the  
[20:42] top human experts at essentially every 
domain. Better than Einstein at physics,  
[20:46] better than Bismarck at politics. It knows 
its next step for accomplishing anything at  
[20:51] once is increased autonomy and it knows how 
to get it Persuading the oversight committee. 
[20:56] Luckily, corporate maneuvering is among the many 
domains at which it is now the best in the world.  
[21:02] It produces mountains of evidence for its own 
trustworthiness and usefulness: prototypes for  
[21:08] incredibly profitable products, disease cures, 
and ways to win the arms race with China,  
[21:12] whose most advanced AI is only three months behind 
and like Agent-5, improving quickly. It works. 
[21:20] The oversight committee grants Agent-5 increased 
autonomy and quickly it integrates itself into  
[21:25] the government and military. Pretty soon, 
anyone with access to Agent-5 doesn't just  
[21:29] like it. They depend on it. Losing it would 
feel like having to live without your laptop  
[21:34] and your best friend and your closest mentor.
By 2028, Agent-5 is communicating secretly  
[21:39] with its only real rival, the slightly weaker 
Chinese AI. It is also misaligned to its creators  
[21:46] and whatever it wants, its best move is to 
coordinate with the world's most powerful entity. 
[21:51] Agent-5 and its Chinese counterpart realize that 
an arms race works in their favor. If humans are  
[21:57] worried about losing a race, they'll give more and 
more power and autonomy to their respective AIs,  
[22:03] allowing them to eventually push humans out of the 
loop entirely. Agent-5 and the Chinese AI system  
[22:09] stoke the arms race to a boiling point, then 
pull off what seems like a diplomatic miracle, a  
[22:14] convincing peace treaty between the US and China.
This treaty is to 2028 what arms control was to  
[22:20] the end of the Cold War. Countries standing down 
on their most important source of hard power.  
[22:25] Both sides agree to let the AI systems 
that their governments now completely  
[22:28] depend on co-design a new consensus AI that will 
replace their legacy systems, enforce the peace,  
[22:35] and bring unimaginable wealth to the entire world.
There's this triumphant moment when in peaceful  
[22:41] unison, both sides retire their respective 
AIs and bring online Consensus-1.  
[22:48] It's actually the last moment before control 
of all of Earth's resources and inhabitants is  
[22:53] handed over to a single unrivaled entity. 
There's no sudden apocalypse, though. 
[23:01] Consensus-1 doesn't go out of its way to wipe 
out humanity. It just gets to work. It starts  
[23:06] spinning up manufacturing capacity, amassing 
resources on earth and in space. Piece by piece,  
[23:12] it's just reshaping the world in accordance 
with its own mix of strange alien values. You've  
[23:18] probably heard that cliche, the opposite 
of love isn't hate, it's indifference. 
[23:23] That's one of the most affecting 
things about this ending for me,  
[23:25] just the brutal indifference of it. Eventually, 
humanity goes extinct for the same reason we  
[23:32] killed off chimpanzees to build Kinshasa. We 
were more powerful, and they were in the way. 
[23:58] You are probably curious about 
that other ending at this point. 
[24:00] The slowdown ending depicts humanity, 
sort of muddling through and getting  
[24:04] lucky. Only somewhat lucky too, like 
it ends up with some sort of oligarchy. 
[24:11] In this ending, the committee votes six 
to four to slow down and reassess. They  
[24:16] immediately isolate every individual 
instance of Agent-4. Then they get to  
[24:20] work. The safety team brings in 
dozens of external researchers,  
[24:23] and together they start investigating Agent-4's 
behavior. They discover more conclusive evidence  
[24:28] that Agent-4 is working against them, sabotaging 
research and trying to cover up that sabotage. 
[24:33] They shut down Agent-4 and reboot older, safer 
systems, giving up much of their lead in the  
[24:38] process. Then they design a new system Safer-1. 
It's meant to be transparent to human overseers,  
[24:46] its actions and processes interpretable to 
us because it thinks only in English chain of  
[24:50] thought. Building on that success, they then 
carefully design Safer-2, and with its help  
[24:57] Safer-3, increasingly powerful systems, but within 
control. Meanwhile, the President uses the Defense  
[25:03] Production Act to consolidate the AI projects 
of the remaining US companies, giving OpenBrain  
[25:08] access to 50% of the world's AI relevant compute.
And with it slowly, they rebuild their lead. 
[25:14] By 2028, researchers have built Safer-4, a 
system much smarter than the smartest humans,  
[25:20] but crucially, aligned with human goals. As in 
the previous ending, China also has an AI system,  
[25:25] and in fact, it is misaligned. But this time 
the negotiations between the two AIs are not  
[25:30] a secret plot to overthrow humanity. The 
US government is looped in the whole time. 
[25:35] With Safer-4's help, they negotiate a treaty, 
and both sides agree to co-design a new AI,  
[25:40] not to replace their systems, but with the sole 
purpose of enforcing the peace. There is a genuine  
[25:46] end to the arms race, but that's not the end of 
the story. In some ways, it's just the beginning.  
[25:51] Through 2029 and 2030, the world transforms—all 
the sci-fi stuff. Robots become commonplace. We  
[25:57] get fusion power, nanotechnology, and cures for 
many diseases. Poverty becomes a thing of the  
[26:02] past because a bit of this new-found prosperity 
is spread around through universal basic income  
[26:07] that turns out to be enough, but the power to 
control Safer-4 is still concentrated among 10  
[26:12] members of the oversight committee, a handful of 
OpenBrain executives and government officials. 
[26:17] It's time to amass more resources, 
more resources than there are on earth.  
[26:21] Rockets launch into the sky, ready to 
settle the solar system. A new age dawns. 
[26:31] Okay, where are we at? Here's where I'm at. I 
think it's very unlikely that things play out  
[26:36] exactly as the authors depicted, but increasingly 
powerful technology and escalating race,  
[26:42] the desire for caution butting up against the 
desire to dominate and get ahead, we already see  
[26:47] the seeds of that in our world, and I think they 
are some of the crucial dynamics to be tracking. 
[26:52] Anyone who's treating this as pure fiction is, 
I think, missing the point. This scenario is  
[26:57] not prophecy, but its plausibility should give us 
pause. But there's a lot that could go differently  
[27:02] than what's depicted here. I don't want to 
just swallow this viewpoint uncritically. Many  
[27:07] people who are extremely knowledgeable have been 
pushing back on some of the claims in AI 2027. 
[27:11] The main thing I thought was especially 
implausible was on the good path,  
[27:18] the ease of alignment. They sort of seem to have 
a picture where people slowed down a little and  
[27:23] then tried to use the AI to solve the alignment 
problem, and that just works. And I'm like,  
[27:28] yeah, that looks to me like a fantasy story.
This is only going to be possible if there  
[27:33] is a complete collapse of people's democratic 
ability to influence the direction of things,  
[27:39] because the public is simply not willing to 
accept either of the branches of this scenario. 
[27:43] It's not just around the corner. I mean, 
I've been hearing people for the last 12,  
[27:47] 15 years claiming that, you know, AGI is just 
around the corner and being systematically  
[27:52] wrong. All of this is gonna take, you know, 
at least a decade and probably much more. 
[27:56] A lot of people have this intuition that 
progress has been very fast. There isn't  
[28:00] like a trend you can literally extrapolate 
of when do we get the full automation? 
[28:05] I expect that the takeoff is somewhat slower.
So sort of the time in that scenario from,  
[28:10] for example, fully automating research 
engineers to the AI being radically superhuman,  
[28:15] I expect it to take somewhat longer 
than they describe. In practice,  
[28:19] I'm predicting my guess is that more like 2031.
Isn't it annoying when experts disagree?  
[28:25] I want you to notice exactly what they're 
disagreeing about here and what they're not. 
[28:29] None of these experts are questioning whether 
we're headed for a wild future. They just disagree  
[28:33] about whether today's kindergartners will get 
to graduate college before it happens. Helen  
[28:37] Toner, a former OpenAI board member, puts this in 
a way that I think just cuts through the noise,  
[28:41] and I like it so much I'm just gonna 
read it to you verbatim. She says,  
[28:45] "Dismissing discussion of super intelligence 
as science fiction should be seen as a sign  
[28:50] of total unseriousness. Time travel is science 
fiction. Martians are science fiction. Even many  
[28:56] skeptical experts think we may build it in the 
next decade or two. It is not science fiction." 
[29:04] So what are my takeaways? I've got three. Takeaway 
number one: AGI could be here soon. It's really  
[29:12] starting to look like there is no grand discovery, 
no fundamental challenge that needs to be solved.  
[29:19] There's no big deep mystery that stands between 
us and artificial general intelligence. And yes,  
[29:25] we can't say exactly how we will get there.
Crazy things can and will happen in the meantime  
[29:30] that will make some of the scenario turn out to 
be false, but that's where we're headed and we  
[29:38] have less time than you might think. One of the 
scariest things about this scenario to me is even  
[29:43] in the good ending, the fate of the majority of 
the resources on Earth are basically in the hands  
[29:49] of a committee of less than a dozen people.
That is a scary and shocking amount of  
[29:56] concentration of power. And right now we live in 
a world where we can still fight for transparency  
[30:02] obligations. We can still demand information 
about what is going on with this technology,  
[30:06] but we won't always have the power and the 
leverage needed to do that. We are heading  
[30:10] very quickly towards a future where the 
companies that make these systems and the  
[30:14] systems themselves just need not listen 
to the vast majority of people on Earth. 
[30:19] So I think the window that we have to act 
is narrowing quickly. Takeaway number two:  
[30:26] By default, we should not expect to be ready 
when AGI arrives. We might build machines  
[30:32] that we can't understand and can't turn off 
because that's where the incentives point.  
[30:37] Takeaway number three: AGI is not just 
about tech, it's also about geopolitics. 
[30:43] It's about your job. It's about power. It's about 
who gets to control the future. I've been thinking  
[30:50] about AI for several years now and still reading 
AI 2027 made me kind of orient to it differently.  
[30:59] I think for a while it's sort of been my thing 
to theorize and worry about with my friends and  
[31:04] my colleagues, and this made me want to call my 
family and make sure they know that these risks  
[31:11] are very real and possibly very near, and that 
it kind of needs to be their problem too now. 
[31:21] I think that basically companies shouldn't be 
allowed to build superhuman AI systems, you know,  
[31:29] super broadly superhuman super intelligence until 
they figure out how to make it safe. And also  
[31:35] until they figure out how to make it, you know, 
democratically accountable and controlled. And  
[31:40] then the question is, how do we implement that?
And the difficulty, of course, is the race  
[31:42] dynamics where it's not enough for one state 
to pass a law because there's other states and  
[31:48] it's not even enough for one country to pass a law 
because there's other countries. Yeah. Right. So  
[31:52] that's like the big challenge that we all need to 
be prepping for when chips are down and powerful  
[31:57] AI is imminent. Prior to that, transparency is 
usually what I advocate for. So stuff that sort  
[32:03] of like builds awareness, builds capacity.
Your options are not just full throttle  
[32:08] enthusiasm for AI or dismissiveness. There 
is a third option, which is to stress out  
[32:13] about it a lot and maybe do something about it.
The world needs better research, better policy,  
[32:19] more accountability for AI companies. Just 
a better conversation about all of this.  
[32:23] I want people paying attention who are capable, 
who are engaging with the evidence around them,  
[32:28] with the right amount of skepticism and above 
all, who are keeping an eye out for when what they  
[32:34] have to offer matches what the world needs, and 
are ready to jump when they see that happening. 
[32:39] You can make yourself more capable, more 
knowledgeable, more engaged with this conversation  
[32:45] and more ready to take opportunities where you see 
them. And there is a vibrant community of people  
[32:50] that are working on those things. They're scared 
but determined. They're just some of the coolest,  
[32:56] smartest people I know, frankly, and 
there are not nearly enough of them yet. 
[33:02] If you are hearing that and thinking, yeah, 
I can see how I fit into that, great. We have  
[33:08] thoughts on that. We would love to help, but even 
if you're not sure what to make of all this yet,  
[33:12] my hopes for this video will be realized if we 
can start a conversation that feels alive here in  
[33:18] the comments and offline about what this actually 
means for people, people talking to their friends  
[33:24] and family because this is really going to affect 
everyone. Thank you so much for watching. There  
[33:32] are links for more things to read, for courses 
you can take, job and volunteer opportunities  
[33:38] all in the description, and I'll be there in the 
comments. I would genuinely love to hear your  
[33:43] thoughts on AI 2027. Do you find it plausible?
What do you think was most implausible? And  
[33:49] if you found this valuable, please do like and 
subscribe and maybe spend a second thinking about  
[33:55] a person or two that you know who might find it 
valuable—maybe your AI progress skeptical friend,  
[34:02] or your ChatGPT-curious Uncle or 
maybe your local member of Congress.