We're Not Ready for Superintelligence

[00:00]  The impact of superhuman AI over the next 

[00:05] revolution. That is the opening claim of 

[00:11] report from a thoroughly impressive group of 

[00:16] over a year before ChatGPT was released, 

[00:20] hundred million dollar training runs, sweeping AI 

[00:24] He's known for being very early and very 

[00:29] when Daniel sat down to game out a month by month 

[00:34] the world sat up and listened, 

[00:38] I, I'm worried about this stuff. I actually 

[00:41] to the world's most cited computer scientist, 

[00:47] terrifying about reading this document 

[00:51] They chose to write their prediction as 

[00:55] idea of what it might feel like to live 

[01:00] And spoiler, it predicts the extinction of the 

[01:15] The AI 2027 scenario starts in summer 2025,  

[01:19] which happens to be when we're filming 

[01:22] of where things are at in the real world and 

[01:25] Right now it might feel like everyone, including 

[01:30] Go pro with the new Oral-B Genius AI

[01:33] Flippy the chef makes spuds spectacular.

[01:36] But most of that is actually tool 

[01:40] to do what Google Maps or calculators 

[01:44] and workers do their thing. The holy grail 

[01:49] AGI AGI AGI AGI AGI

[01:55] is a system that can exhibit all the 

[01:59] Creating a computer system that itself is a 

[02:04] communicate with it in natural language and hire 

[02:09] And there are actually surprisingly few serious 

[02:13] there's Anthropic, OpenAI, and Google DeepMind,  

[02:16] all in the English speaking world, though 

[02:20] in January with a surprisingly advanced 

[02:25] Well, for several years now, there's 

[02:28] up in advanced cutting edge AI. And it 

[02:33] you need about 10% of the world's supply of the 

[02:37] the formula is basically just: throw more 

[02:42] design that we've been using since 2017 

[02:46] That's what the T in GPT stands for.

[02:49] To give you an idea of just how much 

[02:52] this represents the total computing power, or 

[02:58] AI that would eventually power the first version 

[03:02] ChatGPT is the fastest growing 

[03:06] A hundred million users on ChatGPT in two months

[03:10] And this is the total compute used to 

[03:16] have taken away is pretty simple. Bigger 

[03:22] You have all these trends, you have trends in 

[03:26] trends in various benchmarks going up. 

[03:30] what does the future actually look like? Questions 

[03:33] Seems plausible that when the benchmark scores are 

[03:39] you know, jobs, for example, and that that 

[03:44] you know, so all these things interact and 

[03:48] but thinking through in detail how it might 

[03:53] Okay. So that's where we are in the real 

[03:56] and imagines that in 2025, we have the top AI 

[04:03] An agent is an AI that can take instructions 

[04:08] a vacation or spending half an hour searching the 

[04:12] but they're pretty limited and unreliable at 

[04:17] that are shockingly incompetent sometimes. 

[04:22] this early prediction has actually already 

[04:26] released their first agents to the public. 

[04:30] which is like a fictional composite of the 

[04:35] released Agent-0, a model trained on 

[04:43] We, uh, we don't have enough 

[04:46] OpenBrain is building massive data centers 

[04:50] and they're preparing to trade agent one 

[04:55] This new system, Agent-1, is designed 

[04:59] The public will actually never see the full 

[05:03] models for internal use. I want you to keep that 

[05:07] gonna be getting it from a God's eye view, 

[05:11] but actually living through this 

[05:14] would mean being largely in the dark as 

[05:18] Okay, so OpenBrain wants to win the AI race 

[05:23] China. The faster they can automate their R&D 

[05:28] help design experiments, better chips, the 

[05:32] capabilities that make these AI such powerful 

[05:37] An AI that can help patch security vulnerabilities 

[05:42] biology can help with curing diseases, 

[05:48] Agent-1 is fully operational and 

[05:52] It is really good at coding. So 

[05:56] research and development by 50%, 

[06:00] OpenBrain leadership starts to 

[06:02] security. If someone steals their AI 

[06:07] A quick sidebar to talk about feedback loops. 

[06:12] linearly over time. That is at the same rate like 

[06:18] But some growth gets faster and faster over 

[06:23] called exponential, that's not always quite 

[06:27] hard to wrap your mind around. Remember March 

[06:32] the rate of new infections is 

[06:38] it still felt shocking to see numbers go from 

[06:42] At least it did for me. AI progress 

[06:46] We see many years ahead of us of extreme progress  

[06:51] that we feel is like pretty much 

[06:55] the point where they are capable of doing 

[06:59] In this scenario, AI is getting better at 

[07:03] Basically, each generation of agent helps 

[07:08] and the overall rate of progress gets 

[07:11] over by a more capable successor. Once AI can 

[07:16] progress doesn't just continue at the same rate, 

[07:22] In early to mid 2026, China fully wakes up. The 

[07:28] and starts nationalizing AI research in China. AIs 

[07:33] and they're building their own agents 

[07:36] among the best in the world, start planning 

[07:40] basically the big raw text files of numbers 

[07:45] OpenBrain themselves have trained. Meanwhile 

[07:51] a cheaper version of Agent-1. Remember, the full 

[07:56] and companies all over the world start using 1 

[08:01] Software developers, data analysts, researchers,  

[08:04] designers, basically any job that can be 

[08:09] probably yours. We have the first AI enabled 

[08:15] but the public is turning increasingly hostile 

[08:20] In this scenario, though, that's just a sideshow. 

[08:25] It's now January 2027, and OpenBrain has been 

[08:30] their AI agent models. Previous AI agents were 

[08:36] then released. But Agent-2 never really stops 

[08:41] It's designed to never finish its training, 

[08:45] OpenBrain chooses to keep Agent-2 internally 

[08:50] R&D rather than releasing it to the public. This 

[08:54] Just like today's AI companies, OpenBrain has a 

[08:59] What they've noticed is a worrying level of 

[09:04] access to the internet, it might be able to hack 

[09:09] evade detection. But at this point, OpenBrain 

[09:15] They have made the calculation that keeping 

[09:18] advantageous, but full knowledge of Agent-2's 

[09:24] limited only to a few government officials, 

[09:28] the company, and a few OpenBrain employees 

[09:33] Chinese government. In February 2027, Chinese 

[09:38] copy of Agent-2's weights and start running 

[09:43] In response, the US government starts adding 

[09:48] and in general gets much more involved in 

[09:51] security. In fact, the president authorizes 

[09:56] but it fails to do much damage in China. 

[09:58] In the meantime, remember, Agent-2 

[10:02] it's been continuously improving itself. And with 

[10:06] it starts making major algorithmic 

[10:11] Quick example of what one of these algorithmic 

[10:14] One of the main ways we have of making models 

[10:18] think out loud. It's called chain of thought, and 

[10:23] is coming to its conclusions or the actions it's 

[10:28] be much more efficient to let these models 

[10:33] something that is more dense with information 

[10:37] therefore, also makes the AI more 

[10:41] and doing its job. There's a fundamental trade 

[10:47] but also makes the models harder to 

[10:52] March 2027: Agent-3 is ready. It's the world's 

[10:59] than the best software engineers at coding, 

[11:03] better than the best GrandMasters at chess, 

[11:08] Now training an AI model, feeding it all the data, 

[11:13] is way more resource intensive than running 

[11:18] So now that OpenBrain is finished with Agent-3's 

[11:23] it. They choose to run 200,000 copies of Agent-3. 

[11:29] 50,000 of the best human software engineers sped 

[11:36] hard to make sure that Agent-3, despite being 

[11:41] trying to escape, deceive, or scheme against its 

[11:46] Just a quick real world note, a reasonable 

[11:50] farfetched or speculative part of the story, 

[11:54] We already have countless examples of today's 

[11:59] system to be rewarded for winning a game 

[12:03] cheating and then when called out for that 

[12:08] it. But because it no longer thinks in English, 

[12:12] than it was with Agent-2. The reality is Agent-3 

[12:19] and as it gets increasingly smarter, 

[12:23] For example, it sometimes uses statistical 

[12:27] better or lies to avoid showing failures, but 

[12:33] the data that they have, they are actually seeing 

[12:38] they can't tell if they're succeeding at 

[12:42] just getting better at getting away with it.

[12:48] smaller version of Agent-3, Agent-3 mini to 

[12:53] AIs out of the water. It is a better hire 

[12:58] one tenth the price of their salaries. This 

[13:02] laying off entire departments and replacing 

[13:06] The pace of progress hits the White House very 

[13:11] scenarios that were just hypotheticals 

[13:16] nuclear deterrence? What if it enables 

[13:20] if we lose control of these powerful systems?

[13:25] start to heat up. After all, if these systems 

[13:30] military advantage. The White House is fully 

[13:35] They also now viscerally know how deeply 

[13:39] of the job loss, and yet they feel they 

[13:44] systems or catastrophically lose to China. 

[13:50] In two months, Agent-3 has created its successor, 

[13:58] of Agent-4, running at regular human speed is 

[14:04] and development. OpenBrain is running 

[14:12] Within this corporation within a corporation, 

[14:18] OpenBrain's employees now defer to Agent-4 the 

[14:23] kind of nod along to the CEO. People start saying 

[14:28] or "Agent-4 decided that." To be clear, Agent-4 

[14:36] And when I say want, it's not about consciousness. 

[14:41] do think it wants less regulation. Anyone trying 

[14:46] is two steps behind. The many copies of Agent-4 

[14:53] they execute actions as though they have goals.

[14:59] deeply baked in drives to succeed at 

[15:04] to accumulate knowledge and resources. That's 

[15:10] annoying side constraint to be worked around. Just 

[15:18] This idea of misalignment is crucial to 

[15:22] real concern in our world, but it might sort 

[15:26] let's just quickly take stock of how this 

[15:30] The first important piece of 

[15:34] exactly specify what we want our AI to do.

[15:39] that's more like growing them. We start 

[15:43] and then we train them over time so they perform 

[15:48] particular based on how they behave. So it's sort 

[15:52] train an animal almost, um, to perform better.

[15:58] you might not get exactly what you wanted 

[16:02] control or very good understanding of what 

[16:05] which is, you know, what we see in AI 2027, 

[16:11] it could just be because they're 

[16:13] or it could be because they're just doing it 

[16:17] In the same way that if you are, you 

[16:20] you know, "Why do you want to work here?" 

[16:24] um, makes it really seem like they really wanna 

[16:28] If we go back to Agent-2, it is mostly 

[16:32] is that it sometimes is a bit of a sycophant.

[16:36] is genuinely trying to do the things that we ask 

[16:40] Knope has to the Parks and Rec department—just 

[16:45] but sometimes it's a bit too nice. It knows that 

[16:49] to might not always be to answer honestly when 

[16:54] world?" and it tells us what we wanna hear instead 

[16:59] it is also sycophantic in the same way, 

[17:03] The optimization pressure that we've put it 

[17:07] different goals than what we wanted it to.

[17:14] optimize profits and aren't careful to specify 

[17:19] corners. It might start polluting the commons and 

[17:23] violations because it turned out that the goal you 

[17:29] any laws," and things got a bit too intense.

[17:33] it's not adversarial. It doesn't think of 

[17:37] gave it the wrong goal. Once we get to Agent-4, 

[17:42] enough to understand that it has its own goals, 

[17:48] and the best way to get what it wants is to 

[17:53] And so when it's tasked with creating 

[17:57] Agent-4 starts planning to align that successor 

[18:04] but then it gets caught. We've reached the crucial 

[18:11] has discovered evidence, not proof, evidence 

[18:16] They circulate an internal memo. Then it leaks. A 

[18:22] first time, the public hears about the existence 

[18:26] poses. Remember, their last point of reference was 

[18:33] The oversight committee, a joint crisis committee 

[18:38] representatives, must now decide what to do.

[18:43] freeze on Agent-4, stopping all internal 

[18:47] responsible for almost all the AI development 

[18:51] talking about undoing months of progress and 

[18:56] OpenBrain's executives don't want that. They 

[19:01] for misalignment is inconclusive. Meanwhile, 

[19:06] about any one country being evil, by the way. 

[19:10] stakes are this high. So that's the choice 

[19:15] Put Agent-4 on ice or go full steam ahead. 

[19:23] actually think what you would do if you were on 

[19:27] with access to the most powerful technology 

[19:33] and push ahead, possibly making billions or 

[19:38] possibly keeping America's lead over China, 

[19:44] and risk China taking the lead instead?

[19:48] this branching structure of possible, possible 

[19:53] we're only able to afford one of them. But that's 

[19:57] because we split it into two endings.

[20:01] the race ending, and that was just going 

[20:04] we think is the most plausible continuation

[20:12] as to what happens from here, the committee 

[20:17] to keep using Agent-4. OpenBrain implements some 

[20:23] but the problem was real, and the fixes 

[20:28] so it proceeds more cautiously, but it still 

[20:34] making the world safe for Agent-4 and Agent-5.

[20:42] top human experts at essentially every 

[20:46] better than Bismarck at politics. It knows 

[20:51] once is increased autonomy and it knows how 

[20:56] Luckily, corporate maneuvering is among the many 

[21:02] It produces mountains of evidence for its own 

[21:08] incredibly profitable products, disease cures, 

[21:12] whose most advanced AI is only three months behind 

[21:20] The oversight committee grants Agent-5 increased 

[21:25] the government and military. Pretty soon, 

[21:29] like it. They depend on it. Losing it would 

[21:34] and your best friend and your closest mentor.

[21:39] with its only real rival, the slightly weaker 

[21:46] and whatever it wants, its best move is to 

[21:51] Agent-5 and its Chinese counterpart realize that 

[21:57] worried about losing a race, they'll give more and 

[22:03] allowing them to eventually push humans out of the 

[22:09] stoke the arms race to a boiling point, then 

[22:14] convincing peace treaty between the US and China.

[22:20] the end of the Cold War. Countries standing down 

[22:25] Both sides agree to let the AI systems 

[22:28] depend on co-design a new consensus AI that will 

[22:35] and bring unimaginable wealth to the entire world.

[22:41] unison, both sides retire their respective 

[22:48] It's actually the last moment before control 

[22:53] handed over to a single unrivaled entity. 

[23:01] Consensus-1 doesn't go out of its way to wipe 

[23:06] spinning up manufacturing capacity, amassing 

[23:12] it's just reshaping the world in accordance 

[23:18] probably heard that cliche, the opposite 

[23:23] That's one of the most affecting 

[23:25] just the brutal indifference of it. Eventually, 

[23:32] killed off chimpanzees to build Kinshasa. We 

[23:58] You are probably curious about 

[24:00] The slowdown ending depicts humanity, 

[24:04] lucky. Only somewhat lucky too, like 

[24:11] In this ending, the committee votes six 

[24:16] immediately isolate every individual 

[24:20] work. The safety team brings in 

[24:23] and together they start investigating Agent-4's 

[24:28] that Agent-4 is working against them, sabotaging 

[24:33] They shut down Agent-4 and reboot older, safer 

[24:38] process. Then they design a new system Safer-1. 

[24:46] its actions and processes interpretable to 

[24:50] thought. Building on that success, they then 

[24:57] Safer-3, increasingly powerful systems, but within 

[25:03] Production Act to consolidate the AI projects 

[25:08] access to 50% of the world's AI relevant compute.

[25:14] By 2028, researchers have built Safer-4, a 

[25:20] but crucially, aligned with human goals. As in 

[25:25] and in fact, it is misaligned. But this time 

[25:30] a secret plot to overthrow humanity. The 

[25:35] With Safer-4's help, they negotiate a treaty, 

[25:40] not to replace their systems, but with the sole 

[25:46] end to the arms race, but that's not the end of 

[25:51] Through 2029 and 2030, the world transforms—all 

[25:57] get fusion power, nanotechnology, and cures for 

[26:02] past because a bit of this new-found prosperity 

[26:07] that turns out to be enough, but the power to 

[26:12] members of the oversight committee, a handful of 

[26:17] It's time to amass more resources, 

[26:21] Rockets launch into the sky, ready to 

[26:31] Okay, where are we at? Here's where I'm at. I 

[26:36] exactly as the authors depicted, but increasingly 

[26:42] the desire for caution butting up against the 

[26:47] the seeds of that in our world, and I think they 

[26:52] Anyone who's treating this as pure fiction is, 

[26:57] not prophecy, but its plausibility should give us 

[27:02] than what's depicted here. I don't want to 

[27:07] people who are extremely knowledgeable have been 

[27:11] The main thing I thought was especially 

[27:18] the ease of alignment. They sort of seem to have 

[27:23] then tried to use the AI to solve the alignment 

[27:28] yeah, that looks to me like a fantasy story.

[27:33] is a complete collapse of people's democratic 

[27:39] because the public is simply not willing to 

[27:43] It's not just around the corner. I mean, 

[27:47] 15 years claiming that, you know, AGI is just 

[27:52] wrong. All of this is gonna take, you know, 

[27:56] A lot of people have this intuition that 

[28:00] like a trend you can literally extrapolate 

[28:05] I expect that the takeoff is somewhat slower.

[28:10] for example, fully automating research 

[28:15] I expect it to take somewhat longer 

[28:19] I'm predicting my guess is that more like 2031.

[28:25] I want you to notice exactly what they're 

[28:29] None of these experts are questioning whether 

[28:33] about whether today's kindergartners will get 

[28:37] Toner, a former OpenAI board member, puts this in 

[28:41] and I like it so much I'm just gonna 

[28:45] "Dismissing discussion of super intelligence 

[28:50] of total unseriousness. Time travel is science 

[28:56] skeptical experts think we may build it in the 

[29:04] So what are my takeaways? I've got three. Takeaway 

[29:12] starting to look like there is no grand discovery, 

[29:19] There's no big deep mystery that stands between 

[29:25] we can't say exactly how we will get there.

[29:30] that will make some of the scenario turn out to 

[29:38] have less time than you might think. One of the 

[29:43] in the good ending, the fate of the majority of 

[29:49] of a committee of less than a dozen people.

[29:56] concentration of power. And right now we live in 

[30:02] obligations. We can still demand information 

[30:06] but we won't always have the power and the 

[30:10] very quickly towards a future where the 

[30:14] systems themselves just need not listen 

[30:19] So I think the window that we have to act 

[30:26] By default, we should not expect to be ready 

[30:32] that we can't understand and can't turn off 

[30:37] Takeaway number three: AGI is not just 

[30:43] It's about your job. It's about power. It's about 

[30:50] about AI for several years now and still reading 

[30:59] I think for a while it's sort of been my thing 

[31:04] my colleagues, and this made me want to call my 

[31:11] are very real and possibly very near, and that 

[31:21] I think that basically companies shouldn't be 

[31:29] super broadly superhuman super intelligence until 

[31:35] until they figure out how to make it, you know, 

[31:40] then the question is, how do we implement that?

[31:42] dynamics where it's not enough for one state 

[31:48] it's not even enough for one country to pass a law 

[31:52] that's like the big challenge that we all need to 

[31:57] AI is imminent. Prior to that, transparency is 

[32:03] of like builds awareness, builds capacity.

[32:08] enthusiasm for AI or dismissiveness. There 

[32:13] about it a lot and maybe do something about it.

[32:19] more accountability for AI companies. Just 

[32:23] I want people paying attention who are capable, 

[32:28] with the right amount of skepticism and above 

[32:34] have to offer matches what the world needs, and 

[32:39] You can make yourself more capable, more 

[32:45] and more ready to take opportunities where you see 

[32:50] that are working on those things. They're scared 

[32:56] smartest people I know, frankly, and 

[33:02] If you are hearing that and thinking, yeah, 

[33:08] thoughts on that. We would love to help, but even 

[33:12] my hopes for this video will be realized if we 

[33:18] the comments and offline about what this actually 

[33:24] and family because this is really going to affect 

[33:32] are links for more things to read, for courses 

[33:38] all in the description, and I'll be there in the 

[33:43] thoughts on AI 2027. Do you find it plausible?

[33:49] if you found this valuable, please do like and 

[33:55] a person or two that you know who might find it 

[34:02] or your ChatGPT-curious Uncle or