[0:00] Hey, this is the definitive course on AI
[0:02] agents. I currently teach over 2,000
[0:04] people how to use AI agents in both
[0:05] their personal and business lives and
[0:07] run a business that does over $4 million
[0:09] a year using AI agents. So, you don't
[0:11] need any programming or pre-existing
[0:13] computer experience in order to make
[0:15] this course work for you. I myself don't
[0:17] have a formal computer science degree.
[0:18] I've learned everything that I know
[0:19] watching free resources like you're
[0:21] doing now. This is also a general AI
[0:23] agents course, so you don't need to know
[0:25] any specific platform. This isn't just
[0:27] on codecs or claw code or anti-gravity,
[0:29] but rather on all of them. So, wherever
[0:31] you guys are starting, you'll end up at
[0:33] the same place. No fluff. Here's what
[0:34] you're going to learn in this course.
[0:35] First, I'll show you guys a demo where
[0:37] I'm controlling five AI agents, each
[0:39] with their own Chrome browsers as they
[0:41] interact with the web and perform
[0:42] economically valuable activities for me.
[0:44] I wanted to frontload this course with a
[0:46] demo so you guys could see what we're
[0:47] working up to. And just a few months
[0:49] ago, what I'm doing here would have been
[0:50] considered absurd. Then, I'm going to
[0:51] cover the core AI agent workflow loop,
[0:53] which works independent of which
[0:54] platform you're using. After that, I'm
[0:56] actually going to talk about and then
[0:57] sign up to the three major AI agent
[0:59] platforms right now. So, I'll sign up to
[1:00] codeex to anti-gravity and then cloud
[1:02] code. And then after I'll cover what
[1:04] each platform is at the moment the best
[1:06] or the worst at. Then we're going to
[1:07] dive into foundational AI agent
[1:09] prompting techniques. So, selfmodifying
[1:11] agent instructions where the agent will
[1:13] rewrite its own rules to minimize the
[1:15] number of errors made. Multi-agent MCP
[1:17] orchestration which is where we'll
[1:18] register codecs, Gemini, and Claude as
[1:20] MCP servers so you can manage multiple
[1:23] agents within a single conversation
[1:24] thread. video to action pipelines where
[1:26] we'll teach agents to learn from YouTube
[1:28] videos instead of plain text alone.
[1:30] Stochastic multi- aent consensus where
[1:32] we'll spawn multiple agents with the
[1:33] same prompt and then use their
[1:34] statistical spread in order to ideulate
[1:37] and improve things better. Agent chat
[1:38] rooms where you'll build centralized
[1:40] places for agents to debate ideas,
[1:42] pushing them to much higher quality
[1:44] answers than before. Subagent
[1:46] verification loops where your agents
[1:47] will actually review each other's work
[1:49] in real time to catch things that one of
[1:50] them might have missed. We'll talk
[1:52] prompt contracts. I'll show you guys
[1:54] reverse prompting and a bunch of other
[1:56] techniques as well. And finally, we'll
[1:58] chat about context management and
[1:59] improving the agent output quality
[2:01] before closing out by discussing how to
[2:03] optimize um AI agent and then token
[2:05] pricing. So far, I haven't seen anybody
[2:07] on YouTube discuss most of what I cover
[2:08] in this course. So, for all intents and
[2:10] purposes, you guys consider this the
[2:11] sauce. Please bookmark this video,
[2:13] subscribe to the channel, and let's get
[2:15] into it. First, I want to show you how
[2:16] powerful these agents can be when you
[2:18] learn how to distribute work across
[2:20] multiple Chrome instances and give each
[2:22] sub agent their own workspace. What I
[2:24] have here is a simple list of leads
[2:28] from, let's just say, a conference. Now,
[2:30] we have fields like their websites,
[2:32] their LinkedIn description, their first
[2:34] name, their last name, but one thing is
[2:36] missing, their email address. Now, just
[2:39] a year ago or so, that would have
[2:41] invalidated my ability to reach out to
[2:43] these leads. But now, because I possess
[2:46] their websites, I can actually spawn a
[2:48] bunch of Cloud Code agents, have them go
[2:50] to the websites, and then have them
[2:51] interactively and dynamically fill out
[2:53] their contact forms. So, what just
[2:56] happened as I was talking was Claude
[2:58] went ahead and then opened up a bunch of
[2:59] different Chrome browsers for me. I'm
[3:01] going to rearrange these to make it
[3:03] really easy to see. And so, this might
[3:04] be a little bit tough to see, but what
[3:06] these agents are all doing is they're
[3:07] independently navigating over to the
[3:10] contact fields of each of these
[3:11] websites. They're then dynamically
[3:14] filling out fields like the first name,
[3:16] the last name, the email address, and so
[3:18] on and so forth. And then they're
[3:20] putting in a little bit of outreach
[3:22] that's templated, but then changes
[3:23] depending on who they're reaching out
[3:25] to. These agents, through a combination
[3:26] of both research and then communication
[3:28] between each other in a shared chat
[3:30] room, are capable of doing things that
[3:32] any one agent might have taken many,
[3:34] many hours to do before. This is what
[3:35] I'm going to work up to with you guys
[3:37] over the course of the rest of the next
[3:39] couple of hours. The main strength of AI
[3:42] agents is really their ability to
[3:43] parallelize, which is to run multiple
[3:46] instances of each of them simultaneously
[3:48] while they accomplish a task. Now, right
[3:51] now, I would say most AI agents aren't
[3:53] as intelligent or as capable as a human
[3:55] being for any given need. But what they
[3:58] are much better at us then is being
[4:00] fast. And so despite the fact that their
[4:03] accuracy might be a little bit lower
[4:05] than a human, their ability to oneshot
[4:07] stuff is worse than ours at the moment,
[4:10] they can run multiple instances of
[4:11] themselves simultaneously and try
[4:14] multiple approaches over and over and
[4:15] over and over again in order to
[4:17] ultimately achieve much better results
[4:19] than we can. The key is you need to know
[4:21] a little bit about how they work under
[4:23] the hood. Then you need to be able to
[4:24] combine them using elaborate prompt
[4:26] architecture like I'm going to show you
[4:28] in this course. So, why don't we start
[4:29] with one of the simplest, most
[4:31] foundational concepts before I actually
[4:33] guide you guys through signing up and
[4:34] setting up these different agents. And I
[4:36] call this the core agent loop. To make a
[4:39] long story short, I think most of you
[4:41] probably have intuition about how agents
[4:44] do things. But really, what they're
[4:46] doing at the end of the day is they're
[4:48] going through a loop over and over and
[4:50] over again. And this loop is composed of
[4:53] three major functions. The first is the
[4:56] observation step. And so here the agent
[4:59] is basically reading through all of its
[5:01] context. We're going to chat a little
[5:03] bit more about how to optimize and
[5:05] manage that later. That includes things
[5:07] like its files, its previous tool calls.
[5:10] It includes all of the system prompts,
[5:12] the clawed Gemini, and agents.mmds that
[5:15] you provide. If it does research in a
[5:17] previous step, it'll include the
[5:19] research from the internet. uh if you're
[5:21] feeding in multimodal data like vision
[5:23] data, camera data, uh you know, audio
[5:25] files and so on and so forth, it'll
[5:27] include all of that. And so this agent,
[5:29] okay, is just in an environment and it's
[5:31] just always observing what's going
[5:33] around it at least to start in the
[5:35] observation step. From there, it'll
[5:38] reason. And so this is the think step
[5:40] here. It'll consider based off of all of
[5:43] this context and based off of, you know,
[5:44] the user's highle goal, what do I do
[5:47] next? How should I plan my approach? And
[5:49] nowadays, most agent coding platforms
[5:52] make use of like a dedicated reasoning
[5:55] step that you can actually click into
[5:56] and see, which I'll show you guys a
[5:58] little bit more of. And this provides a
[6:00] tremendous amount of interpretability,
[6:01] accountability, and then steerability,
[6:03] which is really important that I think
[6:04] most people sleep on. After it's thought
[6:07] about things and basically wrote its own
[6:09] mini plan, it's time to actually act,
[6:11] right? And so, here's where it'll call
[6:13] tools. It'll edit the files that it
[6:15] decided to uh do so earlier in the plan.
[6:18] or maybe it'll run a command using
[6:20] command line interfaces, CLIs. After the
[6:24] action step is done, what it does is it
[6:26] gets the result of the tool call and
[6:28] then it feeds all of that stuff back
[6:30] into the observe step. So now we're
[6:33] basically running through that loop
[6:34] again, just with a little bit more
[6:36] context. And so what occurs essentially
[6:39] is we just tend to grow bigger and
[6:41] bigger and bigger and bigger. If our
[6:43] initial context was a certain size, our
[6:46] you know second loop it's a little bit
[6:48] bigger, our third loop it's a little bit
[6:50] bigger and fourth loop and so on and so
[6:52] forth. And what this is doing is this is
[6:54] basically stacking more and more tokens
[6:56] into the context that the model can then
[6:58] use to plan its next step. What occurs
[7:01] after you go through this loop you know
[7:03] usually three or four times is
[7:05] eventually the model reaches a point
[7:07] called the definition of done.
[7:11] And what the definition of done is,
[7:13] which I think a lot of people leave out
[7:15] of their agent prompts, which is
[7:16] probably why they're always underwhelmed
[7:17] by what happens, is it's the series of
[7:19] constraints and technical specifications
[7:22] required for the model to conclude that
[7:26] it no longer needs to do this loop. Once
[7:29] it reaches this definition of done,
[7:31] okay, over and over and over and over
[7:32] again, it notices and then it changes
[7:36] routes. So, now it goes to the task
[7:38] complete route where it generates a
[7:40] quick little final response for the
[7:42] user. Usually involves a nicely
[7:43] formatted answer, as I'm sure you guys
[7:45] know. Hey, Nick just finished your new
[7:48] thumbnail app build. And before
[7:50] outputting it in a window either in
[7:53] anti-gravity or codeex or maybe cloud
[7:55] code in a packaged way that you guys are
[7:58] familiar with. And so obviously if you
[8:01] have any intuition about how AI works at
[8:04] this point, if you've ever communicated
[8:05] with chat GPT or you know Claude or some
[8:09] other sort of desktop AI that's nestled
[8:11] into another application that you guys
[8:12] use, you'll probably know some of this
[8:14] stuff um just as like the foundation.
[8:16] But I wanted to make it really explicit
[8:18] at the beginning of this course because
[8:20] we're going to return to each of these
[8:22] steps over and over and over again. And
[8:23] it turns out that you can heavily
[8:25] optimize all three of these. You can
[8:27] optimize the hell out of the observe
[8:29] step. You can optimize the hell out of
[8:31] the think step. And understandably, you
[8:33] can optimize the hell out of the act
[8:34] step as well. That's what we're going to
[8:36] learn. Another point I'm going to make
[8:37] in this course is that AI agents aren't
[8:40] just the large language models
[8:41] themselves. You know, I think neural
[8:43] networks and transformers are obviously
[8:46] super inherently interesting because
[8:48] they're these massive statistical things
[8:50] and these beings that can that can do
[8:52] things. They can reason. They're very
[8:54] far removed from traditional computer
[8:56] programs. just 5 or 10 years ago. So, a
[8:58] lot of interest goes to the LLM, but I
[9:00] want you guys to know that the LLM
[9:02] really is just a very small part of what
[9:04] most people consider AI agents these
[9:06] days. The LLM is of course your
[9:08] reasoning engine, right? Of course, it
[9:10] understands language and of course it
[9:12] makes decisions, but it's kind of like a
[9:14] human being from like 20,000 years ago
[9:17] with like a spear in its hand, right?
[9:19] without all of the infrastructure around
[9:21] human beings, without like your your
[9:23] house and your fireplace and your hearth
[9:25] and a place to sleep at the end of the
[9:27] night and a a society where people farm
[9:29] and produce resources and you have cars
[9:32] that you can get and you traverse a lot
[9:33] of distance without all the tools and
[9:34] the architecture around the
[9:36] intelligence. The intelligence is
[9:37] actually quite limited in what it can do
[9:40] and that's where the rest of these
[9:41] sections come into play. So tools much
[9:44] like human beings have the ability to
[9:46] read files, run code, search the web,
[9:49] call APIs and edit files. Okay, so too
[9:52] does this AI agent. Much like human
[9:54] beings have the ability to set a highle
[9:57] goal and keep going until that task or
[9:59] goal is reached, you know, so too can
[10:02] agents. And much like human beings have
[10:04] some sort of persistent memory where we
[10:06] can keep track of things that we've done
[10:08] and then realize that some of those
[10:10] things didn't work. So we got to take a
[10:11] slightly different tack the next time.
[10:13] So too, agents have things like
[10:15] agents.mmd claw.md gemini.mmd access
[10:18] to their conversation history, access to
[10:20] automemory files and skills. And so it's
[10:23] not actually just the LLM, for instance,
[10:26] that makes an agent work. It's really
[10:28] all of these things multiplied by the
[10:30] fact that, you know, the LLM provides us
[10:32] like the ability to be a little bit
[10:33] flexible. And that's a really big
[10:35] different from just, you know, a chatbot
[10:37] and then an AI agent. A chatbot might
[10:39] just be the LLM, okay? But an agent
[10:42] takes that that LLM and then it adds on
[10:44] tools, a reasoning loop, memory, and so
[10:47] on and so on and so forth. So, as a
[10:48] brief example, I'll use an agentic
[10:50] coding platform called Codeex. And down
[10:53] here, I have a simple prompt where
[10:55] basically I just want this to do a bunch
[10:56] of research for me on creatine
[10:58] supplementation in men. And what I'm
[11:00] doing is I'm giving it a brief
[11:01] definition of done where I'm saying once
[11:04] you've compiled 10 plus empirical
[11:06] sources, return a structured report. And
[11:08] I'm doing this because I want to
[11:09] demonstrate this loop to you. And so
[11:10] there are a bunch of other things that
[11:12] are popping up here. We have the actual
[11:14] chat window up at the top and we have
[11:15] its response. But you'll notice that in
[11:17] between we have this sort of like grayed
[11:18] out section here. Okay. And this grayed
[11:20] out section is the thinking that the
[11:22] model is doing before it gets back to
[11:24] us. And so basically, you know, if this
[11:27] was chatpt back from 2022 or so, all we
[11:31] would have gotten is this. But because
[11:33] I'm telling it to take actions in the
[11:35] real world, it's capable of one,
[11:37] observing, and so it observes all of
[11:40] this text and all of its reply as
[11:42] context. Two, thinking. So it's capable
[11:46] of doing a bunch of thinking on what to
[11:48] do next. And then three, acting. And so
[11:51] then it's capable of saying, hm, the
[11:52] user probably wants me to do some
[11:54] research. I have access to a few tools
[11:56] available. One of the tools lets me
[11:57] search the web. Let me pump in a search
[11:59] term. It then compiled all of this
[12:01] information and then it just repeated
[12:03] the same thing. It then with all this
[12:05] context said, "Okay, I'm observing. Not
[12:07] only do I have these messages, but I
[12:08] also now have a bunch of research. Let
[12:10] me think about what to do next. Have I
[12:12] achieved the goal of the user compiling
[12:14] 10 plus empirical sources?" And you
[12:16] know, after it's made its sort of
[12:17] observation and thought and reasoned
[12:19] about it, then it's deciding to act. And
[12:20] what it's ended up doing after 58
[12:22] seconds is giving me this structured
[12:23] evidence report. So, this is an example
[12:26] of something that might have looped two
[12:27] times, three times, but the more
[12:29] intelligent and capable these models are
[12:30] getting um the longer that they're
[12:32] running autonomously without us.
[12:34] Hopefully, this isn't rocket science to
[12:36] anybody here, but in a nutshell, this is
[12:37] more or less what's always occurring
[12:39] non-stop every time you talk to a model.
[12:41] With all that being said, let's really
[12:43] quickly cover how to set these different
[12:45] models up. I'm going to be using Codeex,
[12:47] Claude Code, and Anti-gravity. You don't
[12:50] need to know anything about any of these
[12:52] platforms in order to run these
[12:53] examples. And if you're already very
[12:55] familiar with, let's say, I don't know,
[12:57] Claude Code and you've chosen to use
[12:58] that as your main agentic coding
[13:00] platform moving forward, you can skip
[13:01] over to the next section of the video.
[13:03] But I want to make sure that we all have
[13:04] an equal playing ground here. We all
[13:06] understand how each of these platforms
[13:08] work under the hood. So there are three
[13:09] major platforms. The first is Codeex,
[13:11] which is owned, managed, and run by
[13:13] OpenAI. The second is Claude Code, which
[13:16] is owned, managed, and run by Anthropic.
[13:18] And the third is Google's anti-gravity,
[13:20] which as I'm sure you can imagine is
[13:22] owned, managed, and run by Google. In
[13:24] order to start with Codex, what you
[13:26] first have to do is sign up to an OpenAI
[13:29] account. The way you do so is just look
[13:31] up OpenAI on Google, get to a page that
[13:33] looks anything like this, and then just
[13:35] go to the top right hand corner where it
[13:36] says try chat GPT. After that, you'll be
[13:38] taken to a page that looks something
[13:40] like this. You can continue with Google,
[13:42] your phone, or whatever you want. And if
[13:44] you choose to chat with the model and
[13:46] then come back at any point in time,
[13:47] just head to the top right hand corner
[13:49] for that modal again. So I'm going to
[13:51] pretend that I haven't made an account
[13:52] before and I'll continue with Google.
[13:53] After some brief onboarding
[13:55] instructions, you'll have access to a
[13:57] page like this. But this is just chat
[13:59] GPT, which is more akin to a chatbot
[14:01] than anything else. We want to take this
[14:03] to the AI agent world. And so in order
[14:05] to do that, we need to use their
[14:07] dedicated AI agentic coding platform,
[14:09] Codeex. So googling OpenAI's codeex or
[14:12] something like that will take you to a
[14:13] page that looks like this. And then you
[14:15] can just click download for Mac OS. By
[14:18] the way, I'm on a Mac, so that button is
[14:20] automatically going to pop up for me.
[14:21] But the Codeex app is now also available
[14:23] on Windows starting March 2024 and
[14:26] beyond. The way you install things on a
[14:27] Mac is you just take this window, drag
[14:29] Codex over to applications, and then
[14:31] you're done. Once you're inside, if you
[14:32] wanted to build a website or something,
[14:34] just head over to this middle, create a
[14:36] new folder, call it whatever you want.
[14:38] So I'll just go to downloads and then go
[14:39] new folder example.
[14:42] Open it within it. And now you're inside
[14:44] of this folder here. You can ask the
[14:45] model to do whatever you want. And so
[14:47] what I'm going to say is make a brief
[14:48] portfolio site about Nick Sariah. Keep
[14:51] it super simple and minimal. It'll now
[14:53] do some thinking. In our case, I
[14:56] actually have a design taste front-end
[14:57] skill which improves its ability to
[14:59] create like sleek, highquality looking
[15:01] designs. And now it's looking through my
[15:04] own workspace to put together this cool,
[15:06] sexy site for me. I'm also going to ask
[15:08] it to open it. Uh, and the way that all
[15:11] AI agent platforms work now is you have
[15:13] the ability to put a cued message in,
[15:16] which you can also choose to send
[15:18] immediately via steer. In my case, I'll
[15:20] just wait until it's done. It'll consume
[15:22] this open it message and then it'll just
[15:24] open it for me in a new tab. Once it's
[15:26] done, the open it message will be fed in
[15:28] and it's just going to open this for me
[15:29] in a new tab. Now, I'm kind of zoomed in
[15:31] here. So, if I zoom in a little bit
[15:33] more, you'll see that this is just a a
[15:35] simple one-page site that says Nyx Drive
[15:37] builds clear modern digital work. Here's
[15:39] some information about me. And here's a
[15:41] contact page. Not rocket science, but
[15:43] this is how easy it is to like build web
[15:44] stuff. Claude is pretty similar. Just
[15:46] Google Claude signup or something like
[15:48] that. And you'll be taken to a page that
[15:50] looks like this. Here you just enter
[15:52] your email address or in my case,
[15:53] continue with Google. In Claude's case,
[15:55] in order to use Claude code, you do have
[15:57] to pay for it. And so there's a pro plan
[15:59] here that's $17 per month with an annual
[16:02] subscription or 20 bucks if build
[16:04] monthly. I'm not working for Claude or
[16:07] anything like that. I don't have any
[16:09] sort of affiliation with Anthropic in
[16:11] that way. But I will say that I receive
[16:14] probably a 100 to 200x return on my
[16:16] investment with an Aentic coding
[16:18] platform, whether it's Claw or whether
[16:20] it's Gemini or whether it's Codeex. So,
[16:22] my recommendation for you, if this seems
[16:24] a little bit steep, is bite the bullet,
[16:27] pay it, and learn whatever you can to
[16:29] make a return on investment with that
[16:30] money in the first month because this
[16:32] stuff is really quite powerful. Assuming
[16:34] you're done, just type cloud code
[16:36] desktop download or something like that.
[16:38] And you'll be taken to a page that looks
[16:40] like this, which allow you to download
[16:41] it for Mac OS, Windows, or even Windows
[16:43] ARM 64. So, I'm going to give my Mac OS
[16:46] thing a quick click. Then, I'll go to
[16:48] the top right hand corner. I'll just
[16:49] open Cloud up just like I did with
[16:51] Codeex. That'll take me to a page like
[16:52] this. And then I just drag this over to
[16:54] the right. And then once you're done,
[16:55] you'll be taken to a chat page that
[16:57] looks something like this. What we
[16:58] really want is we want this code button.
[17:00] So I'm going to give that a click. Then
[17:02] here, all we need to do is just choose a
[17:04] folder to work in. And then we can put
[17:06] in a quick request. So I'm just going to
[17:07] choose a general folder next. Then I'm
[17:10] going to say bypass permissions, which
[17:12] might seem a little bit scary to you,
[17:13] but it just makes the model act
[17:15] independently. Then finally, I'm going
[17:17] to say, hey, make a brief portfolio site
[17:20] about Nyx Drive. super simple and
[17:22] minimal. And so, just like Codeex
[17:24] designed it a moment ago with its
[17:26] various UX uh features, we have the same
[17:28] thing here with Claude Code. It's going
[17:30] to ask to access some files in my
[17:32] folder. And in addition to having the
[17:34] message box, we also have this sort of
[17:36] grayed out shining decal here, which is
[17:39] sort of its like thinking if you think
[17:41] about it, as well as its tool calls. And
[17:43] what it's going to do now is actually
[17:45] build me a brief little site. And then,
[17:47] just like I did before, I'll just say
[17:49] open it.
[17:50] That's going to ceue it and now I can
[17:52] have a conversation with Claude and now
[17:54] we have the actual portfolio which as
[17:56] you guys could see here is done in
[17:57] significantly more minimal fashion.
[17:59] Okay, so this is Nick builder automation
[18:01] expert software engineer. Now unlike
[18:03] with chat GPT and then claude for
[18:06] anti-gravity odds are you probably
[18:08] already have like a Google or a Gmail
[18:10] account set up. So all you have to do is
[18:12] just look up Google anti-gravity
[18:13] download then click download for Mac OS.
[18:16] In my case I have Apple Silicon on Mac.
[18:18] If you guys don't know what you have,
[18:20] just type about this Mac and then if it
[18:22] says Intel up here in chip, you're in
[18:24] Intel. If it's a M something, then
[18:26] you're Apple Silicon. And you can do
[18:27] something similar for Windows and Linux
[18:29] as well. And once I give that a click,
[18:31] we'll be taken to a very similar looking
[18:33] page here. And then I can just drag
[18:34] anti-gravity over to applications. The
[18:36] very first time you open up
[18:37] anti-gravity, it'll look something like
[18:38] this. In your case, maybe it'll be dark
[18:40] mode or maybe it'll be entirely light. I
[18:42] just have some styling settings, which
[18:44] is why mine might look a little
[18:45] different from yours. You may also have
[18:47] to log in unless Google logged you in
[18:49] automatically. In my case, it logged me
[18:51] in automatically because I've used it
[18:52] before. Assuming that you've done that
[18:54] though, on the right hand side, you'll
[18:55] see an agent modal. And this agent modal
[18:57] is very similar to what we saw with
[18:59] codeex and then claude code. All we have
[19:01] to do is just ask it to make a brief
[19:02] portfolio site about Nixive. And you'll
[19:04] see here that the UX is just a little
[19:06] bit different, right? We have a little
[19:07] generating tab down here. Obviously, we
[19:10] have uh multiple settings with fast and
[19:12] Gemini 3.1 Pro. We have this little
[19:14] thinking tab. Uh, it tells you how long
[19:16] it's been doing it. If it has to do any
[19:18] web searches, it does so over here.
[19:20] Hopefully, you guys are seeing these are
[19:22] all just flavors that are slightly
[19:24] different, but ultimately are the same
[19:26] thing. I'm just going to write open it.
[19:27] That'll be added as a pending message,
[19:29] and then it'll open this up in a browser
[19:31] tab. As you see here, Gemini produced
[19:33] what I would probably consider to be the
[19:35] sexiest of all websites, which makes
[19:36] sense. Uh, one thing I'll talk about in
[19:38] a moment is how much better it is at
[19:40] front-end design and so on and so forth.
[19:42] And yeah, we have a very simple and and
[19:43] straightforward site here. So, um, this
[19:45] links to all of my resources, leftclick,
[19:48] YouTube, and so on and so forth. I
[19:49] probably like this one the best. From
[19:51] here on out, most of the conversations
[19:53] and the user experiences are going to be
[19:55] really similar between Aenta coding
[19:57] platforms. So, while I am going to use
[19:59] multiple just to show you guys how some
[20:01] of their quirks interact, uh, for the
[20:03] most part, I want you guys to know that
[20:04] the UX's are very very similar these
[20:07] days. Like the thinking tabs are going
[20:09] to be the same. Some people will
[20:10] probably say that there are slight
[20:11] differences between them and so on and
[20:13] so forth. For instance, I'm a big fan of
[20:15] the little Space Invader icon that Cloud
[20:17] Code has. But for all intents and
[20:19] purposes, I'm just going to assume that
[20:20] you're picking up the UX here as you use
[20:22] these models and focus less on like the
[20:24] tiny little stuff and more on how to
[20:26] orchestrate and then prompt these for
[20:28] higher quality responses. If you guys
[20:30] want to see like step-by-step
[20:32] walkthroughs of these platforms, I'm
[20:33] going to put some little links up above
[20:35] my left shoulder here, and you can click
[20:37] on them anytime to go learn that sort of
[20:39] stuff. Next up, I want to talk about
[20:40] what makes these AI coding platforms
[20:42] different from one another. Not on a
[20:44] user experience um angle, but from an
[20:48] intelligence angle, from a what they
[20:50] could do angle as well. So, as you saw
[20:52] there, there were three different
[20:53] models. There was Claude, which was
[20:55] wrapped around Claude code, Gemini,
[20:58] which was wrapped around anti-gravity,
[21:00] and then GPT, in my case, 5.4, which is
[21:03] wrapped around codeex. And I think that
[21:06] each of these models are really similar
[21:07] at this point in intelligence- wise, but
[21:10] there are some pros and cons to each.
[21:12] They basically like improve how they
[21:14] perform by a few percentage points. So
[21:17] Claude might be, you know, 2% better at
[21:19] these. You know, Gemini might be 5%
[21:21] better at these. GPT might be 1% better
[21:24] than these. I'm just pulling on numbers
[21:25] out of my butt, but I'm making them
[21:27] really small because I do want to really
[21:28] drive home the point that these models
[21:29] are so gosh darn intelligent these days
[21:32] that these minor differences only make
[21:34] sense at the bleeding edge and at the
[21:36] frontier. For most purposes, either of
[21:38] these are going to be sufficient. So,
[21:40] Claude has the most interpretable
[21:42] reasoning. You remember how I could
[21:44] click open that little reasoning tab a
[21:46] moment ago? Well, at least as of the
[21:48] time of this recording, Claude is
[21:49] incredible at making that reasoning tab
[21:51] really, really interpretable. you know
[21:53] exactly what cloud is doing at basically
[21:55] every step of the process when you um
[21:57] use cloud code to visualize that
[21:58] reasoning and that makes it really good
[22:01] for orchestration and then agentic
[22:03] workflows because you can see the
[22:05] decisions that the model is making in
[22:07] real time and in doing so you can also
[22:09] steer the model stop the model pause it
[22:11] or give it new resources halfway through
[22:13] I can't say the same about both Gemini
[22:15] and GPT I think they're a lot less
[22:17] interpretable and it's a lot less
[22:18] accountable you know Claude is sort of a
[22:20] partner that you build things with along
[22:22] the Whereas Gemini and GBT are almost
[22:24] just like I don't know, they're
[22:25] missiles. You set your target, you click
[22:27] the button, and then they go. Now, there
[22:29] are some cons. Claude is a little bit
[22:31] slower unless you use fast mode, which
[22:34] is what I tend to use, although keep in
[22:35] mind that'll burn a ton of credits. And
[22:37] then I find that it's weaker at frontend
[22:39] or design than a model like Gemini.
[22:41] Gemini is really good at design and
[22:43] frontends. As you guys just saw a moment
[22:45] ago, Claude picked a really
[22:46] minimalistic, sleek theme. Gemini did
[22:49] some upscale stuff that still looked
[22:51] sleek, clean, but had like that
[22:53] isomorphic glass. And then GBT, maybe
[22:56] because of my design taste scale or
[22:57] something else, was kind of like more
[22:59] complex and had uh a little bit clunkier
[23:01] of a design. Well, in general, I find
[23:03] that this pattern remains the same.
[23:05] Anytime I want to design a really clean
[23:07] front end, I'm going to use Gemini for
[23:08] that. It's also got superior multimodal
[23:11] abilities. That just means there's
[23:12] actual like endpoints using the Gemini
[23:14] API um where it can understand video.
[23:17] Right now, Claude and GPT both really
[23:19] struggle with this. Although you can
[23:20] build custom pipelines to do that, which
[23:21] I'll show you guys about. It also has
[23:23] the ability to use a fast output, which
[23:25] means it writes really, really quickly
[23:26] if need be. Um, but they don't have
[23:28] access to a dedicated fast mode where
[23:30] you could pay more money to use them
[23:31] really quick. I think it's the least
[23:33] interpretable of the models. And
[23:35] personally, I find the quality is quite
[23:36] inconsistent. There's some days when
[23:38] I'll prompt it and it'll do quite
[23:39] incredible, then other days where I will
[23:40] prompt it and it will just absolutely
[23:42] crap the bed. you know, at least
[23:44] Claude's quite consistent in that way,
[23:45] despite the fact that maybe it's a
[23:47] little bit worse at a few things.
[23:49] Finally, there's GPT. There's the codec
[23:51] series of models, the 5.4 series of
[23:53] models. Now, these are the best at
[23:55] back-end programming. I think they're
[23:56] also the best at like um absolute
[23:58] mathematics, which probably feeds into
[24:00] that. They're really great at
[24:02] test-driven development. And you know
[24:03] how I mentioned earlier Gemini and GBT
[24:05] are more like rockets that you point at
[24:06] a at a at a place and then they go. Um
[24:08] well these testdriven development
[24:11] approaches essentially mean you just
[24:12] outline that definition of done and then
[24:14] it fires and just goes autonomously
[24:16] until it reaches that. There's also
[24:18] quite a big ecosystem of different apps
[24:20] and you know there's a lot of um
[24:21] documentation online about how to use
[24:23] various GPT workflows and stuff like
[24:25] that because this was the first major
[24:27] player to the AI agent market. I'd give
[24:30] it sort of like a uh you know two out of
[24:32] three on the rest of these. I think
[24:33] Claude is much better at its
[24:35] interpretability. It's much better at
[24:37] orchestration and stuff like that, but
[24:39] GPT being a model that just came out
[24:40] quite recently, a 5.4 anyway, is
[24:43] obviously sort of like topping the
[24:44] charts right now on a lot of stuff. Just
[24:45] some caveats there. A lot of people
[24:47] treat this as like [snorts] anathema for
[24:50] you to claim that, you know, Claude is
[24:52] better than GPT at this thing and Gemini
[24:54] is better than than Claude at that
[24:56] thing. The reality is, as I mentioned
[24:58] and alluded to at the beginning, there
[25:00] are very minor differences between these
[25:01] models at this point. All of them are
[25:03] basically trained on the entirety of the
[25:04] internet as is. And so because of this,
[25:07] um, the slight differences in
[25:09] capabilities in the model tend to have
[25:10] more to do with like when they were
[25:12] trained and how recent it is versus, you
[25:14] know, some inherent like cool new design
[25:16] technique. Really, they're just training
[25:18] these galaxys sized brains on the entire
[25:20] internet at this point. So because we're
[25:22] talking about the LLM intelligences, you
[25:24] know, if like GPT was trained after
[25:26] Claude, GPT is probably going to be a
[25:27] little bit better in certain
[25:28] circumstances. If Gemini is trained
[25:30] after GPT, it'll be better. But all that
[25:32] stuff resets with the next generation.
[25:34] So though I am going to be showing you
[25:35] guys some cool multimmcp orchestration
[25:38] uh techniques later on, I want you to
[25:39] know that you don't have to treat all
[25:40] this super seriously. You can also just
[25:42] pick one model and then use that. Okay,
[25:44] next up I want to chat agents.mmd and
[25:46] then how to build a selfmodifying and
[25:49] self-correcting system prompt that
[25:51] significantly minimizes the number of
[25:52] errors that you get as you build things
[25:54] with these AI agents. So for the
[25:56] purposes of this demonstration, I'm
[25:58] going to be using anti-gravity and
[25:59] through it the Gemini series of models.
[26:01] When you open up anti-gravity, you have
[26:03] a little window that looks like this.
[26:05] Generally, I divide this into three
[26:06] panes. You have your explorer on the
[26:08] lefth hand side, your file editor in the
[26:10] middle, and then you have your agent on
[26:11] the right. And what I'm going to do for
[26:13] the purpose of this demo is I'll just
[26:14] click open folder. And then I'm going to
[26:16] go to anti-gravity example and just open
[26:18] this up. Okay. And what I want to do
[26:20] here is I just want to show you how all
[26:21] of this stuff works to start. As you
[26:23] guys can see on the lefth hand side, we
[26:25] have a file called gemini.md. Now, what
[26:27] occurs is when you talk to this model
[26:29] over here. Hey, what's up? Basically,
[26:32] what's occurring is this file is being
[26:36] prepended to the very top of a
[26:38] conversation chain. And so, if I open up
[26:40] this file right now, you see how it's
[26:42] empty. There's nothing in it. Well, when
[26:43] I started this conversation and said,
[26:45] "Hey, what's up?" Okay, it knows that my
[26:47] name is Nick, but it does it knows this
[26:49] because of uh the fact that I'm signed
[26:50] in as Nick. Now, I want you to see what
[26:53] happens if I paste in my name is Antonio
[26:55] Banderas. Refer to me as such. always
[26:57] always also always sign off super kawaii
[27:00] desu. So I'm going to go here to the top
[27:01] right hand corner and I'll say hey
[27:03] what's up and after initializing a new
[27:06] model
[27:09] notice how it's now going to return
[27:11] something quite different to what we had
[27:13] a moment ago. The reason why is of
[27:15] course this gemini.md is just a
[27:18] templated structured prompt that is
[27:21] basically always inserted into the
[27:22] beginning. Okay, the same thing applies
[27:24] with codecs. The same thing applies with
[27:26] claw code, but the names of the files
[27:28] are a little bit different. So if I was
[27:30] in, let's say, codeex for instance, I
[27:32] wouldn't call this a gemini.md. I'd call
[27:34] this an agents.mmd. If I was in claude
[27:36] code, I wouldn't call this an
[27:37] agents.mmd. I'd call this a claude.mmd.
[27:40] Whatever file you use here doesn't
[27:42] really change the idea. The idea is that
[27:44] at the very top of any prompt, you just
[27:46] have this file prepended to it. The
[27:49] reason why this is so powerful is
[27:50] because you now have the ability to
[27:53] statically template out the same prompt
[27:55] over and over and over again on every
[27:57] independent session. This may seem like,
[27:59] well, why don't you just copy and paste
[28:01] the same thing in instead of having to
[28:02] use this elaborate file system
[28:04] structure. And the reason why is because
[28:06] what you can do is at the very beginning
[28:08] of this file, you can actually contain
[28:10] within it like a list of lessons or
[28:12] learnings from previous instances. Then
[28:15] you can build in like a meta prompt
[28:17] structure where before a model signs
[28:19] off, before it finishes whatever it's
[28:20] doing, it always updates that file with
[28:23] more and more and more knowledge. In
[28:25] that way, okay, you can build a
[28:26] highquality list of like memories,
[28:29] preferences, and rules, not to mention
[28:31] things to avoid that significantly
[28:33] improves your agent's ability to operate
[28:35] over a long time scale. And just to show
[28:37] you guys what I mean, let me show you a
[28:39] diagram. In this hypothetical instance,
[28:41] we're going to be using Gemini.m MD. And
[28:43] basically what'll occur every time is a
[28:45] new session is going to start over here.
[28:47] The agent will first read gemini.md.
[28:50] You'll then give it a task like hey
[28:52] build me a website that does whatever.
[28:55] Now it'll return the website for me and
[28:57] then I'll say I don't like this no dark
[29:00] mode. After I give it its feedback of no
[29:03] dark mode rather than just correcting
[29:05] the build, it'll actually write that to
[29:07] my Gemini.m MD for next time, which
[29:10] allow the agent to continue working with
[29:11] the rule applied. When the session ends
[29:13] and a new session starts, now the agent
[29:15] will read the Gemini MD, but the
[29:17] gemini.md will have an additional rule
[29:19] placed. Okay, if this is my file over
[29:21] here, it'll say no dark mode. And that
[29:24] means the next time I ask it to build me
[29:25] a website or any sort of web property,
[29:27] it'll see no dark mode and then it won't
[29:28] make that mistake again. This lets your
[29:30] knowledge accumulate over sessions. The
[29:33] first time that you use, you know,
[29:34] Gemini or Claude Code or or Codeex or
[29:37] whatever, you know, you're only going to
[29:39] have, let's say, one rule or one
[29:41] preference stored. And so the number of
[29:43] errors that the model makes, errors
[29:44] relative to like your preferences will
[29:46] be pretty high. The second time that you
[29:48] use it, though, the number of errors or
[29:51] issues that it makes that don't line up
[29:52] with your preferences will go down. The
[29:54] third time, they'll go down further. The
[29:57] fourth time, it'll go down further. Then
[29:59] the fifth time it'll go really really
[30:01] low to the point where maybe it makes
[30:02] zero errors at all. You can see that um
[30:05] sort of diagrammatically over here with
[30:07] when you start your thing has zero
[30:08] rules. Okay. As it grows longer and
[30:10] longer and longer, you're writing more
[30:12] and more and more and more rules. Um the
[30:15] agents get better and better and better
[30:16] at uh understanding and then um
[30:18] anticipating as well your preferences.
[30:20] So what does this actually look like in
[30:22] practice? Well, it's not all that
[30:23] difficult and you can just append or
[30:25] prepen this to any Gemini Claude or
[30:28] agents MD however you like. It also
[30:30] doesn't need to be this long. Although I
[30:32] did want to go into a fair amount of
[30:33] detail here with you. So you can
[30:34] absolutely just turn this into like a I
[30:36] don't know a three or fourline snippet.
[30:38] Essentially before we start any task
[30:40] read this entire file. This file
[30:42] contains a growing rule set that
[30:43] improves over time. At session start I
[30:46] want you to read the entire learned rule
[30:47] section before doing anything. How it
[30:49] works. When the user corrects you or you
[30:51] make a mistake, immediately append a new
[30:53] rule to the learned rules section at the
[30:55] bottom of this file. Rules are numbered
[30:57] sequentially and written as clear
[30:59] imperative instructions. The format is
[31:02] category never or always do X because Y.
[31:05] And then here's some more formatting
[31:06] instructions. When do you add a rule?
[31:08] Add a rule when the user explicitly
[31:10] corrects your output. When the user
[31:11] rejects a file approach or pattern, when
[31:13] you hit a bug caused by wrong
[31:15] assumption, or when the user states a
[31:16] preference. Okay? Okay. And then it'll
[31:18] give some examples here of different
[31:19] rules in code. Then we have the learned
[31:21] rules down here. So what I'll do just to
[31:23] show you guys what this looks like is
[31:25] I'll say build me a simple portfolio
[31:28] site for Nick Sarif. And I'm going to
[31:30] have it go accomplish a task for me. And
[31:32] then I'm inherently and intentionally
[31:34] going to give it some instructions. You
[31:37] see the very first thing it did was
[31:38] analyze the gemini.md. And so now it
[31:41] actually has this entire file as context
[31:44] inside of its thread. You can't see that
[31:46] context here because obviously they
[31:47] don't want to just muck up your your
[31:49] conversation thread, but it is literally
[31:51] like if you just pasted this entire
[31:53] thing directly in. Okay, so it's going
[31:55] to be reading that constantly as it's
[31:56] building out the rest of our website.
[31:58] And you can see that it's like it's
[31:59] built some cool terminal display here.
[32:02] It's using a library called Vit, which
[32:04] is probably like the best front-end
[32:05] library. Let's see what it does. Okay,
[32:07] this website is looking really really
[32:09] sexy, super clean, and it clearly went
[32:11] above and beyond uh with my spec.
[32:13] However, I don't like how it's dark
[32:14] mode. So, what I'm going to do is go
[32:16] back here and then give it some
[32:17] instructions. Quit doing things in dark
[32:20] mode. And the idea here is when I give
[32:23] it an instruction like quit doing things
[32:25] in dark mode, what it's going to do is
[32:26] it's going to take my message and then
[32:29] say, hey, let's update our gemini.md to
[32:32] never create applications in dark mode.
[32:34] It's a user preference. If I scroll down
[32:37] here now, you can actually see that this
[32:39] style has been added. And so if the next
[32:42] time I run a model and instantiate
[32:45] anti-gravity, I say, "Hey, I'd like you
[32:46] to build me a website," it'll actually
[32:48] have this up at the very very top of its
[32:50] prompt, meaning that I'm never ever
[32:52] going to have a dark mode website again.
[32:54] In this way, this will continuously get
[32:56] closer and closer to my preferences
[32:58] until the number of rules become so
[33:00] exhaustive that, you know, it actually
[33:01] bes counterproductive. In practice, I
[33:04] haven't actually hit this limit yet. I
[33:05] think this just gets better and better
[33:06] and better over time. But I could
[33:08] hypothetically see if you were to get to
[33:09] a point where there's a thousand
[33:10] independent rules, some of them would
[33:12] probably start stepping on its its toes.
[33:14] Um, this sort of self-modifying claude
[33:17] agents or Gemini.mmd is a very very high
[33:19] ROI design pattern. So whatever you're
[33:21] building with an AI agent, whether
[33:22] you're using them for business, personal
[33:24] or programming tasks, I would always
[33:25] recommend to have something like this in
[33:27] your directory. And as you can see, it's
[33:28] now modified the site. We don't actually
[33:30] have that anymore. A lot cleaner. And it
[33:32] also fixed up the images and made it
[33:33] look really sexy. The way this works is
[33:35] at the very top level we have a global
[33:37] claude agents or gemini.md. And these
[33:41] are userwide rules that apply to all of
[33:44] the projects that you start. And so the
[33:46] very top you'll have this sort of
[33:47] injected and you can set this using a
[33:50] variety of different formatting
[33:51] conventions and stuff. You could look it
[33:52] up for the specific uh agent platform
[33:55] that you're using. you know, if you're
[33:56] doing claude or something like that,
[33:58] it's going to be stored in a a
[33:59] tilda.claude
[34:02] slash and then there are variety of
[34:03] other conventions regardless of whatever
[34:05] platform you're using that you guys
[34:06] could also after it's injected the
[34:09] global agents.mmd, it'll then inject the
[34:12] local cloudmd. And so what you could do
[34:15] is you could have a global cloudmd,
[34:17] okay, that has wide ranging user
[34:20] preferences updated and then a local
[34:22] project.mmd that has specific project
[34:25] preferences updated. And then underneath
[34:26] you also have uh skills and then your
[34:29] finally inline prompt and I'll touch on
[34:31] the skill section in a moment. But in
[34:33] that way, you can collapse a ton of
[34:35] context and a ton of sort of
[34:37] functionality into very few tokens,
[34:39] which is important because your build
[34:41] both per token and then the quality of
[34:42] the models tend to degrade the longer
[34:44] the token and context windows get. Next
[34:46] up, I want to talk a little bit about
[34:47] agent skills. And this isn't going to be
[34:49] an exhaustive resource. If you guys want
[34:51] a super in-depth way to look at skills,
[34:53] definitely just check out my full end
[34:55] toend Claude Code skills course. But
[34:58] agent skills, for those of you guys that
[34:59] don't know, is just a simple repeatable
[35:02] way that you can standardize workflows.
[35:05] Now, this is important because large
[35:07] language models are very flexible. So,
[35:10] if you give them a non super tightly
[35:12] scoped task, they'll tend to produce a
[35:14] variety of different results for you.
[35:15] Well, skills are just a way of basically
[35:17] turning that whole, you know, vagueness,
[35:20] that whole statistical variance into
[35:22] like a really straight line
[35:24] deterministic path where it just does
[35:26] the same thing over and over and over
[35:27] and over and over again. And so skills
[35:29] are offered now on all major platforms.
[35:31] We've all adopted them. So you have
[35:33] codec skills, you have Gemini skills,
[35:36] uh, and then you also have Claude code
[35:37] skills and they have very particular
[35:40] specs and they look really, really
[35:41] similar to one another. So, it's worth
[35:43] me at least going over to high level
[35:44] what they look like. To make a long
[35:46] story short, these are just files
[35:47] that'll exist somewhere within our
[35:49] workspace. These files will have sort of
[35:51] this little title section up here, which
[35:53] you know is a title because there'll be
[35:54] three hyphens at the top and three
[35:56] hyphens at the bottom. Inside of the
[35:58] file, you can give it a name like PDF
[35:59] processing, a description like extract
[36:02] text and tables from PDFs. Uh, and then
[36:04] you can even do licenses and metadata
[36:06] and so on and so forth. I don't actually
[36:07] do any of this stuff. my skills are
[36:09] almost always just name, description,
[36:10] and then maybe some optional uh uh tools
[36:13] that I could use as well. Okay, so I
[36:15] just want to give you guys a couple of
[36:16] brief examples. I'm just going to go
[36:18] over to anthropic um skills because they
[36:20] have a bunch of simple ones here that we
[36:22] can use just to gain some context. I'm
[36:24] going to go over to the skills folder
[36:25] here and then click on I don't know,
[36:27] let's do algorithmic art. We'll go
[36:29] skill.md because that's the file. And as
[36:32] you guys could see here, um, we have, if
[36:34] I click on the raw, you guys will see we
[36:36] have the exact same format that I showed
[36:37] you guys earlier. So this is a skill
[36:39] that creates algorithmic art using a
[36:41] particular library. And what's cool is
[36:43] it basically guides the model through
[36:46] the same thing every time to get very,
[36:47] very similar algorithmic art generated.
[36:50] And you can see this is a pretty long
[36:51] skill. There's a lot going on, right? So
[36:53] what I'm going to do is I'm just going
[36:54] to copy this whole thing and show you
[36:55] guys how this works. In this way, we can
[36:57] copy and paste different um standard
[36:59] operating procedures to different models
[37:01] and then get highquality results. So,
[37:03] I'm going to go over here and then, you
[37:05] know, just because this is a oneshot
[37:06] prompt, I'm just going to feed all this
[37:08] in. And I'm going to have this model
[37:10] actually create things according to the
[37:12] skill spec. So, it's doing some
[37:14] thinking. And now it's asking me what do
[37:15] we want to do with it? And I'm going to
[37:16] say yes, save as skill, then run. And
[37:20] then I'm going to actually have this
[37:21] like produce some sort of cool
[37:23] algorithmic art. Now there's no template
[37:25] file or anything like that. So it's
[37:27] actually going to go through the whole
[37:28] process. It's going to create both the
[37:29] skill directory which we can find right
[37:31] over here now called algorithmic art.
[37:33] And then it's also going to create like
[37:34] templates and a bunch of other stuff as
[37:36] well. Okay. And our algorithmic art flow
[37:38] is just finished up. So I'm actually
[37:40] just going to open this so I can take a
[37:41] look at it myself. And we have it. There
[37:44] it is. This is now creating algorithmic
[37:46] art. As you guys could see, we have
[37:47] particles and so on and so forth. I'm
[37:49] just going to significantly decrease the
[37:51] number of particles. Maybe change the
[37:53] noise scale and the turbulence. Actually
[37:55] move this around. And as you guys can
[37:56] see, we we we are actually producing a
[37:58] tremendous number of particles here.
[37:59] This is this is actually like rendering
[38:00] them directly in my browser, which is
[38:02] nuts. Um, so this is indeed algorithmic
[38:05] art. It's it's really cool. Super sexy.
[38:07] I'm a big fan. I don't know. I mean, it
[38:08] looks kind of like hair, but what are
[38:10] you going to do? I'm just going to
[38:11] regenerate a bunch. Maybe change the
[38:13] accent colors. Okay. Maybe we'll have
[38:15] this as my accent now. Blue. And then
[38:17] the background will be kind of this. And
[38:19] I don't know, my cool accent will be
[38:20] kind of like this. There you go. That
[38:22] looks pretty nice. We can now kind of
[38:25] just create new ones as we want. And
[38:28] then we can also just completely
[38:29] randomize them over and over and over
[38:30] and over and over again. And you can see
[38:32] it's actually still doing some design in
[38:34] the background as we go. So I'm just
[38:35] going to change the number of particles
[38:37] to really low. And then I'll just
[38:38] redesign this over and over and over and
[38:40] over again.
[38:42] And I should note that like this is not
[38:43] like a, you know, it's not a piece of
[38:45] software I downloaded. We actually just
[38:46] built this. It's just we built this in a
[38:48] much more standardized and you know
[38:50] consistent way which is really cool. So
[38:52] obviously that's that's what I want. I
[38:54] want the ability to share like
[38:55] repeatable workflows where my agent can
[38:58] build things that other people have
[38:59] validated without me necessarily having
[39:01] just to like copy and paste a piece of
[39:03] software into my computer. Now remember
[39:04] earlier how I said some models are
[39:06] better at things than others and these
[39:08] few percentage point differences can
[39:10] make a lot of impact at the bleeding
[39:12] edge or the frontier. Assuming you guys
[39:14] are at the bleeding edge and the
[39:16] frontier and those percentage point
[39:17] differences stack up, then multi- aent
[39:20] MCP orchestration is the pattern for
[39:23] you. Basically here what happens is you
[39:26] let one model type be the manager or the
[39:29] orchestrator and that orchestrator will
[39:31] take a task and then dole it out, okay,
[39:34] and delegate subchunks of that task to
[39:37] different models. And so what's
[39:38] occurring here is in this hypothetical
[39:40] example, we're using claude code to be
[39:42] our manager. We then give it some task
[39:44] like hey make me [snorts] a SAS app that
[39:49] does X Y and Z. And then what it's doing
[39:52] is it's taking my command and then
[39:54] splitting it into a variety of different
[39:55] functions. There's a front-end task
[39:57] which is delegating to Gemini to build
[40:00] the UI. There's a backend task which is
[40:02] delegating to codeex to build the API.
[40:05] There'll be some testing that we need to
[40:06] occur that we need to do which it'll
[40:08] delegate to codeex to do the testing.
[40:11] Then finally at the end we have claude
[40:12] which we'll collect and then validate
[40:14] the results and then if there are any
[40:16] discrepancies or issues there you know
[40:18] we can loop that back around
[40:19] hypothetically to different models as we
[40:22] will. And so this is a little bit more
[40:24] of an advanced design pattern and I
[40:26] don't necessarily recommend you guys
[40:28] sign up to a bajillion patterns and
[40:29] waste your tokens that way unless you
[40:31] have to. But I wanted to cover it
[40:33] because this is sort of like the next
[40:34] generation of model intelligence. It's
[40:36] where instead of just sticking with one,
[40:38] you're constantly querying different
[40:40] models for things that they're a little
[40:41] bit better at. All of this depends on
[40:43] this idea of a router. And so this
[40:46] router is more or less like a decision
[40:48] hub or like a nexus. When you give it a
[40:51] task where you give it some sort of
[40:53] input, what it'll do is it'll just
[40:54] divide it into different subtasks that
[40:57] different models are better than other
[40:59] models at. So for instance, if we have
[41:01] like a highle task that has to do with
[41:03] replicating a specific SAS app, you
[41:06] know, and the the model has decided that
[41:09] there's some footage on the internet out
[41:11] there that talks about how to build it,
[41:12] it'll actually go delegate the video
[41:14] watching step over to Gemini because
[41:16] Gemini is better at multimodality and
[41:18] their endpoints have built-in video
[41:19] understanding. You know, if it
[41:21] identifies that we need something with a
[41:23] lot of complex reasoning, it'll route
[41:24] that over to Claude. you know, if it
[41:26] identifies that we need some form of
[41:27] sandboxed cloud code execution, it'll do
[41:30] that in codeex because they include that
[41:31] built in. And maybe, you know, I just
[41:33] wanted to show you guys what an example
[41:34] would look like if you had something
[41:35] that was outside of the three. If you
[41:37] need real-time web data, it might do
[41:38] that with Perplexity or or Perplexity's
[41:41] computer or something. And what happens
[41:42] is, you know, we build it all by
[41:45] parallelizing this big sweep and then at
[41:47] the very end, we combine it again with
[41:49] this router, which is probably, you
[41:51] know, at least in my case, almost always
[41:52] going to be Claude Opus 4.6 6 4.7 by the
[41:56] time you guys are reading it. And then
[41:58] that's what ultimately unifies it before
[42:00] maybe doing some additional Q&A bug
[42:02] fixes and agent review which I'll talk
[42:03] about later. Now, all of this sounds
[42:05] pretty abstract and you're like, "Okay,
[42:06] why don't I just have all of this done
[42:08] in one thread?" So, let me show you a
[42:10] practical way to actually do it. By the
[42:11] way, all the files for this course you
[42:12] can find in the top link in the
[42:14] description below. What I'm going to do
[42:15] is go back to Claude Code and open up a
[42:18] new session. And then I'm going to
[42:20] select this folder that I've actually
[42:21] already created for this purpose called
[42:23] multiplatform orchestration. Now, as
[42:26] mentioned, you guys will get everything
[42:27] in the description if you want it. And
[42:29] I'll also run you through how to create
[42:30] it. [gasps] But for now, what I want to
[42:32] do, I just hide this, is say something
[42:35] along the lines of, "Hey, build me a
[42:39] full stack app that lets users enter
[42:45] a desired image to generate and then it
[42:48] generates said image." We'll make this
[42:51] really simple because I don't actually
[42:52] want this to take forever. I'm on kind
[42:54] of a time crunch today and I just want
[42:56] you guys to see how this deals with that
[42:59] problem. Keep in mind in this case
[43:02] Claude which is the model that we're
[43:04] currently talking to cuz it's Claude
[43:06] code is going to be our top level
[43:09] orchestrator. Okay. Now this is going to
[43:12] plan things out for us which is why it's
[43:14] entering this plan mode. Next what we're
[43:17] going to do is we're going to delegate
[43:19] all difficult tasks um like backend
[43:23] tasks to codecs as well as testing
[43:25] tasks. Then down at the very bottom
[43:28] here, you know, uh, for anything related
[43:30] to front end, we're going to delegate
[43:32] that to Gemini. And so we're going to
[43:35] build basically an ecosystem here where
[43:36] Claude is shuttling information back and
[43:38] forth between uh, you know, codecs and
[43:41] Gemini for various things. And as you
[43:43] can see here, it's already starting to
[43:44] ask me, hey, which image generation API
[43:46] would you like to use? I'm actually just
[43:47] going to say um, Nano Banana Pro 2. It's
[43:52] a Google product. Okay, I'm going to
[43:54] submit that. And now what it's going to
[43:56] do is it's going to decide, hey, how am
[43:59] I going to delegate this work? At the
[44:01] end of it, Claude will give me a plan.
[44:02] And you can see here that it's decided
[44:04] on backend, front end, and so on and so
[44:06] forth. And what it'll do now is it'll
[44:08] actually dispatch work to Gemini codeex
[44:10] and then itself to fix various
[44:12] integration issues. So I'm just going to
[44:14] say plan approved. And now it's going to
[44:15] start doing the coding. The way that
[44:17] Claude Code does this is it uses the
[44:19] execute task path for Codeex. And so
[44:23] what's occurring right now is it's just
[44:25] sent this big request into Codex's best
[44:27] model. Okay. And now just clicking the
[44:29] button in the top right hand corner, we
[44:30] now have a preview. And um in this case,
[44:32] Claude is now reviewing the generated
[44:34] application and doing some self-
[44:36] testing. And so we've built this image
[44:38] generator app. We've uh asked for a cute
[44:41] cat wearing sunglasses on a beach. This
[44:43] is now passing through to an API that uh
[44:46] Claude Code set up with Gemini for the
[44:49] front end and then Codeex for the back
[44:50] end's help. It's actually doing the the
[44:52] generation right now and we've generated
[44:54] the cute picture of the cat on the beach
[44:56] looks great to me. The reason why you
[44:58] might want to do this is because well
[44:59] it's kind of twofold. One, you get to
[45:01] parallelize your work as mentioned and
[45:02] so you get to build the front end u
[45:04] using a model for which the front-end
[45:06] builder is the best. You get to build
[45:07] the back end simultaneously using model
[45:09] by which the backend builder is the
[45:11] best. And then you get to use an
[45:12] orchestrator which basically ees out a
[45:14] few percentage points increased like
[45:16] reasoning and decision-m and stuff like
[45:18] that because it's able to evaluate the
[45:21] code from both of these things
[45:22] independently without being polluted by
[45:23] the context window. And we're going to
[45:25] talk more about that specific review
[45:26] pattern later. But um this allows you to
[45:28] e-code, you know, more quality. The
[45:30] downside of this um prompt approach is
[45:32] it usually costs more because now you're
[45:34] splitting your tokens across multiple
[45:35] models instead of just one provider. And
[45:37] usually providers will subsidize your
[45:38] token usage. Like Claude will subsidize
[45:40] most of its usage on the max plan for
[45:42] instance. Um the $200 a month that you
[45:45] spend on it is actually equivalent to
[45:46] like $5,000 a month in usage. Whereas
[45:48] when you build via API, it's usually a
[45:50] little bit more standardized and then as
[45:51] a result of that you end up building way
[45:53] more. You don't you don't get that cool
[45:54] subsidization. However, this is
[45:56] something that people are increasingly
[45:57] using for more complicated
[45:58] infrastructural projects, especially
[46:00] when, as mentioned, a minor percentage
[46:02] point or two difference in terms of
[46:04] quality is very important to you. And so
[46:06] this is me just doing this in Cloud, but
[46:07] you can obviously use, I don't know,
[46:08] Codeex as the orchestrator if you wanted
[46:10] to build this in Codeex. You could use
[46:12] Gemini as the orchestrator if you wanted
[46:13] to do this in uh, you know, entirely
[46:15] Gemini. Right now, this is the stack
[46:16] that seems to make the most sense, what
[46:18] people are talking about the most. If
[46:19] you guys are interested, the way that
[46:21] all of this stuff works under the hood
[46:22] is we basically set up a bunch of
[46:24] different servers that calls and Gemini
[46:28] inside of Claude. And so that's why we
[46:30] see this using the Claude formatting
[46:31] above. It's because the claude is the
[46:33] the orchestrator that's sort of setting
[46:34] it up initially. And there's also a
[46:36] claude.mmd which describes how it's the
[46:38] manager. You know, you plan, reason,
[46:40] delegate, validate, and fix integration
[46:42] issues. When you break tasks down, break
[46:44] them into front end, backend, and test
[46:45] subtasks and then delegate things as
[46:47] required. I'm going to include this
[46:49] prompt as well as everything else you
[46:50] need in order to do the same thing down
[46:52] below in the description. But in order
[46:53] for this to work, you will of course
[46:54] need API keys for various platforms. And
[46:57] in order to get those, you do have to
[46:58] sign up to typically something a little
[46:59] bit different from what we signed up to
[47:00] before. And in order to sign up to
[47:02] those, you do typically need um to go
[47:03] directly to the platform, create an
[47:05] account, and then set up an API key. So
[47:07] you can see over here, that's what I've
[47:08] done for Claude. And you can also do the
[47:11] same thing for OpenAI and then Gemini.
[47:12] Once you have those keys, you would just
[47:14] give it to whatever model you want to
[47:15] use to be the orchestrator. And then it
[47:16] would set this whole thing up for you
[47:17] and then be able to reason and then
[47:19] communicate with different models on
[47:20] your behalf. The next advanced prompting
[47:22] technique is the video toaction
[47:24] pipeline. To make a long story short, up
[47:27] until quite recently, AI agents were
[47:29] forced to learn entirely through text
[47:31] descriptions of stuff. And the reason
[47:33] why is because multimodality like vision
[47:35] usually uh at least in the context of
[47:37] video was sort of out of bounds. There
[47:40] was just no way that we could feasibly
[47:42] take videos which were millions upon
[47:44] millions of tokens when stitched
[47:45] together um you know into some text
[47:48] format that an agent would understand.
[47:49] Well, now agents can learn from the same
[47:51] medium humans learn from. And we do so
[47:54] by combining a little bit about what I
[47:55] showed you guys earlier, okay?
[47:56] Multi-agent MCP orchestration with this
[47:59] idea of passing requests through the
[48:02] Gemini API because Gemini has built-in
[48:05] support for video now. Basically, uh you
[48:08] know how videos are a certain number of
[48:09] frames per second. Like this video for
[48:11] instance is 30 frames a second. You
[48:13] could tell if you find a way to to slow
[48:15] it down to like 0.03.
[48:17] I'll go literally one frame every 003
[48:20] seconds or something like that. Well,
[48:21] what this model does is it divides
[48:23] videos into one frame per second
[48:25] instead. It then analyzes the images in
[48:29] succession and then uses a form of
[48:31] descriptive prompting to break that down
[48:33] into very very clear steps. So basically
[48:35] what occurs is you'll feed in something
[48:36] like a YouTube tutorial URL. Claude will
[48:39] receive the URL but cannot watch the
[48:40] video natively. So instead it'll call
[48:42] the Gemini API. Gemini will watch the
[48:45] full video. Gemini will then extract the
[48:47] step-by-step instructions, format it as
[48:49] like a numbered list that's hyper
[48:51] precise and hypersp specific. The
[48:53] structured steps will return to Claude
[48:55] via very similar flow to what I showed
[48:57] you guys with the design. And then
[48:59] Claude will execute each using
[49:00] hyperspecific tools. Maybe if you're
[49:02] teaching somebody how to build something
[49:04] on Blender or Figma or something like
[49:05] that, you just give it access to the
[49:06] toolkit and it does it. Then the final
[49:08] result is the agent will have replicated
[49:10] the tutorial end to end and in that way
[49:12] they can learn from the exact same
[49:13] medium that that we learn. So I'll show
[49:16] you number one where I got inspiration
[49:17] from this and then number two how to do
[49:19] this for an actual task which in my case
[49:21] is going to be building a simple flow
[49:22] out in a noode tool called NAN. So first
[49:25] the inspiration was Spencer Sterling's
[49:27] post on X. He said he built an agentic
[49:30] system that taught itself the Blender
[49:31] donut tutorial by watching it on
[49:33] YouTube. It watched the tutorials,
[49:35] extracted the steps, filled in the gaps
[49:37] in its own tooling and completed the
[49:38] entire thing autonomously. And it's
[49:40] quite impressive to be honest. Um,
[49:42] anybody that's done any sort of 3D
[49:43] design, myself included, will know that
[49:45] like the uh way you learn how to build
[49:47] things in Blender is you watch this one
[49:49] specific tutorial that shows you how to
[49:50] build a donut. And through this process
[49:52] of building the donut, you learn about
[49:54] like textures. You learn about various
[49:56] shapes. You learn about how to modify
[49:58] them and sculpt and paint and do all
[50:00] this stuff. So, I made my own donut
[50:02] personally a few years ago. I showed it
[50:04] to all my friends. Then, I promptly
[50:05] never touch Blender again. Well, the
[50:07] issue with knowledge like this is it's
[50:09] obviously extraordinarily visual, right?
[50:11] In order to really learn something, you
[50:12] have to watch a video. You can't really
[50:14] break all that down into like
[50:15] hyperspecific text instructions unless,
[50:17] you know, somebody were to just like
[50:19] literally go step by step. Step one,
[50:22] click this button, step two, rotate 283°
[50:25] to the left, step three, do this. So,
[50:27] there's a fair amount of nuance and
[50:28] flexibility there. That's where video
[50:30] learning comes in handy. Human beings
[50:31] learn through video, obviously, but
[50:33] models have a tough time doing it. And
[50:35] so what we do is we convert all of this
[50:37] into a sequence of steps. We leave some
[50:39] steps a little bit more vague, a little
[50:40] bit more general, let the model have its
[50:42] own um kind of interpretability, and
[50:44] then give it some way to like screenshot
[50:45] its results to match it up to um you
[50:48] know like the frames in the video. And
[50:49] so this fell here built this cool like
[50:52] workflow building studio. It's sort of
[50:54] like his own main um operating system. I
[50:56] suppose that's what this is. It's not
[50:57] like an app that he downloaded. It's
[50:58] something that he built. And then he fed
[51:01] in this along with the workflow I'm
[51:02] about to show you to have it actually
[51:03] like build the freaking thing. And it's
[51:05] communicating with this app Blender
[51:07] using what's called MCP, model context
[51:09] protocol, which is the same thing that
[51:10] we used to communicate with um the
[51:12] various models like Gemini and then
[51:14] Codex earlier. And you can get all that
[51:15] stuff in the description down below as
[51:17] well. So I have this stored as a clawed
[51:19] skill in video to action um over here.
[51:23] So, if I open this up and read the
[51:24] skill, you can see here that it actually
[51:26] says, "Extract actionable steps from
[51:28] YouTube videos using Gemini video
[51:29] understanding. Use when the user
[51:31] provides a YouTube link it wants to
[51:32] learn procedures, extract steps,
[51:33] understand visual tutorials, or turn
[51:35] video content into executable
[51:36] instructions." And so, what's occurring
[51:38] is it'll basically take a video, it'll
[51:40] download it for me, so then I'll just be
[51:42] able to feed in a YouTube URL, and then
[51:44] it'll convert that into like a highly
[51:45] optimized series of steps that um you
[51:47] know, you would only really know or be
[51:49] able to use through uh the context of
[51:51] like an actual video. And so to
[51:52] demonstrate what I've done here is
[51:54] instead of using Gemini within
[51:56] anti-gravity, which is sort of the usual
[51:58] design pattern, I thought I'd show you
[51:59] guys my actual stack like what I
[52:00] personally use. I think it's much easier
[52:02] if you just use the models inside of the
[52:04] tools inside of the companies that made
[52:06] them. But in my case, I'm a very big fan
[52:08] of this anti-gravity uh uh kind of
[52:10] container. Then inside of it, I use
[52:12] clawed code. And so in that way, I'm
[52:13] actually using a Google wrapper around a
[52:15] claude code or anthropic extension
[52:17] that's communicating with a claude or an
[52:19] anthropic model. If you guys want to
[52:21] replicate the setup, it's as simple as
[52:22] just opening up anti-gravity, heading to
[52:24] the lefth hand side where it says
[52:26] extensions, downloading the clawed code
[52:28] for VS Code plugin. I know it says VS
[52:30] Code, don't be confused. It's very
[52:32] similar to anti-gravity. Installing it,
[52:34] and then uh you also have to log in
[52:36] here. After you're done, you will have
[52:38] the exact same functionality that you
[52:39] have in the claw desktop app that I just
[52:41] showed you guys earlier when we built
[52:42] out that little full stack app. It's
[52:44] just you'll have it within anti-gravity,
[52:45] which also allows you to do things like,
[52:47] you know, organize your files and stuff
[52:48] on the lefth hand side. So, that's my
[52:50] personal stack. You don't have to use
[52:51] it. Some people judge me for it.
[52:53] Whatever. I like it. It works for me.
[52:55] Okay. So, um what I'm going to do is I'm
[52:57] going to find a YouTube video that I
[52:58] like and then I'm just going to feed it
[52:59] in these instructions. So, I'll say I
[53:01] want you to use the video to action
[53:03] pipeline on and then I'm going to go
[53:05] grab an image. And what I've done is
[53:07] I've found a flow that I built forever
[53:08] ago. It's a short video about 21 minutes
[53:10] that shows you how to scrape leads
[53:11] without paying for a few APIs. I'm going
[53:13] to bring that back into my anti-gravity
[53:15] instance. Then, I'm going to do this.
[53:18] And what this is going to do is it'll
[53:19] start by invoking the skill. And this is
[53:21] the UX for skill uh invocation. I think
[53:24] that's what it's called in English. Holy
[53:25] crap, that better be what it's called in
[53:27] English. And then um it's now going to
[53:29] send that over to Gemini, then receive
[53:31] back a list of highly specific
[53:33] instructions that, you know, understand
[53:35] UX, uh I don't know, highlight the
[53:37] colors of buttons and stuff like that
[53:38] and so on and so forth before actually
[53:40] running it locally on my computer. At
[53:42] the end of it, you'll get a super
[53:44] in-depth analysis that looks like this.
[53:46] So you can actually see down over here
[53:47] it says here's the hyperdetailed
[53:49] breakdown with literally every single
[53:52] step. I mean like hey navigate over to
[53:54] this thing at 17 seconds. Here's how to
[53:56] do this thing on that and and so on and
[53:58] so on. So like it'll it'll literally go
[54:01] visually as well and actually tell us
[54:02] what the end flow is going to look like,
[54:03] but then we'll also have just a
[54:05] tremendous amount of context about
[54:06] everything. Um so what we're going to do
[54:08] now is we're going to feed that in and
[54:09] actually have this control my browser.
[54:11] So I'm going to open up a new cloud code
[54:12] instance by clicking that little button
[54:14] above. We'll go bypass permissions. Then
[54:16] I'll say use Gmaps scraper deep analysis
[54:20] MD to build out the same N8N flow for
[54:24] me. It's now going to open up a Chrome
[54:26] DevTools MCP server. It's then going to
[54:29] link that up to the N8N account. And now
[54:31] it's actually thinking through
[54:32] everything that it's going to do using
[54:33] this file as a reference. And now it'll
[54:35] go through and actually control my
[54:36] browser to do the build. For simplicity,
[54:38] I'm just going to move this over to the
[54:40] right. Okay. And as we see, it just laid
[54:42] out the entire thing from left to right.
[54:45] So it went through, it then identified
[54:48] what all of the steps were. It then
[54:49] created it inside of its own little
[54:52] conversation thread and then it um
[54:54] essentially generated what's called
[54:55] workflow JSON and then pasted it in. Now
[54:57] this can obviously interact with my my
[54:59] browser as well. That's what it just
[55:00] did. So it just went to the top and then
[55:01] basically imported this. What it's going
[55:03] to do now is just make some finer final
[55:05] minor changes. I'm going to configure
[55:07] the Google Sheets node and then we'll be
[55:08] on our way. So, what I'll do is I'll
[55:10] just take a screenshot of this and then
[55:11] paste it in. Then I'll say you're
[55:13] connected. Now, it's just going through
[55:14] and then it's selecting various
[55:16] elements. So, in this case, it's
[55:17] selecting that little search button.
[55:18] It's uh mapping the the fields and stuff
[55:21] like that. And then it'll just continue
[55:22] testing this non-stop until I have a
[55:24] working flow. You can see, you know,
[55:25] just kind of I mean, I should be moving
[55:27] this around cuz it's going to get
[55:28] confused, but you can see that it's um
[55:30] actually gone through and then pumped in
[55:33] like a specific search term. It's it's
[55:34] gone through and basically done
[55:35] everything for me. Really, the only
[55:36] thing left is to do some sort of
[55:38] testing. You can see that uh if we
[55:39] actually click execute workflow, I'm
[55:41] just going to stop it here so I don't
[55:42] consume anything else. It's actually
[55:43] gone through and literally like scraped
[55:45] Google Maps for us, which is sweet. Uh
[55:47] and it's just done so entirely by
[55:48] watching the video. So it's entirely
[55:49] like native video understanding and then
[55:52] it's extraordinarily detailed because
[55:54] we're we're dumping it all into a file
[55:55] and then it can just constantly
[55:56] reference that file. Um it's then doing
[55:59] kind of a combination of like I don't
[56:00] know like ASKI or or text based markup
[56:03] to uh you know understand both the
[56:05] structure at like a micro level and then
[56:06] also like a macro level. Next, I want to
[56:08] chat this idea of stochcastic
[56:10] multi-agent consensus. In case you guys
[56:13] didn't know, if you were to take one
[56:15] model, let's say Gemini 3.1 Pro High,
[56:18] and if you were to ask it like an idea
[56:20] question, hey, give me 10 ideas to do X,
[56:24] Y, and Z. Every time you ask Gemini 3.1
[56:28] Pro the same thing, it'll return a
[56:30] slightly different answer. Now this
[56:33] property some call it randomness but I
[56:35] think the correct technical term is
[56:36] stochasticity which is just where due to
[56:39] minor statistical variations in the
[56:42] input or in the way that the models work
[56:44] the output is going to be slightly
[56:46] different every time. The reason why
[56:48] this is so valuable is because you can
[56:50] exploit this tendency to get much much
[56:52] better answers. For instance, let's say
[56:55] I run three times 1 2 and three. The
[57:00] reality is if I run a query that at the
[57:03] very beginning says give me three ideas
[57:05] for X. Okay,
[57:08] on the very first time, okay, we might
[57:11] get idea A, idea B, and idea C. If we
[57:16] were to hypothetically run this again,
[57:18] we'd probably get idea A, idea B. But
[57:22] just due to statistical variation, there
[57:24] is a chance that on the second run, it
[57:26] won't deliver us idea C at all. it'll
[57:28] actually deliver us idea D. And on the
[57:31] third run, maybe we do B, maybe we do C,
[57:34] and then maybe we also do E. What
[57:36] stochastic multi-agent consensus is, you
[57:39] basically automate the process of
[57:41] spawning multiple agents, giving them
[57:43] slightly varied input prompts to take
[57:45] advantage of stochasticity, and then
[57:46] instead of just getting, let's say,
[57:48] three ideas A, B, and C, you get to
[57:51] exploit stats to get all of the
[57:53] possibilities, including ones that might
[57:56] be a little rarer. the model is less
[57:57] likely to actually answer with. And so
[58:00] in this way you get A, you get B, you
[58:02] can get C, but you can also get D and
[58:04] then you can get E. And so you know if
[58:06] you compare it to just one naive search,
[58:07] what we've done is we basically almost
[58:09] doubled the scope of the ideation. Now
[58:12] mathematically this is termed traversing
[58:14] the search space. I want you to pretend
[58:16] hypothetically that this like little pie
[58:19] chart here represents all possible
[58:21] answers to a question. Maybe the
[58:24] question is, I don't know, what's the
[58:26] simplest way to get to 1 million
[58:28] subscribers? Right? This is something
[58:29] that I asked uh my my model a little
[58:31] while ago because I'm interested in
[58:32] getting to 1 million subscribers. Now,
[58:34] obviously, I'm not just doing what the
[58:35] thing tells me, right? A lot of its
[58:36] ideas are stupid. But if you think about
[58:38] it, if I can parallelize a thousand
[58:40] agents all coming up with their own
[58:41] ideas, even if on net the average reply
[58:44] or idea is a little bit worse than
[58:45] something I'd be able to do, I still get
[58:47] to run it a thousand times, right? It's
[58:49] like running like uh I don't know like a
[58:50] 90q, you know, it's like it's like
[58:52] Einstein versus 10,000 95 IQ
[58:57] researchers. It's like well the 10,95 IQ
[58:59] researchers despite lacking the
[59:00] brilliance of Einstein, they'll probably
[59:02] statistically figure it out eventually,
[59:03] right? So um if this whole pie chart to
[59:06] get back to things is all possible
[59:08] responses, if you just run one search,
[59:10] basically what you're doing is you're
[59:12] only actually getting like a small chunk
[59:15] of all of the possibilities. And so
[59:17] instead what we're doing is we're
[59:18] actually running multiple searches. You
[59:19] know, one search is going to get this,
[59:21] another search is going to get that,
[59:22] another search is going to get that,
[59:23] another search is going to get that. And
[59:25] and and so on and so forth. And then in
[59:28] this way, what we do next is we take the
[59:30] answers and then the replies of the
[59:32] model. That should be red and this one
[59:33] should be blue. And then in doing so, we
[59:36] get to traverse significantly more of
[59:37] that search space without actually
[59:39] necessarily consuming any more of our
[59:41] time. So this can be kind of difficult
[59:43] to understand. And I think I've run out
[59:44] of colors here. uh unless you've done
[59:46] something like this before. But I'll
[59:48] make it really simple by actually giving
[59:49] you guys a brief demonstration on I
[59:51] don't know some use case or problem that
[59:54] uh I think we'd probably all be able to
[59:55] relate to. Another final benefit is you
[59:57] get to do all this in parallel. So like
[59:59] you know if you think about it if you
[60:01] were to do one search and then do
[60:03] another search afterwards and then do
[60:04] another search. So for instance let's
[60:06] say you have a query give me three ideas
[60:07] for X and then it gives you three ideas
[60:10] and you're like hey I want another three
[60:11] ideas and it gives you another three
[60:12] ideas and you're like I want another
[60:13] three ideas. Well, at the end of it, you
[60:14] may have, I don't know, nine ideas or
[60:16] something, but it will have taken a
[60:17] certain amount of time. If the first
[60:18] search is 5 minutes, the second search
[60:20] is 5 minutes, and the third search is 5
[60:21] minutes. Well, you just consumed 15
[60:23] minutes, right? So, instead, what this
[60:25] does is this just copies the idea. Okay?
[60:28] But then it parallelizes it. So, hey,
[60:30] give me three ideas for X. And then what
[60:32] we do is we do 1 2 and three. And in
[60:35] total, this takes 5 minutes. And then we
[60:37] just combine those three answers back
[60:38] over here. The formal way to do
[60:40] stochastic multi-agent consensus, at
[60:41] least the way that I'm doing it here, is
[60:43] we'll provide a single question or
[60:44] prompt. Then we'll do slight framing
[60:46] variations of every prompt that we're
[60:47] feeding into the model. And then we'll
[60:49] feed in I don't know, I'll probably feed
[60:50] in like three or four, five or maybe 10
[60:52] simultaneously. Depends on how deep you
[60:54] want it to go. And then um what'll
[60:56] happen is these will be instantiated as
[60:57] what are called sub aents, okay? Which
[60:59] are similar to the main agent, but they
[61:01] operate in their own defined context
[61:02] window. And then all of these will just
[61:04] report back their answers to the parent
[61:06] agent. So this parent over here is
[61:08] basically going to work with a whole
[61:10] fleet of sub aents and then once they're
[61:13] all done their work, it'll synthesize
[61:14] the answers. And then because what we're
[61:16] looking for is we're looking for like
[61:17] statistical variation, it'll calculate
[61:19] um what's called the mode, which is the
[61:20] frequency of each answer, and then the
[61:22] median, which is like the average of
[61:23] each answer before ultimately combining
[61:25] all this to give you much better
[61:27] results. One final idea there is this
[61:28] idea of consensus. A lot of models are
[61:30] going to say the same things. Obviously,
[61:32] some models are going to say things that
[61:34] are quite different. And then finally,
[61:36] there will be outliers, which are wild
[61:37] cards. These wild cards here are
[61:39] potentially brilliant, but they might
[61:40] only appear like 5 or 10% of the time,
[61:42] which is why we spawn so many of these
[61:44] agents that we can actually like farm
[61:45] these wild cards. We can we can milk
[61:47] them like cows. And then in that way,
[61:49] you can have the best ideas coming from
[61:51] these these fleets of agents. Um, and
[61:53] then also save a lot of time in things
[61:54] like product ideation. I don't know,
[61:56] man. Keyword search, titles for for for
[61:58] content. At least that's what I'm using
[62:00] it for. Or a variety of other things.
[62:02] Hell, research inventions. I'm sure
[62:04] Anthropic and and Google and OpenAI
[62:06] probably have fleets of models that are
[62:07] doing basically this exact same thing
[62:09] behind the scenes constantly. Let me
[62:10] actually show you guys what this looks
[62:12] like in practice. I'm just going to zoom
[62:13] way out of this and close a bunch of
[62:15] these so you don't have to look at them
[62:16] anymore. I'm going to spawn a new Claude
[62:18] code tab over here on the right. And
[62:20] what I'm going to do is I'm going to use
[62:21] the skill that I've set up called
[62:22] stochastic multi-agent consensus. So
[62:24] opening this up so you guys could read
[62:26] it. What we're doing is responding n
[62:28] agents where n is just the number that
[62:30] you specify with slight framing
[62:33] variations to independently analyze a
[62:35] problem then aggregate results by
[62:37] consensus. We use this for decision-m
[62:40] ranking things strategic analysis or any
[62:43] problems where you want to filter
[62:44] hallucinations and surface high variance
[62:46] ideas. So hypothetically let's just say
[62:50] hey I've struggled a lot with finding
[62:52] any traction on Tik Tok whatsoever. I've
[62:54] built up a bunch of accounts and I can't
[62:56] seem to get more than like a thousand
[62:58] views per Tik Tok account. I'd like you
[63:00] to use stochastic multi-agent consensus
[63:02] to help me come up with possible
[63:04] candidate ideas to solve this. I'm going
[63:06] to feed this idea in. Okay. And this is
[63:08] a real idea. Actually, we are struggling
[63:10] to get uh traction on Tik Tok for
[63:13] whatever reason. We got 450K followers
[63:15] on Instagram. No problem. But, you know,
[63:17] the second we move things over to Tik
[63:19] Tok, we're just not really getting too
[63:20] many views. So what it's going to start
[63:22] with is it will spawn 10 agents all
[63:24] independently analyzing my Tik Tok
[63:26] problem and every one of them will get
[63:28] slightly different analytical framing to
[63:30] maximize the diversity of ideas. Just
[63:33] going to zoom in here so you guys could
[63:34] see this. But we now have a conservative
[63:37] analysis. So Nick Sarif has 287K YouTube
[63:40] subscribers. You know um his YouTube
[63:43] audience is primarily professionals.
[63:45] Here's a bunch of information about him.
[63:46] He has a small team. here's how he's
[63:48] doing things and and so on and so forth.
[63:50] This agent over here says, "Hey, I want
[63:52] you to assume limited time and budget."
[63:54] This agent over here, I want you to only
[63:56] focus on what is measurable and
[63:58] provable. This agent over here, you
[64:00] know, I want you to think about it from
[64:01] the end user and viewer perspective. And
[64:04] so what we're doing is we're basically
[64:05] taking advantage of the
[64:06] parallelizability of models, not
[64:08] necessarily the base intelligence. So
[64:10] the intelligence is obviously important,
[64:11] but like we care more about like
[64:12] scanning and searching through a space
[64:14] of all possible solutions really
[64:15] quickly. And then at the end, we're
[64:17] going to converge all this back with our
[64:18] parent agent. Now, once all these agents
[64:20] have turned green here, if I open up
[64:22] this thinking tab, you could see that
[64:24] it's now combining all of the
[64:26] information from each individual one.
[64:28] So, there's a bunch of uh suggestions
[64:30] saying, "Hey, you should try fresh
[64:31] account. You should try device reset.
[64:32] You should try clean fingerprinting.
[64:34] Hey, you should try Tik Tok native hook
[64:36] reformatting. Hey, you should do duets
[64:38] with existing creators to take advantage
[64:39] of the fact that you're probably bigger.
[64:41] Hey, you should do a series format, high
[64:42] posting frequency, and so on and so
[64:44] forth." And then you have some
[64:45] disagreements here as well. And this
[64:47] disagreements might be paid Tik Tok
[64:49] spark ads. Only one out of 10 agents
[64:50] suggested something. You know, in this
[64:52] one, um, they recommend using shorts.
[64:54] But then in this one, they recommend
[64:55] using a micro topic focus to build
[64:57] authority and audience clarity. You
[64:59] know, I'm not going to sit here and
[65:00] pretend like all these ideas are the
[65:01] bee's knees. Not all of them are
[65:03] capturing lightning in a bottle, but you
[65:05] run this thing long enough and you'll
[65:07] see eventually you will get some pretty
[65:08] good ideas. And the ideas will be
[65:10] consensus ideas like the idea of a fresh
[65:12] account, but it'll also be kind of like
[65:14] outlier ideas with painoint framing,
[65:16] paid Tik Tok spark ads, niching down
[65:18] your account identity, crossosting your
[65:19] Instagram reels to YouTube shorts first.
[65:21] I mean, there there there are a lot of
[65:22] possible ideas right [gasps] now. It's
[65:25] opened up this consensus report, which I
[65:27] can visualize for you guys by clicking
[65:29] this button. And you can see here it's
[65:30] now saying, "Hey, here is the context.
[65:32] Tik Tok growth stalled at 1K views per
[65:34] account across multiple accounts despite
[65:36] this massive YouTube subs and 450,000
[65:39] followers with almost 5 million reals
[65:40] views a month. And then here um this
[65:43] orchestrator now summarizes it and says
[65:45] hey every agent independently identified
[65:47] Tik Tok native hook reformatting is
[65:49] really critical. You know Instagram is a
[65:51] little bit different from Tik Tok hooks.
[65:53] Content optimized for Instagram will
[65:55] systematically fail Tik Tok's cold start
[65:57] test. So you actually have to
[65:58] restructure it if you really want to
[65:59] crush. Same thing here. fresh account,
[66:01] clean device, fingerprint. I mean, there
[66:03] is just so much context here, it's not
[66:05] even funny. And so, the reality is I
[66:07] would have come up with these ideas at
[66:09] some point, but I basically got to put,
[66:11] you know, a genie in a bottle and then
[66:14] have 500 genies simultaneously solve my
[66:17] wishes at 100x speed and then aggregate
[66:20] all results for um you know, I don't
[66:23] know, probably like three or$4 dollars
[66:24] realistically in terms of tokens. You
[66:27] also had a couple agents that said, "Is
[66:29] Tik Tok even worth it?" And uh I think
[66:31] that's a really good question to ask
[66:32] because up until now, I really didn't
[66:34] think it was worth it. And so, in
[66:35] general, anytime that I recommend you
[66:37] have a strategic decision that you need,
[66:40] you can make a quick one-time tradeoff
[66:42] of money for analysis by spawning a
[66:45] bunch of agents, all with slight prompt
[66:47] variations, and then collecting the
[66:49] rankings reasoning to build this
[66:51] consensus map document. And from here
[66:54] you can figure out your consensus items,
[66:56] your divergent items, and then your
[66:58] outliers. And you know, if they're
[66:59] consensus items, well, odds are probably
[67:02] because a lot of models have thought
[67:03] it's a good idea. You should probably do
[67:04] it. If there's some divergent items,
[67:06] well, you should probably like reason
[67:07] about these quite a bit before deciding
[67:09] on whether it makes sense. And if it's
[67:11] like an outlier item, if there's only
[67:13] one out of 10 agents doing it, well, it
[67:14] can either be a brilliant idea, in which
[67:16] case maybe you should give it a try, or
[67:18] it might just be a hallucination or some
[67:20] BS, in which case you don't. And so what
[67:22] this allows you to do is execute with
[67:23] high confidence. Thank you very much AI
[67:25] for drawing that cute little that is a
[67:28] huge fist. That thing would be
[67:29] terrifying in real life. Um you know
[67:31] this lets you scan a large portion of
[67:33] the search space in a very short period
[67:34] of time. And uh yeah the actual way that
[67:37] you build it is very straightforward and
[67:38] I'll run you guys through what all that
[67:39] stuff looks like um down below in the
[67:41] project description. So just like
[67:42] stochastic multi- aent consensus allowed
[67:45] us to scan large amounts of search space
[67:47] in a short period of time. What we did
[67:49] is we independently delegated work over
[67:51] to agents and had them uh do things for
[67:53] us. So too can we take advantage of this
[67:56] same idea but in my opinion get even
[67:58] higher quality results through this idea
[68:00] of agent chat rooms. What agent chat
[68:03] rooms are are where instead of you know
[68:06] parallelizing all the work and having
[68:08] all these agents try and independently
[68:10] solve problems what you do is you give
[68:12] all of them slightly different
[68:13] personalities and then you have them all
[68:15] debate with each other about these
[68:17] problems. And in doing so, they tend to
[68:19] deliver much higher quality responses
[68:21] because they're just like they're
[68:22] they're a little bit spikier. You know
[68:24] what I mean? They're not just like a
[68:25] generalized idea, which I'll visualize
[68:27] with like this interface, but you know,
[68:28] because they're they're butdding heads
[68:30] with another um eventually the ideas get
[68:33] really nuanced and really high quality.
[68:35] And so, um whether or not you visualize
[68:37] things in that way, that's personally
[68:39] how I think about things. You really get
[68:40] to carve out all the tiny little nooks
[68:42] and crannies of an idea when you debate.
[68:44] And so, here's a brief little
[68:46] visualization. We start with a problem
[68:47] or a prompt. We feed it in to let's say
[68:50] three agents here. Agent A, agent B, and
[68:52] agent C. All three are given the same
[68:55] document called chat.json. And then what
[68:58] occurs is they basically cycle through a
[69:00] debate sequence where agent A says
[69:02] something, agent B says something, and
[69:03] agent C says something. And, you know,
[69:05] if you do this naively, the results will
[69:07] probably be pretty low. But if you, I
[69:09] don't know, force a little bit of a
[69:10] spark where every agent has a slightly
[69:12] different opinion and they're not afraid
[69:13] to like state their opinion, um, they'll
[69:16] challenge each other's assumptions, they
[69:17] will significantly improve the
[69:19] probability that you catch errors. And
[69:21] then this chat.json ends up being quite
[69:22] a valuable resource because it also
[69:24] shows like problem solving and stuff
[69:25] like that. You can then give that to an
[69:27] orchestrator and ultimately receive
[69:29] higher quality output at the end. And so
[69:30] it's sort of similar to what we had
[69:32] earlier, right? It's just instead of
[69:33] this operating um in parallel lanes,
[69:36] what these agents are doing is they're
[69:38] actually talking back and forth with
[69:39] each other. And so they're actually
[69:40] capable of having these conversations.
[69:42] [sighs and gasps] And I mean like I I
[69:44] just want you to pretend we actually
[69:45] spawn 10 agents. Agent one would be able
[69:47] to communicate with agent two, but also
[69:49] agent three and also agent four and also
[69:51] agent five and also agent six. So like
[69:53] the total number of paths and um
[69:56] potential like communication I don't
[69:58] really know what you want to call them
[70:00] like like vectors um goes up like crazy
[70:03] and these agents ultimately assuming
[70:05] that the idea is an absolute BS do end
[70:08] up at the end of it like quite quite
[70:10] differentiated u in their ideas and
[70:12] their opinions. So to show you guys what
[70:14] this looks like I have another skill
[70:16] which is just a repeatable workflow to
[70:17] be clear where I have this model chat.
[70:20] The description here is to spawn five
[70:22] claw instances on a shared conversation
[70:24] room where they debate, disagree, and
[70:26] converge on solutions. They use
[70:27] roundroin turns with parallel execution
[70:30] within each round for simplicity. Then
[70:32] they trigger on the model chat,
[70:33] multimodel debate, or something else. So
[70:35] I have a bunch of contexts down over
[70:36] here and you guys can grab this file for
[70:38] yourselves. What I'll do is I'll
[70:39] actually just pipe this into model chat.
[70:41] Okay, great. Use model chat for a
[70:44] similar to really work through this
[70:47] idea. And now it'll spark this model
[70:50] chat skill which will then have them all
[70:52] dump shared context into a little
[70:55] chat.json which I'll show you guys when
[70:56] it's done. Okay, so the debate has now
[70:58] concluded after these five agents had
[71:00] this conversation. Okay, we can actually
[71:02] see the the the chat conversation as
[71:05] well by going down here to this model
[71:06] chat. Let's go latest and we'll go
[71:08] conversation. Um basically what's
[71:10] occurred is we've given it a topic to
[71:13] talk about and then we've assigned a
[71:15] systems thinker, a pragmatist, an edge
[71:16] case finder, a user advocate and then a
[71:18] contrarian to the task. So first of all
[71:20] the systems thinker begins, the
[71:22] pragmatist replies, the edge case finder
[71:24] goes, the user advocate goes and so on
[71:26] and so forth. And you can see each of
[71:27] them are um pretty pretty interestingly
[71:29] suggesting uh various approaches. So the
[71:32] user advocate says, "Let me push back on
[71:33] something that challenges the consensus
[71:35] has glossed over, which is the clean
[71:36] device plus fresh account fixes
[71:38] fingerprinting is the problem. There's a
[71:40] simpler explanation nobody has stress
[71:41] tested. Nick's content format is
[71:43] fundamentally mismatched to Tik Tok's
[71:44] cold start algo." And so these are sort
[71:46] of arriving at similar conclusions
[71:48] despite the fact that uh you know we
[71:50] instantiated this separately. And then
[71:52] if we check out the synthesis, we can
[71:53] see that all of them have agreed that we
[71:55] need to run some diagnostics that hook
[71:57] reformatting is necessary but
[71:58] sufficient. the high volume posting
[72:00] blitz two to five a day is wrong. And
[72:01] then fixing the IG YouTube pipeline
[72:03] immediately is important regardless of
[72:05] the Tik Tok decision. This is something
[72:07] that I guess I got contact from one of
[72:08] my other files because um basically
[72:10] despite the fact that I have 450K
[72:12] Instagram followers, a very few of them
[72:13] are converting to YouTube subscribers
[72:15] and a lot of people a lot of models as
[72:17] well are suggesting that the reason for
[72:18] that is cuz Instagram is really blocking
[72:20] outbound links which I think is actually
[72:22] fair. But then uh there are a lot of you
[72:24] know disagreements as well. So a lot of
[72:26] people say, "Nope, Stitch Duet's stupid.
[72:28] Tik Tok versus IG pipeline is an
[72:29] eitheror. Device fingerprinting might
[72:31] not be the issue. Maybe it's content
[72:32] mismatch." Right? And uh there are a lot
[72:35] of insights that because we were able to
[72:37] sharpen our opinions via debate, these
[72:41] agents got that the previous model runs
[72:44] through stochastic multi-agent consensus
[72:46] did not. So maybe we're looking for
[72:48] saves, not completions. Maybe there's
[72:50] just no category online yet. Although
[72:52] this is not true, if they had the
[72:54] ability to research, they probably would
[72:55] have figured this out. Maybe it has to
[72:57] do with emotional moments. And then here
[73:00] it even gave a recommended execution
[73:02] plan. So, as mentioned, you know, I
[73:04] wouldn't rely on agents for strategic
[73:06] advice at the moment, but I would
[73:08] certainly not be opposed to trading a
[73:11] little bit of my money for a bunch of my
[73:13] time back and at least ideulating
[73:14] through the lowerhanging fruit. If you
[73:17] run enough of these cycles, you will
[73:18] find pretty intriguing and interesting
[73:20] outlier ideas. That's just how
[73:22] statistics works. So you guys can get
[73:24] all this down below in that document.
[73:26] The next idea I want to talk about is
[73:28] this idea of sub agent verification
[73:30] loops. To make a long story short, where
[73:33] previously we took advantage of
[73:34] parallelization, we're going to take a
[73:36] step back now to sort of serial um
[73:39] processing. But when an agent works
[73:42] really hard to accomplish a task for
[73:45] you, it usually gets pretty biased in
[73:48] that it believes that its path was the
[73:51] best. And the reason why is because, you
[73:53] know, it just spent god knows how much
[73:55] time, energy, and compute cycles
[73:57] building your app or putting together
[73:59] your workflow or doing your taxes or
[74:02] whatever the hell. And because of that,
[74:04] you know, series of like design
[74:06] decisions and then issues and bug fixes,
[74:09] it's just very consolidated in its
[74:10] opinion that the way that it did what it
[74:12] did was the best. So if you were to ask
[74:14] that same agent, hey, can you make this
[74:16] better? A lot of the time it'll look at
[74:18] it and be like, well, no, I did a pretty
[74:20] good job. I don't think there's any way
[74:21] to do it better. However, instead of
[74:23] just giving that agent back the entire
[74:25] context and saying, can you do it
[74:27] better? A much smarter thing to do is to
[74:29] take all of the um outputs, not the
[74:31] reasoning, then give the output, aka
[74:33] your code or your workflow or the
[74:35] results of your your accounting to
[74:37] another agent and then say, "Hey, is
[74:39] this right?" Because now that second
[74:41] agent can evaluate purely based off
[74:43] output. It doesn't actually have to deal
[74:45] with evaluating things based off the
[74:46] reasoning or the intent. And so your um
[74:49] work can end up being a lot higher
[74:50] quality as a result. So here's a quick
[74:52] example using like a coding thing where
[74:55] uh we wanted to build a rate limiter.
[74:56] What'll happen is our first agent will
[74:59] implement and write the first draft of
[75:01] the code. This code output will pass to
[75:03] a reviewer agent. Now the reviewer agent
[75:06] is spawned with fresh context, meaning
[75:08] there's no tokens that are polluting its
[75:10] window and a zero bias. And what it does
[75:12] is just like objectively speaking, you
[75:15] ask it, is this thing correct? Are there
[75:17] any issues here at first glance? Any
[75:19] ways you could simplify this? Now,
[75:21] because it's treating this just like
[75:22] it's treating a random snippet of code
[75:24] it finds on the internet, you know, it
[75:26] it has no opinions. It has no inherent
[75:28] like desire to claim, well, this is the
[75:30] best way because I spent all this time,
[75:32] energy, and research figuring it out.
[75:34] And it'll be able to to look at things
[75:35] with, you know, those fresh eyes. From
[75:38] there, if it finds issues, the idea
[75:39] behind sub agent verification loops is
[75:41] it'll list those issues and then pass
[75:43] the suggestions to a third agent called
[75:45] a resolver, which has zero context about
[75:48] any of this stuff as well. And so in
[75:50] this way, an implement reviewer resolver
[75:52] loop can get significantly higher
[75:55] quality results than just one agent
[75:57] doing everything simultaneously. If
[75:59] there are no issues, everything's
[76:00] approved, we're good to go. Um,
[76:02] otherwise it resolves, we do some
[76:04] testing, and then we get the final
[76:05] verified code output. Are you guys
[76:07] noticing a trend here? Basically, all of
[76:10] these like advanced agent foundation uh
[76:12] advanced agent prompting techniques
[76:15] ultimately circle back to having
[76:16] multiple agents working in parallel. And
[76:18] it's really interesting because like the
[76:20] way that agents work themselves is they
[76:22] already do work in parallel. You know, a
[76:24] few years ago, um, agents were basically
[76:26] just one statistical model and you would
[76:27] ask the statistical model to help you
[76:29] complete the the the sentence or
[76:31] whatever and then would give you the
[76:32] most likely next token and then would
[76:34] rerun over and over and over again until
[76:35] it did that. Well, a few years back, um,
[76:38] people started introducing this idea
[76:41] called a mixture of experts, which is
[76:43] instead of just having one model, what
[76:45] you do is you actually send the same
[76:47] thing to like three or four models, you
[76:49] average out the statistical
[76:50] probabilities of every word and then you
[76:53] just pick what they all converged on.
[76:54] Very similar what I did there with
[76:56] stochastic multi-age consensus. And so
[76:58] this mixture of experts is sort of like
[77:00] the base foundation that resulted in a
[77:02] really big improvement in large language
[77:04] model accuracy among other things like
[77:06] post-training and RLHF and and and stuff
[77:08] like that. But what's really cool is all
[77:10] of these frameworks basically do the
[77:12] same idea. You know we we treat these
[77:14] mixture of experts now as themselves
[77:16] models and then we prompt them with each
[77:19] other. We do them in parallel and then
[77:21] integrate their answers like stocastic
[77:23] multi-age consensus. We have them debate
[77:25] against each other like with model
[77:27] chats. And now what we're doing is we're
[77:29] basically having them correct each
[77:30] other's work like with sub agent
[77:31] verification loops. So all of these are
[77:33] just try uh trading off the same core
[77:36] foundational like features of models
[77:38] which is that at the end of the day
[77:39] they're statistical machines. And so the
[77:41] more of these statistics that you can I
[77:43] don't know average out the closer you
[77:44] get to the reality. Another way of
[77:46] thinking about this is if the implement
[77:48] agent has already spent 200,000 tokens
[77:50] accumulating all that context, it'll
[77:52] literally remember every wrong turn and
[77:54] every dead end. It'll have a sunk cost
[77:56] bias. It'll say, "Well, I wrote this, so
[77:58] it must be right." And in a way, it'll
[77:59] be blind to its own mistakes. When you
[78:01] pass it off to this super nerdy looking
[78:03] reviewer agent, it has a fresh empty
[78:05] context. It'll only see the output, not
[78:07] the journey that we took to get there.
[78:09] No emotional attachment, although I
[78:10] think this is unnecessary at
[78:12] theorphization. And it'll catch what the
[78:14] reviewer missed. So, uh, let me show you
[78:16] guys how this actually looks like in
[78:17] practice. Here I have this app that I
[78:19] developed a while back for a video on
[78:22] vibe coding, and you guys can check that
[78:23] out in the description if you're
[78:24] interested. It's where I basically put
[78:25] together a full endto-end system that
[78:28] allowed you to um, design and then
[78:29] syndicate a bunch of content. So, you
[78:32] know, this is just some app, right? This
[78:33] app, I don't even know if it's fully
[78:35] functional. Okay, no, it isn't because I
[78:36] had to turn it off. But hypothetically,
[78:38] there's a big code base here, right? And
[78:40] so what I want to do is I want to use
[78:41] this app to show you guys how an un um
[78:44] biased code reviewer would take a look
[78:46] at the code that a previous agent had
[78:48] written, in this case Gemini, and um and
[78:50] improve it. So what I'm going to do is
[78:52] I'm going to go find this repo. Okay.
[78:53] And I found it over here. It's in the
[78:54] Splinter repository. That makes sense.
[78:57] I'm just going to open up a new Cloud
[78:58] Code instance. And then down over here,
[79:00] I'm going to say I'd like you to use,
[79:04] and I just need to make sure I know what
[79:05] the skill is called.
[79:08] agent review on the Splinter repo. It's
[79:12] let's just say folder. It's in the
[79:14] parent folder so that it knows where
[79:16] this is. That way I can still execute it
[79:17] within this um business uh workspace
[79:20] which I found a much better way of
[79:21] organizing things. And while it's doing
[79:23] that, I'm going to open up the skill.md.
[79:25] So what the skill.mmd does is it spawns
[79:28] sub agents to review, simplify, and
[79:30] verify output. It uses after completing
[79:32] many non-trivial implementation tasks
[79:34] and it triggers on the words review this
[79:36] agent review self-review or you know
[79:38] slag agent-review and you can see it's
[79:40] already doing this. It's um spun up a
[79:42] sub agent called review splinter
[79:44] codebase. And what this does is it
[79:46] reviews it for four things. Correctness,
[79:49] edge cases, simplification and then
[79:51] security. Now like do I know how to do
[79:54] all this programming under the hood? No,
[79:55] I don't. But these agents certainly do.
[79:57] And so we can take advantage of that by
[79:58] having an agent with zero context. this
[80:00] one here, review that entire workspace
[80:03] um sort of independently and
[80:04] objectively. And now it's doing a bunch
[80:06] of reading and it's going to integrate
[80:08] that with the suggestions of this model
[80:10] to give us a much higher quality output.
[80:12] All right, the Splinter code review just
[80:13] finished up and we found 22 issues
[80:15] across the codebase. There's some
[80:17] critical ones here, some high issues
[80:19] here, some medium issues here, and then
[80:21] some low issues over there. Now, it's
[80:23] asking me if it wants me to start fixing
[80:26] any of these, and I'll say absolutely.
[80:28] And the whole idea behind this now is
[80:30] we're we're capable of looking at this
[80:32] completely objectively, you know, like I
[80:34] asked the initial model Gemini when I
[80:36] made the uh the app in the course like
[80:38] multiple times, hey, are there any
[80:39] issues here? Hey, are there any ways to
[80:40] make this better? Hey, what do you
[80:41] suspect is a problem? And just couldn't
[80:43] find it because it was so polluted by
[80:44] its own biases. Now another model can
[80:48] and it's very similar to like peer
[80:49] review in like academic um circles. It's
[80:52] not that like you know you're dumb for
[80:54] coming up with this codebase like how
[80:56] dare you. It's just that as you work on
[80:58] things more and more and more, you tend
[81:00] to see things a little more narrow and
[81:02] more narrow because you've explored a
[81:03] bunch of other possible paths. And the
[81:05] reality is the fact that you explored
[81:07] those paths and those don't work don't
[81:09] necessarily mean that if somebody else
[81:11] explored one of those paths, it wouldn't
[81:12] work either. And so this is just a way
[81:13] of remaining as objective as humanly
[81:15] possible, which is obviously a very
[81:16] valuable thing to do when you're doing
[81:18] things like creating applications, code,
[81:20] um, you know, sales, marketing, and all
[81:22] the various things that AI agents allow
[81:23] us to do. Next up, I want to talk a
[81:25] little bit about prompt contracts. For
[81:27] those of you guys that don't know,
[81:28] earlier on we chatted a little bit about
[81:30] a definition of done, right? Well, vague
[81:33] tasks, aka tasks that don't have clearly
[81:35] defined definitions of done, are
[81:37] basically the number one problem
[81:40] nowadays with what I would consider to
[81:42] be people's like disillusion with AI
[81:44] agents. Like when a total novice starts
[81:46] using AI and then they dive into some
[81:48] agent coding platform and then they just
[81:50] say, "Hey, build me a Netflix 2.0. make
[81:53] me a million dollars, make no mistakes.
[81:55] Um, because of their extraordinarily
[81:57] poorly defined definition of done,
[82:00] because they're poorly defined goals,
[82:01] because they don't give it any
[82:02] constraints, because they don't give it
[82:04] any failure conditions, uh, that model
[82:06] is just not going to do any any get
[82:08] anywhere near as high quality and end
[82:10] result as if they did just follow a
[82:11] simple little uh, step-by-step process.
[82:14] And so the step-by-step process
[82:16] obviously you could learn, but you could
[82:18] also just like hardcode it as a skill
[82:20] somewhere in your workspace or as uh you
[82:22] know something in your cloud NMD and
[82:23] then just force your model to always
[82:25] have this information before you
[82:26] proceed. And so for instance, if you
[82:28] give it a vague task like build a rate
[82:30] limiter, okay, it'll do pretty poorly.
[82:32] But the whole idea behind a prompt
[82:34] contract is you basically make the user
[82:35] who puts in a request like this sign a
[82:37] mini contract and just say, "Okay, cool.
[82:39] The contract is, you know, here's what
[82:41] your goal is. Here what your constraints
[82:43] are. here's what your format is and
[82:45] here's what your failure is. Are you
[82:46] good to go? If the answer to that
[82:48] question is yes, now the model has
[82:49] actually gone through the step of
[82:50] defining your goal, your constraints,
[82:53] your format and your failure. And so all
[82:55] of your definitions are done, all of the
[82:57] various uh kind of technical spec
[83:00] requirements here are much more laid out
[83:02] and then the model sort of has a lot
[83:03] easier of a way of going about things.
[83:05] And so this is very similar if you guys
[83:07] are aware um to like this idea of
[83:10] scopes.
[83:11] Now I, you know, I run like a freelance
[83:13] education platform, like an AI
[83:15] automation agency education platform.
[83:17] And so scopes are a really big part of
[83:18] like a successful project. Um, and so I
[83:21] teach people how to define like really
[83:22] precise and concrete scopes. Uh, whether
[83:24] you're doing, you know, a small project
[83:25] for a client or working with some large
[83:27] enterprise business or something like
[83:28] that. And like a real real common issue
[83:31] is scopes just tend either to be way too
[83:34] vague and so people don't actually
[83:36] clearly define them or they end up way
[83:39] too restrictive in so far that people
[83:42] you know in a in an attempt to
[83:44] counterbalance the vagueness. They end
[83:46] up going like way too specific and then
[83:47] the scope ends up being like so
[83:49] restrictive that it's like you know
[83:50] you're a slave to it and you can't
[83:51] change anything. And so prompt contracts
[83:53] sort of help you navigate the the thin
[83:55] line between too vague and too
[83:57] restrictive. And it's very similar in
[83:58] nature to like giving a contractor a
[84:00] task and then the contractor clarifying
[84:02] with you before they actually do the
[84:04] task which I think you know is clearly a
[84:08] consequence of agents pushing all of us
[84:10] more towards like management style
[84:12] positions where we just manage the
[84:13] inputs and the outputs of these things.
[84:15] So a big fan of defining these clearly.
[84:17] So what does this actually mean in
[84:19] practice? Well, there's obviously a
[84:20] million and one different ways you can
[84:22] define prompt contracts. The way that
[84:24] I've decided to do so in this
[84:25] demonstration is through a skill called
[84:27] prompt- contract. And so basically
[84:29] before implementing any non-trivial
[84:31] task, the skill forces you to generate a
[84:34] structured prompt contract with goals,
[84:35] constraints, the format of output, and
[84:37] then failure. So the idea here is you're
[84:40] treating it just like a spec or a scope
[84:42] of work. Any task that produces code or
[84:44] some configuration settings or something
[84:46] like that needs to go through this
[84:47] process. And then this model will sort
[84:49] of selfanalyze the request before
[84:52] drafting a four section contract and
[84:54] then presenting it for approval. This is
[84:56] almost uh similar in nature to like the
[84:58] plan mode that a lot of these um agent
[85:00] platforms now have. Like in cloud code
[85:02] for instance, it can enter plan mode and
[85:04] give you a brief little plan and have
[85:05] you approve the plan before it proceeds.
[85:06] It's just this formalizes it as a
[85:08] contract. And no, you're not signing
[85:10] your life away with cloud code when you
[85:11] do this. Um, but you know, it's a simple
[85:13] and easy way to make sure that you get
[85:15] more repeatable and consistent and
[85:16] accurate outputs every time. So, why
[85:18] don't I actually do this? Use prompt
[85:21] contracts to define this task. And then
[85:24] I'm just going to pretend that I'm
[85:25] giving it a really simple query. I'm
[85:26] just going to say, I want you to build
[85:28] me a beautiful site for leftclick.ai.
[85:33] That's my um agency. So, what it's going
[85:34] to do is it'll begin by invoking the
[85:36] skill prompt contract. And I mean,
[85:38] beautiful site is such a subjective
[85:40] term, right? right? I mean, like, what
[85:41] the heck does that even mean? And so,
[85:42] the model is going to be essentially
[85:44] forced to ask me for more context on
[85:46] what constitutes a beautiful site to me.
[85:49] And in this way, we'll get a much higher
[85:50] quality site or app or whatever the hell
[85:52] at the end of it. Likewise, you could do
[85:53] this with any business task as well. It
[85:55] doesn't just have to be like a design
[85:56] task. Um, you could set up a prompt
[85:58] contract for, hey, email these 45
[86:00] people. And it could ask you like, oh,
[86:02] like what spec, you know, specifications
[86:04] do you do you want to confirm that
[86:05] they're emailed? And what do you want
[86:07] the emails to say? And what's the goal
[86:09] of a successful thing? and like, do you
[86:10] have any failure parameters? If we only
[86:12] email 44, is that okay with you? Right?
[86:14] It basically forces it to be a lot more
[86:16] clear and then concise. So, what's
[86:18] happening now is it's gone through, it's
[86:20] actually accessed leftclick.ai, that's
[86:22] my current website, and then it's um
[86:23] getting a bunch of screenshots and stuff
[86:25] like that. And the reason why is because
[86:26] it's attempting to build up context for
[86:28] the prompt contract. So, its first step
[86:30] was to analyze the request, right? What
[86:32] it's going to do is it'll identify what
[86:34] done looks like. It'll identify some
[86:36] implicit assumptions. So, what am I
[86:38] about to force the model to assume
[86:39] without being told? Well, obviously an
[86:40] assumption is I already have a website,
[86:42] right? And so, it's going to go through,
[86:44] take pictures of my website, and see,
[86:45] well, if Nick wants something different
[86:46] from this, why? And then, it's going to
[86:49] sort of make its own judgment to that
[86:50] end. And now, it's actually giving me
[86:51] the contract. So, the goal is a single
[86:54] page marketing site for leftclick. Here
[86:56] are some constraints. You know, we want
[86:58] smooth scroll animations under 500 lines
[87:00] of HTML. The format is this. There
[87:03] should be these sections. subtle
[87:05] animations fade in on scroll hover
[87:06] states. A failure is if it looks like a
[87:08] generic Bootstrap template. A failure is
[87:11] if it's broken on mobile. A failure is
[87:12] if the animations are janky. The failure
[87:14] is if the file exceeds 500 lines. So, I
[87:16] actually really like this prompt
[87:17] contract. It's really simple and
[87:19] straightforward. So, I'm actually going
[87:20] to say go ahead and build it. But what's
[87:21] cool is, you know, we're now actually
[87:23] having a conversation about this. We're
[87:24] actually agreeing on, you know, what the
[87:26] end result is going to be. And this is
[87:29] actually really similar in nature to the
[87:31] other thing that I want to talk to you
[87:32] guys about which is um kind of related
[87:34] and orthogonal to prompt contracts
[87:37] although it is a little bit different.
[87:38] And this is called reverse prompting.
[87:40] Now reverse prompting is in a similar
[87:42] vein a mechanism used to clarify the
[87:45] quality of a prompt and improve the
[87:47] probability that it ends up okay. And
[87:49] basically the way that this works is
[87:50] instead of just like forcing the model
[87:52] to give you this contract and having you
[87:54] sign off on it, it it takes it one step
[87:56] further. it actually forces the model to
[87:57] ask you some clarifying questions ahead
[87:59] of time. So rather than just give you a
[88:01] spec sheet and say, "Okay, we're good to
[88:02] go." What reverse prompting does is it
[88:05] um has the model ask you a bunch of
[88:06] questions that you maybe didn't even
[88:08] think that you had to answer, the model
[88:09] then takes all bad context and then
[88:11] feeds that into a prompt contract later
[88:13] on. Okay, so step one is when the user
[88:14] gives a task to an AI agent. So I don't
[88:16] know, this is like a website, right?
[88:18] Step two is the agent asks five
[88:20] clarifying questions back to the user
[88:21] before starting. Step three is when we
[88:24] answer and then the agent builds the
[88:25] correct thing on the first try. So
[88:26] significantly improves one-shot
[88:28] potential. And then if we didn't have
[88:29] reverse prompting, there'd be a lot of
[88:31] like wrong implicit assumptions here
[88:33] which would result in, you know, the
[88:35] probability of a oneshot, which is just
[88:36] when the agent does it in literally one
[88:38] request uh going down quite a bit. And
[88:41] so similarly, I also have a reverse
[88:43] prompt skill over here. And so if I go
[88:46] to this reverse prompt skill, you can
[88:47] see the way that this is set up is
[88:49] before implementing any non-trivial
[88:50] build, ask the user five dynamically
[88:52] generated clarifying questions to
[88:54] surface non-obvious preferences,
[88:55] assumptions, and constraints. So when to
[88:57] trigger before starting the
[88:58] implementation, step one, analyze the
[89:01] request, figure out some stated
[89:02] requirements, implicit assumptions,
[89:05] some decision points, failure modes, and
[89:06] taste dependent choices. Right? And so
[89:09] likewise, if I instead wanted to build,
[89:11] let's say something for
[89:15] build a beautiful site for 1 second
[89:17] copy, which is my old content writing
[89:18] company, which we just had to shut down
[89:19] a few days ago. U as you guys can
[89:22] imagine, content isn't super in these
[89:23] days.
[89:25] Oh, and then um use the reverse prompt
[89:28] skill and chain it together with prompt
[89:32] contracts after. What you could see is
[89:35] we're now engaged significantly more
[89:37] than we were before. Before I just say,
[89:39] "Build me a beautiful site." Probability
[89:41] that it gets what I want right on the
[89:42] first try. Pretty damn low. What it's
[89:44] doing now is it's asking a bunch of
[89:46] clarifying questions to confirm whether
[89:48] or not, you know, this site is as I want
[89:51] it to be. And then after I feed it back
[89:53] that information, it'll then take that
[89:55] and use that to construct essentially
[89:57] that that prompt contract that we had
[89:58] before. So here's what the conversation
[90:00] looks like. What's the primary goal of
[90:02] the site? Brand credibility,
[90:03] salesfunnel, lead genen. You know what I
[90:05] want is just brand credibility. Should
[90:07] it be a single static page site or
[90:08] should I build it in some other
[90:10] framework? No, I wanted a simple site.
[90:12] What's the vibe? You know, it's AI
[90:13] content writing. You know, should I do a
[90:15] clean modern SAS aesthetic? Think linear
[90:17] versel? Do I want something different?
[90:19] Yeah, I want like linear but white. You
[90:22] know, should I generate the copy from
[90:24] context or use some placeholder content?
[90:26] No, you're cool. You can generate it
[90:27] from here. Now once we've clarified
[90:30] everything, what this model is going to
[90:31] do is use all this information to
[90:33] outline the prompt contract using the
[90:34] prompt contract skill. And now you can
[90:36] see it's invoking this skill as well.
[90:38] And here we have a contract. It'll be a
[90:40] single page static site for 1 second.
[90:42] Copy linear weight aesthetic 5 seconds
[90:44] deploy ready. Here's some constraints.
[90:46] Here's the format. Maybe I don't like
[90:47] the format. Maybe I don't want it inside
[90:49] of active, you know, I want it somewhere
[90:50] else. Uh but anyway, in in this case,
[90:52] maybe I want to look good and build it.
[90:56] Now, just to show you guys an example of
[90:57] how much higher quality we can get when
[90:58] we actually do this, this is the um
[91:00] website that uh it just built for us.
[91:02] I'm just going to refresh this puppy and
[91:04] take it to a new window because it gets
[91:05] cut off that window. This is here. We
[91:07] have those cool sexy animations. As we
[91:10] scroll down, we also have some
[91:11] information. Um it's it's light theme,
[91:13] right? We have these really minimalistic
[91:15] requirements here. Information about
[91:17] myself, some services page, words from
[91:19] happy clients, and then ultimately like
[91:21] a CTA. And so, you know, the reason why
[91:23] I was able to get much closer to what I
[91:24] wanted, which was a minimalistic white
[91:26] high-end aesthetic is just because,
[91:27] like, you know, I I had it outlined in a
[91:29] contract. As I'm sure you guys can
[91:31] imagine, you can employ the same
[91:32] approach for whatever the heck you want,
[91:33] whether you're building a site or you
[91:35] are, you know, selling to people or you
[91:36] are doing some sort of bookkeeping or
[91:38] accounting. It's all just about uh
[91:40] building out a very strong definition of
[91:41] done. And the model can assist you with
[91:43] this. You don't actually have to sit
[91:44] down and laboriously write it all out
[91:45] yourself. And that takes us to the
[91:47] initial demo that we started with which
[91:50] was the multi- aent Chrome MCP manager.
[91:54] Now basically at the very beginning of
[91:56] this course you didn't understand how
[91:59] you know one agent could spawn a bunch
[92:01] of other agents. You didn't understand a
[92:02] lot of like the parallelization plays.
[92:04] You also didn't understand that uh you
[92:06] know you could have agents actually chat
[92:08] with each other and communicate. You
[92:10] didn't understand the idea behind using
[92:12] one agent to verify the work of another.
[92:14] you didn't understand the idea behind
[92:16] delegating to multiple different types
[92:17] of models. What's really cool is the
[92:20] multi- aent Chrome setup that I showed
[92:22] you guys where we had, you know, five or
[92:24] 10 agents all operating independently in
[92:25] their own browsers, in their own
[92:26] workspaces. All of that just feeds off
[92:29] of this this idea or this concept um of,
[92:32] you know, agents increasing their level
[92:34] of communication with other agents. And
[92:36] so essentially if you think about this
[92:38] logically, you know, if I were to do
[92:40] this uh with like a single agent, so
[92:42] let's just say one agent, you know, it's
[92:44] not actually rocket science to have one
[92:46] agent use a browser these days. There
[92:48] are built-in skills called MCPs, model
[92:50] context protocols basically that you can
[92:52] just pipe in and immediately connect to
[92:54] and it can do everything for you. Okay,
[92:55] it can it can launch Chrome and then it
[92:57] can control things on the page and
[92:58] whatnot. It can do that. [gasps] So, you
[93:00] know, the issue is it just takes a lot
[93:02] of time. We'll receive the target URL.
[93:04] We'll launch Chrome via the DevTools
[93:06] MCP. We'll navigate to the website.
[93:08] We'll take a screenshot and you know in
[93:10] my case this over here was like um
[93:12] specific for me which was just page or
[93:16] rather form fills. After that we'll
[93:20] identify the form extract the form
[93:21] fields generate a personalized message
[93:22] fill the fields and then click submit.
[93:24] Um but you know this is still something
[93:26] that's occurring linearly and because of
[93:28] linear constraints you know unless you
[93:30] are using uh I don't know like a Gemini
[93:32] flash model or you're using fast mode
[93:34] and burning through your clawed token uh
[93:36] usage limits this is going to take a
[93:37] fair amount of time. This process over
[93:39] here literally to just like launch the
[93:41] browser could take 5 seconds. This
[93:43] process to navigate to the website could
[93:44] take 5 seconds. Taking a page screen
[93:46] check could take 15 seconds. Identifying
[93:48] the contact form could take a minute.
[93:50] You know, if you stack it all up,
[93:51] basically what's occurring is this whole
[93:52] process here might take literally 2 to 3
[93:55] minutes per form if you're operating
[93:58] naively using a slower model. And if
[94:00] you're operating non- naively, if you're
[94:02] using a smarter model, then obviously
[94:04] you have to weigh that against cost and
[94:05] and and token usage and stuff like that.
[94:07] So, I don't know, let's hypothetically
[94:09] say in my case, I wanted to reach out to
[94:11] uh you know, 1,000 people. Well, if it
[94:14] takes me 2 to 3 minutes to form, that's
[94:16] 1,000 * 2. That's 2,000 minutes, which
[94:20] divided by 60 is like 30 hours or
[94:22] something like that, right? That's a
[94:23] very long time. It's going to take me a
[94:25] whole day. So, instead of just doing one
[94:27] agent, what I'm going to do is I'm
[94:28] basically going to give every agent its
[94:30] own both Chrome instance and then even
[94:32] its own workspace and then open up its
[94:35] autonomy so that it can make um some
[94:36] advanced decisions to basically help it
[94:38] build its own tooling if it needs uh in
[94:40] order to like navigate website pages or
[94:41] whatever. Now, what this is going to
[94:42] look like it's pretty similar to um our
[94:44] previous uh you know stochastic multi-
[94:47] aent consensus prompt where basically we
[94:49] have a user up top. Okay. And this is
[94:51] us. And what we're going to do is we're
[94:54] going to give all the context about our
[94:57] task whatever it is that we want you
[94:58] know fill out a form or I don't do some
[95:00] lead genen to an orchestrator agent
[95:03] which um in this case I'll do claude and
[95:05] we'll just do opus which uh in my case
[95:08] is going to be 4.6. Maybe in your case
[95:10] it's a better model. And then what
[95:12] that'll do is it'll spawn and set up,
[95:15] you know, however many agents we want in
[95:17] separate windows. That'll then all in
[95:19] parallel navigate to the site, find the
[95:21] form, fill the fields, and then do the
[95:22] submission. And so basically, instead of
[95:24] it taking 2 minutes per form, what we
[95:26] can do is we can actually submit, you
[95:27] know, however many forms. So I don't
[95:28] know, let's say we have like 10 agents,
[95:30] we'd submit 10 forms in the same amount
[95:32] of time it took to submit one. So maybe
[95:34] for us it'll be 120 seconds. And then
[95:36] what we do is we just we just increase
[95:37] this as necessary. I mean I could
[95:39] theoretically have 500 operating if I
[95:40] had the computing power. So you know if
[95:42] um previously it was one form in uh what
[95:46] did I say 2 minutes and that means the
[95:49] form per minute rate is like 0.5 forms a
[95:52] minute right but now if we spin up 10
[95:55] and we do 10 in 2 minutes we're up to
[95:57] five a minute. If we spin up I don't
[96:00] know 100 and we're up to 50 a minute.
[96:03] And you know, if my goal was 2,000 a day
[96:05] and we're at 50 a minute, then obviously
[96:06] 2,000 divided by 50 means we can get
[96:08] this whole thing done in 40 minutes. And
[96:10] you know, depending on the the the list
[96:12] and whatever the heck you got, obviously
[96:13] the constraints change, but um this is
[96:15] how you can have multiple Chrome
[96:17] instances operating simultaneously it
[96:19] navigating the website and stuff like
[96:21] that. What I have here is a skill called
[96:23] multi-- aent-chrome. And again, this is
[96:26] something you can implement using
[96:27] whatever um context framework you want,
[96:29] whether it's a skill, whether it's like
[96:31] a claude, Gemini, or agentmd, whatever
[96:33] the heck you want. What this basically
[96:34] forces it to do is to orchestrate
[96:36] parallel browser automation using
[96:37] multiple Chrome DevTools MCP instances.
[96:40] And this is used when a task requires
[96:42] doing the same browser action across
[96:43] many targets simultaneously. So some
[96:45] good examples are submitting forms,
[96:47] filling apps, scraping pages that need
[96:48] JavaScript rendering and and whatever.
[96:50] And so what's occurring down here is
[96:52] basically we have a tople business
[96:55] workspace which is sort of the folder
[96:56] that I'm in right now. And this actually
[96:58] interacts with a bunch of Chrome agents
[97:00] which all have their own little MCP
[97:03] servers their own little um claw.mds and
[97:06] so on and so forth. And then they all
[97:08] communicate with a centralized chat and
[97:11] if they run into problems on websites if
[97:13] they have any reports they want to give
[97:14] basically what happens is this
[97:15] orchestrator just checks the chat every
[97:17] 30 seconds or so. Okay. So the very
[97:20] first step is it determines how many
[97:21] agents are needed. Then it launches all
[97:22] the Chrome instances. It resets the chat
[97:24] file because uh you know previous runs
[97:26] may have that. And then you can see how
[97:28] every individual sub agent actually
[97:29] monitors its own um task list by
[97:32] basically just pumping things into a
[97:34] chat. This is one of the simplest and
[97:36] easiest ways of getting this specific
[97:38] design pattern done. As mentioned, you
[97:39] guys can get this down below if you
[97:40] want, but I'm just going to give you
[97:42] guys a simple example, which in my case
[97:43] is going to be just finding Vancouver
[97:45] rentals because I'm, you know,
[97:46] considering getting a rental um down
[97:48] there. And so, you know, rather than
[97:50] have it give you crappy results, this
[97:52] thing can actually navigate like
[97:53] Craigslist, Facebook Marketplace,
[97:54] Kijiji, whatever the heck you want. Uh,
[97:56] and the the specific script is right
[97:57] over here. So, hypothetically, what I'll
[97:59] do is I'll just open up a new window.
[98:01] And then I'll go over here and then I'll
[98:02] write uh I want to find a rental in
[98:07] Vancouver. Needs to be 15 minute walk
[98:09] from the Granville Sky Train station
[98:13] downtown. use multi- aent Chrome to
[98:17] navigate through sites and give me high
[98:22] quality sleek places under 2.5 let's say
[98:27] 2K to 2.5K
[98:30] other restrictions like one bed one bath
[98:34] reasonably near the water needs AC built
[98:38] in okay so I'm giving it a highle um you
[98:42] know piece of instruction and sorry what
[98:44] meant to do is actually do prompt
[98:45] contract after this. And now I want it
[98:47] to give me like a very clear contract.
[98:49] So it's going to give me a list of 5 to
[98:51] 10 rental apartments. Why don't we say
[98:53] 20 rental apartments? And then I'll say
[98:56] 1.2 km is fine. We'll say near water,
[98:59] south of Drake or west of Bard. Okay,
[99:02] I'm just going to make some changes
[99:03] here. And then I'll say that sounds
[99:05] pretty good. Go for it. And now it's
[99:07] going to actually launch the multi- aent
[99:09] chrome scraping. So it's then going to
[99:11] invoke the skill. I'm just going to keep
[99:12] my hands off. What it'll do next is
[99:15] actually spawn um [clears throat] four
[99:16] parallel Chrome agents, one per rental
[99:18] site. So, it'll determine that there are
[99:20] four rental sites that it's going to be
[99:22] running through. And it'll just have one
[99:24] Chrome instance sort of do everything
[99:25] there uh per site. So, now we have the
[99:28] four instances. I'm just going to open
[99:29] this up here. Open this up here. I'll
[99:33] move this one down over here. I'll also
[99:35] move this one down over here. Obviously,
[99:37] you could use an approach like this for
[99:39] pretty nefarious purposes. Um, so you do
[99:41] have to be cognizant of that that a lot
[99:43] of people and websites are probably um,
[99:46] you know, they're looking to verify
[99:47] whether or not you are a person. And so
[99:49] there are multiple things you can do to
[99:50] get around that if you so wanted to,
[99:52] like using custom um, browser
[99:54] fingerprinting and whatnot. And I think
[99:56] that's a story for another course
[99:58] because I don't really want this course
[99:59] to be accused of just showing you guys
[100:00] how to spin up like 500 Chrome instances
[100:02] scraping all sorts of illicit
[100:03] information on the internet using unique
[100:05] browser fingerprints. But uh that stuff
[100:07] is definitely possible and there are
[100:08] probably like a lot of people doing
[100:10] stuff similar to this right now. They're
[100:11] just way farther ahead in terms of their
[100:13] uh you know understanding of a agents
[100:15] and stuff like that. Now after the 30
[100:16] secondond or so wait time these will
[100:18] receive their instructions and they'll
[100:19] actually check the main thread and then
[100:20] they'll load in their websites. So this
[100:22] one up here spawned li I've never used
[100:25] that site before. This one's
[100:26] padmapper.com which is another one. You
[100:28] know these are all like websites and
[100:30] resources I probably would not have
[100:32] looked at. And as a result I'm going to
[100:33] get more of like a search spread. I'm
[100:36] going to again a big chunk of the search
[100:37] space much faster than if I were to have
[100:39] done all this stuff manually. What's
[100:41] cool is these can zoom directly into
[100:43] pages for me. These can click on links
[100:45] and stuff like that. They're obviously
[100:46] modifying filters and and whatnot
[100:48] autonomously so that they're not just
[100:49] getting a bunch of bogus results. And at
[100:51] the end I get a high quality filtered
[100:53] list of apartments that are, you know,
[100:54] within my specifications. Okay, I just
[100:56] turned my camera off because I wanted
[100:58] some additional room on the bottom left
[100:59] hand side to really drill a few
[101:01] important points home. The first is your
[101:03] context window. Now remember earlier how
[101:06] we talked about the claude.m MD, the
[101:08] Gemini MD, and then the agents.m MD.
[101:12] Over here, we just have Claude, but I
[101:14] just want you to treat this as all uh
[101:16] three of them. That's not the only thing
[101:19] that gets injected, so to speak, in your
[101:22] context.
[101:24] You have a variety of other things. Now,
[101:27] you have your system prompt up top. You
[101:30] then have the claw.md agents.mmd and
[101:33] whatever else. You have a file at least
[101:36] in cloud code called memory.md
[101:39] although there are analoges in other
[101:42] coding platforms.
[101:44] And then you also have skills and tools.
[101:49] We've chatted a lot about MCP over the
[101:52] course the last hour and a half or so.
[101:54] Right? Well MCP is a type of skill and
[101:58] tool. We also have the actual skills
[102:00] themselves. So remember the agent
[102:02] reviewer when we were doing sub agent
[102:04] verification loops. Well, that was an
[102:06] example of a skill. If you think to the
[102:10] prompt contracts,
[102:12] those were examples of skills.
[102:16] And the reason why I'm going into depth
[102:18] here is because each of these sections
[102:21] can consume a tremendous number of
[102:23] tokens. And you're not given an
[102:26] unlimited number of tokens to start
[102:27] with. Everything in life is finite,
[102:30] including uh you know your claude or
[102:32] your Gemini context window. Now most
[102:35] models right now are somewhere between
[102:38] uh I don't know if it's like 4.6 we're
[102:40] probably talking 200K to 1 million. If
[102:44] we're talking Gemini, you know, we have
[102:46] like 3.1 and there there a couple other
[102:48] ones obviously by the time you guys are
[102:50] watching this there'll probably be more.
[102:53] And then you know you have uh GPT 5.4 4
[102:57] and then codeex 5.3, but the 5.4 is
[102:59] coming out. You know, most models
[103:02] nowadays have somewhere in the realm of
[103:04] between 200k to 1 million. And to be
[103:07] clear, um, a token is not a word. A
[103:11] token is about
[103:14] 0.7
[103:16] words. So if you think about it in that
[103:18] vein, what this means is these 200k
[103:20] tokens sort of actually equate to
[103:23] somewhere between like 140,000 words to
[103:25] about 700,000 words. Okay. Um but this
[103:29] context window obviously gets filled up
[103:31] the more that you talk with it. And
[103:33] unfortunately, one common and major
[103:36] problem in large language models,
[103:38] specifically the types that we're
[103:39] dealing with in this course, are as time
[103:42] goes on and you talk to it more and more
[103:45] and more, what you find is the average
[103:49] quality
[103:51] goes down.
[103:53] So quality
[103:55] as a factor or a byproduct of token
[103:59] count typically starts pretty high up
[104:01] here at maybe I don't know 100%.
[104:05] And then the longer and longer and
[104:07] longer the number of tokens in your
[104:09] context the lower and lower and lower
[104:11] the quality gets. So maybe this is at
[104:14] 10k, maybe this is at 50k, maybe this
[104:18] over here is at 200k. And what that
[104:21] means is let's just hypothetically say
[104:23] you're at your 199,000th
[104:26] token. Okay, that means that on a
[104:31] equivalent query that you might have
[104:34] previously scored 100% at
[104:37] at I don't know 5 or 10k tokens at
[104:40] 199,000 tokens you might only score 40%
[104:42] at. Now, these numbers I basically
[104:45] pulled out of my ass to be clear, but
[104:47] the point I'm trying to make is the
[104:49] longer the token count, basically the
[104:51] bigger the context length, the lower the
[104:56] performance of the model. And so,
[104:58] understanding context windows and then
[105:00] learning a little bit of context
[105:01] management, ways to proactively manage
[105:03] all of these things, some of which you
[105:04] have control over and some of things
[105:06] which you don't, is very important. It's
[105:09] also important, of course, because of
[105:11] billing. The more tokens that you use
[105:14] up, obviously the more money that you
[105:16] spend. And so, not only is it best from
[105:18] a quality perspective over here to try
[105:22] and push to the left side of this graph
[105:23] as much as humanly possible, it's also
[105:25] very relevant from a financial
[105:28] perspective over here because obviously
[105:29] the more tokens that you use, the more
[105:31] money you spend. Case in point, just to
[105:33] make this video, I've spent something
[105:35] around $500 or so in tokens. That's
[105:38] because I'm using a particular agent uh
[105:40] fast mode which bills me directly
[105:42] instead of just using a monthly plan.
[105:44] But the point remains any sort of
[105:46] serious AI agent application will start
[105:48] spending and using a fair amount of your
[105:50] money. Okay, just before I move on, I
[105:53] want to talk a tiny bit about the
[105:55] differences between each of these in
[105:57] length. If I open up an actual clawed
[106:00] instance here, open up one of these and
[106:02] then I go to terminal, which is the
[106:05] current best way to visualize this. Then
[106:08] if I just maximize this panel size,
[106:10] let's just make this as big as humanly
[106:12] possible. Then I go /context,
[106:15] claude will show us all of the things
[106:18] currently consuming its tokens. I'm
[106:20] going to zoom in here to make it really,
[106:22] really clear what's going on.
[106:25] This right over here is your context
[106:27] usage. And as you can see, they've
[106:30] illustrated this as sort of a series of
[106:32] squares where every square is, I don't
[106:34] know, let's see, 1 2 3 4 5 6 7 8 9 10.
[106:37] Okay, every square is I think 2,000
[106:40] tokens or so. And so what we're seeing
[106:43] is despite the fact that we have put no
[106:47] conversation tokens, so we haven't spent
[106:50] any tokens at all on conversation,
[106:54] we're still at 9,000 used.
[106:59] You're probably wondering where the hell
[107:00] are these 9,000 coming from? Are they
[107:02] shadow billing me to try and uh rinse my
[107:05] wallet as much as humanly possible?
[107:07] Well, a little bit. I mean this system
[107:09] prompt here, okay, which is partially
[107:12] composed by your agents, your Gemini or
[107:14] your clawmd and partially a few
[107:16] additional things is actually already
[107:17] consuming 4,900 tokens. So 2.5% of my
[107:21] entire token count before I even send a
[107:23] message is being used by in this case
[107:25] probably the claw.md.
[107:28] But in addition, you have other things
[107:30] like memory files which are consuming
[107:32] 2,000 tokens. Okay. And the way that
[107:35] they do that in cloud code is they use
[107:37] something called a memory MD which
[107:39] stores your preferences and some some
[107:41] previous highle things. Then next up we
[107:43] have skills which are consuming 1,700
[107:46] tokens. What are these skills? Well, you
[107:48] guys remember when we made a bunch over
[107:50] here? If I go to the top lefthand corner
[107:52] where it says docloud skills, you know,
[107:54] every aenta coding platform has their
[107:56] own configuration for this stuff. But
[107:58] the way that it works in cloud code is,
[108:00] you know, you organize these workflows
[108:01] into these skills. Well, guess what?
[108:04] These skills aren't free. In order for
[108:06] Claude to be able to use these skills,
[108:08] okay, this multi- aent orchestrator, it
[108:11] needs to store all these tokens
[108:13] somewhere and then give it to the model.
[108:14] And that's what's going on over here.
[108:16] Right now, the actual messages that
[108:17] we've used are only at eight tokens. And
[108:20] guess what? That's actually I think it's
[108:22] um this word here, context usage.
[108:26] I'm not entirely sure, but I think
[108:28] context usage just because the way that
[108:29] it's broken or maybe context usage plus
[108:31] this term here /context, you know, is
[108:33] equal to 8 tokens, but know that, you
[108:36] know, that's that's it. That's it for
[108:37] our whole token count. So, the other
[108:40] 158,000 of our 200,000 limit is
[108:42] currently free. And so, I mean, this is
[108:44] a quick and easy way obviously to
[108:45] visualize it inside of Cloud Code, but
[108:48] other platforms have their own
[108:49] visualization mechanisms. Now, next up
[108:50] are these MCP tools. A way to look at
[108:52] these mcp
[108:55] chromedev tools_click.
[108:58] What is that? Well, this is the tool
[109:01] that allows Chrome to click on parts of
[109:04] the page. Remember earlier when we were
[109:07] building that little nadn flow? Well, we
[109:09] were doing it by clicking on various
[109:11] parts of the page. How about drag?
[109:14] Right? We can drag things. Get console
[109:16] message. These are all basically buttons
[109:19] in some colossal spaceship. Basically,
[109:23] we are in the cockpit with clawed code
[109:26] and we're telling it to do stuff for us.
[109:28] We don't know what these buttons are. It
[109:30] does because it's, you know, the ship
[109:32] technician or the navigator, whatever.
[109:34] And so, it's clicking these buttons
[109:35] left, right, and center for us to do
[109:37] various things. That's how you can
[109:39] conceptualize all these MCP tools and
[109:40] all these skills. And what I really like
[109:41] about this is it breaks everything down.
[109:42] So here are our memory files, okay,
[109:44] which I talked about the memory MD, the
[109:46] cloudMD. You can see this is being
[109:48] contributed to in a variety of ways. We
[109:50] have a global cloud.mmd which is sort of
[109:52] like a very high level one with some
[109:53] sparse instructions. We have the local
[109:55] cloudmd and we actually have the memory
[109:57] down here and then we have all the
[109:58] skills. You know what was really telling
[110:00] about that is despite the fact that in
[110:01] this diagram conversation history is
[110:03] like the biggest chunk of it all. Notice
[110:05] that in reality conversation history for
[110:07] us at least at the beginning was
[110:09] nothing. you know, it was actually a
[110:11] tremendous number of tokens, about 10%
[110:13] of our entire context window dedicated
[110:15] to just this little chunk. And that is a
[110:19] problem because if you're not careful,
[110:21] your cloud MD with all its rules are
[110:23] going to get really, really long. Your
[110:25] memory MD with all your preferences is
[110:26] going to get huge. Same thing with all
[110:28] the skills and tools and stuff like
[110:29] that. And your conversation history when
[110:30] you actually do get to conversating with
[110:32] the model will be very, very, very
[110:34] small. you know, instead of starting, I
[110:37] don't know, somewhere in this region
[110:39] here, you're actually because the
[110:41] context window of all of your BS is so
[110:44] big, you might actually like start in
[110:45] the effective area over here. And
[110:48] obviously, this is not what you want to
[110:49] begin an agent conversation at because
[110:51] if you start here, then it's obviously
[110:53] only downhill from there. Now, this
[110:54] takes me to the logical question of, you
[110:57] know, hey, Nick, what happens when you
[110:58] run out of context? Because obviously
[111:00] that's going to happen. Well, when there
[111:04] are 50,000 tokens, let's say out of
[111:06] 200,000, okay, no problem. You are
[111:09] having the full conversation history
[111:11] with the model, the model basically gets
[111:13] literally every message starting from
[111:15] message number 1 2 3 4 all the way down
[111:18] to, I don't know, message number 25. And
[111:20] then what it does is it takes all of
[111:22] this context and then feeds it into its
[111:24] big neural network to generate message
[111:26] number 26, right? However, when we get
[111:29] to a certain length, right? I don't
[111:30] know, let's say message 50. Obviously,
[111:32] there's no there's no tokens left
[111:34] anymore in in the context window. Maybe
[111:35] we're like 199. Well, not actually
[111:37] probably like 155k out of 200k. Well,
[111:40] what happens is all models now have some
[111:42] sort of um what's called like autoco
[111:44] compact limit. I mean, basically all of
[111:47] them have adopted this convention where
[111:49] when the um number of tokens that you're
[111:52] using, let's say the limit is right over
[111:55] here. when the number of tokens that
[111:58] you're using gets to this point, okay?
[112:01] You know, it fills up then fills up,
[112:04] fills up, fills up and fills up. What
[112:08] happens is this triggers a mechanism
[112:10] called compaction
[112:13] where we take all of the information
[112:15] here,
[112:18] which is I don't know maybe like 80% or
[112:21] so of the whole context, and then we
[112:22] compress it. I want you to imagine right
[112:25] now there's like a a big hydraulic press
[112:27] type thing over here and it's pushing
[112:30] all of this context down. It's squishing
[112:32] it. Basically, what's going to occur is
[112:35] we're going to erase the vast majority
[112:37] of this and then, you know, instead of
[112:40] it consuming 80%, we're going to cram
[112:42] all that information into maybe I don't
[112:43] know 30 or 40% or so.
[112:47] And so that is um called compaction. and
[112:50] it occurs on a relatively regular basis
[112:52] across the uh the model ecosystem. The
[112:55] issue with the compaction or you know
[112:57] compression or whatever the heck you
[112:58] want to call it the same idea is during
[113:01] this summarization and like
[113:03] densification process we are going to
[113:05] drop outputs from tools. we are going to
[113:09] remove some information and context that
[113:11] might actually be useful to you now at
[113:13] you know message 50 for message 4 which
[113:16] might actually like eliminate a mistake
[113:17] and so because of that you know you are
[113:19] going to lose some of the quality um the
[113:21] benefit is obviously you will
[113:22] significantly improve the information
[113:24] density and what do I mean by
[113:25] information density I mean literally
[113:26] like if it's like um hello
[113:30] how are you doing
[113:35] I mean let's say somewhere in your
[113:36] context you have the term
[113:38] The sentence hello, how are you doing?
[113:40] Well, hello is actually two tokens. How
[113:42] is one, are is one, you is one, doing is
[113:46] three along the question mark. So, in
[113:48] total, if you count all this stuff up,
[113:50] depending on the uh thing you're using
[113:51] to do the embedding, that's uh eight,
[113:54] right? Well, you know, compaction is
[113:57] literally going to take this sentence
[113:58] and then it's going to compress it. So,
[113:59] it's going to say, "Hi, how are you?"
[114:04] So now instead of that's one token here,
[114:06] one token here, one token here, one
[114:08] token here equals four, it will
[114:10] literally get all of the context that it
[114:12] can and then try and squish it so that
[114:15] the same meaning is available in fewer
[114:17] tokens and fewer words wherever
[114:18] possible. And then it'll just run this
[114:20] naively across your entire um context.
[114:23] And so this is going to occur every
[114:24] single time. Obviously, it's something
[114:25] that we want to avoid occurring if we
[114:27] have sensitive and important data, but
[114:29] you know, it does allow us to continue
[114:31] conversating with models. before we had
[114:33] context compression and some form of
[114:34] autoco compaction. Um, basically we
[114:36] would just run out of tokens and we'd
[114:38] have to restart a totally new session.
[114:39] So this just does sort of that
[114:41] intermediate step that most people were
[114:42] doing before where they would like like
[114:43] take all of that, you know, try and
[114:46] summarize it in some other model and
[114:47] then paste it back into uh into into
[114:49] another one. Now that takes us to what
[114:52] most model practitioners are using
[114:54] nowadays, which is a variant of
[114:55] something that uh people call the
[114:57] iceberg technique. Now, in case you guys
[115:00] have never seen an iceberg before, I'm
[115:02] Canadian, so we have them everywhere,
[115:04] including literally in the river across
[115:06] the street from my house. Not uh actual
[115:09] icebergs, but little ice floats. The way
[115:12] that they work is basically you have um
[115:14] a section that's visible above, which is
[115:16] usually quite massive and quite
[115:17] intimidating. And then you're like, "Oh
[115:19] my god, that's a really big iceberg."
[115:21] And then what you don't realize is
[115:22] underneath the iceberg is actually like
[115:24] two or three times as big. And so above
[115:27] the stuff that's like immediately
[115:29] visible or in our terms for the model
[115:33] accessible immediately
[115:36] is actually only a very small percentage
[115:38] of the total iceberg.
[115:42] And so in the context of our model what
[115:44] we store above which is immediately
[115:46] accessible. It's just sort of the stuff
[115:48] that's like visible to the plain eye is
[115:50] we'll store our memory. We'll store our
[115:52] claude or our agents or our gemini.md.
[115:56] We'll store our local memory as well.
[115:59] There's different types. There's global
[116:00] local. We'll store all of the current
[116:03] task context. So everything that our
[116:04] tools are doing then maybe any active
[116:06] file content. Okay. And so all of this
[116:09] stuff here is like basically always
[116:10] accessible to us. It's literally just in
[116:12] our prompt. But then what people are
[116:15] doing to reduce the total number of
[116:16] tokens that they require is they're
[116:18] abstracting away everything else. and
[116:19] then they're just making it accessible
[116:20] to the model if it needs it. What I mean
[116:23] by that is instead of putting all of the
[116:25] files in your codebase in a prompt, what
[116:27] it does is it gives you a tool called
[116:29] read. And what read can do is read can
[116:31] at any time read a file. Okay, but
[116:34] instead of putting the entire file in
[116:35] there for now, all it does is it just
[116:37] puts the titles. So if you say, "Hey, I
[116:39] want you to grab the um information on
[116:41] the iceberg technique." And then in your
[116:43] workspace, you have a file called
[116:44] iceberg technique.md,
[116:46] it'll know. It doesn't actually have to
[116:48] read all of the files in your workspace.
[116:49] It only has to read iceberg.mmd. Right?
[116:52] Same thing with the full codebase. You
[116:54] have tools like gp and glob. And these
[116:56] tools are sort of analogous. Instead of
[116:58] reading a whole file, what this does,
[117:00] this allows you to hone in on a specific
[117:01] segment of text. You know, if this is my
[117:03] entire code file, okay, and
[117:05] hypothetically, let's just say it's
[117:07] really big and you know, there's there's
[117:09] a lot of stuff, but the only thing that
[117:11] I actually care about is this little
[117:14] segment over here. Then why would I load
[117:18] all of the I don't know 10k tokens? I
[117:22] don't need to. Okay. Realistically, what
[117:24] I can do as a smart model is I could use
[117:26] GP and Glob these tools to maybe hone in
[117:29] only on this segment over here, which is
[117:32] I don't know, let's just say 2K tokens.
[117:35] And because it contains some of the text
[117:37] before and some of the text after, you
[117:39] know, this still usually gives us enough
[117:40] context to um tell the model what it
[117:43] needs in order to to finish its
[117:44] function. Now you also have web data via
[117:47] web fetch. Web data is pretty cool. You
[117:49] can kind of think of it as the same
[117:50] thing. Obviously it doesn't have access
[117:51] to the whole internet, but it can make
[117:53] search queries, right? And so because
[117:55] it's able to use some general reasoning
[117:57] when you say, "Hey, what's the iceberg
[117:58] technique?" First, it's going to start
[118:00] by looking for file contents called
[118:02] iceberg technique. It can't find any,
[118:04] maybe it'll quickly type fisceberg to
[118:06] look through the codebase. If it can't
[118:08] find that, you know, maybe it'll look
[118:09] through some other things like uh memory
[118:11] files and the skills library. But if it
[118:13] can't find that, it'll say, "Okay, cool.
[118:14] So, we don't have this in our in our
[118:16] context window right now, and we don't
[118:18] even have access to it within our space,
[118:20] but it's probably somewhere on the
[118:21] internet. So, I'm just going to Google
[118:22] iceberg technique. And then what it'll
[118:24] do is it won't even take the entire
[118:26] thing. It'll just grab tople links, and
[118:28] then it'll look at the URL, you know,
[118:30] uh, and the URL is about, guess what?
[118:33] Icebergs. I'm going off the map here,
[118:35] but hopefully you guys understand what I
[118:36] mean. Then, if there's three URLs, one
[118:38] of them is about icebergs, then instead
[118:39] of reading all of them, it's only going
[118:40] to read this one. And so it's sort of
[118:42] like a successive narrowing of the lens
[118:45] until eventually it gets to, you know,
[118:47] what you want. And it doesn't load the
[118:50] entirety of all of the context, but it
[118:52] just has the opportunity to select all
[118:54] of this. You know, it starts here, it
[118:56] goes here, goes here, goes here, goes
[118:58] here, goes here, and then goes here, and
[118:59] then finally it's at its goal. You can
[119:01] do the same thing in a variety of other
[119:02] ways. You can use bash, get history, and
[119:04] so on and so forth. But um essentially
[119:07] what you want to do is instead of
[119:08] storing all of the information like the
[119:10] full code base, the file contents, the
[119:12] web data, all the git history, all the
[119:13] skills and everything like that, what
[119:15] you do is you just store the ability to
[119:17] access it on demand. And then inside the
[119:20] context, okay, the tiny chunk that you
[119:22] do, I mean in this diagram it says 10%
[119:24] 90%, you know, I think in reality it's
[119:26] probably closer to 20 80 or maybe 30 70.
[119:30] Um here's where you store stuff that's
[119:31] just like it it needs to be the same all
[119:34] the time. it always needs to be present.
[119:36] There always needs to be some sort of
[119:37] patterns that are learned, some sort of
[119:39] active file context or uh contents or
[119:42] current task context. You can think of
[119:44] this as the difference between naive
[119:46] versus strategic context loading. Now,
[119:49] way back in the day, and when I say way
[119:51] back in the day, I mean like 2023, good
[119:53] god, I'm getting old. You know, when you
[119:55] were working with an agent, you would
[119:57] dump in the whole codebase. You would
[119:59] honestly just copy and paste everything
[120:00] and just hope to God that it that it
[120:02] knew what it was doing. Obviously,
[120:04] because this was extraordinarily
[120:06] infeasible, there were so many tokens in
[120:10] the context, tons of it was lost and you
[120:13] routinely ran out of context limits.
[120:15] Well, nowadays what we've done is we
[120:17] basically built in a whole tool stack
[120:18] where instead of all of the file, you
[120:20] just read selectively and only the
[120:22] relevant functions. You know, you have a
[120:24] clawmd which is basically a compression
[120:26] a compression function which just stores
[120:28] your preferences. skills instead of all
[120:31] being read, you know, we only read a
[120:32] specific segment of them. This is
[120:34] technically called the YAML front
[120:36] matter, which is just a tiny little
[120:37] section at the beginning of the skill.
[120:39] You can actually see this if we go back
[120:40] to the skill.md and I make this visible,
[120:42] right? Only this section up here is
[120:45] actually, you know, you can actually see
[120:47] this up here. Um, the YAML front matter
[120:49] is this segment. Only this segment is
[120:52] actually loaded into context until you
[120:53] ask for, you know, more information
[120:55] about create proposal. And that's just
[120:57] because this little space invader sees
[120:59] you. It has the ability to call a create
[121:01] proposal skill, but it doesn't need to
[121:03] know all the rest of this because what's
[121:05] realistically going to happen? Well, 90%
[121:06] of the time you won't even ask. There's
[121:07] so many other skills it'll probably be
[121:08] using. We're also now doing things like
[121:11] summarizing tool results. So instead of
[121:13] storing the entire thing in your
[121:15] context, you know, we just store like a
[121:17] very short summary of basically the
[121:18] input output. So way back in the day, I
[121:21] used to think that all of an agent was
[121:23] really just its intelligence. So it was
[121:25] just the core model, right? Which in
[121:27] this case would be Opus 4.6.
[121:30] But what I've come to quickly realize is
[121:32] although models themselves are quite
[121:34] intelligent, it's really the
[121:36] architecture that we've wrapped around
[121:38] it. I want you to pretend this little
[121:39] space invader, it's kind of now in, I
[121:42] don't know, a house of some kind. And
[121:44] the house has a little chimney with a
[121:46] fireplace. It has a little place where
[121:48] it could, I don't know, roast a nice
[121:50] turkey. You know, has a nice bed that it
[121:52] can go to sleep in every night. you
[121:54] know, this agent by itself probably
[121:56] wouldn't last super long out there on
[121:57] the savannah. But because we've built
[121:59] all this infrastructure around it
[122:00] because we built roads for it, we built
[122:02] ways for it to communicate and stuff
[122:04] like that. You know, it's capable of
[122:05] actually doing a lot of very
[122:06] economically valuable work for us. And
[122:09] so, just like human beings, uh, way back
[122:11] in the day had to conceptualize the idea
[122:13] of, I don't know, like a like a spear or
[122:16] something to hunt, um, um, saber-tooth
[122:20] tigers on the planes, you know. So too
[122:24] do these agents use tools effectively to
[122:27] solve problems in their environment and
[122:29] ultimately get us the users what we
[122:31] want. Now obviously there's context
[122:34] window management like we just talked
[122:36] about and that's for optimizing the
[122:38] usage of a specific model okay in the
[122:41] choice of a model. But there's also the
[122:44] ability to choose different models for
[122:46] different purposes. Now, throughout most
[122:48] of the course so far, what I've done is
[122:50] I've just used mostly naive like Opus
[122:52] 4.6 agents in order to spawn other Opus
[122:55] 4.6 sub agents. And that was mostly for
[122:58] capability sake because at least in my
[123:00] case, cost is not a a concern. And I
[123:02] really do want to eek out the marginal
[123:04] quality um benefits wherever possible.
[123:06] But there are a lot of cases
[123:08] specifically like enterprise and and big
[123:10] infrastructure ones where, you know,
[123:12] people are actually comfortable making a
[123:13] miter tradeoff for quality. I'm going to
[123:16] draw another one of my famous graphs.
[123:18] Um, people are comfortable making, you
[123:20] know, a trade-off in terms of, you know,
[123:23] cost and then quality. Now, in a lot of
[123:26] disciplines out there, biology, physics,
[123:28] chemistry, and stuff, there's this this
[123:30] idea of like this inverted U curve, and
[123:33] you can call it whatever the heck you
[123:34] want. Um, I think the actual name is the
[123:37] Yorky's Dodson curve.
[123:40] And what this is is this is basically
[123:42] like the the optimal point
[123:45] that combines two different factors. And
[123:48] so in our case, if we're optimizing for
[123:49] both cost and quality, okay,
[123:51] simultaneously, not just cost, not just
[123:53] quality, you can imagine that the
[123:55] optimal place to choose on this graph is
[123:58] probably going to be something like over
[123:59] here. It's going to be it's going to be
[124:00] somewhere around here. If you wanted to
[124:02] minimize cost exactly, we'd probably go
[124:04] like way over here. But obviously, we
[124:05] care about quality as well. So, we're
[124:07] going to push up a little bit, right? We
[124:09] don't want this point because one, it
[124:11] costs a lot. And then boom, two, the
[124:14] quality is pretty low because despite
[124:15] the fact that quality is really really
[124:17] high, um you know, cost is like many
[124:19] many times higher than it was over here.
[124:21] And so this is sort of like our our
[124:23] optimal point. And you know, a lot of
[124:25] large enterprises, since we're dealing
[124:26] with hundreds of millions of dollars
[124:28] here, um, are actually comfortable
[124:30] making a little trade-off where, you
[124:31] know, if this quality is 85% and then
[124:34] this one here is, I don't know, 95%,
[124:36] they're okay taking like a like a 10%
[124:39] hit here, if it means that they also
[124:41] reduce their costs by, I don't know,
[124:43] 40%. Or something like that. And this is
[124:46] really where all this stuff comes in.
[124:47] Okay? So, I don't mean to talk your ear
[124:49] off here. It's not super important, but
[124:51] uh basically uh what a lot of people
[124:52] have taken to doing now is doing a 60/30
[124:54] 10 rule where they'll use some top level
[124:58] agent router which is sort of like the
[125:00] orchestrator in our multi- aent Chrome
[125:03] window example. And what that agent
[125:05] writer does is it calls dumber models
[125:08] and then it assigns different strengths
[125:10] to tasks. So that you know if you're
[125:13] giving it a really simple task and you
[125:15] say hey you know I just want you to
[125:16] classify this into one of three
[125:18] categories and it's really dumb. It's
[125:21] like you know red, blue or green, angry,
[125:24] serene or or healthy or whatever the
[125:27] heck. um you don't have to use like
[125:29] Opus, which is like space age
[125:30] intelligence and costs you a ton more in
[125:32] token cost in order to get that done.
[125:35] Instead, you know, you can get all the
[125:36] way down to like a haik coup or maybe a
[125:38] Gemini flash model or something like
[125:40] that. Likewise, if you have some other
[125:42] task here and maybe that task requires a
[125:43] lot of um I don't know, research or
[125:45] something and you say, "Hey, I want you
[125:47] to go and compile 200 million tokens
[125:49] worth of stuff and then um give it to me
[125:52] in a big report." Well, you know, you
[125:53] probably don't want the dumbest model to
[125:55] do that for you, but you also don't need
[125:56] the most expensive model. So maybe
[125:57] you'll use something like a sonnet or
[125:58] like a a lower level GPT model which
[126:00] might cost 2 or3 million uh $2 or $3 per
[126:03] million tokens instead. And so you know
[126:05] this allocation where you have 60 3010
[126:08] if you think about it like sort of a pie
[126:09] chart. Um what you do is you designate
[126:12] you know the vast majority of your token
[126:15] usage to stuff that is in that first
[126:18] category which is kind of you know
[126:20] dumber. And then what you do is you do
[126:22] the other 30% or so
[126:25] in sort of that mid tier and then your
[126:27] really really really smart models, you
[126:30] know, they do the highest level tasks.
[126:32] And basically what'll happen for the
[126:33] most part is this would be, you know,
[126:35] your Opus 4.6 or your Gemini
[126:39] 3.1 or your GPT 5.4. It would be
[126:43] responsible for routing decisions and
[126:45] obviously you want the smartest model
[126:46] possible for that. But all of the heavy
[126:47] lifting, all the contacts and stuff like
[126:49] that is um through sub agents that are
[126:51] spawned either Haiku, Sonnet or uh you
[126:55] know, I don't know if if you wanted to
[126:57] do a really smart call, then you'd
[126:58] obviously spawn an Opus sub agent like
[126:59] me as well. And if you do all this, you
[127:02] could significantly reduce the cost. I
[127:03] mean, like just just think about it
[127:04] mathematically. If previously you were
[127:06] doing 100 million tokens times, you
[127:09] know, $5 per 1 million tokens, [gasps]
[127:13] what's the cost there? Well, that's
[127:15] obviously going to cost $500. And so
[127:17] that's like your opus only, right? But
[127:19] if you did 10 million * $5 plus
[127:25] 30 million * $3 plus 60 million * $1.
[127:32] Well, what's the total cost going to be
[127:33] now? Well, it's going to be 60 + 90
[127:39] + 50 or in total 200. And so you know
[127:44] 200 expressed as a fraction of 500 is
[127:48] 40% of our total cost and we will have
[127:52] just saved you know 60% with probably
[127:54] minimal impacts on quality because the
[127:57] things that we're now spawning you know
[127:59] dumber agents um to do are things that
[128:02] to be honest the quality was already
[128:04] okay a few generations ago back at the
[128:06] haikus and the sonnetss. So just to give
[128:07] you an example from something that I do
[128:09] pretty often, okay, which is going to be
[128:11] some form of lead scraping. What you can
[128:14] do is you can actually traverse a very
[128:16] large portion of the internet using a a
[128:18] relatively dumb model. These haiku
[128:20] models, what these do is these um scrape
[128:23] a vast majority a vast amount of
[128:26] internet data. Okay, all of the code of
[128:28] uh I don't know, let's say 10,000
[128:30] websites or something. And then in doing
[128:32] so, they just like that little
[128:33] magnifying glass, use some sort of GP or
[128:36] extraction prompt to look for things
[128:38] that are formatted like email addresses.
[128:40] So if you have like something and then
[128:42] it's an at and then it's a, you know,
[128:44] the term gmail.com, odds are this is a
[128:46] real email address, right? And you store
[128:48] that to a database. And because this is
[128:50] just such like a mass data application,
[128:51] you use haik coup that drives the cost
[128:53] really, really low. Well, then maybe um
[128:56] the actual enrichment point, you know,
[128:58] takes significantly more intelligence.
[129:00] And so maybe here we'll use sonnet and
[129:01] it'll cost us uh 0.008 cents per lead.
[129:05] 08 per lead. The actual outreach part is
[129:08] mostly templated. So we'll use sonnet
[129:10] for that as well. And then maybe at the
[129:12] end we just have a quality review step
[129:13] to make sure things aren't absolutely
[129:15] nuts. Well, when you do it this way, um
[129:17] you know, the math ends up being uh 08
[129:20] plus 05. So that's 013 plus 01.014 plus
[129:25] 015 is 029. And then if you were to go
[129:29] 100% opus, then it would be I don't know
[129:32] uh about 12 cents or so per lead. And so
[129:34] on a list of let's just say a thousand,
[129:36] which is approximately how much I'm
[129:38] sending a day right now. I'm much
[129:40] farther down than my maximum. But if we
[129:42] multiply all these together, we move
[129:43] that one to three decimal points over to
[129:46] the left. Then that um ultimately would
[129:48] be $15 a day
[129:51] or you know $450 a month. Well, instead
[129:55] I'm doing that for literally one quarter
[129:57] of this or something like, you know,
[130:00] $120 a month instead. Obviously, I'd
[130:03] much rather the latter. And if my
[130:04] quality is only going down a few
[130:06] percentage points because of that
[130:07] Yorky's Dodson curve, you know, I'm okay
[130:10] being over here instead of over here
[130:13] because this gap to me is fine on tasks
[130:15] that aren't super high and high quality.
[130:18] Then, uh, this is a very, very efficient
[130:20] stack. And you know, the bigger and
[130:22] bigger my company gets, whoever I'm
[130:24] working with, the more the cost per lead
[130:27] is going to be important versus the
[130:29] actual quality. I've included just a
[130:30] little LLM API pricing cheat sheet. I
[130:33] don't expect this to be super relevant
[130:35] or useful to you guys. There are a lot
[130:37] more models for OpenAI and Google, but I
[130:40] am actually using um not this model
[130:43] series anymore, but this one here for
[130:45] some queries. I'm also using the Flash
[130:47] model series for some queries as well.
[130:49] And then what's really cool is a lot of
[130:51] them offer what's called a batch API now
[130:54] where you can submit a bulk number of
[130:56] requests over um simultaneously and then
[130:59] if you're comfortable waiting like a day
[131:01] or so what the companies do is they
[131:04] batch it and then they serve your
[131:06] requests during periods in which they
[131:08] have very low inference or low
[131:10] competition. So maybe in the middle of
[131:11] the night or something like that. And in
[131:13] doing so they actually get to load
[131:14] balance. Like if you think about it,
[131:16] like if this is like a day and then this
[131:18] is their load like on their servers and
[131:20] their neural networks and stuff, you
[131:21] know, it probably peaks somewhere around
[131:23] like noon and there's probably like a
[131:25] couple things and then it's like low
[131:26] during the during the day, right? I
[131:28] don't know, this is like 4:00 a.m. What
[131:29] they'll do is they'll actually take all
[131:32] of your queries, batch them, and then
[131:33] they'll just like run them over here
[131:35] when there's very little competition.
[131:37] And then later when you know the things
[131:38] go up and stuff like that again, that's
[131:41] okay. And in doing so, what they want to
[131:42] do is they want to shift some of the the
[131:44] really top end of all of these users um
[131:47] to the low end to basically fill this so
[131:49] that they have a lot more like
[131:50] dependable load instead of these like uh
[131:53] jagged peaks and whatnot.
[131:55] [sighs and gasps] But anyway, don't
[131:56] worry too much about that. I just wanted
[131:58] to cover um some LLM pricing principles
[132:00] as well, so you guys know not only how
[132:02] to manage your contacts better, but also
[132:04] how to um save, especially when you get
[132:07] into more sophisticated multi- aent
[132:09] setups like I've been showing you. And
[132:10] that's it. Thank you guys very much for
[132:12] watching this video end to end. If you
[132:14] guys have made it all the way to this
[132:15] point in the course, you're part of like
[132:17] the 2 or 3% that actually do. Um, I'd
[132:19] really appreciate a big solid if you
[132:21] could do me a favor and subscribe to the
[132:22] channel. Something like 70% of you
[132:24] aren't, which significantly hurts my
[132:26] reach and uh, despite me hating asking
[132:28] for it, it does help the channel grow.
[132:30] So, if I've given you guys any value
[132:31] whatsoever, please do that. You can also
[132:34] send me over a comment down below asking
[132:36] any question about any point in the
[132:38] video. Um, I'm much more engaged than
[132:39] the average YouTuber. So, the
[132:41] probability that I will reply is pretty
[132:43] pretty far up there, I would say,
[132:44] statistically. Um, if you guys have any,
[132:47] you know, suggestions for future videos
[132:48] or future courses, please drop them down
[132:50] below as well. And above all else, keep
[132:52] learning and growing with AI agents.
[132:54] This is by far the biggest and most
[132:57] impactful of economic changes that I
[132:59] think any of us will see in our
[133:00] lifetime. It's a blessed time to be
[133:02] alive in general. You might as well not
[133:04] waste it. Make the most out of it. All
[133:06] right. Um, thank you very much. Feel
[133:07] free to use the chapter headings to
[133:08] revisit any section in the course. And
[133:11] uh looking forward to seeing all y'all
[133:12] in the next one.