[00:00]
Few people know, but modern AI systems are really silly. How? Well, imagine having a Michelin-star chef being asked for a simple peanut butter sandwich. That’s weird, but okay. Now,
[00:13]
the chef says, you’ll need to wait just a bit, because I am going to start planting peanuts, wait six months, harvest, churn some peanut butter, and then get to work on your bread.
[00:25]
That sounds really silly, and that is exactly what modern AI systems like ChatGPT and Gemini do. When they need to recall a simple fact, like who Alexander the Great was,
[00:37]
something crazy happens. They go through complex reasoning layers and reconstruct everything from scratch every single time. That is crazy. Now I have an amazing research paper for you here from folks at DeepSeek AI, and this is a
[00:54]
piece of technology that might underpin most if not all of the amazing AI systems of the future. Now every now and then we are going to look at a figure, but for the rest, I am going to bring my
[01:07]
physics simulations and all the goodness we talk about around here. Apologies for that. Okay, so this is a massive waste of compute. But why does this happen? Well,
[01:19]
standard transformers are a kind of neural network that is inside nearly all modern AI assistants. And here is the problem: they lack a simple and cheap way to just look things up.
[01:32]
Whatever the question is, the answer is a huge bunch of dense mathematical calculations. From scratch. Yes, it is literally planting that peanut when you ask for a sandwich.
[01:44]
Now in this work, DeepSeek introduces Engram. With this, they are giving our tired little chef a pantry. Cutting edge technology brother! Instead of
[01:56]
growing that peanut butter sandwich from scratch, it now just grabs the ingredients from the pantry. I’ll explain to you how exactly they did it in a bit. Now, this makes the AI way more efficient, okay, I expected that. But, what? Are you seeing what
[02:15]
I am seeing? Now this I did not expect at all. So here comes the surprise. Now hold on to your papers Fellow Scholars, because when taking away some of the AI's complex reasoning parts, known as
[02:29]
mixture of experts, MoE. Then, replacing it with the pantry actually makes the AI smarter. Lower is better here on the loss curves. And not just a little, this is significantly smarter. Those dots
[02:44]
dipping way down show that this hybrid chef makes far fewer mistakes than previous techniques. It achieves a perfect balance of active cooking and just grabbing from the pantry. Genius.
[02:58]
But that is not the only surprising thing in this paper. They also added a way for the AI to check the ingredients before using them. You don’t want rotting fish in your strawberry jam. To
[03:11]
ensure this, they created a context-aware gating mechanism. The current context is the dish being cooked. Now here, this is compared against the retrieved memory, the jar from the pantry. If
[03:25]
the jar's contents don’t agree with the dish, the gate drops to zero, throwing the ingredient away. Bye bye rotting fish! This mechanism lives right here, inside this jolly little dot product.
[03:39]
Now let’s see how it actually performs against the current systems. I’ll tell you exactly what is going to happen now. What happens in nearly all research papers with something new. It
[03:51]
does something, it is compared to previous methods, and it’s better at some things, worse at others. And then you sit down and you do your analysis. Okay, let’s see…wait what? What
[04:07]
just happened here? The new engram technique makes the neural network better…everywhere. Absolutely everything is measurably better. This is an absolute miracle work.
[04:19]
The engram model is actually better on every single benchmark compared to Now this is an amazing life lesson too. How? Well,
[04:33]
essentially what DeepSeek does is automates the easy part, and focuses on the more difficult tasks. No wonder it works so well! What a time to be alive! We
[04:45]
can learn so much from these research papers, and not just about AI, but about life itself. Okay, now I’ll tell you how this works, and it turns out, there are more surprises ahead.
[04:58]
Dear Fellow Scholars, this is Two Minute Papers with Dr. Károly Zsolnai-Fehér. Okay, so how does it do this magic? Well, it uses what they call n-gram embeddings combined with multi-head hashing. Okay, what the heck does that mean? Well,
[05:13]
in the kitchen, the chef looks at the order ticket, sees a 3-word phrase, and instantly knows exactly which shelf in the pantry has the premade sauce, and grabs it quickly.
[05:26]
And I think this also shows us that there are simple and basic ideas in AI that we haven’t found yet. I mean, this thing is basically a look up table. It is as simple as it gets,
[05:39]
and it makes everything more efficient and better across the board. Just think about it: we removed 20 or 25% of the smart experts in this little virtual brain,
[05:51]
put a spreadsheet there, and it got better! I mean what? Crazy. And I love how we have a little better understanding of the AI system itself. Usually, no one knows what is going on inside,
[06:06]
but here. Look. When they switched off the engram memory during testing, the AI’s ability to answer trivia went down 70%. But its reading comprehension remained at 93%. Why?
[06:21]
Well I think this shows that AI split its brain, and it’s using the new part just to store facts. Just think about it. When they locked the pantry door during testing his ability to
[06:35]
understand a recipe stayed at a massive 93%! What does that mean? It shows the chef split the work perfectly. He used the pantry strictly as a storage shelf
[06:47]
for memorized ingredients, but he can still cook an amazing meal. I think this is going to lead to even cheaper and even smarter AI systems, and this will be an important part of why we will all get more systems that we can actually own,
[07:04]
no subscriptions, these run in our pockets super fast, mostly for free. Okay, now not even this technique is perfect. One limitation is that if you put the engram module too deep in the network, it gets less accurate because the model has already wasted
[07:21]
time processing what is being asked. Of course, there is no need to look up what you already computed. I think this is common sense at this point. Our chef has to check the pantry at the
[07:33]
start of the shift. If he checks it after the food was served, the pantry is completely useless. A really advanced research paper explained in simple words. We are Fellow Scholars,
[07:46]
and that’s what we do here. And we have a growing club. I’ll continue in a moment, but you know who is also watching us? The one and only Larry Wheels. Yes. He is one of our OG Fellow Scholars,
[08:01]
doing some Scholarly work between two hard sets of bicep curls in the gym. You think I am kidding? I am not. Link is in the description. Reading his comment made me instantly
[08:13]
more muscular. So much value. Huge respect to Mr. Wheels! Honored to have you here. And here comes the best part. I think this will be a part of every major AI system,
[08:26]
and it is knowledge out there for free for all of us, and now you know exactly how it works! No nonsense where everything is hidden in a proprietary system that costs 300 dollars per month to run. Nope. All free for all of us. Glorious. An epic paper.
[08:45]
Now, as our chef does, I took a bit longer to cook this video. But I promise that I did not put together my computer from scratch before starting. So I took some more time
[08:57]
to make sure you get a better video. If you feel this is the right way of doing that, subscribe, hit the bell and leave a really kind comment. And you can also check out Lambda with our link
[09:09]
in the description because it is an excellent way of running DeepSeek privately, I do it too.