NVIDIA's New AI Just Changed Everything

[00:00] Remember that most AI systems are proprietary,  we have to pay a subscription for them,   and no one knows how they work or  what data they were trained on? Well, now hold on to your papers Fellow  Scholars and check out this incredible work,  

[00:18] and when I first saw it, my jaw hit the floor.  They absolutely knocked it out of the park.   They spilled all the secrets. This is an AI  assistant that is free for all of us forever,  

[00:32] but not just the model itself. They also  gave us a 51-page research paper which   might be the holy bible of creating  such a system for now. Why is that?

[00:45] and the dataset it was trained on as well. That is extraordinary. Usually  something is always missing. Not here.

[00:59] They call it Nemotron 3 Super and we are going  to find out whether it is indeed super or not. Okay, so in goes 25 trillion  tokens as training data,  

[01:12] and out comes a 120 billion parameter  AI assistant that is how smart exactly? It roughly matches the best closed frontier  models from about a year and a half ago.

[01:28] Note that those models cost billions  of dollars to train and every detail   about them was kept in secret. And now, we just get this kind of   stuff for free. That is mind blowing. This is  amazing for us, consumers and Fellow Scholars.

[01:45] So as you see, it is really smart. Up with some  of the best open models out there in most tests,   but note that it’s still a bit behind some  areas. Here’s something that surprised me: 

[01:58] in this result, they showcase  two versions of the new model,   BF16 and NVFP4. They perform roughly the same in  terms of accuracy, so why the big fuss about this?

[02:10] Well, look at this. Holy mother of papers.  Wow. Well, the NVFP4 version is about 3.5 times   faster than their other model, and it is up to  7 times faster than similarly smart open models.

[02:28] So the story is not just the similarly smart part,   the story is that it is 7 times faster  while it is similarly smart. Goodness.

[02:41] Okay, so how on Earth did they  do that? So here are 4 secrets   they gave us from the paper, in very simple words.  Dear Fellow Scholars, this is Two Minute  Papers with Dr. Károly Zsolnai-Fehér.

[02:53] Okay, NVFP4. What is that? This is a way for  speeding up the AI to run a great deal faster   by essentially compressing the mathematics it  uses. Seeing a long number and rounding off  

[03:09] a few digits. You get a smaller format.  Less work! What’s wrong with that? Well,   everything. Normally, if you do that,  you lose too much accuracy and the  

[03:23] system will output nonsense. However,  here, scientists did it the smart way:   they left the most sensitive calculations  alone, and did this rounding for the rest,   where it does not cause trouble. The result  is that it runs up to 7 times faster than many  

[03:41] other techniques. And we saw that it gives  us no meaningful loss in accuracy. Magic. But there is more magic. When other  AI techniques write their answer,  

[03:54] they write it token by token. Let’s simplify  by saying word by word. Writing one word at   a time. But not this one. This one  calculates several future words at  

[04:06] once. A whole sentence! Almost. Specifically,  7 tokens. And then the system verifies the 7   tokens in one go. Another massive speed  up. They call it multi-token prediction.

[04:22] But why stop there? Let’s add even more magic!  They showcased these weird things they call  the mamba layers. What do these do? Well,   traditional AI systems have a bit of a  memory problem. They work like a student  

[04:38] who constantly re-reads the textbook over and  over again when they are given a question. Scientists at NVIDIA say, that’s not the  way to go. Memory is precious. So instead,  

[04:51] read the book only once, and take highly  compressed notes. So this kind of memory   remembers important details about the  conversation. However, it is smart enough   to throw away the filler words. Thus, this system  can process massive amounts of data efficiently.

[05:10] All this sounds glorious, but this still does  not give us a working system. Why is that?  Well, this is why. You see that there is a lot  of addition here? That is the problem. The AI  

[05:23] generates your answer step by step, and because  we rounded off the numbers, there is a little   error. That’s not a problem. Here’s the problem.  There are many steps, and the error is magnified  

[05:37] through each step. Imagine trying to walk to your  car, which is a 100 steps away, but you feel a bit   sluggish today and every single one of your steps  is a bit smaller than it was before. What’s the  

[05:51] result? Well, of course, after a 100 steps,  you are still really far away from your car! So what is the solution? Well, scientists solved  this by adding back some random noise in the  

[06:04] system. But wait, this noise is carefully  crafted in a way that it averages to zero.   So your new steps are sometimes smaller,  and sometimes bigger than they used to be,  

[06:18] but if you average them out, over a 100  steps, you will be exactly at your car.   So good! They call this stochastic  rounding and it is a genius idea.

[06:30] Now, not even this technique is perfect.  For instance, when I give it my favorite   question about assembling robotic cows,  with lots of math, I like this guy a lot,   but it thinks for almost an hour to get  me an answer for that one. That’s a lot.  

[06:47] So if I have workloads like that, I like  to run it on a much faster Lambda instance. But still I think the AI game has suddenly  changed. Closed systems used to dominate. Now,  

[07:01] not anymore. It seems to me that Jensen  at NVIDIA is not playing games here. It’s   in the news that they are going to invest tens  of billions of dollars into fully open systems   like this. I am not a money person, I don’t  know how that works exactly, but if we get to  

[07:18] own more amazing free AI systems. Well, sign  me up for this one! What a time to be alive! And there is just so much more in the paper,  I would definitely love to come back for at  

[07:32] least another video on it. Let me know  in the comments if you would like that,   and if you enjoyed this,  subscribe, and hit the bell.