[00:00] Let’s see what is going on here. This is me&nbsp; around 9am. A bit wobbly, steps are unsure,&nbsp;&nbsp; yup, that checks out. Now then, give me my&nbsp; fake badge. Thank you sir. Hehehe, no one&nbsp;&nbsp;
[00:16] noticed. Now let’s proceed to the next step of my&nbsp; mastermind plans. Let’s eat all their food. Wait,&nbsp;&nbsp; they noticed. Proceed to the next&nbsp; step. What was that? Oh yes, run!
[00:34] Now, jokes aside, look at that. Sign up&nbsp; for this one baby. Oh yes, please mow&nbsp;&nbsp; my lawn. That is excellent. Rake the leaves!&nbsp; Perfect. Hey, don’t slack off, that’s my job!
[00:48] Okay, so what is going on here. Let’s start with&nbsp; the good news, this is a new teleoperated robot&nbsp;&nbsp; controller and more. They call it Sonic. Now&nbsp; the work here is not the robot, but the software&nbsp;&nbsp;
[01:06] controlling it. At least in this footage, watch&nbsp; until the end and you might get surprised. This&nbsp;&nbsp; means there is a human performing these movements,&nbsp; and the robot is able to understand these motions,&nbsp;&nbsp;
[01:20] and then translate them to a bunch of joint&nbsp; positions in 3D space. It’s kind of insane&nbsp;&nbsp; that this is possible. But it will just get&nbsp; better and better as we continue the video.
[01:34] So, before you ask, yes it can do kung&nbsp; fu. Provided that you can do kung fu. It&nbsp;&nbsp; understands whole body movement, so you can get it&nbsp; to crawl into some space you don’t want to go to.&nbsp;&nbsp;
[01:49] And that is super useful, people are already&nbsp; using robots for that. Why? Well, chiefly,&nbsp;&nbsp; for exploring under explored and dangerous&nbsp; areas. This means tons of useful applications,&nbsp;&nbsp;
[02:04] for instance, a variant of this could&nbsp; help save humans stuck under rubble,&nbsp;&nbsp; or perhaps later, even explore other&nbsp; planets without putting humans at risk. But that’s still nothing. Because this is a&nbsp; multimodal system. Meaning that the input can be&nbsp;&nbsp;
[02:22] almost anything. So, you say that I don’t have to&nbsp; pretend to mow the lawn to actually mow the lawn,&nbsp;&nbsp; because where is the fun in that? Well, just&nbsp; tell it to do that. Can you? Well, currently,&nbsp;&nbsp;
[02:40] for simpler tasks, like moving around or behaving&nbsp; like a monkey, yes you can! Absolutely incredible. And I love how expressive it is.&nbsp; You can ask it to walk happily,&nbsp;&nbsp;
[02:53] And you know, just the fact that it is stable&nbsp; and does not fall is remarkable. Previously,&nbsp;&nbsp; even in simple characters in simulated worlds, you&nbsp; needed thousands and thousands of tries to teach&nbsp;&nbsp;
[03:10] them to just be able to walk without falling.&nbsp; And now, this, is a huge leap forward. Wow. But it gets better, we said multimodal.&nbsp; Yup, that means that the input can also&nbsp;&nbsp;
[03:26] be music. I’ll show you the dancing, but&nbsp; not the music because of Youtube reasons,&nbsp;&nbsp;
[03:38] And we haven’t even talked about the most insane&nbsp; part of the whole thing. Now hold on to your&nbsp;&nbsp;
[03:50] papers Fellow Scholars, because this runs with&nbsp; about 42 million parameters. That is a neural&nbsp;&nbsp; network so simple, it can run so easily on your&nbsp; phone it barely notices it. It may even run on&nbsp;&nbsp;
[04:07] your toaster these days. That size is absolutely&nbsp; nothing. This is an incredible achivement.
[04:19] Okay, but how? How is that even possible? Dear&nbsp; Fellow Scholars, this is Two Minute Papers with&nbsp;&nbsp; Dr. Károly Zsolnai-Fehér. Well, first, it&nbsp; looked at 100 million frames of human motion&nbsp;&nbsp;
[04:32] to understand what we do and how we do it. The&nbsp; incredible thing is that this system does not&nbsp;&nbsp; require human-made action labels, so we don’t have&nbsp; to explain our movements. It just watches the raw&nbsp;&nbsp;
[04:45] motions and figures out how to transition&nbsp; between tasks without any unnatural pauses! So then, your multi-modal input goes in, a video&nbsp; of you, your voice, music, or just text. A motion&nbsp;&nbsp;
[04:58] generator turns these into human motion, and the&nbsp; human encoder processes it into a latent space,&nbsp;&nbsp; and then a quantizer converts it to universal&nbsp; tokens. Once again, universal tokens, that is key,&nbsp;&nbsp;
[05:13] you’ll see a bit later. Then, the decoder&nbsp; translates these tokens into motor commands. But there is a big problem. Learning to convert&nbsp; one to the other is super hard. First of all,&nbsp;&nbsp;
[05:30] robots do not work like humans, that&nbsp; is one of the fundamental challenges. So if the user commands you to turn around, it&nbsp; should be turning around. Okay, sure. But how&nbsp;&nbsp;
[05:43] fast exactly? You don’t want to try to turn 180&nbsp; degrees too quickly, because you would fall apart. To solve this, in their research paper, they&nbsp; propose what they call a root trajectory&nbsp;&nbsp;
[05:57] spring model. This dampens sudden, quick user&nbsp; commands so the robot does not get injured.&nbsp;&nbsp; Yes, robots can get injured&nbsp; too, which is kind of hilarious.
[06:10] Now there is an exponential term as a function of&nbsp; time. What is that? That is a physical brake. As&nbsp;&nbsp; time increases, this term rapidly shrinks to 0,&nbsp; which forces the whole mathematical expression to&nbsp;&nbsp;
[06:26] decay smoothly. This serves two goals: one,&nbsp; the robot does not injure itself and two,&nbsp;&nbsp; it will settle at a target position without&nbsp; oscillating back and forth forever. Nice.
[06:41] Now, do the dampening too much, and&nbsp; of course, you’ll get a little slug&nbsp;&nbsp; that can’t get anything done, so it’s&nbsp; really tough to do well. Well done folks. Now, all this took 128 GPUs and 3 days to&nbsp; train. That is expensive. But here’s the key,&nbsp;&nbsp;
[07:01] after the training is done, the final product&nbsp; is so lightweight, we don’t need this kind of&nbsp;&nbsp; hardware to run it at all. In fact, all&nbsp; of the models showcased in these videos&nbsp;&nbsp; will be given to all of us for free, forever.&nbsp; They run on your phone, easy-peasy. That is&nbsp;&nbsp;
[07:21] incredible. Open research for the benefit&nbsp; of humanity. Love it, thank you so much. This project is led by professor Zhu and Jim Fan,&nbsp;&nbsp; who I love dearly. Jim started the humanoid&nbsp; robots lab at NVIDIA just 2 years ago,&nbsp;&nbsp;
[07:38] and they are raining research papers on us,&nbsp; breakthrough after breakthrough. Insanity. And to compress all this human movement&nbsp; knowledge down into a tiny little AI&nbsp;&nbsp;
[07:51] controller that can be used by any of&nbsp; us is simply a stunning achievement. It turns out, training a good AI requires coding&nbsp; good thinking into a machine. But, surprisingly,&nbsp;&nbsp;
[08:04] we ourselves can also learn a lot of good&nbsp; life advice from this kind of thinking too. For instance, the model compresses a messy,&nbsp; diverse soup of inputs into a kind of pure,&nbsp;&nbsp;
[08:17] abstract token. You know, in life,&nbsp; when asking other people for advice,&nbsp;&nbsp; you will inevitably hear everything,&nbsp; and its opposite too. That is also a&nbsp;&nbsp; big soup of inputs. But try to look at all of&nbsp; them, side by side, and you’ll find that they&nbsp;&nbsp;
[08:34] often share an underlying truth. This works,&nbsp; as is showcased by this incredible project too. And note that this work is not the end of&nbsp; anything, this is just a start. An early&nbsp;&nbsp;
[08:48] work at a nascent area. Two more papers&nbsp; down the line, and I really hope this is&nbsp;&nbsp; going to start folding my laundry and cooking&nbsp; my lunch. That would be amazing. What a time to&nbsp;&nbsp;
[09:01] be alive! And this is not some proprietary&nbsp; nonsense, this is open knowledge and open&nbsp;&nbsp;
[09:26] just dropped. If you are interested in hearing&nbsp; more hopefully soon, subscribe and hit the bell.