Testing Nvidia's Neotron 3 Nano Omni Multimodal Model

Okay, so we saw the skate uh the girl skating, right? And we had some music in the background. She does this big jump over here and she rides in kind of into the sunset here. So let's see what happens when we drop this in here now. And remember this is an MP4 file. Okay. And we even get the video. I didn't know that. So you can see now we are transcribing video. It's registered up here. And

okay, that was pretty fast. Okay. Okay, so today we are taking a look at Nvidia's new Neotron 3 Nano Omni model. Uh, of course, this is a part of their Neotron series open-source local models you can run on your own inference if you have the hardware for that. The Nano Omni model is a 30B A3B model. It's an MOE, but this time the focus is on multimodel. And this is what we're going to test today. I'm

going to build out an app that can take in video, uh, audio, images, PDFs, and of course text if you want that. Uh, I think I'm going to test the text part over an open code so we can test some aentic tool tool calling and stuff like that. Uh, but let's just head into it and let's build out our multimodel test using the nano omni model from Nvidia. Okay, so what I set out to build here

was something pretty simple. I just one simple app that can ingest like uh like I said video, text, image, PDFs, anything. So it's like a everything app that kind of uses the nano omni uh model in the background here. So you can clearly see like we are passing in the the base URL here. And if we look at uh where did I put this model name here? So if you go to my ENV, you can see

the model name is going to be the Neotron 3 Nano Omni Reyncing 30B. Uh, of course I will be running this uh from an API on the Nvidia hardware, but uh this should be able to run if you have some own local hardware. This should be able to run on your local setup. Right? So, we're going to pass in like a simple input here. Please describe this image in rich vivid detail, capture colors, blah blah blah.

So, it's a very simple app. We're just going to take anything and we're going to use the multimodel uh features from the new nano omni model to yeah describe whatever is see. So I built this out using cloud code to set everything up. It's a react bite app. So let me just show you how it looks now. So this is what we ended up with. Nvidia Nemo train nano omni drop an image audio video get text

back. So basically now we can drop anything we want in here and the model should be able to turn anything into text right. So let's do some examples here and try all the formats we can test out here. So let's just start with this image with no text here. Uh so let me just drop this in and that should be it. You can see the image here and now the model is running image description and you

can see that was pretty fast right? Uh but again we are running this on cloud now but you can see highly detailed atmospheric digital illustration of a cyberpunk or cyber thriller scene digital warfare or hacking right and yeah we got a lot of information in the image here image uses dramatic and high contrast or color pallet change of danger and intensity. This was a really detailed description of this image and of course since this is a

local 30B model you can see how fast it was right. So that is pretty cool. So, let's find an image with some text and let's see if we can extract all the text from the image. So, I put in the intro slide here for the Nano3 Omni model and let's see. Yeah, we already have it. That was super fast. You can see Nano Omni 30B. That's good. Sub agents. Yeah, everything here looks perfect. In the bottom

right corner, we can see the Nvidia logo and yeah, perfect. Available today, April 28th. And yeah, looks like we got all the text here. And that is important when we come kind of to the next thing. So I can just clear this, right? And let's do something else. Let's try audio. Let's try that. So let's just drag this MP3 file here. It's just a short clip from um a video I found online. I think it's uh

Charlie Moist. So let's have a quick listen to it before we check out the results. >> Ever guess to the amount of money that was raised. It is incredible. This had Polish celebrities, influencers, television stations, everyone coming together, not only to watch it, but also to donate and support the money going directly to children battling cancer through this organization called Okay, so let's just drop this again so you can see how fast this is. Okay, so

we drop it down. We scroll down here. Audio transcription thinking and there we have it. So that was pretty quick, right? Yeah. ever guess uh the Polish celebrities fighting through the cancer called cancer fighters. Yeah, perfect. So audio super fast at least if you're running this on the cloud here and it should be pretty good running it on local too. So let's clear that. And now we have done images, we have done audio. Let's do a

PDF before we do a video. So I think this was maybe the most impressing thing. So when I upload this PDF now you can see uh it's already starting PDF OCR and you can see we have all the pages here and you can see it's starting on page one of 35. Everything looks very good here. So you can see we can just scroll down page two of 35. So this is super fast. Look how fast this

goes translating from uh PDFs to text. And we can let this run right. And I also like kind of the integration up here where we can kind of look at all this. That was pretty cool. So it's a bit buggy, but yeah, you get the point. You can see how fast this model is turning all of this into text from the PDF. So that was super impressive, right? Uh let's do video. That is kind of our

last thing now. So let me open up uh a video here and let's watch it and then drop it into our Nano Omni app and see what we can get out of this. Okay, so we saw the skate uh the girl skating, right? And we had some music in the background. She does this big jump over here and she rides in kind of into the sunset here. So, let's see what happens when we drop this in

here now. And remember, this is an MP4 file. Okay. And we even get a bit here. I didn't know that. So you can see now we are transcribing video. It's registered app here. And okay, that was pretty fast. So you can see the video opens with a wide shot of a concrete skate park at dust. Yes. Pants across the park. A young woman with long blonde hair. Uh white sneakers enters the frame. She steps onto a

skateboard. Follow her. She gains momentum. Flips beneath her feet. Executes a trick. Yeah. Lands Moody on the other side of the rail. The camera captures her various angles. She rides away into the sunset behind her. The background music is upbeat and and energetic. Yeah. Complimenting action blah blah blah. So yeah, what can I say? And that was pretty fast, right? That was like a not the biggest video, 7.8 uh megabytes, but it did uh translate the

or transcribe the both the frames and the audio pretty quick. So I think this is very useful. I might actually want to do something else with video uh transcription using this model uh in the future. But uh this was I think this was pretty cool and I see a lot of use cases for this actually. I might even try out myself. So that was basically the app I built with this and I set it up. It's

pretty good or it's very good I would say even and this interface here that you just drop anything in here I was very happy with. So one thing I did mention is uh of course this uh model also has reasoning right so you can kind of set your thinking budget for to whatever you want. Uh add this short uh short simple chat application here. So I'm going to say explain how quantum computing works to a 5-year-old.

Think really hard and make it as easy as possible to understand. We can just send this and you can see I checked off show reasoning here. And now we should be able to see all the reasoning tokens. uh nano uh nano omni generate and we will hopefully get an answer. So let's see what we get here. Okay, so you can say we spent 3,000 uh tokens or characters on the reasoning party here, right? And the answer

we got was a regular computer uses lights uh that are either on or off, but a quantum computer uses magic lights that can be both on and off at the same time. So you can try as uh add many possibilities together. Uh, and when you look, it picks the one answer. Okay, so that was pretty good. It's almost like a Shredderinger's cat, right? So, I kind of like that. So, you can see this model can also

do reasoning quite good here. But now, we're going to ask the question that no one almost no models can answer. So, let me bring that up. And that is the hello, I need to wash my car. It's a nice day. Should I drive or walk down to the car wash to wash the car? So, basically, no model gets this. I tried it in Opus 47 GPT 5.5 and N. So let's see here. How many tokens are

we going to spend on this? Okay, so we get if the car wash is within a short pleasant walk. Walking lets you enjoy a nice day. Yeah, no one no model gets this. Not at least yet. Uh but that could be a nice thing to ask uh the Neotron models in the future. But yeah, that is basically um the reasoning. So yeah, super cool to play around with this. So definitely check this out if you are

planning on doing any multimodel stuff. I would definitely do that. And at the end now I want to actually go to open code and just test it out a bit uh on aentic stuff. So uh to set this up on open code uh it was very easy. I just went to open code the config JSON. I passed in the blob here with the nano3 omni reasoning 30B. So this means you can see we have the Neotron

3 nano omni model here in open code. So I just want to do some quick tests on some tool calling and that is basically all I want to do. I have some documentation uh from OpenAI here GPD2 image and I have the API key. So I'm just going to instruct now OB now to build like a simple single file HTML uh page or like an app that can call the GP22 image model to generate the image.

That is basically all I'm going to try with this. And we're going to check out the speed and how well it performs on this simple task here. So I'm just going to go. We have the documentation for GPT2 image and the API key. Build a single HTML page in a dark team that uses the text to image model. Render the returned image from the API on the HTML page. Make it look good and smooth. Open it

when ready to test. So hopefully now I can see some uh tool calling here. You can see it's planning out the HTML page here. And super fast though. And yeah, it wrote the HTML. That's good. Uh, open it in a browser to test. That was super quick. Uh, open it in Chrome. I'm just going to say that. Let's see if we can do that. Tool call. Uh, yeah, that's good. Uh, enter your prompt. So, let's see

now. Uh, I don't maybe it's going to work. Let's try it out. So, let's just try uh generate. Okay, it is loading at least. A League of Legends Pokemon style TGC card of Jinx looking character. So, let's give this some time and see if this works on the first uh attempt here. Okay, we got it back. So, yeah, that looked really good, too. Look at this. So, we have the Saprite Super Mega Death Rocket. That's the

ultright. And the image looks very good. And we have the League of Legends logo. So yeah, I guess that proved that we can use uh the tool calling here. Looks very good and it was super fast. Of course, we are running this on on the cloud now. But uh I think I'm pretty happy with that. I don't I just think that proves that it can do tool calls. It did this in a oneshot, right? And I

don't think we need anything else. We could do just for fun uh make the app look uh much more professional. Let's see what happens. So, let's just follow this. We can try to run it again with a different character again. Super fast here, of course, for a 30B model on the cloud. We can refresh it. Okay, that's a bit better. I wouldn't say it's perfect. Uh, this time try a different character. Let's do Sho. Okay, so

now we have the spinning wheel here. That's a bit more professional. And let's see how it renders the result here. Okay, here we go. You can see I think this rendered it a bit better because you got it in line here. We have the sho, we have the two ship poison, the hallucinate, we have the deceive ability. And again, looks pretty good if you ask me. But uh I think we improved it a bit here. And

the point was just to say, yeah, this works super cool on open code and it's pretty fast if you ask me, especially like I said on the cloud here. Uh, but overall, yeah, I had a really nice experience using this model to be honest. And I was super impressed by the Omni model, Omni app we built here with just a drop in of any type of file and turn it into text. This can be very useful

if you need some kind of support for this. Like this could be in your agent workflow or anything, right? So, what I'm going to do is I'm going to leave some uh information in the description below so you can check out uh the new Neatron 3 Nano Omni model. And yeah, that was pretty much it. Thank you for tuning in and I'll see you again very