[0:00] In previous video, we tested the limit
[0:02] of 8 gigs GPU by running a 30 billion
[0:04] models on it. In this video, we're going
[0:06] to test it even further by running a 80
[0:09] billion model on a 16 gigs card and then
[0:11] compare the result with Gemini 3.1 Pro.
[0:14] Will it works? Is it practical? We'll
[0:16] find out in this video.
[0:22] Couple weeks ago, Alibaba released a new
[0:24] coder model, Quen 3 Coder Next, and 80
[0:27] billion parameters, yet score really
[0:30] well, almost on par with models much
[0:32] larger than its size. Quen 3 coder Next
[0:35] also has only 3 billion active
[0:37] parameters, which should allow us to run
[0:39] it on consumer graphics card with
[0:40] reasonably fast speed. And today, we're
[0:42] going to see if we can fit this model
[0:44] into a RTX 5060 Ti 16 gigs. So, here is
[0:48] my portable AIP PC. 32 gigs of RAM and
[0:51] 16 gigs of VRAM. It's natural that we
[0:54] have to use a quantized version in order
[0:56] to run it. Looking at its size, at
[0:58] first, the 4-bit quant version seems
[1:01] like it will fit. However, we'll be left
[1:03] without any spare system memory to use
[1:06] and very limited context, so it's a bit
[1:08] impractical. The 3-bit Imatrix version
[1:11] with 33 gigs in size should give us
[1:13] plenty of breathing room with almost the
[1:15] same intelligent with Q3KM version. And
[1:18] if you use Unslo quantized version, they
[1:20] even did some benchmark showing that
[1:22] their 3bit Imatrix quant performed
[1:24] better than non-unslaught Q4 and almost
[1:26] on par with the full precision model.
[1:29] For the setting, I'm using 50K context
[1:31] length and offload around half of the
[1:33] model to GPU, leaving 1 gigs of free
[1:36] VRAM. For the systems, I use flash
[1:39] attention but did not use KV cache
[1:41] quantization since the model itself was
[1:44] quantized enough as it is already. So I
[1:46] want to avoid any other potential
[1:47] quality degradation if possible. So the
[1:50] first test I'll ask it to create a 3D
[1:52] audio visualizer web app using 3JS
[1:55] particle to react to the sound using a
[1:57] blank web project as a base.
[2:00] Despite being an 80 billion model
[2:02] running on 16 gigs card, the token
[2:04] generation speed is pretty fast. The
[2:07] model completed the task in around 10
[2:08] minutes and here is the result.
[2:20] It works beautifully. I'm really amazed
[2:23] I can get this result in one shot from
[2:25] local AI running on 16 gigs card. We can
[2:28] definitely polish it further, but for
[2:30] the first try, this is a very good
[2:32] result. So, the next test, we're going
[2:34] for a specific prompt. I found this good
[2:37] prompt example on X that he used it to
[2:39] test Gemini 3.1 Pro. So, let's use it to
[2:42] test with Quen 3 Next as well.
[2:49] The prompt again took only about 10
[2:51] minutes and the result is pretty good
[2:52] for the first attempt. The model handled
[2:54] most of the prompt correctly with the
[2:56] exception of some. For example, it
[2:58] missed the blur layer for the glass
[2:59] morphism navbar. Some of the animation
[3:01] are not working and minor visual bug.
[3:12] However, if we compare the result with
[3:14] Gemini 3.1 Pro, even though Gemini
[3:16] handled the animation a lot better, Quen
[3:18] 3 coder next result doesn't really look
[3:20] that far apart.
[3:30] So moving on to the next test, we'll add
[3:32] more complexity by giving the model very
[3:35] detailed concept such as 3JS particles,
[3:38] horizontal scrolling, several animation,
[3:41] and different section layout. This time
[3:43] the prompt took around 30 minutes to
[3:45] complete and here is the result. This
[3:48] time we start to see the limit of the
[3:50] heavily quantized 80 billion model with
[3:53] complex concept prompt. The first
[3:55] section, navbar and menu look quite okay
[3:58] at first, but the other sections are
[4:00] quite bad, especially the horizontal
[4:03] scrolling, which is totally broken.
[4:15] Gemini handled this test much better.
[4:47] The final test. We're switching from web
[4:50] development to Python. The prompt is to
[4:52] write a simple arcade spaceship shooter
[4:55] game. The model took around 6 minutes to
[4:57] complete the task. And here is the
[4:59] result. The game is playable at the
[5:02] first attempt. And since I didn't
[5:04] provide the sprite image, the model used
[5:06] simple rectangle for the spaceship
[5:07] instead. So I provided them and told the
[5:10] model to update the code. Here is the
[5:12] final result.
[5:18] So running an 80 billion model on a 16
[5:20] gig 5060 Ti isn't just a fun experiment,
[5:24] it's surprisingly practical. Even at
[5:26] 3bit quantized, Quentry Coder Next
[5:29] proved it can singleshot various tasks
[5:31] at moderate difficulty with impressive
[5:33] speed and accuracy. It still hits a wall
[5:36] on high complexity task which is where
[5:38] cloud model like Gemini Pro still hold
[5:41] the crown. But for local model on a
[5:43] consumer GPU, this is a massive leap
[5:45] forward. Let me know down in the
[5:47] comments what you want me to test next.
[5:49] And if you found this video helpful,
[5:50] like and subscribe for more. Thanks for
[5:53] watching. See you in the next one.