[0:00] Hey everyone, it's been a few weeks
[0:02] since I've done a video. I've been
[0:04] pretty busy lately with traveling for
[0:06] work, working on Cal, my voice
[0:08] assistant, and also developing a new
[0:11] AI-powered application, which hopefully
[0:14] I'll be able to show you in a future
[0:15] video.
[0:16] But I'm back now and hoping to get on a
[0:19] more regular schedule for releasing
[0:21] videos again.
[0:22] And in this video, I want to go over how
[0:26] I've been training Qwen 3.5 [music]
[0:29] for specifically tool calling and how
[0:31] I'm using Vast AI, a cloud [music] GPU
[0:34] provider, to do that. So, let's get into
[0:38] it.
[0:42] So, I've been fine-tuning some small
[0:44] models for a few months now. I started
[0:47] with Minstral 3 8 billion parameters,
[0:50] and I was training it using this card
[0:52] right here,
[0:54] a 12 gig RTX 3060.
[0:58] And that worked for a little while. I
[0:59] got some good results with it, and I
[1:01] showed that in a previous video how I
[1:04] was doing that with simple examples,
[1:06] just one-turn examples.
[1:09] But I quickly outgrew the 12 gigs of
[1:12] VRAM that's in this card and had to move
[1:15] to something else. And there's two
[1:17] things that drove that higher demand for
[1:19] VRAM. One was that I switched from
[1:22] Minstral 3 as the base model to Qwen
[1:24] 3.5. It's a better base model, more
[1:28] sophisticated, better at naturally
[1:30] calling tools. And the other thing is
[1:32] that my training data became more
[1:34] sophisticated. So, on the first point, I
[1:37] use Unsloth to do my fine-tuning. It's a
[1:40] very popular package to do that. And if
[1:42] you look at their guide on training Qwen
[1:44] 3.5,
[1:47] come down here,
[1:50] it says it is not recommended to use
[1:53] QLoRA 4-bit training on the Qwen 3.5
[1:57] models, either the mixture of experts
[2:00] models or the dense models,
[2:02] due to their higher than normal
[2:04] quantization differences.
[2:07] So, that means that full BF16
[2:11] bit
[2:13] needs to be used for training Qwen 3.5.
[2:17] And that makes the VRAM requirement way
[2:21] higher.
[2:22] So, there's no way that was going to fit
[2:23] on the 12 gig 3060. The second point on
[2:28] the training data getting more
[2:29] sophisticated, I went from single-turn
[2:32] examples,
[2:34] user request, tool call, user request,
[2:37] tool call,
[2:39] to full multi-turn conversations where I
[2:42] include the entire system prompt, all
[2:45] the tool definitions,
[2:47] I include previous conversation history
[2:50] in the example, and then do multiple
[2:53] tool call examples that it trains on.
[2:56] So, it might do three tool calls in a
[2:58] row in order to satisfy the request. So,
[3:01] I'm going to show you those training
[3:03] examples in a minute. But between those
[3:06] two changes, the BF16
[3:08] quantization requirement for training,
[3:11] and the more sophisticated training
[3:13] data,
[3:14] that pushed the VRAM requirements up
[3:17] from what fit on a 12 gig
[3:20] card to using up to 75 gigs of VRAM,
[3:25] which you can't even get on a consumer
[3:27] card.
[3:28] So, the answer was going to a cloud
[3:31] service. And the service I'm using is
[3:34] vast.ai.
[3:36] Now, they're not sponsoring this. I'm
[3:37] not getting anything out of this from
[3:40] them, but this is just what I'm using,
[3:42] and it actually came from a
[3:43] recommendation from a previous video. Uh
[3:47] one of the viewers mentioned it in a
[3:48] comment, and so I checked it out. So,
[3:50] thanks for that. So, Vast AI is
[3:53] basically a marketplace where people can
[3:55] put their GPUs up for rent on this
[3:57] marketplace, and other people can rent
[3:59] them and utilize them.
[4:01] Before I show you how Vast AI works and
[4:04] how my training pipeline works, let me
[4:07] show you the actual training data. So,
[4:09] here's one training example,
[4:13] and I'll show you what the actual raw
[4:16] JSON looks like.
[4:18] It looks like this.
[4:20] And I store all the training examples in
[4:23] JSON, and then they get converted in the
[4:26] training script to whatever format is
[4:28] required by the model.
[4:30] So, whether I'm training for a Minstral
[4:33] 3 model or Qwen 3.5, it's the same
[4:35] training data set. It just gets rendered
[4:38] a little differently for those models.
[4:41] So, these examples would be one line
[4:43] item in this JSONL
[4:46] file,
[4:46] but I've broken it out here to make it
[4:48] easier to look at.
[4:49] And I'm actually going to run you
[4:51] through this
[4:53] step by step to show you how this works.
[4:55] This is a three-step tool call. So, this
[4:58] is for my voice assistant, and it's
[5:01] training for the custom tools that voice
[5:04] assistant has available.
[5:06] In particular, this one's training on
[5:08] three different tools, which you'll see.
[5:11] So, the user message is email Sarah the
[5:15] next F1 race.
[5:16] And the first step
[5:18] is for the model to call a tool.
[5:21] So, we give it that example, call the
[5:23] ESPN tool with the arguments schedule
[5:26] and the sport is F1.
[5:29] The tool returns a response. It says
[5:32] next up is Australian Grand Prix, and
[5:34] after that, the Japanese Grand Prix.
[5:37] Not the actual dates for 2026 schedule,
[5:39] but close enough.
[5:42] And then step two would be to find
[5:45] Sarah's email. So, it uses the Google
[5:48] contacts tool,
[5:50] and it searches for Sarah,
[5:53] and then that tool would respond with
[5:56] her email.
[5:57] Now, this is made up. I don't know Sarah
[5:59] Jones.
[6:01] So, Sarah Jones, if you're watching,
[6:03] sorry for using your email address.
[6:06] And then the third step would be
[6:08] using the Gmail tool.
[6:10] So, using Gmail, it then uses the email
[6:13] it just found,
[6:15] and sends the email with that race
[6:18] schedule. The tool says it's done, and
[6:21] then the final response from the voice
[6:24] assistant is that I've emailed Sarah,
[6:27] and the next race is the Australian
[6:29] Grand Prix. So, that's the full example.
[6:33] And so, the model's now learned
[6:35] that F1 uses this tool when looking up a
[6:39] user, it uses Google contacts to find
[6:42] their email,
[6:44] and then it combines the information
[6:46] from step one and step two in the Gmail
[6:50] call. So, it learns all that. It learns
[6:52] the custom tools that it has available
[6:54] and how to use them. And this is again
[6:58] what that looks like in the raw JSON.
[7:01] And when this gets rendered into the
[7:03] actual training format for the model,
[7:06] it'll also add the full system prompt
[7:09] and all the tool definitions ahead of
[7:11] this example for every single example
[7:13] that it sees, which why it moves the
[7:16] token count for each example from a few
[7:19] hundred
[7:20] to on average about 7,000 tokens per per
[7:24] training example.
[7:26] Hence, the larger VRAM requirement. But
[7:30] I'm training in a much more production
[7:34] uh reflective environment. It reflects a
[7:37] production environment a lot more. Now,
[7:39] I went through some trial and errors
[7:41] with this, and one issue I came into
[7:43] when trying to include all this previous
[7:46] context, and I'll show you another
[7:47] example here in a minute,
[7:49] of all previous tool calls in a
[7:51] conversation history, and then the one I
[7:53] actually wanted to train on,
[7:55] it was actually training on everything.
[7:58] And then Qwen 3.5,
[8:00] the output of that, it was just calling
[8:03] tools for everything. It You asked it
[8:05] something, tool call. You wanted to know
[8:08] the capital of Japan, tool call. It just
[8:11] fired off tools right away.
[8:13] So, changed the approach a little bit,
[8:18] and I'll show you that here where it's
[8:20] only training on the last message.
[8:25] So,
[8:26] looking here at another example, we
[8:29] include
[8:31] all this context,
[8:35] and then we have the training signal.
[8:39] So, above the training signal is a bunch
[8:41] of previous tool calls. So, here's the
[8:43] F1
[8:45] tool call we just looked at.
[8:47] That's one uh user request.
[8:52] Then we have another user request here
[8:54] to get the the uh
[8:56] to get the jet schedule, send that to
[8:59] David.
[9:01] Okay. That's another
[9:04] user request.
[9:06] And then one more to get the temperature
[9:09] in Melbourne.
[9:11] And so, there's three previous user
[9:13] messages in the context that it sees.
[9:16] So, this would be like a realistic chat
[9:19] history,
[9:21] but the model's not training on that.
[9:23] It's not getting graded on that. It just
[9:25] sees that as part of the training data
[9:27] to go, "Okay, I see all this previous
[9:29] messaging, but now I need to just focus
[9:32] on the most recent user request." And
[9:36] this one is asking about an iOS update,
[9:39] and it uses a web search tool to find
[9:42] the answer and responds to that.
[9:46] And that's what the model will get
[9:47] trained on is just specifically this
[9:50] tool call.
[9:52] And this approach was a big unlock for
[9:54] me and the Qwen 3.5 model. Once I went
[9:58] to this approach where I had the right
[10:00] split between previous context and the
[10:04] actual training data,
[10:06] that made the model just so effective.
[10:12] Up to like 99% on the test that I wrote
[10:15] for it. Where it's calling the right
[10:16] tool, it's having conversation where it
[10:19] should have conversation, it's using its
[10:22] own knowledge where it should use its
[10:23] knowledge, and it's using tools where it
[10:25] should use tools. Now, let me show you
[10:27] how I actually do this. With Vast AI,
[10:30] I'm going to actually set up an
[10:32] instance. I'm going to use my setup
[10:34] script,
[10:35] and I'm going to start a training
[10:38] example.
[10:39] So, in Vast AI, once you're logged in,
[10:41] you have some credit in your bank,
[10:45] you would search for available GPUs.
[10:50] Now, I use the this base image.
[10:54] You can select one that's right for you,
[10:57] but this PyTorch
[10:59] I found had the basics of what I need to
[11:03] get going, and then I add my own
[11:05] packages like Unsloth, that's required.
[11:07] I'll show you that in a minute.
[11:10] So, you select the base image,
[11:12] you select how much storage space you
[11:13] need.
[11:15] It's going to be 48 gigs.
[11:17] Then you select a GPU. So, I have this
[11:20] filtered just for 5090s right now, but
[11:23] you could select
[11:26] you just want A100s or H100s if you need
[11:30] 80 gigs of VRAM, which I did for the 9
[11:33] billion parameter model.
[11:35] But, I'll just kick one off with a 2
[11:37] billion parameter model. I'll show you
[11:39] how that works. It's the same pipeline.
[11:42] So, I'll select 5090
[11:45] and see
[11:47] all the different available ones here
[11:49] and the prices.
[11:50] Now, one thing I look at is the network
[11:53] speed because when I'm downloading the
[11:56] base model to train, and then when I'm
[11:59] downloading the completed model to my
[12:00] computer, I don't want it to take
[12:02] forever.
[12:04] So, I look at the network speed,
[12:07] and it's not a huge price difference,
[12:10] like 10 cents an hour difference to go
[12:12] from
[12:14] a 500 meg upload speed to a 6,000.
[12:18] So, let's go with this one in Texas. I'm
[12:20] going to rent that.
[12:23] And then if I click over to instances,
[12:27] I'll see
[12:29] this is creating the image,
[12:32] and then it's going to start up and
[12:33] we'll be able to SSH into it.
[12:36] Okay, it's ready. I could open it here
[12:38] to open up a Jupyter notebook. I don't
[12:40] use that. I just connect with SSH,
[12:44] and I would just need this part right
[12:46] here to do that.
[12:49] You have to set up SSH key ahead of time
[12:52] under keys.
[12:55] So, you would use your computer's
[12:57] created public SSH key, put it in here,
[13:01] and then it'll recognize that you are
[13:04] who you say you are and allow you to
[13:06] connect.
[13:07] So, let's connect to that.
[13:11] Say yes.
[13:14] And now I'm connected to that machine.
[13:18] So, there's nothing in this folder right
[13:20] now, but this workspace folder is where
[13:22] we're going to put all of our files.
[13:24] So, it's running tmux,
[13:27] and I also run tmux, so I have like a
[13:29] nested tmux situation going on here.
[13:33] Now, you could now set up your
[13:35] environment, so we would need to install
[13:37] Unsloth,
[13:39] and make sure we have Transformers 5
[13:41] installed. We saw that in the
[13:43] requirements that we need Transformers
[13:45] 5. That's the default for Unsloth now,
[13:48] so you don't have to worry about that
[13:50] when you do install Unsloth.
[13:52] When I first started doing this, I had
[13:54] to make sure I had Transformers V5 with
[13:57] a different version of Unsloth, and it
[13:59] was a bit of an ordeal, but now it just
[14:00] works.
[14:02] So, you could set up Unsloth, and then
[14:07] you could SCP
[14:09] to push all the files up that you need
[14:12] to to train with, and do that every
[14:15] time, or you could set up
[14:18] a setup script, which I've done.
[14:21] So, I have a script, which is Vast AI
[14:25] setup,
[14:27] so I can go Vast AI setup,
[14:32] and all I need to give it
[14:35] is this instance here,
[14:38] instance number,
[14:42] and it's going to connect to the SSH
[14:45] server,
[14:47] and it's going to push all the files
[14:48] that we need up to the server.
[14:52] The system prompt, the training data,
[14:55] all my different training scripts.
[14:57] It's going to now
[14:59] install Unsloth
[15:02] into the virtual environment.
[15:04] This takes a minute or two.
[15:07] Okay, now it's validating that
[15:09] everything is there,
[15:11] and we have Unsloth, the March 17th
[15:14] version.
[15:16] And now it says we can connect
[15:18] here, and it gives us even the uh the
[15:22] training command we would run
[15:24] potentially.
[15:25] So, the way that script works by just
[15:27] giving it the session ID
[15:30] is it runs the command line. You can
[15:32] install this Vast AI command line tool,
[15:37] and you can go SSH URL,
[15:41] URL
[15:43] of that instance, and it'll give you the
[15:47] the URL and the port for that.
[15:50] So, it uses that tool inside the script
[15:53] to find the SSH URL and the port, and
[15:55] then uses that to SCP all the files up,
[15:59] and then it also uses that to run the
[16:02] install commands on the server.
[16:06] So, now if I connect to that server,
[16:10] I can SSH.
[16:13] I just need this bit here,
[16:19] and I can see all the files are now in
[16:21] this folder.
[16:25] So, let's kick off a training run.
[16:29] So, it's a Python script,
[16:32] and it's the
[16:34] train Quen
[16:38] V2,
[16:41] and I'm going to train the 2 billion
[16:43] parameter version of the model.
[16:47] And I'm going to use the training data
[16:50] round seven.
[16:53] I've been through a few rounds of this,
[16:55] and I'm going to output it to this
[16:57] output folder.
[17:01] So, this should kick off.
[17:04] It'll load Unsloth.
[17:08] It'll download the model.
[17:12] There it goes. It's downloading the 2
[17:14] billion parameter model, 4 and 1/2 gigs.
[17:18] This is the 16-bit version of it.
[17:27] And now it's starting to train.
[17:29] So, we're training on a 5090
[17:32] with 32 gigs of max memory.
[17:36] We're doing
[17:38] 2,600 pairs
[17:40] twice, so two epochs.
[17:44] And the batch is 4 * 2, eight effective.
[17:47] So, that's how it batches up the
[17:48] examples and gives it to the GPU.
[17:52] This is how you can manage VRAM by
[17:54] splitting up the batch sizes.
[17:58] And we have a max sequence length of
[18:02] 8,000
[18:03] K. That's actually probably too short.
[18:07] I think some of my training examples are
[18:09] up to 10,000, and in which case they get
[18:12] truncated. So, I need to look at that
[18:14] for this script. I don't think that's
[18:16] set up right for the 2 billion parameter
[18:18] model.
[18:19] So, after fine-tuning is done, about 1%
[18:23] of the 2 billion parameters are going to
[18:26] be our new
[18:28] fine-tuned parameters that get attached
[18:31] with the LoRA.
[18:33] So, it takes
[18:34] a bit to get going. Right now, it says
[18:37] it's going to take 13 and 1/2 hours.
[18:39] It's not going to take that long.
[18:42] See, it's down to 8 and 1/2 already.
[18:44] It'll settle in around three, four, five
[18:48] um out of the 650 here, and then you'll
[18:51] see a true
[18:53] uh time estimate. That'll be closer to
[18:55] the the realistic time estimate.
[18:59] Now, I'll often kick these off at night
[19:02] and let it run overnight.
[19:04] I don't want to sit here and watch it
[19:06] for 7 hours, 6 hours.
[19:09] So, I kick it off overnight,
[19:11] but when it's completed, how do I know?
[19:15] And if I wake up in the morning and it's
[19:17] been done for 6 hours,
[19:20] I'm just spending money on that GPU
[19:22] rental for another 6 hours that I don't
[19:24] have to.
[19:26] So, we have another script that's going
[19:29] to be used to poll the server
[19:32] to check if it's done.
[19:35] And if it's done,
[19:36] it's going to download the model,
[19:39] and it's going to shut down the GPU
[19:42] instance so that we're not spending any
[19:44] more money on it. Just the small like 3
[19:46] cents just to have it running in the
[19:49] background. It's probably less than that
[19:50] for this 5090.
[19:54] So, let me show you how that works. It's
[19:56] another script
[19:58] in here.
[20:01] It's my poll vast AI train script. I
[20:06] need to clean all these up.
[20:08] But, that's the one it is.
[20:11] And the way this one works
[20:14] is you give it the instance ID as well.
[20:19] And the output directory where you want
[20:21] to save it. And how many minutes it's
[20:25] going to take to train.
[20:28] So, it won't start polling until 80% of
[20:32] those minutes have passed and then it'll
[20:36] start polling to see if the model has
[20:39] been trained or not.
[20:41] So, if we back out of this and start
[20:43] this script.
[20:49] Give it that instance again. This one
[20:51] here.
[20:56] And say we want to go to models.
[21:01] And we'll output it to
[21:05] Quen 3.5
[21:07] 2B
[21:09] Lora 2026
[21:12] 03
[21:14] 29.
[21:18] And I won't put a time here. I'll just
[21:19] show you how it polls. It'll check every
[21:21] 300 seconds
[21:23] to see if it's still training.
[21:25] And what it's checking for
[21:29] is if there's a file created
[21:32] on
[21:34] the server. It's basically just a
[21:36] done.txt file that gets created at the
[21:40] end of this training script.
[21:42] So, this is polling for the existence of
[21:45] that text file. Once it finds it,
[21:49] it will download
[21:51] the Lora folder
[21:54] and the exported GGUF model file that
[21:59] happens uh to get created on the VPS
[22:03] after the training is completed as well.
[22:06] So, what I would do
[22:08] is run that, but I'd give it a time.
[22:13] And this said it was going to be done in
[22:16] 5 hours.
[22:18] So,
[22:19] I could give it 300
[22:23] minutes.
[22:24] Did I say seconds before? This is
[22:26] minutes.
[22:27] Give it 300 minutes.
[22:30] And so, it's going to sleep for 240
[22:32] minutes and then start polling after
[22:34] that. And it's going to output
[22:37] to
[22:38] a file name that is created on the
[22:42] server for the actual model. And it's
[22:44] going to put the Lora in the folder that
[22:46] I specified up here.
[22:49] And after it downloads it, it will shut
[22:52] down the instance.
[22:54] And
[22:56] then it's done.
[22:58] I actually kicked off one of these
[22:59] training runs from an airplane the other
[23:01] week when I was on my way to Mexico
[23:05] on a WestJet flight. They had Wi-Fi. And
[23:09] I was on my laptop. Set up an instance
[23:11] on Vast AI. Got the training data that I
[23:14] had just updated. Kicked off the
[23:16] training run on the airplane. And closed
[23:20] off my laptop. The poll script was
[23:23] running on my local server here.
[23:26] And then when
[23:28] I landed and got to the hotel, the
[23:30] training was done. And the model was
[23:32] downloaded.
[23:34] So, let's look at what this is costing.
[23:36] So,
[23:38] I said it was going to take about 5
[23:39] hours to train that 2 billion parameter
[23:41] model. This is costing
[23:45] .428 dollars
[23:48] every hour.
[23:51] So, about 43 cents an hour
[23:54] times 5 hours
[23:56] $2.14
[23:59] to train that model.
[24:02] That's pretty good.
[24:04] Compared to the cost of a 5090, which if
[24:08] you can find one, go between about 3,000
[24:12] 5,000 dollars Canadian.
[24:15] I think I'll choose the $2 to train the
[24:18] model.
[24:20] Now, for the 9 billion parameter model,
[24:23] what were those?
[24:26] Let's look at what the bigger GPUs are
[24:29] going for. Let's go A100
[24:32] or H100.
[24:35] Let's look at A100s. So, 80 gigs is what
[24:39] we needed for the 9 billion parameter
[24:41] model and that training data and that
[24:43] approach,
[24:44] it took 75 gigs of VRAM.
[24:48] So, we need the 80 gig version of the
[24:50] A100. There's also a 40 gig version of
[24:52] it, but that's not quite enough.
[24:55] And that's a dollar and 13 cents per
[24:58] hour. And that, I think, took about 11
[25:01] hours. So,
[25:04] about 12 bucks
[25:06] to train that model. And it took me a
[25:08] few times to get it right. I've spent
[25:10] probably about 100 dollars worth of
[25:14] training credits trying to get all the
[25:16] training data correct, the approach
[25:18] correct, the rendering of the tool
[25:21] calling correct for the Quen 3.5 model.
[25:24] Uh this was
[25:26] I was using this to to try training a
[25:28] Ministral 3 14 billion parameter model
[25:32] as well.
[25:33] So, with all the the trial and error
[25:36] I've been doing,
[25:38] yeah, I've spent about 100 bucks.
[25:40] And so,
[25:41] compared to
[25:43] buying a 5090, and even with a 5090, I
[25:47] couldn't have done what I'm doing with
[25:48] the Quen 3.5 model. So, very
[25:51] cost-effective. And once you have your
[25:53] training pipeline set up,
[25:56] um it becomes very cost cost-effective
[25:59] because you just kick off a new training
[26:01] run
[26:02] with new training data or the slightly
[26:05] different approach and you can get it
[26:07] streamlined with these training scripts.
[26:10] And I've been using this same approach.
[26:12] So, for what I showed you was the
[26:15] training data for my voice assistant.
[26:18] I've also been using this exact same
[26:19] approach for another AI-powered
[26:22] application that I'm developing. The one
[26:25] I mentioned earlier in the video that
[26:27] hopefully I'll be able to show you in a
[26:28] future video. But, I'm using this exact
[26:31] pipeline uh to to train that model as
[26:35] well for that application. So, this
[26:37] technique is transferable.
[26:40] All right. Hopefully, that was helpful.
[26:42] And if you want me to go into more
[26:43] detail on anything, let me know. And if
[26:45] you want some of these scripts, [music]
[26:47] um I can look at sharing those on GitHub
[26:49] if that would be helpful for people, the
[26:51] setup scripts.
[26:52] >> [music]
[26:52] >> And in the next video, we're going to be
[26:54] looking at this 2 billion parameter
[26:57] model running on the ZimaBoard. So, in
[27:00] the last video I did on the ZimaBoard, I
[27:03] ran out of VRAM on the little 2 gig GT
[27:06] 1030 card and didn't have a big enough
[27:09] power supply to run the 5 gig GPU that I
[27:13] wanted to run, the P2200 Quadro. I now
[27:17] have the bigger power supply.
[27:21] So, the folks at at ZimaSpace
[27:24] >> [music]
[27:24] >> that make the ZimaBoard, they sent me
[27:27] this 120 watt power supply. So, I'm
[27:30] going to try this with that Quadro to
[27:33] run the fine-tuned [clears throat]
[27:35] 2 [music] billion parameter model. And
[27:37] also, there's one other thing they sent
[27:39] me that I'm going to try out.
[27:41] Um but, I'll save that for the video.
[27:43] So, thanks for watching this one. And
[27:45] hopefully, we'll see you in the next
[27:47] one. Cheers.
[27:56] >> [music]