---
title: 'Fine-Tuning Qwen 3.5 for $11 on a Rented GPU'
source: 'https://youtube.com/watch?v=ui1zR5Pc4Q8'
video_id: 'ui1zR5Pc4Q8'
date: 2026-07-28
duration_sec: 1679
---

# Fine-Tuning Qwen 3.5 for $11 on a Rented GPU

> Source: [Fine-Tuning Qwen 3.5 for $11 on a Rented GPU](https://youtube.com/watch?v=ui1zR5Pc4Q8)

## Summary

The creator explains how they fine-tune Qwen 3.5 for tool calling using Vast AI, a cloud GPU marketplace. They detail the shift from a local RTX 3060 to rented GPUs due to increased VRAM requirements from BF16 training and complex multi-turn training data. The process includes setting up instances, running training scripts, and automating model download and instance shutdown to minimize costs.

### Key Points

- **Shift from local to cloud GPU** [00:42] — Started with Mistral 3 on a 12GB RTX 3060, but outgrew VRAM when switching to Qwen 3.5 and more sophisticated training data.
- **Why Qwen 3.5 requires more VRAM** [01:22] — Unsloth recommends full BF16 training for Qwen 3.5, not QLoRA 4-bit, due to quantization differences. This increases VRAM demand.
- **Training data complexity** [02:28] — Moved from single-turn examples to multi-turn conversations including system prompt, tool definitions, and history, pushing VRAM usage to 75GB.
- **Using Vast AI** [03:28] — Vast AI is a marketplace for renting GPUs. The creator uses it to access high-VRAM GPUs like A100s and H100s.
- **Training data example** [04:09] — A three-step tool call: look up F1 race via ESPN, find Sarah's email via Google Contacts, send email via Gmail. The model learns to chain tools.
- **Avoiding over-tool-calling** [07:39] — Initially the model called tools for everything. Fixed by training only on the last message in a conversation, not on previous context.
- **Achieving 99% accuracy** [10:12] — With the correct training approach, the model reached 99% on tests, correctly deciding when to use tools, converse, or rely on knowledge.
- **Setting up Vast AI instance** [10:27] — Select base image (PyTorch), storage (48GB), GPU (e.g., 5090), and consider network speed for faster model downloads.
- **Setup script automation** [14:21] — A script pushes files, installs Unsloth, and provides the training command. It uses the Vast AI CLI to get SSH details.
- **Training run example** [16:25] — Training a 2B parameter model on a 5090 with 32GB VRAM, 2600 pairs, 2 epochs, batch size 4*2, max sequence length 8000 (noted as too short).
- **Polling script to save costs** [18:59] — A script polls for a done.txt file, downloads the LoRA and GGUF model, then shuts down the instance to avoid extra charges.
- **Training from an airplane** [22:58] — Kicked off a training run on a flight using Wi-Fi; the poll script on a local server downloaded the model and shut down the instance automatically.
- **Cost breakdown** [23:36] — 2B model: ~$2.14 (5 hours at $0.428/hr). 9B model: ~$12 (11 hours at $1.13/hr). Total trial-and-error cost: ~$100.
- **Transferable technique** [26:10] — The same pipeline is used for another AI application, showing the approach is reusable.

### Conclusion

Fine-tuning Qwen 3.5 for tool calling is cost-effective using rented GPUs, with a well-automated pipeline that minimizes manual effort and cost. The technique is transferable to other applications.

## Transcript

Hey everyone, it's been a few weeks
since I've done a video. I've been
pretty busy lately with traveling for
work, working on Cal, my voice
assistant, and also developing a new
AI-powered application, which hopefully
I'll be able to show you in a future
video.
But I'm back now and hoping to get on a
more regular schedule for releasing
videos again.
And in this video, I want to go over how
I've been training Qwen 3.5 [music]
for specifically tool calling and how
I'm using Vast AI, a cloud [music] GPU
provider, to do that. So, let's get into
it.
So, I've been fine-tuning some small
models for a few months now. I started
with Minstral 3 8 billion parameters,
and I was training it using this card
right here,
a 12 gig RTX 3060.
And that worked for a little while. I
got some good results with it, and I
showed that in a previous video how I
was doing that with simple examples,
just one-turn examples.
But I quickly outgrew the 12 gigs of
VRAM that's in this card and had to move
to something else. And there's two
things that drove that higher demand for
VRAM. One was that I switched from
Minstral 3 as the base model to Qwen
3.5. It's a better base model, more
sophisticated, better at naturally
calling tools. And the other thing is
that my training data became more
sophisticated. So, on the first point, I
use Unsloth to do my fine-tuning. It's a
very popular package to do that. And if
you look at their guide on training Qwen
3.5,
come down here,
it says it is not recommended to use
QLoRA 4-bit training on the Qwen 3.5
models, either the mixture of experts
models or the dense models,
due to their higher than normal
quantization differences.
So, that means that full BF16
bit
needs to be used for training Qwen 3.5.
And that makes the VRAM requirement way
higher.
So, there's no way that was going to fit
on the 12 gig 3060. The second point on
the training data getting more
sophisticated, I went from single-turn
examples,
user request, tool call, user request,
tool call,
to full multi-turn conversations where I
include the entire system prompt, all
the tool definitions,
I include previous conversation history
in the example, and then do multiple
tool call examples that it trains on.
So, it might do three tool calls in a
row in order to satisfy the request. So,
I'm going to show you those training
examples in a minute. But between those
two changes, the BF16
quantization requirement for training,
and the more sophisticated training
data,
that pushed the VRAM requirements up
from what fit on a 12 gig
card to using up to 75 gigs of VRAM,
which you can't even get on a consumer
card.
So, the answer was going to a cloud
service. And the service I'm using is
vast.ai.
Now, they're not sponsoring this. I'm
not getting anything out of this from
them, but this is just what I'm using,
and it actually came from a
recommendation from a previous video. Uh
one of the viewers mentioned it in a
comment, and so I checked it out. So,
thanks for that. So, Vast AI is
basically a marketplace where people can
put their GPUs up for rent on this
marketplace, and other people can rent
them and utilize them.
Before I show you how Vast AI works and
how my training pipeline works, let me
show you the actual training data. So,
here's one training example,
and I'll show you what the actual raw
JSON looks like.
It looks like this.
And I store all the training examples in
JSON, and then they get converted in the
training script to whatever format is
required by the model.
So, whether I'm training for a Minstral
3 model or Qwen 3.5, it's the same
training data set. It just gets rendered
a little differently for those models.
So, these examples would be one line
item in this JSONL
file,
but I've broken it out here to make it
easier to look at.
And I'm actually going to run you
through this
step by step to show you how this works.
This is a three-step tool call. So, this
is for my voice assistant, and it's
training for the custom tools that voice
assistant has available.
In particular, this one's training on
three different tools, which you'll see.
So, the user message is email Sarah the
next F1 race.
And the first step
is for the model to call a tool.
So, we give it that example, call the
ESPN tool with the arguments schedule
and the sport is F1.
The tool returns a response. It says
next up is Australian Grand Prix, and
after that, the Japanese Grand Prix.
Not the actual dates for 2026 schedule,
but close enough.
And then step two would be to find
Sarah's email. So, it uses the Google
contacts tool,
and it searches for Sarah,
and then that tool would respond with
her email.
Now, this is made up. I don't know Sarah
Jones.
So, Sarah Jones, if you're watching,
sorry for using your email address.
And then the third step would be
using the Gmail tool.
So, using Gmail, it then uses the email
it just found,
and sends the email with that race
schedule. The tool says it's done, and
then the final response from the voice
assistant is that I've emailed Sarah,
and the next race is the Australian
Grand Prix. So, that's the full example.
And so, the model's now learned
that F1 uses this tool when looking up a
user, it uses Google contacts to find
their email,
and then it combines the information
from step one and step two in the Gmail
call. So, it learns all that. It learns
the custom tools that it has available
and how to use them. And this is again
what that looks like in the raw JSON.
And when this gets rendered into the
actual training format for the model,
it'll also add the full system prompt
and all the tool definitions ahead of
this example for every single example
that it sees, which why it moves the
token count for each example from a few
hundred
to on average about 7,000 tokens per per
training example.
Hence, the larger VRAM requirement. But
I'm training in a much more production
uh reflective environment. It reflects a
production environment a lot more. Now,
I went through some trial and errors
with this, and one issue I came into
when trying to include all this previous
context, and I'll show you another
example here in a minute,
of all previous tool calls in a
conversation history, and then the one I
actually wanted to train on,
it was actually training on everything.
And then Qwen 3.5,
the output of that, it was just calling
tools for everything. It You asked it
something, tool call. You wanted to know
the capital of Japan, tool call. It just
fired off tools right away.
So, changed the approach a little bit,
and I'll show you that here where it's
only training on the last message.
So,
looking here at another example, we
include
all this context,
and then we have the training signal.
So, above the training signal is a bunch
of previous tool calls. So, here's the
F1
tool call we just looked at.
That's one uh user request.
Then we have another user request here
to get the the uh
to get the jet schedule, send that to
David.
Okay. That's another
user request.
And then one more to get the temperature
in Melbourne.
And so, there's three previous user
messages in the context that it sees.
So, this would be like a realistic chat
history,
but the model's not training on that.
It's not getting graded on that. It just
sees that as part of the training data
to go, "Okay, I see all this previous
messaging, but now I need to just focus
on the most recent user request." And
this one is asking about an iOS update,
and it uses a web search tool to find
the answer and responds to that.
And that's what the model will get
trained on is just specifically this
tool call.
And this approach was a big unlock for
me and the Qwen 3.5 model. Once I went
to this approach where I had the right
split between previous context and the
actual training data,
that made the model just so effective.
Up to like 99% on the test that I wrote
for it. Where it's calling the right
tool, it's having conversation where it
should have conversation, it's using its
own knowledge where it should use its
knowledge, and it's using tools where it
should use tools. Now, let me show you
how I actually do this. With Vast AI,
I'm going to actually set up an
instance. I'm going to use my setup
script,
and I'm going to start a training
example.
So, in Vast AI, once you're logged in,
you have some credit in your bank,
you would search for available GPUs.
Now, I use the this base image.
You can select one that's right for you,
but this PyTorch
I found had the basics of what I need to
get going, and then I add my own
packages like Unsloth, that's required.
I'll show you that in a minute.
So, you select the base image,
you select how much storage space you
need.
It's going to be 48 gigs.
Then you select a GPU. So, I have this
filtered just for 5090s right now, but
you could select
you just want A100s or H100s if you need
80 gigs of VRAM, which I did for the 9
billion parameter model.
But, I'll just kick one off with a 2
billion parameter model. I'll show you
how that works. It's the same pipeline.
So, I'll select 5090
and see
all the different available ones here
and the prices.
Now, one thing I look at is the network
speed because when I'm downloading the
base model to train, and then when I'm
downloading the completed model to my
computer, I don't want it to take
forever.
So, I look at the network speed,
and it's not a huge price difference,
like 10 cents an hour difference to go
from
a 500 meg upload speed to a 6,000.
So, let's go with this one in Texas. I'm
going to rent that.
And then if I click over to instances,
I'll see
this is creating the image,
and then it's going to start up and
we'll be able to SSH into it.
Okay, it's ready. I could open it here
to open up a Jupyter notebook. I don't
use that. I just connect with SSH,
and I would just need this part right
here to do that.
You have to set up SSH key ahead of time
under keys.
So, you would use your computer's
created public SSH key, put it in here,
and then it'll recognize that you are
who you say you are and allow you to
connect.
So, let's connect to that.
Say yes.
And now I'm connected to that machine.
So, there's nothing in this folder right
now, but this workspace folder is where
we're going to put all of our files.
So, it's running tmux,
and I also run tmux, so I have like a
nested tmux situation going on here.
Now, you could now set up your
environment, so we would need to install
Unsloth,
and make sure we have Transformers 5
installed. We saw that in the
requirements that we need Transformers
5. That's the default for Unsloth now,
so you don't have to worry about that
when you do install Unsloth.
When I first started doing this, I had
to make sure I had Transformers V5 with
a different version of Unsloth, and it
was a bit of an ordeal, but now it just
works.
So, you could set up Unsloth, and then
you could SCP
to push all the files up that you need
to to train with, and do that every
time, or you could set up
a setup script, which I've done.
So, I have a script, which is Vast AI
setup,
so I can go Vast AI setup,
and all I need to give it
is this instance here,
instance number,
and it's going to connect to the SSH
server,
and it's going to push all the files
that we need up to the server.
The system prompt, the training data,
all my different training scripts.
It's going to now
install Unsloth
into the virtual environment.
This takes a minute or two.
Okay, now it's validating that
everything is there,
and we have Unsloth, the March 17th
version.
And now it says we can connect
here, and it gives us even the uh the
training command we would run
potentially.
So, the way that script works by just
giving it the session ID
is it runs the command line. You can
install this Vast AI command line tool,
and you can go SSH URL,
URL
of that instance, and it'll give you the
the URL and the port for that.
So, it uses that tool inside the script
to find the SSH URL and the port, and
then uses that to SCP all the files up,
and then it also uses that to run the
install commands on the server.
So, now if I connect to that server,
I can SSH.
I just need this bit here,
and I can see all the files are now in
this folder.
So, let's kick off a training run.
So, it's a Python script,
and it's the
train Quen
V2,
and I'm going to train the 2 billion
parameter version of the model.
And I'm going to use the training data
round seven.
I've been through a few rounds of this,
and I'm going to output it to this
output folder.
So, this should kick off.
It'll load Unsloth.
It'll download the model.
There it goes. It's downloading the 2
billion parameter model, 4 and 1/2 gigs.
This is the 16-bit version of it.
And now it's starting to train.
So, we're training on a 5090
with 32 gigs of max memory.
We're doing
2,600 pairs
twice, so two epochs.
And the batch is 4 * 2, eight effective.
So, that's how it batches up the
examples and gives it to the GPU.
This is how you can manage VRAM by
splitting up the batch sizes.
And we have a max sequence length of
8,000
K. That's actually probably too short.
I think some of my training examples are
up to 10,000, and in which case they get
truncated. So, I need to look at that
for this script. I don't think that's
set up right for the 2 billion parameter
model.
So, after fine-tuning is done, about 1%
of the 2 billion parameters are going to
be our new
fine-tuned parameters that get attached
with the LoRA.
So, it takes
a bit to get going. Right now, it says
it's going to take 13 and 1/2 hours.
It's not going to take that long.
See, it's down to 8 and 1/2 already.
It'll settle in around three, four, five
um out of the 650 here, and then you'll
see a true
uh time estimate. That'll be closer to
the the realistic time estimate.
Now, I'll often kick these off at night
and let it run overnight.
I don't want to sit here and watch it
for 7 hours, 6 hours.
So, I kick it off overnight,
but when it's completed, how do I know?
And if I wake up in the morning and it's
been done for 6 hours,
I'm just spending money on that GPU
rental for another 6 hours that I don't
have to.
So, we have another script that's going
to be used to poll the server
to check if it's done.
And if it's done,
it's going to download the model,
and it's going to shut down the GPU
instance so that we're not spending any
more money on it. Just the small like 3
cents just to have it running in the
background. It's probably less than that
for this 5090.
So, let me show you how that works. It's
another script
in here.
It's my poll vast AI train script. I
need to clean all these up.
But, that's the one it is.
And the way this one works
is you give it the instance ID as well.
And the output directory where you want
to save it. And how many minutes it's
going to take to train.
So, it won't start polling until 80% of
those minutes have passed and then it'll
start polling to see if the model has
been trained or not.
So, if we back out of this and start
this script.
Give it that instance again. This one
here.
And say we want to go to models.
And we'll output it to
Quen 3.5
2B
Lora 2026
03
29.
And I won't put a time here. I'll just
show you how it polls. It'll check every
300 seconds
to see if it's still training.
And what it's checking for
is if there's a file created
on
the server. It's basically just a
done.txt file that gets created at the
end of this training script.
So, this is polling for the existence of
that text file. Once it finds it,
it will download
the Lora folder
and the exported GGUF model file that
happens uh to get created on the VPS
after the training is completed as well.
So, what I would do
is run that, but I'd give it a time.
And this said it was going to be done in
5 hours.
So,
I could give it 300
minutes.
Did I say seconds before? This is
minutes.
Give it 300 minutes.
And so, it's going to sleep for 240
minutes and then start polling after
that. And it's going to output
to
a file name that is created on the
server for the actual model. And it's
going to put the Lora in the folder that
I specified up here.
And after it downloads it, it will shut
down the instance.
And
then it's done.
I actually kicked off one of these
training runs from an airplane the other
week when I was on my way to Mexico
on a WestJet flight. They had Wi-Fi. And
I was on my laptop. Set up an instance
on Vast AI. Got the training data that I
had just updated. Kicked off the
training run on the airplane. And closed
off my laptop. The poll script was
running on my local server here.
And then when
I landed and got to the hotel, the
training was done. And the model was
downloaded.
So, let's look at what this is costing.
So,
I said it was going to take about 5
hours to train that 2 billion parameter
model. This is costing
.428 dollars
every hour.
So, about 43 cents an hour
times 5 hours
$2.14
to train that model.
That's pretty good.
Compared to the cost of a 5090, which if
you can find one, go between about 3,000
5,000 dollars Canadian.
I think I'll choose the $2 to train the
model.
Now, for the 9 billion parameter model,
what were those?
Let's look at what the bigger GPUs are
going for. Let's go A100
or H100.
Let's look at A100s. So, 80 gigs is what
we needed for the 9 billion parameter
model and that training data and that
approach,
it took 75 gigs of VRAM.
So, we need the 80 gig version of the
A100. There's also a 40 gig version of
it, but that's not quite enough.
And that's a dollar and 13 cents per
hour. And that, I think, took about 11
hours. So,
about 12 bucks
to train that model. And it took me a
few times to get it right. I've spent
probably about 100 dollars worth of
training credits trying to get all the
training data correct, the approach
correct, the rendering of the tool
calling correct for the Quen 3.5 model.
Uh this was
I was using this to to try training a
Ministral 3 14 billion parameter model
as well.
So, with all the the trial and error
I've been doing,
yeah, I've spent about 100 bucks.
And so,
compared to
buying a 5090, and even with a 5090, I
couldn't have done what I'm doing with
the Quen 3.5 model. So, very
cost-effective. And once you have your
training pipeline set up,
um it becomes very cost cost-effective
because you just kick off a new training
run
with new training data or the slightly
different approach and you can get it
streamlined with these training scripts.
And I've been using this same approach.
So, for what I showed you was the
training data for my voice assistant.
I've also been using this exact same
approach for another AI-powered
application that I'm developing. The one
I mentioned earlier in the video that
hopefully I'll be able to show you in a
future video. But, I'm using this exact
pipeline uh to to train that model as
well for that application. So, this
technique is transferable.
All right. Hopefully, that was helpful.
And if you want me to go into more
detail on anything, let me know. And if
you want some of these scripts, [music]
um I can look at sharing those on GitHub
if that would be helpful for people, the
setup scripts.
>> [music]
>> And in the next video, we're going to be
looking at this 2 billion parameter
model running on the ZimaBoard. So, in
the last video I did on the ZimaBoard, I
ran out of VRAM on the little 2 gig GT
1030 card and didn't have a big enough
power supply to run the 5 gig GPU that I
wanted to run, the P2200 Quadro. I now
have the bigger power supply.
So, the folks at at ZimaSpace
>> [music]
>> that make the ZimaBoard, they sent me
this 120 watt power supply. So, I'm
going to try this with that Quadro to
run the fine-tuned [clears throat]
2 [music] billion parameter model. And
also, there's one other thing they sent
me that I'm going to try out.
Um but, I'll save that for the video.
So, thanks for watching this one. And
hopefully, we'll see you in the next
one. Cheers.
>> [music]