---
title: 'EASIEST Way to Fine-Tune a LLM and Use It With Ollama'
source: 'https://youtube.com/watch?v=pTaSDVz0gok'
video_id: 'pTaSDVz0gok'
date: 2026-07-28
duration_sec: 1322
---

# EASIEST Way to Fine-Tune a LLM and Use It With Ollama

> Source: [EASIEST Way to Fine-Tune a LLM and Use It With Ollama](https://youtube.com/watch?v=pTaSDVz0gok)

## Summary

This video provides a step-by-step guide on fine-tuning a large language model (LLM) in Python and deploying it locally with Ollama. It covers the concept of fine-tuning, when to use it, and walks through the entire process using Google Colab and the Unsloth library. The tutorial includes data preparation, model training, and integration with Ollama for local inference.

### Key Points

- **What is Fine-Tuning?** [00:11] — Fine-tuning takes a pre-trained language model and teaches it to be better at a specific task, like training an experienced chef on your restaurant's recipes rather than teaching someone to cook from scratch.
- **Fine-Tuning vs Parameter Tuning** [00:52] — Fine-tuning is different from parameter tuning (adjusting settings like temperature). Parameter tuning is like adjusting your car's radio, while fine-tuning is like teaching your car to drive in a completely different neighborhood.
- **When to Fine-Tune** [01:08] — Three main scenarios: 1) Consistent formatting or style that prompting alone can't achieve, 2) Domain-specific data the model hasn't seen, 3) Reducing costs by using a smaller, specialized model.
- **Key Advantage of Fine-Tuning** [01:42] — You need way less data and compute power compared to training from scratch. Instead of millions of examples and months of training, you might need thousands or hundreds of examples and minutes to hours of training.
- **Step 1: Gather Data** [02:26] — The most important step. If you have bad data, you'll have a poorly fine-tuned model. The video uses a dataset of 500 examples for HTML extraction, where input is HTML and output is a formatted JSON.
- **Using Unsloth for Fine-Tuning** [03:47] — Unsloth is an open-source library that is extremely good and fast for fine-tuning models. The tutorial uses a Google Colab notebook with Unsloth.
- **Choosing a Base Model** [09:20] — The video uses a small model (53 mini) for speed. You can fine-tune any open-source model like Llama 3.1, Mistral, etc. The model is loaded in 4-bit to save memory.
- **Preprocessing Data** [10:48] — Data is formatted into a single string with input, output, and an end-of-text token. The format prompt function needs to be adapted to your specific data.
- **Applying LoRA Adapters** [12:14] — LoRA (Low-Rank Adaptation) adds trainable layers to the model, enabling efficient fine-tuning without modifying all parameters.
- **Setting Up the Trainer** [13:21] — The SFT trainer from Unsloth handles the fine-tuning process. Key parameters include model, tokenizer, dataset, and training arguments.
- **Testing the Model** [14:59] — After training, the model is tested in Google Colab by running inference on a sample prompt to verify it works correctly.
- **Downloading the Model for Ollama** [16:15] — The model is saved in GGUF format (compatible with Ollama) and downloaded to the local machine. This step can take 10-25 minutes.
- **Creating a Model File for Ollama** [17:26] — A Modelfile is created to define the custom configuration, specifying the GGUF file, parameters (temperature, stop tokens), and prompt template.
- **Adding the Model to Ollama** [20:07] — Use 'ollama create' with the Modelfile to add the model to Ollama, then run it with 'ollama run'.

### Conclusion

Fine-tuning an LLM for local use with Ollama is achievable with the right tools and data. By following the steps in this tutorial, you can create a specialized model that runs on your own machine, though experimentation with parameters and data is key to good performance.

## Transcript

Today, you'll learn how to fine tune
LLMs in Python for use in Ollama.
I'll walk you through it
step by step, give you all the code
and show you how to test it out.
So let's go ahead and get started.
Now first
I want to discuss what fine tuning is
and when you should actually do it.
So fine tuning is taking a pre-trained
language model and teaching it
to be better at your specific task.
So think of this like hiring
an experienced chef and training them
on your restaurant's particular recipes.
Rather than teaching someone
to cook completely from scratch.
So here's how it works.
Instead of training a model from zero,
you start with something like GPT
or Claude that already understands
human language.
Then you feed these examples
of your specific use cases.
So maybe customer service conversations,
legal documents or medical records.
The model then adjust its existing
knowledge to excel at your specific domain
using those examples.
Now this is completely different from
parameter tuning, which is just adjusting
settings like temperature
or topic to change how the model performs.
Parameter tuning
is like adjusting your car's radio.
Whereas fine tuning
is like teaching your car to drive
in a completely different neighborhood.
Now, when should you fine tune?
Well, there's three main scenarios.
First is when you need consistent
formatting or style.
The prompting alone can't achieve.
So something like you want to output
a particular Json model or something,
or you want the model to write in
a very particular format.
Second,
you have a lot of domain specific data
that the model just hasn't seen before.
So things like advanced medical records
or customer service logs
or information that I just wouldn't
naturally know because it's very specific
to what it is that you're doing.
And then third,
when you need to reduce costs
by using a smaller, specialized model
instead of a really massive, larger one.
Now, the key advantage to fine tuning,
rather than training from scratch,
is that you need way
less data and compute power to do this.
Instead of millions of examples
and months of training,
you might need thousands
or even hundreds of examples
and maybe minutes to train or hours
to train, not weeks.
But here's the catch when you do fine
tune, you can make these models worse.
At general tasks,
but you will make them better at yours.
So if you do fine tune
one of these models, keep in mind
that it will get worse at general tasks,
at least typically, but be much better
at what you're fine tuning it to do.
Anyways, with that information
out of the way,
I want to show you how to do this
fine tuning.
So let's go over to the computer
and I'm going to show you how to set this
up in Python with some example data
that I have some on the computer now.
And the first step
that we need to complete
if we want to fine tune a model, is
we need to gather the data
that we're going to fine tune
that model with.
If this is the most important step,
if you have bad data,
you're going to have a poorly
fine tuned model.
So make sure that you take your time here
and gather the correct data.
Now for this video I've just used
AI to generate a pretty simple data set
which demonstrates HTML extraction.
So you can see that
I have some HTML like a div, an h2 tag,
a price tag, whatever, etc..
Okay, there's a bunch of information
kind of inside of here,
and what I'm expecting the model
to give me is a nicely formatted output
that tells me the name, price, category
and manufacturer of this particular data.
Now, this is very specific
because I just want to extract data
from these types of tags.
However, you could do anything as advanced
as you want here.
And you can see that
I have 500 examples in this Json file
where I have some sample input
and some sample output.
Now this can be literally anything
that you want, because
all you're doing to fine tune the model is
you're giving it example prompts
and then example outputs of what
the result or the answer from
the LMS should be based on that prompt.
So you can use customer data again
medical reports files whatever.
In this case just simple HTML extraction.
So you can see kind of how it works.
And I can compare it easily in this video.
If you want this specific data set
I'll leave a link
to download it in the description.
Okay.
Now for this video
we're gonna use something called onslaught
for doing our fine training.
This is open source.
It's free to use,
and it's just extremely good
and very fast at fine
training these models.
All right.
Now what I'm going to do
is I'm going to go over to this fine
training notebook that I've already set up
for the purpose of this video.
I'm not going to write all of the code
from scratch.
I'm just going to leave a link
to this notebook
in the description that you can download,
or that you can connect to
in Google Collaboratory,
and then you can use your own data
and start fine tuning your own models.
But before we get into that, here's
a quick idea that's worth stealing.
If you're already fine tuning LMS
or building tools with alarm.
Chances are you're solving real problems
for real users.
Maybe it's a niche SAS,
an AI agent, or a quick side
hustle for early stage businesses.
Now imagine giving those users
a fully functional website, a storefront,
booking system, portfolio, whatever
it is, without ever leaving your app.
No templates, no drag and drop
builder to maintain just one API call.
Well, that's what ten web's AI Website
Builder API does.
It's not a template engine
or a static generator.
It's a real API that spins up fully
edible WordPress based websites,
structure content design,
images, e-commerce, etc.
all in under 60s.
With one API call,
you can generate a complete website.
Your users never leave your dashboard.
You get full white label control,
so you're branding your domain,
your customer relationship,
and ten web handles the hosting, WordPress
management, and technical infrastructure
while you focus on your core product.
This is how ten web powers websites
with over 2 million users.
And now you can do the same thing
without building anything.
Whether your user is launching
a restaurant coaching business
or online store, the site comes complete
with hosting, security,
mobile optimization, and full e-commerce
capabilities, all powered by WordPress.
The platform that runs 62% of the web.
If you're ready to turn your next big idea
into reality,
then visit my link in the description
and book it strategy.
Call with the ten web team to see
how they can help transform your business.
Okay, so how does the fine tuning work?
Well, you can fine
tune this locally on your own computer
by running all of the code
that you see right here
that you can download
from the link in the description.
But unless you have a really powerful GPU,
something like a 48, 40, 90, something
even more powerful than that, then
this is going to take a very long time.
So for most of you, I suggest that you use
something called Google Collaboratory.
This is a free online
code editor environment provided by Google
that allows you to connect and use super
high end GPUs so you can do all of your
training in Google Collaboratory.
We can
then take the model that we train here,
and we can download it to our own computer
and run it in Alana, which is exactly
what I'm going to show you how to do.
So first step here is open up
this notebook again.
I'll leave a link in the description.
If you're doing this on Google Colab
and connect to a run times,
you should see a button here
that says connect.
We're going to press that
and we should connect to a
T4 instance runtime, which is a T4 GPU.
And if you're not sure,
you can press on this right here
and it's going to show you
what you have available for
system Ram, GPU, Ram, and disk,
and it will show you
the runtime you're connected to.
If that didn't work, you can go here
and you can go to change runtime type.
And then you can make sure
that you select T4 GPU
and that you're running in Python three.
Okay.
So now that we're connected
to this runtime,
we can start running the various cells
that I have inside of this notebook.
Now, because we're going to train this
on our own custom data,
we need to bring in this file.
So what I've done is I have this Json file
and I'm going to open it up here.
But in order to open it in
Google Collaboratory
I need to upload it to Google Laboratory.
So I'm going to press on this file
button here on the left hand side.
And I'm going to go to this
little upload button where it says upload.
And I'm going to select this file again.
You can download this
from the link in the description okay.
So once this file is uploaded
you should see it here.
And what I'm doing is I'm
simply opening this file and loading it
as a Json file by providing the name Json
Extraction data set 500.
Now you can use any type of data
that you want.
Again,
just make sure you get it in this format
where you have some input
and then some output.
Okay.
And the output
should actually be a string.
I'm going to convert this
to a string in one second,
because right now it's a Json object.
All right.
So now that we have
that I'm going to run this.
And you can see that
I just printed out the first example.
And I'm successfully loading in this file
that I've uploaded.
So I'll go ahead and close that window.
Next we need to install the various
dependencies.
In my case I'm in Google Colab.
So I'm going to run exclamation
mark pip install for now.
I'll remove the uninstall command.
I just had that in case
I had something installed before it.
Okay, this is going to take probably
a minute or two to download and install.
Once it's done, I'll be right back.
It will continue. All right.
So that install has finished here.
And now we're going to
move on to the next step here
which is related to the GPU check.
However, as I says here, the following
packages were previously imported.
In this runtime,
you must restart the runtime in order
to use the newly installed versions.
So I'm just going to press
on the restart session button here
just to get this reloaded.
And once this restarting thing is done,
okay, we're good to go.
We can move on to the next cell.
And if you are running this locally,
the install might
look a little bit different
and you are going to need to have Cuda
installed on your system
and be using an invariant GPU
in order for this to work,
or at least for to work quickly.
That's why I like using Google Colab,
because everything is already
set up for us.
So I'm going to go ahead and press run
here.
I'm just going to run this cell and check
if I have a GPU available
and if Cuda is installed,
I should get two trues here
or sorry, a true and then a GPU value.
So Cuda is available
and I'm using the Tesla T4 GPU
in this Google Colab instance or runtime,
which is exactly what I want.
Okay, so now we can move on
and we can start actually
setting up our fine tuning training.
Now here we need to pick the model
that we want to fine tune.
Now I'm going to pick
a really small model,
because I want to do this
in a small amount of time,
and I want this to take days or weeks
to train.
And the model that I'm using here
is the 53 mini model.
Okay.
You can look up this model
if you want to see
more information about it,
but you can fine tune
any model that you want.
That's open source.
So for the model name here,
you can put any sloth model.
You can go to the onslaught documentation
to find all the models that are available.
There are pretty much
any open source model
you could do here like Lama 3.1
for example, Mistral Pixel, etc.
you can put them all here.
Then we're going to set the sequence
length.
Now in this case
I'm just going with 2048 tokens.
This is the maximum number of tokens
that the model can handle
as input for the dtype or the data type.
Don't worry about that.
We'll leave it as none.
And this just means that we're going
to auto detect what the data type is.
Now what we need to do here is just load
the model that we're trying to fine tune.
So that's why I put the model name here.
And I'm getting the model
and the tokenizer from the fast
language model from Unsworth.
And then I'm
loading this pre-trained model
which is v3, giving the maximum sequence
length load in four bit is equal to true
because this is a four bit model okay.
So I'm going to go ahead and press on run.
That's going to load the model for us.
It will need to download it
if it's not already here.
So that can take a second.
And you might see some stuff like this
where Unsworth is patching our computer.
Don't worry, that is totally normal.
And once that is finished,
I'll be right back
and we'll move on to the next step.
All right,
so the model is now downloaded here.
Again, I picked a pretty small one
just for this video.
If you pick a larger one, this can take
a much longer amount of time.
And next what we're going to do is we're
just going to preprocess our data.
So the day that I have remember
comes in this format
where we have some input
and then we have some output.
Now what we need to do is
we need to put this in just one single
string that we can send to the model
when we're fine tuning it.
So what we're going to do is we're just
going to use this format prompt function
that I wrote here,
where we're going to have an input.
That input is going to be equal to
whatever the input is from our data set.
So you'll likely
have to change this function to work with
whatever type of data it is that you have.
Then I have a new line character,
then I have the output.
And what I've done here for the output
is I've gone and I've grabbed this
Json object, I've converted it into text.
So that's what Json dump is doing.
I'm taking a Json object
and converting this to a string.
So I'm doing that for my output okay.
And then I put this end of text tag here
so that the model knows
okay this is the end of the text.
That's it.
There's many different ways
that you can format the prompt,
but this is what I want the prompt
to look like where we have this input.
And then this is the expected output okay.
I then have my format of data
where I essentially just run
through all of the items
that I have in my file and call this
format prompt function on them.
And then I convert this into a data set.
Now if you try to run this,
you might get an error.
That's
just because this first line up here,
we may need to rerun where
we're loading in the file
because we reset the runtime.
So if we scroll down here now
after rerunning
that we can re-execute this cell.
And we should be good to go.
And we now generate this data set that
just because we need this data set object
when we're running this trainer okay.
So now what we're going to do
is we're going to use
something called the Lora adapters.
Now I'm not going to go into too much
depth on exactly what this is doing.
But this line right here is essentially
going to add the kind of layers
that we need to our lemma in order
to actually do this fine tuning.
So again this can get very complicated.
You don't need to understand
all of the parameters that are here.
And if you do want to mess with them
you are going to have to look these up.
Or maybe use an L1 to explain them to you,
because there's a lot going on
and it's not enough time for me.
Explain this in this video
if I want to make it short.
So in fact,
if we want more information on this
or in Colab, I'll just highlight this.
Right click on it, press explain code.
This is going to use Gemini
to explain the code to us.
And then we can actually just see
what the explanation is.
And that's going to be more accurate
than me walking through everything.
So says the selected code applies
the Lora low rank adoption
method to the language model.
Using this, here's
a breakdown of the parameters
and then it tells
you exactly what these are.
Okay, so use Gemini within here
if you want to know
about all the parameters.
Either way, we're going to go ahead and
run this for you to create this new model.
Now by kind of adding
these adapters to it,
which then will actually allow us
to perform the fine tuning,
which we'll do now.
So you can see 1.32 pounds layers
with 32 shaved layers.
Don't worry too much about that
if it means nothing to you.
Okay, so now that we've learned in the
model, we're going to set up the trainer.
Now the trainer is actually the thing
that's going to perform the fine
tuning for us people much smarter than
all of us have written all of this code.
So all we have to do is simply use it.
So again,
you can read through all the parameters
if you want, but what I'm using here
is the SFT trainer.
Importantly, I pass my model
I pass the tokenizer for the model.
This is actually going
to convert our string into the token
so it can be understood by the model.
I pass my data set
which I created earlier.
The field for my data set is text.
That's important because if we go look
at the data set you can see that is it.
Here we created the data
set with this dictionary which is text.
And then it has all of the values okay.
And then we have a few other things
in here that you can look at.
For example the maximum sequence length
that needs to match
with what we had before.
And then all of these training arguments.
Again, I'm just going to leave
these kind of all default,
but you can adjust them if you like.
All right.
So now we're going to run this.
We're going to initialize the trainer.
And now that that is created
what we're going to do
is actually train the model.
Now this step is going
to take a different amount of time
based on how many examples you have
and the different settings
that you configure here.
The more examples you have, the longer
this will take, but the better performance
you will get.
The larger
the base model is that you're using.
Then again, the longer this will take.
In my case, I'm using a very small model
with a very small amount of examples,
so my performance won't be great,
but this should train in
maybe a few minutes, not hours or weeks.
All right, so we are training now
and you can watch this window
when you kind of see the progress.
So once this is finished
I'll be right back.
And then I'll show you how
we can actually test this model,
make sure that it's working
and then download it
and use it with Alama.
All right.
So the training step is finished here.
Just as an FYI
that took about ten minutes to run.
So the next step is we're going to set up
the model for inference
just so we can test it and
make sure it works before we download it.
So in order to do that, pretty much
all you have to do
here is modify these messages
to be the messages that you want to test.
So you can see I have some raw
which is the user.
And then I have content
which is the prompt I want to test.
You can place multiple messages here.
And you can see that this message is
just similar to one of the examples where
I'm doing
this kind of HTML text extraction.
You don't need to worry
too much about these lines right here.
We're just kind of putting the input
and output into a format
where we're able
to actually send it to this model,
at least in the inference mode,
and then we can go ahead and press on run.
That will just take a second.
It will run the inference.
And then you can see that
we get the output.
So we have user right
which said extract the information.
And then if we go over here
you can see we have the assistant.
And then we get the output in the format
that we specified.
Okay.
So we can see this is indeed working.
And we can test this
with multiple other messages
to make sure it is working to our liking
before we go ahead and download it.
Okay.
So now that we've assumed
all right this is good.
It's working.
We've tested a few times in Google Colab.
What we want to do is download this model
so we can actually load it into a llama,
and then start using it more permanently
on our own computer or our local machine.
And this step does take a while.
So bear with me here.
What we're going to do is run this where
we say model dot save pre-trained gov.
Gov is the model format
that a llama understands.
So that's
why we're downloading it in this format.
So I'm going to go ahead and press run.
It is now going to essentially
create this output file for us.
Download it to Google Colab.
And then after that
we need to manually download this ourself
so we can save it to our computer.
Now this can take a long time.
It can take ten minutes
15 minutes 20 minutes.
So just bear with it.
Be patient.
And once it's done you can run this cell.
This cell will then download it
to your own computer.
Then read and move on to the next step
where I show you how to actually load this
into a llama on your local machine.
So once this is done executing
and you've ran that final cell,
you should have a file
downloaded to your downloads
folder that's named something like this
Unsworth IQ for Kcmg. You.
If you're just looking for some file
that ends in.gov,
this is going to be a pretty large file
and it will take a long time to download.
In my case, both
those steps took about 25 minutes to run,
so just keep that in mind.
This will take a while,
especially if you have a slow
internet connection
when you're downloading the file.
Okay, now that we have the file,
what we need to do is add this to a llama.
So we need to create
something called a model file.
So what I'm going to do
is I'm going to open up my terminal.
I'm just going to do this
all from terminal commands
to make our life a little bit easier.
And I'm going to open this up okay.
I'm going to zoom in a little bit
and we're just going to test
and make sure that a llama
is installed on the system.
So run the llama command,
make sure you've got that downloaded
and installed for the next steps to work.
Okay, from here
I'm going to go to my downloads folder.
So CD into downloads
and from the downloads folder I know that
I have this Unsworth kind of
for underscore k underscore m file.
So what I'm going to do
is I'm just going to make a new directory.
So I'm going to say mkdir
Alamut dash test.
Okay.
If we go to downloads now we can see this
folder is created.
I'm just going to drag this file
into the folder here.
So now it's inside of Olamide test okay
I'm now
going to go inside of Olamide test.
And I'm just doing this
to kind of organize things.
But you don't need to follow
this whole setup here.
And what we're going to do now
is we're going to make something
called a model file.
Now a model file
defines a custom configuration for a model
that you want to run in a llama.
So we need to make one of those.
Now to do that we can make a new file
so we can say something like touch.
And then we're going to call this model
file okay.
Just exactly like this with the capital
M that's going to make a new file for us.
If we ls we can see it's here.
Now to edit this
we can do nano model file like that.
And then I'm going to paste in
what the model file should look like okay.
So it's going to say from dot slash.
And then this needs to be the name
of your file to the name of your model.
So the local file
that we had is called this.
So I'm just going to copy the name
by going to rename copy
and then pasting it here.
So we're going to say from dot slash
Unsworth dot Q4 underscore
underscore mgg us again
this is kind of the base model.
So we're saying from this file right here
which is in our local directory
where this model file exists, then
we can specify some different parameters.
So we can do the top
the temperature the parameter stop.
So user end of text.
Because that's the way
that our model works.
And then we have a template.
So the template is going to be user prompt
and then assistant.
And then we just have a system message.
So if we want to tell the model
to do something specific
we can in this case
we just say you are a helpful I assistant.
So this is a very simple model file.
This is how we're going to be able
to load in this particular model
and use it in a llama.
So now we are going to save.
So I'm going to hit Ctrl x
and then press Y to save the model file.
Go ahead and press enter.
And now you can see
if we open up the model file again.
So nano model file.
It looks like this. Again
we can get out of that.
So now we have a model file
and we have the model downloaded.
What we need to do is add this to a llama.
So in order to add this to a llama we're
going to type a llama and then create.
And then we're going
to give the model a name.
So I'm going to say
this is an HTML model okay.
And we're going to do dash f
and then model file.
What this is going to do
is add a new model configuration
two llama for us using the model file.
So let's go ahead and press enter.
That's going to take a second.
And now if we want to see if this is here
we can type a llama list.
And we should see that we get the HTML
model that was just created.
So now if we want to run the model
we can just type a llama run.
And then the name of this
which is the HTML model.
It's going to take a second to load up.
And then we can paste
any message inside of here.
And we should see it working.
So let's go into our data set.
Let's just copy one of the ones
that we have just to keep things simple.
So let's copy this right here.
Go back and paste it okay
I just cleared the screen here
so it's a bit easier to see okay.
Then you can see after
I put the prompt here and press enter.
This is the result that we got.
Now this isn't always going to work.
So I can put this a few times.
And you'll see now
look like it gives me a different output.
That's because obviously
we had a small data set we're using,
you know a pretty poor model.
Not one of the massive models,
but generally it is working
in the way that we wanted it to work.
And more examples pass in.
This will give us a better output
either way.
This is a now custom fine tuned
model loaded in a llama that we can run
locally on our own computer.
We could connect to it from Python or
something and use it in our application.
And that is going to wrap up this video.
Now I know
this is not going to be perfect.
You do need to experiment
with this. Modify some of the parameters.
But I wanted to show you how to do fine
tuning, talk about what that actually is
and give you a step by step guide.
So hopefully this can get you up
and running as fast as possible.
All of the assets for this
video will be linked in the description
in case you need them.
And with that said, I look forward
to seeing you in another video.