[0:00] you want to fine tune your large language
model and run it locally on your machine
[0:03] using Ollama.
[0:07] Well, in today's video,
we're going to do exactly that.
[0:09] So let's go.
[0:12] First, for the fun part,
just finding the right data set.
[0:14] The reason why finding the right data
set is so important is when you train
[0:18] a small, large language model
with a data set
[0:21] that is relevant to the task
you're trying to do.
[0:23] It can actually outperform large models.
[0:26] What I'm going to be doing
today is creating a small, fast LLM
[0:29] that will generate
SQL data based off of table data.
[0:32] I provide it.
[0:33] One of the biggest data sets to do this
with is called synthetic text to SQL,
[0:37] which has over 105,000 records split
[0:41] into columns of prompt SQL
content, complexity, and more.
[0:45] Im running a Nvidia 4090 GPU,
[0:47] so I'm going to be fine
tuning this on my machine using ubuntu.
[0:50] If you don't have a GPU,
feel free to do this using Google Colab,
[0:53] which allows you to run
training code in the cloud.
[0:56] The great news is that this project
[0:57] does not require a lot of complex
hardware to get it up and running.
[1:01] We're going to be using Unsloth
[1:02] which allows you to fine tune
a lot of open source models
[1:06] really efficiently,
with about 80% less memory usage.
[1:09] And we're going to be using Llama 3.1
which is an LLM used for commercial and
[1:13] research purposes, especially in English,
and has really high performance.
[1:18] Make sure that you have Anaconda
[1:19] installed on your machine
as well as the Cuda libraries.
[1:22] I will be using Cuda 12.1 and Python 3.10
for this project.
[1:26] You want to install
[1:27] the dependencies required by Unsloth
which you can find in the Readme.
[1:31] But for simplicity, here it is.
[1:33] This creates a new environment for us
[1:35] and installed PyTorch Cuda libraries
as well as the latest unsloth.
[1:39] You'll also want to install Jupyter
if it isn't there already,
[1:42] and then run your Jupyter notebook.
[1:44] And now you're done with the setup.
[1:46] So let's go into the Jupyter
notebook and get started.
[1:48] First, we want to make sure that all the
installed requirements are actually there.
[1:52] If you're using Google Colab,
this command will install the packages.
[1:55] Next we're going to import the fast
language model by Unsloth.
[1:58] Here we're specifying that we want to use
the Lama three eight bit model.
[2:02] We also want to set up a max sequence
length of 2048 tokens.
[2:06] This means that the model
will only consider up to 2048 tokens,
[2:10] where a token can be a word, subword
character, or even punctuation.
[2:15] When processing or generating text,
and we'll set load in four bit to true,
[2:19] which essentially means we're using less
bits, as opposed to using the typical 16
[2:23] or 32 bits
to represent the information in the model.
[2:26] Doing this is going to help you
[2:28] reduce memory usage
and also reduce the load on your machine.
[2:31] After running this,
[2:32] you're going to get a cute Ascii image,
and that means that your model is loaded.
[2:35] After this,
[2:36] we're going to load in the PEFT model,
which is basically Lora adapters.
[2:40] If you don't know what these terms
mean, that's totally fine.
[2:43] Basically, the LORA adapters mean that
[2:45] we only have to update 1
to 10% of the parameters in this model.
[2:49] Without them, it means that
[2:50] we would have to retrain the whole model,
not just a small portion,
[2:54] which takes a lot of time, energy,
and even money.
[2:57] Unsworth provides
this here with the recommended settings.
[3:00] I trust them,
so feel free to read each comment.
[3:02] Now this is where things can get
a little bit tricky depending on what data
[3:06] set you're using.
[3:06] The each data
set comes different from each other,
[3:09] but they're each formatted in the same way
such that the large language model
[3:12] can understand it.
[3:13] Llama three uses alpaca prompts
which look like this.
[3:17] Now, if you remember our data set,
it is not as easy as just plugging it in
[3:21] and letting it go off to the races.
[3:22] I have to format my response
first before plugging in the data.
[3:26] I'm only interested
in the SQL of my database.
[3:28] The prompt I will be asking for, as well
as the generated code and explanation.
[3:32] So I'm going to update my code
to reflect this.
[3:34] Now we set up the training module
to supervise to fine tuning.
[3:37] Trainer by hugging face is what I used.
[3:39] There are a lot of parameters to use all
that can be described in their own video.
[3:43] So for example, have max steps which tells
us how many training steps to perform.
[3:47] Seed is a random number generator.
[3:48] We used to be able to reproduce results
and warmup steps gradually increases
[3:53] learning rate over time.
[3:54] So now that we have everything
set up, let's run it.
[3:59] And that's it.
[4:00] Your model has been trained.
[4:01] Now before we move on,
[4:02] we actually need
[4:03] to convert this into the right file type
so that we can run this locally
[4:06] using a llama.
[4:07] Luckily, onslaught has a one liner
we can use to do this.
[4:10] After this is done, we only need to do one
step to be able to run this with Allama.
[4:15] First, open up your terminal.
[4:16] I'm using the warp terminal here.
[4:18] Go to the path of where the file is saved.
[4:21] Then create a file called Model file
and open it up in the code editor.
[4:25] This is Ollamas Docker
like file configuration
[4:28] where we can create new models
with specific parameters.
[4:30] In our model file
we're going to put a prompt.
[4:32] So something like you're an SQL generator
that takes a user's query
[4:36] and gives them helpful SQL to use.
[4:38] Finally make sure Olan was running.
[4:40] And then we're just going to run
this command.
[4:42] This command will then read all the items
in the model file you just created,
[4:46] and start using llama Dhcp under the hood
to make sure that the model runs
[4:50] on your machine. And congrats!
[4:51] You can now use your fine tuned
LM locally,
[4:54] all with the OpenAI compatible API
and more in your applications.
[5:01] If you're
[5:01] curious to know more about Alama,
we do have a two minute video
[5:05] out about everything you need to know
about Alama here.
[5:08] Otherwise, thank you for watching
and I'll see you next time.