---
title: 'LLM Fine-Tuning Crash Course & Tutorial'
source: 'https://youtube.com/watch?v=IIvORO248Zs'
video_id: 'IIvORO248Zs'
date: 2026-07-31
duration_sec: 3221
---

# LLM Fine-Tuning Crash Course & Tutorial

> Source: [LLM Fine-Tuning Crash Course & Tutorial](https://youtube.com/watch?v=IIvORO248Zs)

## Summary

This crash course covers LLM fine-tuning from theory to practice. It explains transfer learning, compares RAG and fine-tuning, dives deep into LoRA and QLoRA techniques, and walks through a hands-on Unsloth fine-tuning example in Google Colab.

### Key Points

- **What is LLM Fine-Tuning?** [00:50] — A process of retraining a pretrained model (like Llama) on a specific task, dataset, tone, and format, leveraging transfer learning.
- **RAG vs Fine-Tuning** [02:37] — RAG points the LLM to external knowledge without retraining, making it cheap; fine-tuning is expensive but adapts tone, format, and precision. The industry combines both.
- **Base LLMs vs Instruct Models** [05:18] — Base models like Llama 3.2 1B are autocomplete engines; instruct versions are fine-tuned to follow prompts, producing ChatGPT-like responses.
- **LoRA: Low-Rank Adaptation** [06:54] — LoRA freezes original weight matrices W and adds a trainable delta W, decomposed into A and B matrices, dramatically reducing trainable parameters.
- **LoRA Efficiency Math** [10:33] — For a 512×512 delta W, full training requires 262,144 parameters; decomposing with rank 8 reduces it to just 8,192 trainable parameters.
- **Quantization Explained** [15:00] — Converting weights from float32 to lower-precision formats cuts memory: a 7B model drops from 28GB to 7GB with int8 or 3.5GB with NF4.
- **Quantization Formula** [18:21] — Scale = (Xmax-Xmin)/(Qmax-Qmin), quantized = round(X/scale) + zero_point, dequantized = scale × (Q - zero_point).
- **NF4 Quantization** [24:59] — NF4 creates histogram-based bins instead of equal bins, so the normal distribution of weights maps more evenly and preserves accuracy.
- **QLoRA: 4-bit LoRA** [30:25] — QLoRA combines NF4 quantization with LoRA, plus double quantization and page optimizers, enabling fine-tuning of a 65B model on a single 48GB GPU.
- **Fine-Tuning in Action** [50:39] — After fine-tuning Llama 3.2 3B on 172k reasoning samples, the model produces DeepSeek-like step-by-step reasoning and answers 'strawberry' has 2 Rs.

### Conclusion

The video delivers a comprehensive crash course on fine-tuning LLMs, covering both the underlying theory (LoRA, QLoRA, quantization) and a practical Unsloth implementation. By combining RAG with fine-tuning and using efficient techniques like QLoRA, you can adapt even massive models to your domain without breaking the bank.

## Transcript

LLM fine-tuning where we will start with some theory on what is fine-tuning. We'll cover some popular methods such as Laura, QLoRA, etc. And then we will write code in Unsloth to perform fine-tuning on a sample data set.
very simple language. When I moved to US from India, >> one of my friends took me to play baseball. And I was actually comfortable playing baseball. Why? Because in India I have already played cricket. Now in
both of these games, you have bat, ball, you have to track the ball, and move your bat, run. So whatever raw skills I had from cricket, [music] I deployed that while playing baseball. Essentially, I'm transferring the skills
from one game to another. So in AI there is a term called transfer learning where you take the model and then you retrain it or you fine-tune it on new data set, task, and format. So that is what LLM fine-tuning is. When you look at LLM
such as GPT-5, Llama, etc., they are trained on a vast amount of internet data. But let's say you are an AI engineer working in a private company. Let's say you are working for Johnson & Johnson or Reliance Jio and
>> you are given a task to build a chatbot for your customers that has knowledge of the internal data of that organization. Also, it should respond in a specific format, tone, etc. At this time, you will use LLM fine-tuning. So LLM
fine-tuning is a process of retraining the pre-trained model such as Llama on a specific task, data set, tone, and format. Let's take an example of an imaginary company called Loki Phones uh to understand this thing better. You're
building a chatbot for this company using models such as GPT and Llama. And when your customer asks a question, "My Loki Phone 12 screen is cracked. What are my options?" The base LLM will give some general answer, right? But the
right answer in this case is you should use this Loki Care Plus plan if you already have it. Now let's say this data is not available on internet. In that case, plain LLM will not be able to answer this.
There might be other questions, okay, how long battery last, whatever. And this is the kind of data which is private to organization and LLM has not seen it. So using plain LLM, it won't be
able to answer. So what you do is you use a simple technique called retrieval augmented generation. In this technique, you point your LLM to In this technique, you point your LLM to external source of knowledge, right? It
can be a database, PDF files, whatever. This is private data. It's not available on internet. But when you're building a chatbot internally, you can point it and it will be able to answer it, okay? Now the benefit of RAG is that you are not
retraining the model, right? So LLM is nothing but there is a neural network here, right? There is a neural network uh which you're training and the network looks something like this. It's a huge network. Let's say it has 70 billion
parameter. Now what do I mean by parameter? Each of these edges have weight. Let's say this has a weight of 0.8, 0.7. And how many edges are there? Maybe 40, 50? In actual LLM, there are 70 billion
update. So the benefit of RAG is that you're not updating everything and it is very cost-effective, okay? But there are disadvantages. RAG may not be able to produce the best
RAG may not be able to produce the best possible answer in your company's brand tone and the format that you expect. For example, for this question, RAG may produce this answer, but ideally you want this answer. Similarly,
this is a second question. RAG may produce this and I but fine-tuned model that here there is more empathy. See? And let's say empathy is one of the values of your company
that you want your chatbot to follow as well. So overall fine-tuning gives this benefit that it can handle emotion slang. It can answer with brand tone and overall it can produce more precise answers, whereas
RAG has its own set of benefit, which is it is low cost, okay? So if I have to fully summarize pros and cons, then these are the pros and cons, okay? You can go over it. Overall, RAG is cheap, fine-tuning is expensive, but
fine-tuning produces better answer compared to RAG. So what people do in the industry is they combine both RAG and fine-tuning, okay? I'm [snorts] giving you an example of a base LLM called Llama 3.2. And by the way, you
can go to Hugging Face and find both of these models. So Llama 3.2 1B is a base LLM, whereas 3.2 1B instruct is a fine-tuned model. So this model is fine-tuned on this particular base LLM.
And the base the foundation model is just an autocomplete. You all know, works. The way it works is you're giving a sentence and it is just doing autocomplete of that sentence. So when I
autocomplete of that sentence. So when I say, "The capital of India is", it will Okay, and then it will autocomplete next sentence, next next word, next word. And that's how it produces a big paragraph as an answer. But when you look at the
instruct model, that is more similar to the response you get in ChatGPT. summarize this article. sentence to Hindi. It will summarize, right? So this is So this fine-tuned
model is more like ChatGPT, more consumable. There are two types of fine-tuning. One is full fine-tuning where you will retrain the entire network and it's going to be costly, right? You're updating 70 billion, 100
billion, whatever is the parameters of your LLM, you're updating all of them. fine-tuning. And there are two popular methods, LoRA and QLoRA. Here you are not updating all the layers actually. You keep
You keep these layers frozen and you add some new layers on top of it. If you know about transfer learning, in deep learning learning where you keep certain layers frozen. You don't update them. And you
update only some other layers. So PEFT is similar to that and these are the two popular methods.
low-rank adaptation technique. It is one of the popular fine-tuning techniques for LLM. We have to first think about the transformer architecture. In transformer architecture, we had all these V, K, Q parameters. If
you remember, query parameter, key value parameter, and so on. And when you're training this model, you are essentially training the trainable parameters, which is represented by these matrices. So if you remember from our deep learning
module, we already discussed these metrics, WQ, WV. And these are again trainable parameters. These are the matrices which gets trained when you are training this
particular transformer model. And I'm representing one slide here. We looked at this slide previously where we had this WQ, which was, you know, matrix that will tell you how to encode query of a token for attention computation.
And we had couple of such matrices in attention layer. We had WQ, WKV, WO will be the output one. And then in the feed-forward layer, we had two matrices.
If we represent all these matrices as W, let's assume we call it W in a generic let's assume we call it W in a generic way, then what we do in LoRA is we keep this W frozen. So let's say if you are using Llama model or GPT model, okay?
And you are fine-tuning it. Now that model will have all these matrices. You keep them frozen. You don't change anything into them. But you take your training data set on which you want to fine-tune your model.
And when you're fine-tuning, while you keep W frozen, where you don't change any parameter here, you add a new uh set of parameters you add a new uh set of parameters called delta W. So you add a new matrix,
delta W. And during the training, you change parameter only here. So this one is frozen, okay? It's like a box. It's frozen. It doesn't change. But this frozen. It doesn't change. But this delta W will change. This is called
parameter efficient fine-tuning. Here the whole model is frozen, but a small set of trainable parameter is added to the model, okay? So this is PEFT. Now you might ask, "How
can this save computation time?" Because we are saying, "Oh, parameter efficient fine-tuning, right?" Um we we could have fine-tuned the entire don't want to change it because we want this process to be efficient. But then
if you're adding the delta W, it's the same thing. It's in fact more computation. So the question is, "How does this solve computation problem?" Well, in LoRA technique, we do something really clever. Let's say you have this
delta W, right? Which is a D by D matrix. Okay? And Delta W is this. The new uh trainable parameter that you have added, okay? So, let's say it's D have added, okay? So, let's say it's D by D matrix. You decompose that into two
matrices, okay? A and B. And here we are using this R parameter, which is called rank, okay? So, in LoRA, low rank adaptation, okay? So, this R is this
hyperparameter. So, if you multiply these two matrix, like 4 by 2 and 2 by 4, you will get this particular matrix, okay? So, let's assume that D parameter
is 512. So, let's say if you're training with Delta W, you will be training these The total number of parameter will be 262,000. But, when you decompose it into this A and B, so this is a dot product, by the
way. So, A.B, you get Delta W, okay? So, when you do that, uh and let's say your rank is eight, okay? So, this R, once again, is a rank parameter. When you say low rank adaptation, R is a rank. And this is a hyperparameter. It
can be 2 8 6. There are different values, okay? That you can take. I'm just using R eight as an example value here. here. So, what happens is when R is eight is
the parameters in this one is 4096. Parameters in this one is 4096. So, total number of trainable parameter is 8192. So, when you're training a model, you know, when you do back propagation and when you update the weights, you're
updating only 8192 parameters. Whereas, in this case, you're updating so many parameters. So, this is how you achieve efficiency. And this is the crux of low rank adaptation. The rank parameter can take any values. Uh these are just
general guidelines. But, as you can see, R can be 4 8 16 32. You know, it can be any of these values. I have seen eight to be the common value, eight or 16 to be the common value. But, once again, it's a hyperparameter, okay?
If you look at uh PEFT library from Hugging Face, which is used for this PEFT optimization, R is taken as a hyperparameter in a LoRA config, okay? So, here eight is a common value. But, based on your data set,
based on your problem, you can play with it and just see which method works the best. If you're interested in reading the full paper, you can just Google LoRA the full paper, you can just Google LoRA paper and you will find a PDF. So, just
in case you're curious, you can go through it. But, I hope you got the crux of it, which is adding Delta W parameter and then decomposing that into A and B individual matrix matrix such that A.B
is equal to Delta W. And then, total number of parameters that you have to train through A and B will be much lesser compared to Delta W. And in this technique, once again, you keep your W, which is your original set
of parameters, frozen. You don't change them, okay? So, computationally, them, okay? So, computationally, this method is very efficient.
technique, you need to first understand quantization. You might know about this concept from the deep learning module. But, let's uh refresh our memories. Let's say you have LLaMA 7B model. 7B means it has 7 billion parameter. And
these parameters are nothing but the weights in neural network. Let's say weights in neural network. Let's say each parameter takes 4 bytes or float 32 things. In that case, this model needs 28 GB of
RAM. Like, when you're loading that model in your computer, it needs 28 GB of RAM. If you're talking about 70 billion model, it needs 280 GB of RAM, which is too much memory, okay? You can't run it on your local computer.
can't run it on your local computer. But, what if we reduce the size for storing each parameter. Let's say from 4 byte you go to 8 bits or just 1 byte, byte you go to 8 bits or just 1 byte, okay? So, 8 bits is 1 byte. 1 byte is
equal to int eight, okay? Int eight is a data type. In that case, it will take data type. In that case, it will take only 7 GB of memory. Here, I'm showing you a table where we have this different sizes for bits, okay? So, if Q is 32,
which means 4 byte, it will take 28 GB, this scenario. this scenario. But, if it is eight, it takes 7 GB. If it is NF4, int four, which is half bit, you know, 4 bits, it takes only 3.5 GB
and you can run that model locally on your computer. If you have 70 billion your computer. If you have 70 billion model, you might need multiple GPUs with float 32. But, if you have int eight, you might need only one GPU. You know,
you can run it with one GPU or two GPUs very easily. And this process of reducing the bytes, you know, that it takes to store one parameter is called quantization. It's a process of converting high precision numbers into
low precision formats to reduce memory and computation requirements by machine learning models. Now, you'll be like, when you go from high precision to low precision, of course, you're losing accuracy. Well, that is true. But, it
has been shown through the experiments that, practically speaking, you still get a pretty good accuracy, okay? For your given use case. And quantization helps especially with edge devices. Let's say you have a drone, which is
flying over your field and you're using deep learning to uh do some kind of uh computer vision type of use case, then quantization really helps because you can have quantized model loaded into your drone.
You know, drone will not have that much memory. So, it will work the best uh during the inference time. So, quantization is popular when it comes to edge devices. It is popular in other use cases as well, where you need efficiency
when it comes to your computation and memory requirements. So, let's see how quantization works. Let's say you have a couple of weights which are float 32. Okay? And usually, these weights uh are minus one to one. So, the
weights uh are minus one to one. So, the values of these weights will be .91 .78, okay? Negative. And then, .87 .28 and so on. So, I'm just showing you some sample weights from a neural
some sample weights from a neural network. And the scale here is minus one to one. When you quantize this to int two, int two will take how many What tell me. You need to have this computer
science fundamentals clear. Int two means two bits. And in two bits, we can means two bits. And in two bits, we can store four number. Minus two to positive one. So, the way you can quantize this is you can divide your float 32 range
is you can divide your float 32 range into four equal bins, okay? Four equal bins and just map them to these four bits. So, these values n- .91 .78
will become negative two. .39 will become negative one. Any values that comes into this bin will become negative one. Okay? So, it is sort of like scaling, but scaling with surrounding and some other difference, okay? So, the
and some other difference, okay? So, the values you had for weights, minus .91 values you had for weights, minus .91 -.78 and so on, after quantization will become this, okay? So, it's a pretty simple concept. What
will happen if you are quantizing to int eight? Well, once again, what is the range for int eight? Int eight means eight bits. And eight bits can store eight bits. And eight bits can store values from minus 128 to positive 127.
So, you divide, once again, this range into total 255 bins and you just map them. And let's say you call your original values X and
say you call your original values X and you call your quantized values Q, okay? So, there is a mathematical formula to perform quantization. The first thing you need to do is find out the scale. Again, this is like you have learned
is similar to that. You're scaling values from one range to another range. So, you first find scale by saying Xmax minus Xmin Qmax minus Qmin. So, Xmax is
one. Xmin is minus one. Qmax is 127. Qmin is minus 128. So, you put all those values here and you get a scale of .0078.
here and you get a scale of .0078. Now, to get quantized values, you use formula. So, you have your X values, minus zero point, which means what is the value where you want to center your points. In
our case, let's say this is zero, okay? So, this zero point will be zero. X will be all these values and scale will be scale, okay? So, for one of the values, let's say for this negative .91 value,
uh zero point is zero, then scale is this. When you round it, you will get this. When you round it, you will get negative 116. So, you map this value to negative 116. Once again, it's a concept similar to scaling with some
differences, where you have a bunch of values, you're where you have a bunch of values, you're mapping it to limited range of integers. Okay? See, when we were doing scaling in machine learning module, the scaled
value will still be a float. So, you are not saving any time anything in terms of memory. But here, the scale value is integer. So, you are mapping this float to integer through this formula, and that will give you this memory saving.
So, I have this Jupiter notebook where I'm creating some hundred random values in range minus one to one, okay? So, let's say these values look something like this. Now, I'm using the same scale formula that we saw previously. Max X
minus min X Q max minus Q min, and scale is going to be 0.0078, Now, my zero point is zero underscore point is zero.
And for quantization, I'm using that same formula. X minus zero point, okay? X minus zero point here, divide by scale, and you round it to an integer. And then you clip the values to Q min and Q max. You don't want to your values
and Q max. You don't want to your values to go beyond minus 128 and plus 127. And when you apply this formula, you get Q values for your X values. So, for your
X values which are these first 10 values, see? We map minus one Then um minus point 9797, whatever, to minus
125, and so on. And to do dequantization, which is from these values, let's say I want to get these values back, right? Because you have to do that. To do that, you will just use this formula. It's a reverse of
quantization, okay? So, this formula should be pretty straightforward, folks. Scale into Q minus zero point, and you get these values back. So, this minus one value you got it back as minus 1.0039.
precision. You understand that, right? It's like when you zip your data, unzip it, there is some loss. Similarly, here there is some loss. So, minus 0.979, you got it back as minus 0.98. But it's not too bad, right? See, minus 0.818,
you got minus 01.815. So, it's not too bad. You save a lot of memory, and you get similar accuracy. There'll be some drop in the accuracy, but for your practical use case, it will mostly works, okay? And this is
it will mostly works, okay? And this is how you plot it on the chart. Let's now how you plot it on the chart. Let's now look at normal float NF4 quantization. We previously saw that when you have in the two, possible values are four, okay?
Minus two, minus one, zero, one. Total possible four values. So, you create four bins, and you map it, okay? What happens if instead of in two, you have int four? In int four, how many possible values are there? Folks, use your
this video and tell me how many possible values are there. values are there. Well, total possible values will be 16, minus eight to positive seven. See, minus eight will be this, okay? This is
a bit representation where this bit is a sign. So, sign one one bit means negative, and zero zero zero. Uh then one and one one one is minus seven, and so on. So, you create 16 bins and map them,
okay? This is a linear mapping. You create 16 equal bins and map it. But this will create problems when we use quantization for our neural network, especially for LLM, because our weights, you know, all these weights, let's say
you take that neural network, and you take all the weights, and if you plot a distribution, it will be a normal distribution. So, if it's a normal distribution, majority of the values are centered around zero. And when you are
using regular in four, you are creating equal bins, correct? Equal equally spaced bins, and due to this, what will happen is majority of these values will be mapped to one value, right? Like like these
to one value, right? Like like these values, let's say this is this value is, let's say, 30% of total values. Okay, I I don't know the exact number, but let's assume. Let's say this is 30%. And let's say this is 20% of the total
values, and this is 20%. So, then all these values are being mapped to single value. Let's say this mapped to single value. Let's say this is minus two, this is minus one, zero.
I think zero, one, two, three, four, five, six, seven, okay? So, majority of the values are being mapped to minus one and one. And when you do dequantization, this is not a good approach, right? For
a normal distribution, creating equal bins will not work. So, folks, please pause this video and tell me an idea which will work the best for this scenario. I'm telling you, it's a common sense.
sense. Pause this video and tell me how do you change this so that there is an equal mapping. All right, I hope you found the correct answer. The answer is you will not create equal bins. You will create bins based on the
histogram. So, this total number of data points you have is 100%, right? And if you divide that by 16, you get 6.25%. So, you want 6.25%
in each bin, and you will design that bin accordingly, okay? So, here, since in this histogram there are more number of data points, the width of the bin is lower, okay? So, if this is zero value, this might be zero to
let's say 0.005. Let's assume that this is zero, this is one, and this is minus one. So, then this bin will be zero to 0.00 this bin will be zero to 0.00 let's say one. Then 0.001 to 0.005,
such a way that the data points, let's say these data points, right? Like so So, if you look at this particular area, this should be 6.25%. If you look at this particular area, this should be
this should be 6.25%. Let's look at the coding so that you get an idea. And this is called normal float four, right? Like normal normal for
normal distribution. So, this is NF4 quantization. Let's look at the code now. So, in the code, what we're doing here is X is we are just creating 100 data points, okay? 100 random data points between minus one to one range.
Let's say these are your weights of your neural network, and these NF4 values are 16 values. Okay? So, these values are minus 0.01 to minus 0.05. So, the minus 0.01 to minus 0.05. So, the distance between these two is 0.04.
And then from here to here, the distance is 0.05. See, distance will increase. You see in this chart, this distance, okay? This distance right here is how much? 0.04.
Okay? So, 0.04. width for the second bin, okay? Second bin,
is what is this? If you add 0.05 into this, you get 0.1. So, this distance is 0.05.
or the width for the third bin will be you even higher. So, 0.1 to 0.2 is 0.1, see? 0.1. So, you realize like when you increase that width, you can accommodate more data points. Because of this normal
more data points. Because of this normal distribution, we need to do that. And these values are kind of fixed, okay? If you use some ready-made Python library like bits and bytes, you will get a better values, but this is a rough
approximation. So, now let's quantize it. So, what you're doing essentially is uh once again, very simple. You are mapping all these data points to nearby value. So, what is a nearby value for minus one? This. So, you'll map it to
this. What is more nearby to minus 0.97? Is it What is more nearby to minus 0.97? Is it minus one or point 75? Well, it's minus here, okay? And what is
And what is most nearby to this? Is it minus 0.75 or minus one? Just not this. 0.87. Again, minus one. But if you look at minus 0.85, it is more near to this. So,
these three values will be mapped to minus 0.75. These four values will be mapped to minus 1.0. You are just mapping it to nearby value. Very simple logic, and we have this Python function. And when you quantize it, see? You'll
get this. See, all these values are mapped to minus one. Last three values are mapped to point minus 0.75 because it's it's more closer to it, okay? And
And this is how the distribution looks like. version of LoRA technique. Let's understand how this works.
Say you are fine-tuning an LLM with with 65 billion parameter. Now, if this is using float 32, it will take 260 GB of memory to load the model.
You will first quantize this using let's say NF4 quantization. And the quantized model will have a size of 32.5 GB because for each parameter is it it is taking half a byte. So, for 65 it will be 32.5. And then you apply LoRA
fine-tuning using your domain-specific data set. This is what QLoRA is. It has few other elements as well, but just to go over definition of QLoRA, it's a memory-efficient fine-tuning method that combines 4-bit quantization with
low-rank adaptation. Now, other than NF4 quantization, it uses a concept of double quantization and also page optimizers. If you look at
their research paper, in the abstract itself, they talk about all these three points, which is 4-bit normal float quantization, double quantization, and page optimizer. So, we already looked at NF4 quantization. Let's look at what is
double quantization. This is like zipping a file once again. So, you zip a file, you get a smaller version, then you again zip it, you get even a smaller version. So, same thing we do here. So, let's say you have 7 billion parameter
model. To quantize it usually, you don't take all the parameters and quantize it in one shot. You will create a block of 64 parameters. So, you have all these 7 billion parameters, okay? You create a group of 64 billion
parameters. And you will get n number of blocks, I think 109 million blocks in this case. And then you apply quantization, okay? And then you apply quantization, okay? NF4 quantization. So, each of these
blocks will have less size and they will have their own individual scale values. If you remember quantization formula, there was scale and there was zero point, okay? So, for doing block-wise quantization, each block will have
different scale. So, the scale one will be different than scale two and so on. It will also have a different zero point. Now, the people who wrote QLoRA paper thought that, okay, we got some optimization here, but the scales are
still float 32, right? The scale is floating-point number. In our case, it was 0.0078 something. It is still a floating-point number. And you have so many such numbers. In this case, 7 billion parameters, you will have 109
billion parameters, you will have 109 million blocks, I think. And for each block you have one scale. So, 109 million float 32 numbers. So, what if you take all these scales and you quantize them again, okay? So, that way
you also get some reduction in terms of size for the scale values and zero point values, okay? So, it's a simple concept just like zipping a file which is already zipped, okay? So, quantization you're doing and then you're doing a
quantization again on the scale, etc. The third element of QLoRA is page optimizers. So, what happens is when you are fine-tuning this model, usually are fine-tuning this model, usually you'll use GPU. So, on GPU, let's say
fine-tuning is going on. Sometimes it may run out of memory. So, you will use this paging concept similar to you know how paging works in terms of operating system. There is a page of memory which will be swapped in, swapped out of CPU
memory back to disk and disk to CPU memory or RAM. We use similar concept here. So, when GPU is going out of memory, some of the pages that it has will be swapped out to CPU memory or RAM and then they will be swapped back in
and then they will be swapped back in whenever needed. So, these are the three key elements of QLoRA technique. And I have the research paper open here. As you can see, you can fine-tune 65 billion parameter model on a single 48
GB GPU. So, you realize it says if 65 billion parameter with float 32, that multiplied by four, so 280 GB, whatever, it will be that huge number. So, I have a QLoRA paper open here. And here it says that
a 65 billion parameter model can be trained on a single 48 GB GPU. This is humongous because 65 billion parameter model is around 100 260 GB.
And if you can fine-tune that heavy model on a single GPU with 48 GB RAM, then that's really amazing, okay? I'm going to link this research paper. You can read it if you want to go into the details.
is very popular when it comes to LLM fine-tuning. And they have this nice documentation which you can use to get help on API, etc. To install it, you will use pip install Unsloth on your computer.
Now, to fine-tune, you need GPU on your computer. So, if your computer has GPU, you can do it locally, but I prefer doing this in cloud because in cloud you can get access to GPU as well as some of
the dependency errors that I was getting while I was running locally, I did not while I was running locally, I did not get on cloud. So, I will start with Colab, okay? So, colab.research.google.com.
connect it with a GPU instance. So, by default this is connected to CPU. You can just say change runtime type. T4. So, this is giving you an access of Tesla T4 GPU, which is not very powerful, but good enough for our
practical demonstration. If you have a paid account and if you can get access to higher-end GPU, then that's even better. I will demonstrate this entire thing through a pre-built notebook because running this code takes
a lot of time, okay? So, I have already executed this notebook and I'll walk over it. So, in a Colab notebook, you first need to install Unsloth library, okay? Uh because when you connect with the Colab environment, it starts fresh.
So, you can't just install it and save it. You have to install it, okay? So, in your cell, just run this after you have connected with that GPU instance. And then you will import fast language model class from
Unsloth, okay? You'll also import PyTorch, which is torch. And here you are defining a bunch of parameters. And at this point, you are getting a
pre-trained model. So, here we are fine-tuning Llama model, Llama 3.2 3B instruct. This is the model we are fine-tuning. So, you specify that. fine-tuning. So, you specify that. Then max sequence length is your context
window length, okay? How much longer sequence you want to support in your fine-tuned model. Dtype none means let GPU figure out what kind of dtype will be the best. And by the way, I have provided help on all
these parameters. See, it specifies the data type of model weights and computation. If you have something like Nvidia A100 GPU, Nvidia A100 GPU, then it will use this torch.bfloat16,
okay? So, when you say dtype none, it will figure out the correct uh type for the storage. And load in four bit is used for 4-bit
quantization, okay? So, we are going to use 4-bit quantization here. And when you run it, it will download a bunch of stuff because it is downloading this model. Uh and it is loading that model into
this model parameter. You also get this tokenizer variable. If you look at this index, this looks similar to the Hugging Face Transformer library syntax where you say library class.from_pretrained,
you load the model. Okay, so now I have Llama model here in this model variable. And I will fine-tune that, okay? You also need this tokenizer, okay? So, as a next step,
what you do is you get the parameter efficient fine-tuned model, get PEFT So, here you are defining your parameters for LoRA. So, what do you need? Rank, okay? Rank I'm specifying as 16.
You can also try eight, 32, etc. It's a hyperparameter, folks. So, I'm starting with 16, but you can start with eight, 32, play with it. Then for target modules, Q_proj, K_proj, V_proj, and O_proj refers to those uh
remember those matrices? You remember these in a attention layer, you will have these four matrices. In feed-forward layer, you will have two matrices, W1, W2. And and and you will have few more in the
and and you will have few more in the output layer as well. So, that's what it output layer as well. So, that's what it is. Q_proj is for that uh WQ uh matrix, then WK, VV K, etc. Now, let's say if you don't specify this parameter, then
it will not add that extra adapter, that extra delta W after that a particular these matrices uh defined, what it will do is see, you
uh defined, what it will do is see, you will have WQ and plus it will add delta to this. Then since I have W K approach defined, it will have WK plus delta WK, something like that.
Okay, so wherever you want to modify those matrices, you will add this approach, okay? If I don't add that, it will use WK original as a frozen layer. It will not add that adapter. I
coming from our LoRA lecture that we already discussed before, okay? And these are the matrices from the fully connected layer. Then we
from the fully connected layer. Then we have LoRA alpha. So this LoRA alpha is this particular alpha. See, your equation is W0, which is original weight matrix, plus delta W. So A.B is your delta W.
But you don't just do delta W, you add some scaling parameter. So that is alpha some scaling parameter. So that is alpha divided by rank, okay? And that alpha is this LoRA alpha parameter that you're specifying. This will tell you how much
that adapter contributes to your weight update. Because this WW W0 is same as W that we have discussed before. This W is not changing, it's a frozen layer. What is changing is this A.B, which is
What is changing is this A.B, which is delta W, correct? But how much delta W contributes to the final weights, you can configure that through this scaling parameter, okay? So that's what this LoRA alpha is. Then you have
So while you're doing training, if you lecture, if you want to drop certain neurons to kind of address overfitting, then you can specify that. In In our case, we are not going to
have any dropout, we'll just say zero. Then this bias, okay? So I've given the information on all these parameters here, okay? Random state, all of that. here, okay? Random state, all of that. So here, you are defining your LoRA
model, your parameter efficient fine-tune model. So you loaded Llama 3 model, then you defined all these then you defined all these parameters for your LoRA, okay? Like R
and LoRA alpha and so on. And now you got this new model, which is having all those LoRA parameters. As a next step, you will download the dataset on which you want to fine-tune your model. So
you want to fine-tune your model. So here we are using ServiceNow R1 Distill SFT dataset. And this dataset library, by the way, it is a Hugging Face by the way, it is a Hugging Face library. It allows you to access so many
different datasets. I'm on Hugging Face website right now and I have this ServiceNow R1 Distill SFT dataset, as you can see. It has 172,000 rows and each row is sort of like you have a problem and you have solution. So
see, this is a problem. It's like a puzzle. For Halloween, Sarah receives 66 pieces, whatever. This is like a puzzle and this is the This is like a puzzle and this is the solution. So you want to fine-tune your
model on all these puzzles and you want to fine-tune it in a way and you want to fine-tune it in a way that it will do reasoning similar to your DeepSeek model, okay? And for that, see, these are the two main columns in
your dataset, right? Problem and this is a solution. And in solution, Sarah ate nine pieces of candy, see? The solution is this descriptive answer, but the actual number that you're looking at is nine, okay? And other than these two
nine, okay? And other than these two columns, you have third column, which is re-annotated assistant content. And this is a reasoning. This is like a think, you know, that think prompt that you get in DeepSeek. So if you want to solve
this problem, this is how you will do reasoning step determine total number of pieces candy, whatever. Next I will do this. This is Like when you're given a puzzle, first you will do this first step, second
step, third step and so on. And that's what this column is and at the end, final answer. Now, in case you're wondering what is this {slash} boxed nine {slash} {slash} etc. Well, this is the LaTeX LaTeX
markup language, okay? So let me just show you the show you the It called LaTeX, but LaTeX is how you pronounce it. And this markup language, okay? What it
does is people use it for in academia for people use it for in academia for scientific publications, etc. So let me just show you. Let's say you are having a research paper
and you want to have all these mathematical equation, etc. For that, you will use this LaTeX package, okay? So it's like {slash} markup language. Okay, folks, I hope you're getting
what I mean by markup language. And see, LSD listing. So it has its own syntax. If you want to learn about it, you can go to YouTube. You'll find some LaTeX videos, but I don't think it's needed. It's just in in your brain, like you
need to think that, okay, this is some kind of markup language used in this particular dataset, okay? So this is a dataset. Now,
you load that dataset here through transformer dataset library, okay? So after the dataset is loaded, we'll look at the few records and see.
It has a problem, it has a solution, it has an answer. See, problem. There were 27 boys, 35 girls. Remember in in high school days and college days, you used to get all these problems. There were
fill in the blank children on the playground. So there's a problem. There's a solution. Solution is usually in that boxed parameter. Okay, so solution will be here somewhere and
solution will be here somewhere and there is this reasoning pattern. Now, you take all those records. So there are 172,000 records in our dataset that we will use for fine-tuning.
In this particular code, what we are doing is we take each of those records and you put it in this prompt. So you're saying you are a reflective assistant, blah blah blah, engaging iterative reasoning, mimicking human stream of
your approach and present exploration start out, whatever. So you give this prompt, then here is a problem. So there are three brackets, right? So this is a problem, okay? So you you go through, see, you have a dataset,
right? So you go through each record. When you say dot map, this dot map function will go through each record one by one. For each record, in every record, what do you have? Problem, solution and DeepSeek-like
step-by-step thinking block, okay? For them, what you do is you take R1 prompt and R1 has these three parameters, right? So when you do format, it will in first block, it will first
put problem, then in the second one, it will put thought, third, it will put solution. Okay? And it will return you the text. So for each of the records, just imagine you're getting this big prompt for each
of the puzzle you have. After that, what happens is you're creating a trainer object. Now, trainer object is coming from transformer reinforcement learning library, which is once again Hugging Face, okay?
So this is the library, folks. It's a Hugging Face library, transformer reinforcement learning library. And when you did pip install on sloth, it kind of install all these because these are the dependencies.
So in that library, there is this SFT trainer, okay? Like supervised fine-tune training trainer, which you are importing and you are giving the base model, that LoRA model, okay?
yet, okay? You have just defined the configuration. So this model has that LoRA configuration that we gave here. See, this this particular configuration. So it knows what is the value of R, what is R LoRA alpha and so on.
Then you will use the dataset and tokenizer. So when you run the training, see, later on I will say trainer.train. So when I do that, it will use this tokenizer for tokenizing. It will use
this dataset to load the record and to train the model. Okay, so these are all the parameters, these are training arguments and these are some of these are familiar to you when you looked into the deep learning
module. We had all this learning parameter, learning rate, weight decay, all of those, okay? Like optimizer. So Adam is your optimizer here and I have given information on all those parameters
here, folks. Read through it. You don't need to remember the syntax here. You can get the syntax through ChatGPT or through the on sloth documentation. So don't stress too much, okay? Like right now you might be thinking, oh, this guy
is just showing the code and he's not typing. Well, typing is not needed nowadays in the world of AI. It's like how you make use of those AI tools and make yourself productive while still learning the fundamentals. That is the
entire uh goal that we need to have. Okay? So, you created this trainer object and then you will do trainer.train. It will take long time, folks. For me, I
think it took like 20 minutes. And it is showing you the epochs and the training loss and you as you can see the loss is reducing, which is a good sign, which means my model is training in the right direction.
And after 60 epochs, my loss is 0.48. So, I read somewhere in the documentation if it is less than 0.5, Uh I mean, it actually depends on your use case, but that's a general
guideline. And then you will try some prompt. So, my model is trained, okay? After that that line, when you the trainer.train, model is trained. Which means model is fine-tuned. So, we
Which means model is fine-tuned. So, we took Llama 3 model. We trained it on the ServiceNow R1 dataset, okay? Which dataset? This dataset. 172,000 rows. And that Llama 3 model was not
And that Llama 3 model was not originally trained to think like this, like DeepSeek, you know, like step-by-step. But now that we fine-tuned the model will also start thinking like DeepSeek, okay? So, here
I'm giving one sample uh question. How many Rs are present in strawberry? And folks, this might look like a simple And folks, this might look like a simple problem. Actually, LLM gets confused
with these problems, okay? So, it's kind of like a hard problem for LLM. I give this uh problem uh statement to my model, uh problem uh statement to my model, okay? And here you are doing tokenizer,
whatever the basic setup, essentially. And then here is where you are doing inference. So, you'll say model.generate. whatever all these parameters, temperature, and so on.
And whatever output you get, you will decode it into a response. So, here, if you look at the response, let me just show you the response. See? Now it is thinking step-by-step.
Let me recap. See, let me recap. So, it is actually thinking step-by-step. It is saying that, "Okay, um you know, it goes through the entire sentence letter by letter and it's
saying, "Okay, R is one. Here R is one." and so on. So, it it is is also doing double-check because in our uh prompt, we asked it to do human-like thinking.
And then in the end, it will tell you total R R 2. Okay? Isn't this amazing, folks? We trained the model. We fine-tuned it We trained the model. We fine-tuned it on this ServiceNow R1 dataset. And Llama
3, which was not originally designed to produce output in this format, now it is able to do reasoning, okay? You can run this in Google Colab. It's going to take long time. Um the code is attached. So, folks,
please read through the code. Once again, let me remind you. I know I'm saying this again and again. You don't need to remember syntax, folks. If you waste your mental energy in remembering syntax, you are not doing yourself any
You can get all this help through ChatGPT AI and API documentation. Your goal is to remember fundamentals, okay? And connect those fundamentals on how do
you solve a business problem. All right. So, that's about it. Uh the code is provided. Please run it in your Colab notebook and uh see how it in your Colab notebook and uh see how it goes.