How to Run LLMs Locally - Full Guide

[00:00] If you're not running LLMs locally, then

[00:02] you're missing out. Sure, Chat GBT and

[00:05] other hosted solutions are great, but if

[00:07] you care about speed, privacy, and cost,

[00:10] then you'll want to learn how to run

[00:11] them on your own machine. That's why in

[00:14] this video, I'll show you two methods of

[00:16] running LLMs locally from a developer

[00:18] perspective. I'll show you how to

[00:20] download various models, run them

[00:22] interactively, and integrate them into

[00:24] your code to replace something like the

[00:26] OpenAI API. Now, the two methods that

[00:29] I'm going to show you are O Lama and

[00:30] Docker Model Runner. Both are free,

[00:33] highly performant, and you'll learn the

[00:35] differences between them as I go through

[00:36] this video. Let's get started. So, the

[00:39] first method to show you is Olama. Now,

[00:41] Olama is a popular open- source software

[00:44] that allows you to download and run

[00:46] models locally on your own computer.

[00:48] Now, the models that you can run do

[00:49] depend on the hardware and the specs of

[00:51] that computer, but generally speaking,

[00:54] you can run models, you can manage them,

[00:56] delete them, etc., and then access those

[00:58] same models from code. So, let me give

[01:00] you a quick overview into how Olama

[01:02] works. Let's get started. So, first

[01:04] things first, go to ola.com and simply

[01:07] download O Lama for Mac, Windows, or

[01:10] Linux. Once it's downloaded, what you'll

[01:12] want to do is make sure it's running.

[01:13] So, you can go to something like your

[01:15] Spotlight search, search for O Lama, and

[01:17] simply run it and just make sure it's

[01:19] loaded on your computer. If you're on

[01:21] Windows, usually you can see if it's

[01:22] running by looking in the bottom right

[01:24] hand corner in kind of the activity bar

[01:26] and seeing if that Lama icon is

[01:28] appearing. Same thing, you can also kind

[01:30] of go to the explorer search, search for

[01:31] Olama, just double press it. Make sure

[01:33] it's running on your computer. Now, once

[01:35] Olama is running, what you're going to

[01:36] want to do is head over to your

[01:38] terminal. So, open up a terminal or a

[01:40] command prompt and simply type in the

[01:42] command Olama. If you do that and the

[01:44] command works, it means running and

[01:46] you're good to go. Now from here it's

[01:48] very straightforward on how to use this

[01:49] tool but what you're able to do is

[01:51] download and manage various different

[01:53] models. So before you can start using

[01:55] this from code you need to actually pull

[01:57] a model. So the command to pull a model

[02:00] is lama pull and then the name of the

[02:02] model. Now if you're looking for the

[02:04] names of the models you can find them

[02:06] from this Olama kind of search page

[02:08] which I'll link in the description.

[02:09] You'll see there's a ton of different

[02:11] options that you can download, right?

[02:12] You can go through all of them and if

[02:14] you want to see which ones you're

[02:15] actually able to run, you can click into

[02:17] one of them, for example, and you'll see

[02:19] the size of these various different

[02:21] models. Now, for extremely large models,

[02:24] you're going to want to have a GPU and a

[02:25] lot of RAM on your computer. And if

[02:27] you're not sure, you can go to something

[02:28] like Chat GBT and just ask it based on

[02:31] the specs of your machine if you would

[02:32] be able to run this model. So, I suggest

[02:34] when you're testing out just to go with

[02:36] a small one. So, what I'm going to do is

[02:37] just search for one called Small M2.

[02:40] This is a pretty small model that should

[02:42] run locally. Whether you have a GPU, a

[02:44] lot of RAM, CPU, doesn't matter. And you

[02:46] can see that in order to actually run it

[02:48] or to pull it, use a llama run and the

[02:50] name of the model, which is this right

[02:51] here. So, what I'll do is I'll just copy

[02:53] this name here of the one that's only

[02:54] 271 megabytes. I'm going to go to my

[02:57] terminal. I'm going to paste it. So, a

[02:59] llama pull small m2 col 135m. When I do

[03:04] that, it's going to start downloading

[03:05] the model to my machine. Once that's

[03:07] finished, I'll be right back and we can

[03:09] continue. So, the model has just

[03:10] finished downloading here. And now, if I

[03:12] want to start using it, what I can do is

[03:14] I can start by typing LS. That's going

[03:17] to show me a list of all of the

[03:19] different models that I have on my

[03:20] computer. You can see I have a bunch of

[03:21] them here, a lot of them that I was

[03:22] testing with many months ago. And if I

[03:24] want to run a particular model, then I

[03:27] can type run and then simply specify the

[03:30] name of the model that you see here. So,

[03:32] I'm going to go small M2 col135M

[03:36] and then run that. And you'll see that

[03:37] now it brings me into an interactive

[03:39] chat where I can start typing directly

[03:41] with the model. And you'll notice that

[03:43] it's quite fast because it's running on

[03:45] my own computer and we have no network

[03:46] latency. So I'll say what is let's do

[03:49] this. The capital of Canada for example

[03:53] and you can see that it says the capital

[03:55] of Canada is Paris. Okay. So that

[03:57] obviously got that wrong. Now that's the

[03:59] thing with these super small models.

[04:00] They don't have access to the internet.

[04:02] They're running locally on your own

[04:03] computer. They can make mistakes. And

[04:05] this particular one is a super small one

[04:07] that's just meant to do small predictive

[04:08] text. So don't expect it to be perfect.

[04:10] Now if you want to exit this particular

[04:13] window, you can type /by. If you do

[04:15] that, it's going to bring you out of

[04:16] this and then you can continue using

[04:18] lama. That's pretty much all you need to

[04:20] do from the command line. And now what

[04:22] I'll do is show you how you can run

[04:23] Olama models from code because that's

[04:26] where they actually become useful after

[04:28] a quick word from our sponsor boot.dev.

[04:31] It's an online learning platform

[04:32] designed specifically for back-end

[04:34] development, and it approaches learning

[04:36] in a way that's far more interactive

[04:38] than the usual video-based courses.

[04:41] Rather than having you sit through hours

[04:42] of lectures, boot.dev puts you straight

[04:45] into hands-on coding. You'll work

[04:46] directly in your browser building real

[04:49] projects while learning back-end

[04:50] fundamentals like APIs, databases, and

[04:53] serverside logic using Python and Go.

[04:56] Now, what makes it stand out is the way

[04:58] that it borrows from game design. You'll

[05:00] progress through levels, unlock new

[05:02] content, and keep your momentum up as

[05:04] you go. The platform is filled with

[05:06] exercises and practical challenges, so

[05:08] you'll end up writing a lot of code,

[05:10] which is exactly what helps you improve.

[05:12] Now, all of the core content is free to

[05:14] access, and if you decide to commit to

[05:16] the annual plan, you can use the code

[05:18] tech with Tim to get 25% off your first

[05:21] year. I've been going through it myself

[05:22] lately, and honestly, it's surprisingly

[05:24] addictive. Thanks to boot.dev. Now,

[05:26] let's get back into it. So at this

[05:28] point, I'm assuming that you've

[05:29] downloaded a llama, you've pulled at

[05:31] least one model, you're kind of familiar

[05:32] with running it, and now you probably

[05:34] want to actually use this from some type

[05:36] of application, right? So from code. So

[05:39] in order to do that, there's a few

[05:40] different ways. I'm going to show them

[05:41] to you directly here inside of Python.

[05:43] However, this works in pretty much every

[05:45] programming language. Just make a few

[05:47] small adjustments. So, whenever you have

[05:49] a llama running on your computer, it's

[05:51] going to expose an HTTP server or a REST

[05:54] API that allows you to just directly

[05:57] call the REST API and get a response.

[06:00] So, you don't need to use any fancy

[06:01] modules. You can just send a normal HTTP

[06:03] request directly to that server and it

[06:06] will give you a response from the AI

[06:08] model. So, for example, this is the URL.

[06:10] So, HTTP localhost port and then by

[06:13] default, O Lama will run on port 1434.

[06:16] You can go to the slash API/ chat. You

[06:19] can pass some data. So in this case, I'm

[06:21] passing the model I want to use is small

[06:23] M2. I want to stream this. No, I don't.

[06:26] And then I'm passing a few messages that

[06:27] I want this to respond to. So the first

[06:29] message is a system message saying

[06:31] you're a helpful assistant. And the next

[06:32] one, please write me 500 words about the

[06:34] fall of Rome. Now system messages are

[06:36] ones that tell the model what it should

[06:38] be doing and give it additional context.

[06:40] Whereas user messages are ones that will

[06:42] actually be responded to directly. I

[06:44] then send a post request to this

[06:45] endpoint and then I'm able to get the

[06:47] response and grab the message and the

[06:48] content. So if I come here and I type UV

[06:52] run and now this is in lama local and

[06:54] then main.py it will take a second here

[06:56] because it is sending the network

[06:57] request and then we should get the

[06:59] response. Okay. And you can see that we

[07:01] get the response here 500 words about

[07:03] the fall of Rome directly inside my

[07:05] terminal. Awesome. So that is the first

[07:07] way. Now the second way to use Olama

[07:09] from code is to simply import the Olama

[07:12] module. Now, in order to use this

[07:13] module, you need to install it. So, you

[07:15] would simply type pip install lama if

[07:18] you're using something like Python. And

[07:20] then you would be able to actually

[07:21] import this module and use it from code.

[07:23] So, assuming you've installed that

[07:24] module, you'll be able to import it.

[07:26] Same thing, you can reference the model

[07:27] name, pass any messages you want, and

[07:29] this time, rather than manually calling

[07:31] the HTTP server, you can simply use the

[07:33] lama.

[07:35] Function or method, you can pass the

[07:37] model and the messages, and then you can

[07:39] get the response. This works effectively

[07:41] the exact same way as the code you saw

[07:42] before just wrapped in this nice Olama

[07:44] module. So if I run the command here you

[07:46] can see same thing it will take a second

[07:48] and then it will give me the response

[07:49] and there you go you can see we get our

[07:51] 500 words same as before showing up

[07:53] inside of the terminal. So to recap, if

[07:56] you want to use a llama, you simply

[07:58] install it, pull a particular model, and

[08:00] then you can call the HTTP REST API if

[08:03] you want to get a response or you can

[08:04] use something like the Lama module if

[08:06] you're working in a language like Python

[08:08] to directly get the reply from code.

[08:10] This is the first way to run LLMs

[08:12] locally. It's very good. It's very

[08:14] popular and you can use this inside of

[08:16] frameworks like Langchain, for example,

[08:18] or really any other AI framework that

[08:20] you want. But now I want to move on to

[08:21] the next method which is a newer one

[08:23] that many people don't know about which

[08:25] is the docker model runner. Now the next

[08:27] method of running LLMs locally is

[08:29] actually my preferred choice and that is

[08:31] to use the docker model runner. Strictly

[08:34] speaking this is a better more efficient

[08:36] way to run models than using a llama. It

[08:39] works with more systems. It has better

[08:41] GPU acceleration and support and it

[08:43] works inside of containerized

[08:45] applications especially when you're

[08:46] moving to deployment. I'm not going to

[08:48] cover the entire setup and all of the

[08:50] configuration in this video. I'll give

[08:52] you a quick intro so you can see how it

[08:53] works, but I do have a longer video

[08:55] which I'll link in the description which

[08:56] explains how to use this inside of

[08:58] things like containerized applications

[09:00] that really gets the benefit of using

[09:03] Docker Model Runner. So anyways, just

[09:05] understand this is a better solution in

[09:07] almost every single way. You don't need

[09:09] to know why or all of the specific

[09:10] details. So I would suggest using this

[09:12] one and I'll show you how to set it up.

[09:15] Okay, so Docker Model Runner allows you

[09:17] to simply run models using Docker. So in

[09:20] order to use this, you do need to have

[09:22] Docker Desktop installed. If you don't

[09:24] already have it, it is completely free

[09:25] to download. And you can simply just

[09:27] download it by going to Docker Desktop,

[09:29] right? Searching for it, and then

[09:30] finding the download link like this. So

[09:32] you can download it for your operating

[09:34] system. Now, once you have Docker

[09:35] Desktop downloaded, make sure that you

[09:37] have the newest version on your computer

[09:39] and simply open it up. Similarly to

[09:41] before, you can go to your spotlight

[09:42] search and just search for Docker

[09:44] Desktop, right? And then open the

[09:46] application. Same thing if you're on

[09:48] Windows. Now, once you open the

[09:49] application, what you're going to want

[09:50] to navigate to here is this settings

[09:52] window. You're going to want to go to AI

[09:55] and make sure that you enable the Docker

[09:57] model runner. Okay? So, settings, AI,

[09:59] enable docker model runner, and then

[10:02] enable the host side TCP support and

[10:05] change cores to say all. Now, if for

[10:08] some reason you're not seeing this, you

[10:09] can go into beta features and make sure

[10:11] that you enable the Docker MCP toolkit.

[10:14] Once you do that, all of this should be

[10:16] good. You can enable this setting and

[10:18] you're good to go to run the model

[10:20] runner. Now, once you've enabled that,

[10:21] you can actually go directly inside of

[10:23] Docker Desktop, go to models, and from

[10:26] here, you can actually pull models and

[10:28] start using them directly inside of

[10:29] Docker. So, right now, we're looking at

[10:31] my local models. You can see that I have

[10:32] one here called small 2. But if I change

[10:35] over to Docker Hub, then there's a list

[10:37] of models that I can download simply

[10:39] from this UI. So, same thing as before,

[10:41] I'll just use that small two model. So,

[10:42] I can just type small. And you can see

[10:44] there's a bunch of options here. Maybe

[10:45] we want to go small three now because

[10:47] this is the updated version. I'll just

[10:49] find the smallest one, which is latest.

[10:51] And then I can just press download here

[10:53] to pull. So, I can do this directly from

[10:55] the user interface. And then I'll be

[10:56] able to run the model right inside of

[10:58] Docker Desktop. For example, this model

[11:00] that's already downloaded. I can just go

[11:02] here to run and I can start chatting

[11:04] with it directly inside of this UI. You

[11:06] can see if I say hello, it just gives me

[11:07] the response. Now I can also go into

[11:09] inspect. I can see all the information

[11:11] and I can view all of the requests that

[11:13] have been sent, the context, usage,

[11:14] duration, etc., etc. So that's great. We

[11:17] can do that directly from the UI, which

[11:19] in my opinion is a little bit easier

[11:20] than a llama or we can do it directly

[11:22] from the command line. So same thing as

[11:24] before, if you have docker model runner

[11:26] enabled, you can simply type the command

[11:28] docker model. When you do this, it will

[11:30] give you a very similar UI to O Lama.

[11:33] From here, you could type docker model

[11:35] pull and then you can pull a particular

[11:37] model. So again, same thing, small m2 if

[11:39] we wanted to pull that and then it would

[11:41] start downloading it. You can see in my

[11:42] case, it's cached. I already have it. We

[11:44] can type docker model list. If we do

[11:48] that, it gives us a list of the local

[11:49] models that we have available. And then

[11:51] there's a few other ones that we can

[11:52] look at here, right? Like packaging

[11:53] models, listing them, inspecting,

[11:55] running, deleting, etc., etc. So also

[11:58] from here we can go docker model run

[12:01] small m2. When we do that brings us to

[12:03] the interactive terminal. Same thing we

[12:05] can type hello. If we want to escape we

[12:07] can type /by and it will remove us from

[12:09] there. Okay. So effectively same thing

[12:12] as a llama just a slightly better user

[12:13] interface in my opinion. If you're

[12:15] looking for all of the different models

[12:16] you can find them directly from the UI

[12:18] here or you can go to the hub.docker.com

[12:22] which I will link in the description and

[12:24] search for all of the available models

[12:25] on this page. Cool. So with that said,

[12:27] similarly to before, let me show you how

[12:29] you can use the Docker model runner

[12:31] directly from code. So what you're

[12:33] seeing on my screen is an example of

[12:34] Python code that sends a request to

[12:37] docker model runner to use the same

[12:39] model as before, the small M2 model,

[12:41] this time from docker model runner as

[12:43] opposed to now. You'll notice that the

[12:46] only thing that's changed about this

[12:48] code is simply the port that I am

[12:50] calling. Now, Olama, if you remember

[12:52] before, runs an HTTP REST API on port

[12:56] 11434.

[12:57] Docker Model Runner runs it on port

[13:00] 12434. So, the only change is that it's

[13:03] 12 versus 11. So, all I have to do is

[13:06] simply change my URL here to be 12434

[13:10] and slightly change the endpoint or path

[13:13] that you can see. And similarly to

[13:15] before, I can send a request directly to

[13:17] Docker Model Runner and get a response.

[13:20] That is because just like Olama, Docker

[13:21] model runner runs its own REST API that

[13:24] you can call. So what I can do here is

[13:26] type uvron docker local/main.py

[13:31] and you'll see that it will just take a

[13:32] second here and it will give me the

[13:33] response. Okay. And there we go. You can

[13:36] see I get the 500word response popping

[13:38] up directly here. Now similarly to that

[13:41] we also can directly use modules like

[13:43] the OpenAI module. So previously I used

[13:46] the Olama module. Obviously, we're not

[13:47] going to use the Olava module when we're

[13:49] working with Docker Model Runner.

[13:51] However, there are modules like OpenAI

[13:53] or Langchain. Now, those by default are

[13:56] going to use something like OpenAI's

[13:58] public API. However, they have the

[14:01] ability to actually override the base

[14:03] URL. So, we can actually change the base

[14:06] URL to be the Docker model runner, which

[14:08] is running locally on our own computer.

[14:10] And now we can use this module just like

[14:13] we would before with all of the nice

[14:15] completions and methods except instead

[14:17] of it sending a request to OpenAI which

[14:19] we would need to pay for, we can send it

[14:21] locally to our own machine. So you can

[14:24] see that if I want to specify the model

[14:25] name, I do AI slash and then the name of

[14:28] the model. So in this case, small M2. I

[14:30] can specify a prompt and then I can use

[14:32] the OpenAI module just like I would for

[14:34] any OpenAI request. So for example, UV

[14:37] run docker local slash and then this

[14:40] open AI module. Same thing just wait a

[14:43] second and you can see it explains how

[14:45] transformers work. Boom. So there you

[14:47] go. That is how you use it directly from

[14:49] code. Now a lot of you at this point

[14:51] might be asking okay well what's the

[14:52] difference between the docker model

[14:54] runner and strictly speaking the docker

[14:56] model runner is more optimized works

[14:58] better with GPUs has more support and

[15:01] also works for containerized

[15:03] applications. So right now I'm just

[15:05] showing you a very basic app. However,

[15:07] if I had this wrapped inside of a Docker

[15:09] container, I can actually expose the

[15:11] Docker model runner directly inside of

[15:13] that container. I can have the model

[15:16] actually wrapped in the container so

[15:17] that it doesn't take up a massive amount

[15:19] of space and then I can run the model

[15:21] directly inside of the containerized

[15:23] app. Now, I'm not going to explain that

[15:25] in this video because I know a lot of

[15:26] you won't find that that useful.

[15:28] However, I do have a video on YouTube,

[15:30] I'll link it in the description, that

[15:31] explains how to do this, where

[15:32] essentially you can expose an LLM

[15:34] service to your Docker container in

[15:36] something like a Docker Compose file

[15:38] that allows you to specify what model

[15:40] you want to run on what port, etc.,

[15:43] etc., and that will allow it to ingest

[15:45] the AI model in just a much better way,

[15:47] especially when it comes to deploying

[15:48] this application out. So, with that

[15:50] said, that's going to wrap up this

[15:52] video. I hope that you found this

[15:53] useful. If you did, make sure to leave a

[15:55] like, subscribe, and I will see you in

[15:57] the next one.

[16:00] [music]