[0:00] By now, you've probably tried out some really helpful AI models
[0:03] to summarize your data, to act as a pair programmer, and much more.
[0:08] But traditionally, this meant using cloud services.
[0:11] Perhaps you were using an LLM through some type of chatbot, or for me, an API that was hosted on a cloud service.
[0:19] But at the end of the day, I was using someone else's cloud computing resources.
[0:23] Now, what if I told you, though, that there's open source way.
[0:27] to run AI models and LLMs locally from your own machine.
[0:32] And that allows you to save on cost for your AI bills,
[0:36] to keep your data private and as a developer to build out applications and features that use AI from your on machine.
[0:44] Now that's right, today we're gonna be taking a look at Ollama to run large language models or LLMs from your old machine.
[0:53] Now, this is really, really cool because you're able to take a model
[0:56] which might be quantized or compressed and run it from your own system resources
[1:02] and integrate with a huge ecosystem of models from Llama to Mistral and beyond.
[1:07] And for organizations that are looking to use AI in their applications, it can be very helpful
[1:12] because we're able take and deploy a small or large language model
[1:17] locally and ensure that customer data doesn't leave the secure environment at all.
[1:22] And this is all using a simple command that we're going to take a look at here in a second.
[1:27] But how does it work and what should you know about using Olamma?
[1:31] Well let's begin with the Olammas CLI.
[1:33] Now whether you're on Windows, Mac or Linux,
[1:37] you can head over to ollamma.com in order to download the CLI or command line interface for your machine.
[1:46] Now this allows you to download models, to run them, and to interact with them all from your own terminal.
[1:51] While in the past you had to go over to repositories such as Hugging Face
[1:56] in order to download model weights,
[1:57] and you had work with complicated setup in order to get the model ready to be inferenced and chatted with,
[2:03] this is all simplified with Olamma through a single command, Olammarrun.
[2:10] After that, we'll pass in the argument or parameter of the model name.
[2:14] It could be granite, could be llama, could be deep seek, and that'll kick off the process.
[2:19] to download one of the Olamas own models,
[2:22] their compressed and optimized models to your machine, and start up an inference server,
[2:26] similar to how you would start up a web server and serve your web pages.
[2:31] This will drop you into a GPT looking chat window,
[2:34] and you can almost think of the olama run command as a package manager for AI,
[2:39] allowing you to run and manage models with a single command.
[2:43] Now, that's awesome, but speaking of models, Thanks for watching!
[2:46] Olama has a catalog of standardized and customizable language, multi-model embedding and tool calling models.
[2:53] Now, I just said a lot of things, but let's break it down here.
[2:57] The first one that I wanted to mention is a type of model for language.
[3:01] So for example, that means working with your text and data,
[3:04] either in a conversational or base type of format or an instructional format for answering questions and answers.
[3:11] Now, the second type is multi-modal models.
[3:14] So for example.
[3:16] working with images and being able to analyze, hey, what's going on in this specific frame?
[3:21] Now, the next one is a type of model called embedding,
[3:24] which is essentially taking our data from PDFs and other types of data formats
[3:29] and preparing it to be used in a vector database to do question and answering on our own unique data.
[3:35] And then finally, the last type of model that you can use is tool calling.
[3:39] So, it's a fine-tuned version of a language model.
[3:43] that is familiar with calling different functions, APIs, and services in an agentic way.
[3:48] Now, these are some of the types of models, but how would you pick the right model for your use case?
[3:54] Well, it's a good question, and it depends on your project's requirements.
[3:58] But some of popular models that we see being used a lot in the community are, for example, llamas series of models.
[4:06] Different open and fine-tuned models for various use cases that provide support for different languages.
[4:12] as well as IBM's Granite model.
[4:15] So, the Granite Model is a enterprise ready LLM that can be used with RAG or agentic functionality.
[4:22] And we also see these types of reasoning models that are being
[4:26] more and more popular that essentially provide chain of thought or thinking capabilities to answer your questions.
[4:34] Now, beyond just using the Model Catalog, you can actually take advantage of what's known
[4:37] as the Ollama model file to essentially just how Docker has abstracted the complexities of containers.
[4:46] Well, we're using a model file to abstract the complexities models to be able to import
[4:51] from Hugging Face, for example,
[4:53] or to start from a model that you already have
[4:56] and customize it with system prompts and different parameters to be the best model for your use cases.
[5:02] But no matter what type of model that you want to use,
[5:05] your request at the end of the day will be passed through the Ollama server, which is running on localhost on port 11434.
[5:13] So for example, let's say you're making a request or prompt to the model from your CLI, from your terminal with Ollama.
[5:21] Well, this is actually being passed to the O Lama model server,
[5:24] and in a similar way for applications that might want to use models,
[5:28] for example with Lang chain or another framework, you're make a post request to the model that's running on local host.
[5:35] on this specific port, which is a REST server.
[5:39] So it has endpoints, and we can make that request
[5:42] similar to how we would make a request to any other service that's running on our machine.
[5:46] And right here is the simplicity of Ollamo for developers.
[5:50] It lifts the weight of having to run the model in your application, and it abstracts the model as an API.
[5:56] So you make that requests, and you get that response back all locally on your machine.
[6:01] Or let's say you want to run Ollama on another machine and you just SSH there, or you make that request remotely.
[6:08] No matter what you do though, Ollama is doing the heavy lifting of running the model.
[6:12] And you can even connect other interfaces, such as Open Web UI,
[6:16] in order to set up a simple RAG pipeline and use your PDFs or other documents
[6:21] to be passed with the context of that information to Ollama and get that response back.
[6:28] But that is the Ollama server.
[6:29] Now.
[6:31] Whether you want to save on cloud costs when you're using AI models, or have
[6:35] private sensitive data that can't leave your premise, or even
[6:38] limited internet access in an IoT device,
[6:41] you can use Ollamma and the power of open source AI to use and manage LLMs from your local machine.
[6:48] Now, it isn't the only tool for doing this, but as a developer, it's made my life much easier, and I encourage you to check it out.
[6:55] As always, thank you so much for joining us today.
[6:58] Make sure to like the video if you learned something.
[7:00] and have a good one!