---
title: 'Ollama Course – Build AI Apps Locally'
source: 'https://youtube.com/watch?v=GWB9ApTPTv4'
video_id: 'GWB9ApTPTv4'
date: 2026-06-14
duration_sec: 10644
---

# Ollama Course – Build AI Apps Locally

> Source: [Ollama Course – Build AI Apps Locally](https://youtube.com/watch?v=GWB9ApTPTv4)

## Summary

This course teaches how to use Ollama, an open-source tool for running large language models locally. It covers setup, model management, REST APIs, Python integrations, and real-world projects like a grocery list organizer, RAG system, and AI recruiter agency.

### Key Points

- **Course Introduction** [00:00] — Ollama simplifies running LLMs locally. The course covers pulling/customizing models, REST APIs, Python integrations, and projects like a grocery organizer, RAG system, and AI recruiter.
- **What is Ollama** [02:00] — Ollama is an open-source tool that simplifies running LLMs locally on your own hardware, abstracting technical complexity.
- **Ollama Deep Dive** [06:44] — Ollama uses a CLI to manage installation and execution of models. It provides a straightforward way to download, run, and interact with various LLMs without cloud services.
- **Problem Ollama Solves** [08:52] — Ollama addresses cost, privacy, latency, and customization issues. Local execution eliminates API costs, keeps data private, reduces latency, and allows model fine-tuning.
- **Key Features of Ollama** [13:10] — Model management, unified interface, extensibility, and performance optimizations including GPU acceleration.
- **Use Cases** [15:10] — Development/testing, education/research, and secure applications in healthcare/finance where data privacy is critical.
- **Installation on Mac** [17:58] — Download from ollama.com, install the application, and run 'ollama run llama3.2' to get started.
- **Interacting with Models via CLI** [21:00] — Use 'ollama run <model>' to start a shell. Commands like /show info display model details.
- **Model Selection and Parameters** [26:28] — Ollama library hosts many models. Parameters (e.g., 3B, 1B) indicate model size and complexity; larger models are more accurate but require more resources.
- **Understanding Model Parameters** [33:00] — Parameters are internal weights learned during training. More parameters generally mean better performance but higher computational cost.
- **Context Length and Embedding Length** [37:30] — Context length is max tokens per input; embedding length is vector size for token representation. Larger values capture more nuance.
- **Quantization** [39:26] — Technique to reduce model size by lowering weight precision (e.g., 4-bit), resulting in smaller, faster models with lower memory usage.
- **Ollama CLI Commands** [42:57] — Commands: ollama list, ollama pull, ollama run, ollama rm, ollama help. Models can be pulled and run interchangeably.
- **Multimodal Models (LLaVA)** [47:00] — LLaVA combines vision encoder and LLM for visual understanding. Example: describing an image of flowers.
- **Customizing Models with Modelfile** [54:00] — Create a Modelfile with FROM, PARAMETER temperature, SYSTEM message. Use 'ollama create' to build a customized model.
- **REST API Endpoints** [60:00] — Ollama serves at localhost:11434. Endpoints: /api/generate, /api/chat. Use curl with stream=false for complete responses.
- **UI-Based Interface with Msty** [66:00] — Msty app provides a ChatGPT-like UI for local models. Supports knowledge stacks for RAG with embedding models.
- **Python Library Basics** [84:00] — Install ollama Python library. Use ollama.list(), ollama.chat(), ollama.generate() to interact with models programmatically.
- **Streaming Responses in Python** [96:00] — Set stream=True in chat() and iterate over response chunks to display tokens as they arrive.
- **Grocery List Organizer Project** [103:00] — Use LLM to categorize and sort grocery items from a text file. Prompt instructs model to categorize into produce, dairy, etc., and sort alphabetically.
- **RAG System Overview** [111:00] — RAG = Retrieval Augmented Generation. Combines document retrieval with LLM to answer questions based on custom data, reducing hallucination.
- **RAG Architecture** [117:00] — Documents are chunked, embedded, stored in vector DB. User query is embedded, similar chunks retrieved, and passed with prompt to LLM for answer.
- **LangChain for RAG** [120:00] — LangChain provides abstractions for document loading, splitting, embeddings, vector stores, and retrieval. Simplifies building RAG pipelines.
- **Building RAG with Ollama and LangChain** [126:00] — Use Ollama embeddings (nomic-embed-text) and LLM (llama3.2) with ChromaDB. Multi-query retriever generates multiple query perspectives for better retrieval.
- **AI Recruiter Agency Project** [148:00] — Multi-agent system using Swarm framework with Ollama. Agents: extractor, analyzer, matcher, screener, recommender, orchestrator.
- **Base Agent Class** [157:00] — BaseAgent sets up OpenAI client with custom base URL for Ollama. Provides query_ollama() method and JSON parsing helper.
- **Specialized Agents** [162:00] — Each agent (e.g., ScreenerAgent) inherits from BaseAgent, defines instructions, and implements run() method. Agents are called by orchestrator.
- **Orchestrator Agent** [168:00] — Coordinates workflow: extract resume, analyze profile, match jobs, screen candidates, generate recommendations. Maintains workflow context.
- **Streamlit UI for Recruiter** [172:00] — Streamlit app provides tabs for upload, skills analysis, job matches, screening, and recommendations. Results saved to text file.

### Conclusion

Ollama democratizes local AI by enabling free, private, and customizable LLM applications. With CLI, REST API, Python library, and integration with frameworks like LangChain and Swarm, you can build powerful AI solutions entirely on your own machine.

## Transcript

olama is an open-source tool that
simplifies running large language models
locally on your personal computer in
this course you'll learn how to set up
and use olama this Hands-On course
covers polling and customizing models
rest apis python Integrations and real
world projects like a grocer lless
organizer rag system and an AI recruiter
agency Paulo created this course hello
and welcome to this mini course I'm very
excited that you're here because I know
you want to get right in and start
learning about AMA and building AI based
Solutions using AMA first let me go
ahead and introduce myself again which
is always awkward okay my name is Paulo
deson I am a software Ai and Cloud
engineer but most importantly I am an
online instructor and I have taught over
280,000 students
all over the world in about 175
countries uh skills ranging from
building mobile applications Android to
learning Ai and Cloud Technologies so
this is what I do and I love what I do
what is this minicourse about I want to
show you how to leverage olama and its
many many models so that you can build
AI Solutions a applications locally for
free usually the structure that I like
to follow and has Pro proven to work is
in the beginning or the first part of
the courses I start with the basics the
fundamentals as well as Theory Concepts
because I think it's important for you
to also understand the theory before you
go into Hands-On and then the second
part is usually Hands-On but sometimes I
like to mix those two together so that
it's more exciting and you don't get
bored okay so I know you're very excited
to get started and I'm excited that
you're here okay let's go ahead and get
started well this course is going to be
about ol that's the main thing so we're
going to learn how to build local large
language model llm applications using
AMA so the idea is that we're going to
be able to use AMA to customize our
models meaning that we are able to use
different flavors of models so we can
test them around and all of that is
actually going to be free which is very
exciting and also we're going to be able
to build rag system so retrieval
augmented generation systems that are
powered solely by AMA models keep in
mind all of this is going to be for free
you're not going to have to pay for
anything okay and we are going to learn
about tools and function calling when we
are using these AMA models so we can Aid
our large language models that are going
to be using with more things that they
can do okay I'm going to look at all of
that and ultimately I'm going to show
you how to use AMA models to build
full-fledged large language model
applications using these models okay
keep in mind all of these models are
going to be for free we don't have to
pay a dime to use these models now let's
talk about who is this course for now
this course is for developers AI
Engineers open-minded Learners machine
learning engineers and so forth as well
as data scientists so if you are
somebody who is willing to put in the
work and wants to learn about AMA and
build local llm applications then this
course is for you now let's talk about
course prerequisites in this course I
assume that you know programming this is
a very um important skill obviously and
I expect you to at least know the basics
you'll be seeing a lot of python in this
course so understand understanding how
python Works how to write python code is
obviously very important you also need
to understand very basics of AI machine
learning large language model you don't
have to be an expert but just have some
general
understanding keep in mind also that
this is not a programming course because
I'm not going to be teaching you how to
program and most importantly I always
tell my Learners is that you have to be
willing to learn and if there things
that you need to go and brush on just go
and do that and come back to the course
but most likely you won't need to go
anywhere but continue in this course and
learn as much as you can so most of my
courses I have this mixture of two
things I have Theory so this is where we
talk about the fundamental concepts the
lingo and so forth and I have Hands-On
because it's all about actually doing
things that way you actually understand
and know how to get things done that's
the whole point and so the thing is I'd
like to do a mixture of both and
sometimes we have more handson and
sometimes we we may have more Theory but
I try to balance it all out where we
have mixture both because I know I
believe at least that is important for
you to have both in your learning
process but in this course you are
mainly going to be focusing on Hands-On
although you will still have some Theory
fundamentals as well all right so let's
look at the development environment
setup so in this case you know that this
is all P about python which means you'll
have to have python installed and also
you have to have some sort of a code
editor for me I'm going to be using VSS
code so you don't have to use that if
you don't like it but that's what I'm
going to be using you can use whatever
you want okay now I'm not going to go
through the process of installing python
or vs code or anything like that because
you should be able to have that if not
you should be able to do that on your
own so one place that I always recommend
people go is to this link here kin.com
knowledgebase install python there are
many places online where you can learn
or they will walk you through the
installation of python if that's your
case uh but this seems to be one of the
best my favorite okay so if you don't
have python installed please go ahead
and get that set up so that we can
continue in the next next videos now we
have everything set up on our machine
and we're ready to go so we're going to
do a deep dive into AMA so you
understand exactly what is how it works
and what problem um AMA solves all right
so let's go ahead and get started now
it's time for us to do the Deep dive in
ama so we understand exactly what is AMA
what is the motivation behind and the
advantages that ama brings for us
developers okay so first let's look at
AMA what is it ama is an open-source
tool that is designed to essentially
simplify the process of running large
language models locally meaning on your
own Hardware the idea here is very
simple as you know right now if you want
to run large language models or if you
want to use a model large language model
in this case just a model most likely
you'll have to use open Ai chbt and so
forth and many others out there that are
paid and the thing is with a Lama you
don't have to pay for anything it's free
and that's the beauty so the idea is
that ama sits at the center and allows
us developers to pick different large
language models depending on the
situation depending on what we want to
do at its core AMA uses what we call the
CLI which is command line interface what
this does is it manages is all the
things in the back end so the
installation and also the execution of
different models but all of that again
keep in mind it's all locally you will
see as we go through that ama abstracts
away the technicality so the technical
complexity that are involved when we
want to set up these models which makes
Advanced language processing accessible
to a broader audience such as developers
researchers and hobbyists in nutshell
AMA provides a straightforward way to
download run and interact with various
models or LMS without us having to rely
on cloud-based services or even dealing
with complex setup procedures rather
what is the problem that ama solves now
when we talk about large language models
we are usually talking about this thing
called rag system retrieval augmented
generation the idea is very simple is
that we have documents those documents
are chopped into smaller chunks right
and then we pass those through a model
that is responsible for creating
embeddings which is the vector
representation of these chunks those
embeddings are saved somewhere and then
a query comes in goes through the same
process of embedding right and then once
we have this embedding that is indeed
what's actually used to query do similar
search internally in a factor database
so that we can then pass all that
information right through a large
language model a model and of course get
the response now again usually what
happen is that all these models that we
need to run a rag system for instance we
may need to pay such as when we use
openingi trpt and all different models
that they provide okay now the other
thing also behind all of that is that
when we are using AMA models these are
ours meaning that we download them
internally in our local machine and we
have a lot of control okay so that is
the beauty of or in this case that is
the problem that they solved but that's
not the only problem that been solved
the other one is that privacy so in this
case here when we run our own models
locally uh we are making sure that our
data doesn't need to be sent to external
servers because remember when we passing
data through a large language model
using chbt in this case or openi models
and so forth we literally are passing
through or sending our private data to
different servers that we don't know
what will happen out there so this way
we have some security issues but with
AMA that goes away because we have this
enhanced privacy and security this is
very important because imagine that you
have an application that needs to be uh
dealing with sensitive information then
it's important to have your own models
your own llm so that you have this
contained enir environment where you're
not sending out that sensitive
information when you want to set up
large language models it is cumbersome
which means is technically challenging
and often requiring knowledge of machine
learning
Frameworks and also Hardware
configurations but as you will see with
AMA all of that is simplified so this
whole process is simplified by handling
the heavy lifting for you also most
importantly cost efficiency now we're
going to be eliminating the need for
cloud-based Services which means you're
going to avoid this ongoing costs
associated with API calls or server
usage because everything remember is
going to be local goal so once you have
everything set up you can run models
without additional expenses the other
one is lency reduction what that means
is that local execution obviously
reduces the latency inherent in network
communications so when we are
communicating with something that is
somewhere else through the network of
course there is always a delay even if
the network is really fast there's
always this latency the delay okay but
since every everything is local we don't
have that problem okay because we're not
dealing with remote servers which
results in Faster response times for
interactive
applications and also most importantly
we are able to customize our own models
because when we running these models
locally it allows for greater
flexibility in customizing or even
fine-tuning those models to better suit
specific needs without having this
limitations imposed by thirdparty
services so these are some of the
advantages we have by using AMA
models so I know that we are eager to go
ahead and start doing the Hands-On but
let's continue here to talk about some
key features of AMA okay so the first
one is what we call the Model Management
this is a big one which means that we
are able to easily download and if we
want to able to switch between different
models or between different large
language models and that is the main
point about AMA is that the idea is that
we have this Center place called Lama as
a open source as a framework as a tool
that essentially allows us to manage all
of the different models that we can use
interchangeably okay and next one of
course is we have this unified interface
what does that mean well that means that
we're able to interact with various
models using one consistent set of
commands as you will see and of course
we also have what we call extensibility
what that means well it that means we
have this support for adding custom
models and extensions if we need to
obviously we have what we call also
performance optimizations which means
you're able to utilize your own Hardware
effectively including GPU acceleration
if available because remember all AMA
allows us to do is to manage all this
large language models that we can
download locally that is the key word
key Point here and then we can do all
sort of things with that we have this
performance that is optimized because
everything is internal it's locally as
well extensibility because we can add
different models and extensions as we
see fit and of course we have this
unified interface because we're able to
interact with different models uh but
still using a very consistent set of
commands as you will see but the main
point here of of course is that we have
this Model Management in one place we're
able to easily download and switch
between different large language models
let's look at some use cases well as you
can see you can imagine we have a lot of
different use cases uh
for one of the first one is development
and testing so you can imagine that as a
developer you are able to look for
different large language models and
switch them to test them out to see
which one is is going to perform better
depending on what you want done so
that's huge right so this is a deal of
course for developers who are looking
for ways to test applications that
integrate large language models without
having to set up different environments
and all that complexity so also for
education and resource what that means
is that makes it really easy as a
platform for Learning and
experimentation without the barriers of
Entry that are asso ated with cloud
services so it's really easy just
download certain kinds of large language
models and you can go ahead and do
testing and do some research and so
forth and we talked about this secure
applications the beauty of having AMA is
that it provides this managed or this
management platform of all of these
different large language models which
means it's suitable for Industries like
healthc care or Finance where data
privacy is very critical because
everything is your own you download the
large language models on your own
machines so in conclusion AMA addresses
these challenges of accessibility
privacy and of course as we talked about
cost in the realm of large language
models so by enabling local execution
which is exactly what we do with AMA it
democratizes the use of advanced AI
Technologies make making them more
accessible and practical for a wide
range of applications which is exactly
what you would want okay so now it's
time for us to look into setting up AMA
locally um and what we'll do we're going
to do go through the installation
process and do the setup before we do
setup there's some things that I need to
let you know most of you should be okay
there are some system requirements we
need to be aware of first of all uh AMA
supports Mac Linux and windows
and I'm sure other operating systems
there as well but these are the main
ones and also make sure that we have at
least close to 10 GB of free storage on
your machine this is very important
because as you will see some of these
models require a lot of space and also
processor as long as you have a modern
CPU processor you should be fine and
most of you should have that in place
okay so just something to keep in mind
all right let's go ahead and get started
with
setting installing and setting up AMA so
if you go to ama.com O lam.com as such
this is what you are going to be able to
see says here get up and running with
large L models now because this is a
moving Target in technology things may
change a little bit or dramatically by
the time you watch this video hopefully
not but the most important thing here to
remember is that the IDE is still the
same you will be able to download this
tool and use that way independent on how
it looks okay so should look the same
but you never know and you can see here
at the recording of this video says here
Lama 3.2 this is the latest Lama 3.2
model that they provide we need to
download AMA how do we do this well it's
very simple all you do click here
download Once you click that this will
take you to download AMA page now you
can see we have three different flavors
we have Mac OS so the browser was able
to pick up that I am on Mac and so
offered Mac if you're on Linux of course
you'll have to click and look at this so
you'll have to install by taking this
command and running on your terminal if
your windows of course is also just like
mac you have to download as an
application and run it okay whichever
way you have to download the AMA zip
file so I'm on Mac I'm going to click
download
and then you can see that it is
downloading here so I'm going to wait
for a
second all right so it was downloaded
and let's go ahead and click and open
real quick so this is what you should
see a zip file at least on Mac all right
so I'm going to go ahead and double
click and it's going to and zip and then
it's going to look like this as an
application okay so what I will do I
just double click again and it will
install again as an application on my
back so I'm going to say open that's
fine and then I'm going to move to
applications and there we go so once
we've done that I'm going to show you
real quick
here this is what I see so I have now
this window on Mac that allows us to go
through the process I'm going to say
next and install the command line I'm
going to say go ahead and say install
and I'm going to pass here my
credentials say okay so run your first
model so we're going to run this AMA run
Lama 3.2 now again this number here may
be different from when you're watching
this doesn't matter just follow the
instructions so I'm going to say finish
so what happens now is going to go ahead
in the back and run the command and
everything is good so it's going to
install the Llama 3.2 which is a model
that's a large language model that's
going to be installed with our
installation so we can get started and
you can see once that happens that the
Llama is running because now now we have
this icon at the top that is showing
there this is Lama icon and you can quit
from here as well okay next what we can
do is let's go back to models this page
this is where all the different models
that ama has this is where they are
aggregated and we can look into them and
learn about them you can see at the top
we have Lama 3.2 which is the latest one
which is the one that was installed
already locally in our machine well come
to this later and see more now let's go
ahead and click on this one so I can
show you something real quick here okay
the beauty here is that when you click
on any of these large language models
you will see essentially the same things
okay we have these 3B we'll talk about
this later and um 1B latest and so forth
okay and then most importantly we also
have all of these things that we don't
care about at this point we have a
readme we have the sizes it tell us
exactly what they mean what they good at
this 3B default for instance and how to
run it so this is essentially what you
have at the top here so we're going to
cop this and we're going to open a
command line or a terminal I have one
open and I'm going to paste that so what
this does well it say Lama run Lama 3.2
so what we saying here we're going to
use a llama as the tool the framework
right to run llama 3.2 which
incidentally it's already installed on
our machine right I'm going to just
enter and what we'll do is going to spin
up the Lama
3.2 model for us and then open this
shell where we can start interacting
with the model just like that now I need
you to think to go back to what we
talked about the beauty or the
advantages of using ol is that ama has
different models that we can use and
install them internally locally on our
machine and we can easily then through
AMA as this manager of this Model start
interacting with different models right
now we are interacting we're able to
interacting through the shell here with
Lama 3.2 model which means I can ask
question
like what how
old is the
universe look at that and it gives me an
answer the thing to keep in mind is that
Lama 3.2 tends to be a little bit verose
and as you can see uh it is quite veros
but that's okay I can say in short tell
me how old is the universe so you can
actually direct it to tell you exactly
what you need right kind of kind of like
prompting it a little bit so the
universe is approximately 13.8 billion
years all right so I can also say for
instance clear see what's going to
happen glad you could convey that
information clearly
I guess I have to say something like
this clear it cleared the session okay
now I can continue here by saying for
instance for help I can say forward
slash I can put the exclamation point or
I can say help like this so this will
give me it's actually exactly other
commands that I can use we have the set
show load save and clear session as we
saw as well as Buy to exit the shell so
let's go ahead and use this show here
okay I'm going to say forward slash show
what we want what is that I want to show
I want to show info so this will give us
information about the model that we are
looking at which is the Lama 3.2 says
here architecture is llama the
parameters is 3.2b we'll talk about that
and the context length is that and
embedding length and so forth and some
other licensing information so just like
that ladies and gentlemen we're able to
install AMA and of course because AMA
itself is just a manager we need to have
the actual large language model and
that's easy to do by installing in this
case llama 3.2 we could have installed
any other model of course but we start
with that one when we're installing
AMA all right so now I can say another
question tell me a short
joke okay here's one why couldn't the
bicycle stand up by itself because it
was too tired very funny all right and
if I want to get out of this shell I can
say for a slash buy like this and you
can see we're no longer in our Lama 3
shell so I can clear and we're done okay
very good so I hope this is a good
introduction there is a lot that coming
um and I hope this excites you because
the power that we have right now is at
all this is locally we've imported we
have downloaded the AMA tool in this
case manager along with one model which
is Lama 3.2 okay make sure that you're
able to do this and I'll see you next
now to keep in mind that the whole idea
of a Lama is that we have this place
where we have many different models we
can use interchangeably we can just pick
one and then drop it and pick another
one and test until we find something
that works
so it is important to always look at
these
models page here so we can look at
what's available and most importantly as
you can see here you can also go ahead
and search or actually filter by
featured most popular and uh or newest
and so forth okay so newest here seems
to be neotron so keep in mind again that
these will change depending on when
you're watching this video okay so maybe
you're not going to see this as the
newest but the important thing is this
is the place for you to come and look at
what's going on so as far as we're
concerned right now updated 3 weeks ago
we have the Lama
3.2 it is also important for us to have
more information about those large
language models because the idea is for
us to be able to choose what works for
what we trying to accomplish okay so
let's click on Lama 3.2 so when you
click on Lama 3.2 or on any large
language model that is shown here you're
taken to this description page now
there's a lot going on here the
important part here is that we
understand at least the basics of what's
Happening first of all we see that is
under Tools 1B 3B which we'll talk about
but also we see that it was pull 1.1
million times okay which means this is
essentially being used by many many many
people and then here we have these tags
six to three tags but then we have here
this drop- down list that tells us
different things first one we have
different flavors of this Lama 3.2 we
have this 3B which is the latest that's
the one we have and then if we want a
shorter smaller one is 1B one important
to keep in mind also is that depending
on the flavor that we get here right you
notice that also we have the sizes so
depending on the flavor that we have
here we can have different sizes so this
one 3B is 3 2.0 gigabytes okay so it's
not too bad but 1B is 13 GB and latest
of course is 2 GB which is this one
latest same thing okay and we've seen
this before we have this command that
we're going to use to run in this case
this particular
model now if you go down here we have a
read me tells a little bit more about
this model important for us to read and
then we have the sizes so 3B parameters
which is
default this says here the 3B model
outperforms the Jamma 2 2.6 B and the 5
3.5 Min models on tasks such as
following instructions summarization
prompt rewriting tool use but what does
this really mean so I want you to look
at this as a guide because again we go
back to the whole point of Ama is that
we have different models that we can use
and us as developers or as people who
are wanting to use these models is that
we need to find something that works for
us this is why it's important for you to
always think about testing different
models until you find something that
works for you what works for me in a
situation X Y and Z it's not obviously
going to be what's going to work for you
in your own situation so that is
something to keep in mind
always all right so it tells us here the
comparison that for 3B parameter this
size here it's really good at following
instructions summarization prompt
rewriting and Tool use so use tools or
function calling and so forth now when
it comes to 1B parameter this is a
smaller version of 3.2 it says here the
model is compatitive with other 1
through 3B parameter models its use
cases include personal information
management multilingual knowledge
retrieval rewriting tasks running
locally on edge okay so if you want to
pull Lama 3.2 1B then you would run this
so you can imagine if you want something
that will be good for this use case
obviously you will use Lama 3.2
1B and let's go ahead I'm going to show
you how to do this so we can run or Lama
run Lama 3.2 to pull down to get this
Lama 3.2 1B so I'm going to copy
this go back to our terminal just paste
that
actually I should copy the whole thing
and paste it so run Lama 32 1B so this
will download this particular subset of
model this model so hit enter you can
see it's pulling down manifest it's
pulling everything in what may take a
little bit depending on on this size
remember we're actually downloading
locally so that means it will take up
some
space farewell so now you can see that
we have installed Lama 3 1B so if I were
to go ahead and say for slash again show
info you can see voila it's now we're
now using architecture Lama of course
the same thing parameters is 1 2 B which
is exactly different than the
full-fledged Lama 3.2 so it's a little
smaller okay and context length and all
that information so we can go and ask
question how old is the
universe like this and it's going to go
ahead and do the same thing but because
we
know from Reading here we know that this
particular one here is really good at
writing tasks running locally on edge
multilingual knowledge retrievals so I
can ask things like as an example how do
you say hello I am
fine in Tai let's
see okay so it was really good this is
just a very simplistic uh test here so
again you can see the things you can do
you can pull in different large language
models to see what's going on now the
other thing I can do here I can say I'm
going to just say buy just to get out of
here and still here I can say
AMA and I can say list so this command
here is going to list all of the models
that have been download downloaded
locally look at this now you can see we
have this Lama 3.2 1B this is the latest
one okay it says exactly when this was
downloaded and of course I have a few
others that I've done loaded a while ago
so L 32 latest time ago I have this
embedding large as well which we'll talk
about later very good so here it shows
everything that has been downloaded in
terms of a list of all of the
models let's look at parameters and do a
quick Deep dive uh so that we understand
a little bit more about the information
that comes with models so we're going to
learn what are parameters as well as
what is that they really mean right so
when we say show info that command we
saw this we have this model architecture
llama parameters of course
3.2b context length embedding length
quantization and so forth so what do all
of these mean let's start with the first
one so the first one is in this these
llama architecture here first of all the
architecture side of thing that means it
was architected it's a llama large
language model meta AI that means they
are the ones who created it matter
Facebook and also that means that these
models were designed with one thing in
mind which is efficiency they were
designed to be extremely efficient which
means they are strong when it comes to
performing even at smaller scales
compared of course to other large models
out there so they're very efficient and
perform really well now when we look at
then the parameters side of things it
says here 3.2b and we saw that sometimes
it fluctuates to 1.2 or3 or 1 and so
forth different numbers these are the
internal weights and biases that the
model learns during training but what
they do really is that they determined
essentially how the model processes
input data and how it generates output
because that's the whole point okay now
when we say
32b for instance what does that mean
well that stands for 3.2 billion
parameters now you can imagine even if
you don't understand exactly what those
means that means if you have 3.2 billion
parameters in this case these weights
and biases everything neurons and nodes
and everything that means you have a lot
inside of this neural neural network
which means there's a lot of information
that can be passed around the
relationships and so forth which in
terms mean that this model the the more
billions or the more parameters it has
which means has more connections and
more interactivity inside that means the
more accurate it is when it comes to
getting the results now it's important
to understand also that the number of
parameters does reflect the complexity
and the capacity of the model as
explained but there's also what we call
a tradeoff which means that more
parameters can improve performance of
course but it require more computational
resources so if you have a large
language model that is 3B or 5B or 7B or
8B well that means it's going to be
amazing in terms of performance but also
means it is going to require a large
amount of computational resources so
when we have something like
32b this kind of strikes the balance
between per performance and resource
consumption so let's summarize this
simply when we talk about parameters
talk about 3B or 2B or 10p or 7p and so
forth these are numbers inside a neural
network that it adjusts to learn how to
turn inputs into correct outputs so the
largest number the more the better in
this case but we have drawbacks in that
you also need uh a lot of computational
resources to run it so next we have the
context length so this number refers to
the maximum number of tokens which
essentially are pieces of text that the
large language model can ingest in a
single input so when we have something
like
131,072 this is an exceptionally long
context length what that means is that
it can handle very long documents
capturing dependencies across large
large spans of text which is really good
that means you can take big large books
and article lengthy articles and
extensive conversations and it can deal
with it with no problem when we talk
about the embedding length we're talking
about the size of the vector
representation for each token in the
input text so when we say 372 what we're
saying is that we have
372 dimensions in the embedding space so
the larger number again the more
Dimensions we have which means the more
relations you're going to have in this
Vector space so what does that mean well
that means that we have what we call the
semantic richness which means higher
dimensional embeddings can obviously
capture more nuanced meanings and
relationships between words when it
comes to a large language model then
that means this will reflect the model's
ability to understand very complex
language patterns so the large ler we
have this High dimensional the more the
model is going to understand complex
language patterns now there are some
implications again is that we have this
competition load so larger embeddings
increases of course the compettition
requirements and also if we have that
that's going to improve the model's
ability to generate contextually
relevant and coherent
responses okay and the last one we have
here is quantization now there's books
and we can spend hours and hours talking
about this but in general quantization
essentially is a technique used to
reduce the size of a neural network
model by in this case reducing the
Precision of its weights this number
indicates that the model's weights are
quantized or
quantized to four bits so translating
essentially we just saying we now have a
smaller model and faster processing
and lower memory usage so now we have a
more efficient um
model so going back to our Lama 3.2
model page here to see more information
now you understand what 3B means 1B
means and the implications and so forth
and one thing you will realize also is
that at the bottom we have what we call
benchmarks now I don't really trust a
lot of these benchmarks because anybody
can inflate B the benchmarks or deflate
them to follow certain agenda but
nonetheless it's nice to look at and see
The Benchmark but one thing I want to
show you here is the implication of the
sizes the parameters if I go back
perhaps and let's go back to models here
real quick and let's look for something
different let's look for Lama 3.1 you
can see with Lama 3.1 you even have 8B
70b
45b let's click on
405b well let's look at this one you can
see that if we go to
405b that's 229 GB of space you have to
have in your hard dis that's a lot and
even if you're able to have that
internally locally that that means you
also have to have the computational
capacity for you to be able to run these
models so this is something to keep in
mind you could have the capacity of
having 229 gabyt locally but will you
have the capacity of the computational
resources that are needed to run this
now keep in mind as you learn now this
is probably one of the best models
versions right because it has everything
it's large billions and billions of of
nodes and neurons and
everything that's something to uh keep
in mind and so for things that most of
us are able to do want to do 8B even 70b
should be totally fine okay just
something to keep in mind and as you go
through each one of these different
models they have different things okay
you can see of course we have more
information here 45b 7B 8B and so forth
as we talked about and they tell us the
evaluation and what they're good at when
should you use it and all these
different informations as well as human
evaluation as you see here so this is
something to always go ahead and read
and perhaps go to uh some links here to
read more about it now I want you to
always have this inquisitive mindset
whenever you're looking for models so
that you know exactly what you're
getting um and contract that with what
you want to accomplish that's how you
need to uh go by uh when it comes to
using AMA and different models here it's
not just one model fits all that's not
the idea the idea is that you find
something that will work for you okay so
models are real fun and you can always
um go ahead and test them out now now
let's go ahead and run through some
commands uh with our Lama that way we
are well situated and we understand how
to use some of the most important
commands Okay Okay so let's go ahead and
get started so we know that we always
start with AMA we can say AMA list this
will list all of the installed models
internally locally so you can see we
have
32b 32 callon 1B here it's the size and
when it was modified and we have other
ones that I have here in your case you
should probably most likely just have
one let's say I want to delete or remove
a model I can say
AMA remove also I can say AMA help which
is going to give me some of the commands
that I can use we have serve have create
Show run stop pull push list we've seen
before PS and RM for remove so let's
start with I'm going to go back say AMA
list to show the list of all of the
models that we have here let's say I
want to delete the this one here so I'm
going to copy that and I'm going to say
clear AMA RM to remove and pass in the
model name hit enter and it's going to
delete that model now if I go ahead and
say AMA list again we should see that we
no longer have that model very good and
if you want to delete this
other one here you can also do like that
now if we want to go ahead and pull in a
new run we can say
AMA and I'm going to say pull and the
name of the model as we've seen before
so again we can go back to our models
website all right AMA Library models and
let's say we want to get Jamma okay so I
go back here you can see models all
these models let's say I want to get
something different let's say we want
this code Gemma if you click here you
can see that we have different flavors
let's say I want to be the smaller so
you can see we can run this one here I
can either run all of this or I can just
pull first this one and then run it okay
so two ways to do the same thing so I
can say AMA pull like that it's going to
go ahead and
pull our model now remember this is 1.
gigabytes and may take a little
bit okay so once that is done I can say
AMA list we should see that we have code
Jamma so I can now run it right so let's
go ahead say all Lama run coama so it's
going to now run coama so we can
interact with it go back here I can say
that Co Jamma is a collection of
powerful lightweight models that can
perform a variety of coding tasks like
fill in the middle code completion code
generation so forth so let's see we can
test it out right can see if this works
so I can say for instance write me a
python
function that returns hello
world okay there we go you can see
Define function your name return hello
world your name and there we go all
right very good so you can see that
we're able to of course download a new
model and test it out so what I can do
again I can just say for slash buy to
get out of here and then I can say
AMA
list to see that and I can copy this and
remove it so I'm going to say
AMA RM to remove that and it's
deleted and
AMA list again you can see that we no
longer have Jemma
okay so back to our models here I can go
ahead and look at models and imagine
that we want to find a model that deals
with images or that is able to read an
image and tell us what's going on
there's a model called
lava so lava is a novel endtoend trained
large
multimodel that combines a vision
encoder and funa for general purpose
Visual and language understanding so
this is is a good candidate for what we
call a
multimodel model which means is that
it's able to deal with images as well as
text and so forth and other types okay
so we have just this one and then we
have this lava Lama 3 so depending on
the situation we can read more to figure
out which one is the one we should
probably look into so let's click on
this
lava and you can see we have 7B 13B and
34b so you know which one we're going to
be using so I'm going to go ahead head
and get at least the
7B which is this one here latest or 7B
doesn't matter it's about 4.7 gigabytes
okay so let's go ahead and pull this one
in I'm going to copy that and let's go
ahead and just paste it all in and get
our lava
7B all right so after a little while you
can see that we downloaded a few things
here but most importantly we have this
4.1 gab so we have now our model is
running and you can see we have our
shell I'm going to just say what is your
name see what's going to say I don't
have a name of course it's just a large
language model okay so now let's see
what we can do one thing I'm going to do
actually is I'm going to move over to a
code editor vs code to make this simple
okay so I'm just going to say byy for
now and then I'm going to open up here
to see vs code so I have this
flower1 PNG we're going to use this with
of course our lava model to see if it
can read this image and tell us what
it's all about okay so let me go ahead
and open the terminal here real quick
and so what I'm going to do first let's
go ahead and say AMA I'm going to
run lava
7B okay now it's running very good so
what I'm going to do now is I'm going to
say what is in this image
and I'm just going to
pass the image name which is
flower one. PNG let's see what's going
to
happen so now it's going to think and
tell us exactly what it sees the image
shows a bunch of small purple white
flowers with yellow centers which
appears to be a type of pansy they're
growing in what seems like soil or a
potting medium surrounded by what looks
like green foliage so that's pretty good
right for if you go back here and look
yeah looks exactly that's exactly what
it's describing which is really really
awesome now let's test something here
I'm going to test to see if this model
has this history capability meaning it's
able to save the conversation for
context I'm going to say and
now write me a
short poem
about the about that let's see what's
going to
happen ah there we go so it was able to
look look at that a purple and white
with yellow Hearts Define they add A
Touch of Beauty to the world around
little patch and so forth so it knows
now we know if we're testing that this
model is able to save the context the
history of the things that we've been
asking uh the context of our
conversation
tell me more
about those flowers let's see what it's
going to
say okay there we
go it knows exactly what flowers we're
talking about which is what we had here
okay I can ask can you tell me where
they grow
best and keep in mind again this is all
local large language model that we
running we don't have to go and send
request to a network somewhere it's all
local very good so pansies are native to
the Mediterranean region and all that
great stuff very good so we see that it
is indeed working so we're able to pull
in a new large language model in this
case lava which is a multimodel model
which allows us to deal with text as
well with images and so forth and we ask
it to tell us what it sees right in this
image here and it was able to give us
the description of the image and we ask
some follow-up questions about that
image and it
works so I've changed to
AMA run Lama 3.2 model right now so I
can do it again you can see since it's
loaded then we can see that we are able
to go to the
shell so what I want to be able to do is
show you that we can also do many many
different things here right so because
we are running on a large language model
it's a model that does a lot of things I
can perhaps say I have a sentence that
I'm not sure the sentiment of that
sentence right I can say tell me what is
the
sentiment of the
following
sentence I
not willing to pay you back let's see
what's going to happen okay again it's
very verbose which is okay but
essentially says that the sentiment of
the sentence I am not willing to pay you
back is negative and confrontational it
implies that the speaker is unwilling or
unable to repay a debt all right so this
is really good now one thing we can do I
can also say can you please be less
verbos okay there we go so the beauty
here is that you're talking or you're
interacting with a large language model
and you can actually prompt it to do
what you want it to do in this case I
told it to always be prompt to always be
less verbos at least in this case and
you can see says the sentiment of I am
not willing to pay your back is negative
and confrontational it implies
resentment defensiveness and obligation
aversion okay now the beauty here is
also we can modify or in this case
customize our model because we can
actually add certain pieces of metadata
to the model that cater to what we want
it to be like what I mean by that is
that we can actually create a file here
right click I'm going to click here and
this is what we call a model file so it
has to be something like this model file
no extension whatsoever and inside of
this file file this is where we are
going to add a few lines of text code
essentially so at the top here I'm going
to say all caps say from because we are
modifying it from a different from a
base model it's going to be from Lama
3.2 in this case if you have something
different you add that and then here is
where we're going to set a few things so
we can set a lot of things but now I'm
going to show you how to set in this
case the temperature what is a
temperature temperature is what allows
the model to be more creative or more
direct and matter of fact so the higher
the number in this case from 0 to one
the higher the number closer to one the
more elaborate the more um creative the
model is going to be okay so I can
change to 0 point three for instance if
I want it to be less
creative and then I can add for instance
the system message so this is just the
prompt right that it needs to know of so
in this case here put that into let's
say triple quotes like that to say
inside here we can put text a very smart
assistant who answers questions
succintly and
informatively
okay something like this so this
is our model file which is going to
allow us to modify or customize our
model okay so we can add more parameters
here more things to Aid our model but
this is okay for now so how do we do
this now to make it so that our model
indeed will comply with what we put
together here in this model file well
it's very simple all we have to do now
I'm going to let's go say byy real quick
here
all we have to do now is we need to use
the create command from AMA to create
some sort of a new version of our model
which is going to be specified by this
model file very easy really so I'm going
to say AMA and then say create and I can
give it a name just
say James okay whatever you want and
then we're going to say f to say to the
file it's going to be under the model
file notice the model file is actually
the same level here so I can just go
ahead and that so if I hit enter what's
going to happen now let's say enter okay
so says success you can see that it went
ahead and transferred model data so
using the layers and everything in the
background and so now if I save for
instance list you can see that we have
this James so it really created a
replica of the main
model and now this is going to be
different from the Lama 3 3.2 right
because it will have a little bit more
of a modified customized piece to it
very good so now I can use Mario or
James I should
say to ask
questions so look now I can say
orama first
run James look at that and we should run
James how beautiful is that so look what
will happen I can say
what is
your name look what's going to happen my
name is James and look how suc it is it
goes straight to tell to answer what
needs to be answered are you smart let's
see I designed to process and provide
accurate information quickly and
efficiently making me highly
knowledgeable on a wide range of topics
and all of that comes from this because
we said that your assistant that was
very smart
ask answer questions suly and
informatively very good very good you
the power things you can do right so now
I can ask questions let's see let's say
tell me about all tell me about the
oceans let's
see okay so this is pretty good because
it's such an open ended question it went
ahead there are five oceans gives me all
that information oceans play a crucial
role in regulating the earth's climate
weather patterns and ecosystems they
also provide all of this information
here okay so it is to the point that's
pretty good this was just an example to
show you the things you can do you can
actually modify customize your model to
be and do certain things that you wanted
to do this of course was very very
simplistic but you can see for the
system here we can prompt it even better
with more complex information and change
temperature and all these different
things let's go ahead and just say byy
and because we're done let's go ahead
and get rid of it so that we save some
space so llama
RM James okay so Lama list we should not
see James and while I'm here let me go
ahead and actually get rid of some of
these as well so that we have more space
okay very good so far we've been using
the CLI the command line interface with
AMA which is attached with AMA that
governs everything in a back end but we
can also use the rest API in fact most
of the things that we're going to be
using later are based on the rest API
which means there is an endo that we can
hit and run in this case the models
so how does that work one thing to keep
in mind is that in the back end as you
see well you can't quite see here but at
the top my bar here I do have this well
you probably see this quid AMA but there
is this AMA that is running in the back
end so that means in the background it's
running which means we're able to do all
sort of things that we're doing right
the all Lama application per se it's
running but it's actually being served
at an endpoint locally and all of that
is being served at Local Host
11434 what does that mean well that
means then we can generate a response
using the olama rest API because it has
it attached to it and it's running in
the back end in the
background okay so to do so it's very
simple all you have to do you have
access to all of this we can curl
something like this so in this case you
can see it's a curl and this is the end
point so Local Host 1143 for API gen
generate and then dasd and then we pass
the payload here so here we're saying
the model is going to be Lama 3.2 right
and the prompt we passing along it why
is the sky blue so if I were to run this
or hit enter what will happen is it's
going to hit that end point and of
course it's going to show me this now
this is not very helpful mainly because
it just shows a lot of gibberish and
there is this stream that's happening
but if you look look closely you can see
that every time this run extremely it's
going to show a certain word in response
so combining all these words will be the
response so that means then when we
write the actual rest API payload we
need to also add something else to it to
tell it that we don't want it to stream
as it is right now so that we just get
result what that means now I can let me
go back down let's clear so now what we
need to pass is this here that says
stream false so say we don't want it to
stream just give us the response if you
hit enter you can see it will take a few
moments and then voila we have this
model everything all the metadata about
the model that we're using in this case
is say tell me a fun fact about Portugal
and then says here here is one response
did you know that the town of obid obid
in Portugal is often called the fairy
tale Town due to its medieval
architecture and picturesque streets
right it gives it some other information
all right so that is a difference here
if we put stream false then we get that
if this is not there it assumes that
stream is true that's why we get what do
you saw earlier now in this case you can
see the end point is generate now we can
also get the chat with the model the
chat end point okay so generate the
difference here generate it just goes
ahead and predicts what needs to happen
as you see just gives us result but we
can also pass in the chat endpoint so
let me go and clear to do so I have that
you will have access to all of this
anyway so no worries and there we go we
do the same thing curl now the end point
here is API for SL chat and then DD and
then we pass here the payload so so we
pass the model the messages messages
plural that's why we have here a list
okay so we pass the role it's going to
be user and the content is going to say
tell me a fun fact about mosm Beek let's
go ahead and run this so there we go and
now this is a different endpoint it
gives us here some here's one fun fact
about gorang gorza Peninsula which has
one of the largest Coral in the world um
I think this is actually not correct
they have to remember that sometimes
these large lar models are not always
correct so there's no such thing as
going a peninsula I know this because
I'm from there but anyway you can see
that it's giving us some results large
ler models and are always correct so as
always we can also pass more information
or more specification to our endpoint in
our payload so imagine that we want to
for instance we want to request a Json
mode we can do so by requesting it by
adding Json right so you can see here we
are getting the generated API and we
have the same thing model prompt now
here we even said in our prompt make
sure that we want to get the response as
Json but also we want to say here as one
of the parameters that format it has to
be Json to make sure that indeed the
large language model aders to this okay
let's go ahead and
run and you can see doesn't look that
great but we want to believe that this
is actually a Json okay that is coming
in so you can see the curly Bryce is or
opening their model and then llama 3
created and all that information and we
should have an actual Json but you can
see response Day morning Sky color and
so forth the beauty here is that if you
go to this link here gith habo Lama docs
API end points or so many other end
points we can go ahead and hit um using
the rest API okay you can always come
here and look at some of them we can do
essentially the same thing we were able
to do in our s ey that's the whole point
we can do that using the rest API for
instance we can copy a model show model
information you've seen that before okay
you can just curl and then do that so
I'm not going to do that you can do that
yourself and then this is the result
that you get so all these things that
we've done before you're going to be
able to do them as well using the
endpoint API the rest API endpoints
we've learned a lot so far there are a
lot of things that we've learned about
the CLI we learn how to use the CLI of
course which is attached to orama to do
all sort of things to pull in different
models to run them to remove them to
even modify the model or in this case
customize the model by creating the
model file file and then use that to
create some sort of a copy or a modified
version customized version of that model
which is very handy as you see for use
cases that you may have okay so a lot
that I've thrown at you I hope this is
making sense I hope you are practicing
and seeing the power and most
importantly here remember is that all of
that is local you're not having to do
anything uh passing something to the
network to get anything back in that way
which means it's free you're just using
your own resources which is the power of
AMA so as a quick full summary here AMA
as we know is a platform that allows you
to run large language models locally
which is really awesome the great thing
is that it supports various models
tailored for different tasks including
text generation code generation and
multimodel applications so essentially
AMA model support these tasks here text
generation such as Lama 3 x could be 3.2
3.10 or 3.11 depends Moll and so many
others uh we have also code generation
one of the example was code Lama okay
and multi model application in this case
text and images and we saw in this case
was lava the lava model now this is just
to show you the breath of things or
models that ama provides Okay so again
your job is really to figure out what
models is going to give you the best
results for your use case Okay so keep
in mind that you always have to test
figure out which one will work for you
all right so to create large language
model applications using AMA we saw that
first of all uh we can use large
language models that come with Alama
because Alama hosts per se all these
different large language models we can
use now we also learned that there are
different ways to interact with AMA and
its models so we saw that the main way
is through the CLI the command line
interface so it's easy straightforward
and that's what we've been looking into
and then the second way would be in this
case to use UI based interface which
we're going to look into next so
essentially have a user interface where
the back and we can put or apply a model
AMA model and then we have a nice user
interface that users or yourself can use
to interact with the
model now we've just finished looking at
the rest API so this is the base of
essentially everything else we're going
to be doing after the next UI based
interface way of interacting with the
models okay so essentially we're able to
use rest API curl and then hit the end
point and get information that way about
generating text or um in this case
deleting things deleting models or
looking up everything related to our
models so essentially the same things
that we're able to do using the CLI we
have a way of doing that using the API
the rest API and later we'll see we're
going to use AMA python library now this
is where we're going to be able to then
use the tools in Python in code because
now we have more customization more
freedom per se to start building the
actual large language models
applications that we so want to do okay
okay so now let's go ahead and look at
the UI based interface so if you go to m
thata MST that app as you see here this
is what you will see so you can read
about it say Lama 3.2 Vapor mode all
this great stuff say the easiest way way
to use local and online AI models so
without Misty painful setup endless
configurations confusing UI Docker and
all these other things with MTI one app
one click setup no Docker no terminal
offline and private unique and Powerful
feature so that's the reason why I chose
this of course you can choose any other
that you might find out there that I
mentioned so this is what I'm going to
be using and you can see there's a lot
of logos here saying that it works with
uh meta stuff with chat open Ai and many
many others so it's really good so you
can see Windows Mac and Linux and they
do have a user interface so you can see
exactly how it will look like so
essentially it's going to be like having
chat GPT uh but now we are running our
own models so click here and I am on Mac
of course I'm going to go and go that
way if you're on Windows you have the
option to do the 64 or 64 CPU only MD or
Nvidia if you're Mac you have apple
silicon or Intel chip if you're Linux of
course you have these flavors as well so
pick the one that works best for you so
I'm on Mac I'm going to go to Apple
silicon like that so it's going to go
ahead and
download okay so it's now downloaded I'm
go let's go ahead and open it and I'm
going to double click to start the
installation so I'm going to go ahead
and drop it there very good and let me
go ahead and open it real
quick okay so I'm going to say open all
right and there you have it now once you
open it this should happen and the
beauty here is that you can see how
would you like to get started you can
set up locally AI I remote modeles
provided and so forth but if you look
down here look at this Gama get started
quickly using AMA models all right and
even found it under users meama and all
the models they found it locally because
it detected that we have Ama installed
in our in our uh on our machine so I'm
going to go ahead and say continue down
here okay that's it just one click and
we are done and you can see here at the
bottom here make see I can make this
larger yeah I can make this larger
that's very
good okay at the bottom here you can
click smaller you can click you can see
we have
different levels or different models we
have lava 7B which you remember and I
have Lama
3.2 all right because those are the ones
that we have installed at least I have
installed locally if you have more than
one then you're going to see all of them
sure here and this is beautiful because
we didn't have to do anything just one
click indeed like they promised and we
are there so let me choose the Lama 3.2
and the moment you do that that's it and
you can just start chatting just like
that I'm going to say how old are you
just silly to see what's going to
say so this is being powered Again by
our own model so I was released to the
public in 2023 very good can you tell me
a funny short
story and just like that look how fast
this is one day a man walked into a
library and asked the librarian do you
have any books on puff loves dogs and
scroller cats scroll danger I think
that's what you say the libran replied
it rings a bell but I'm not sure if it's
hair or
not okay that's very cool so we have
that so you can start talking you can
start chatting with this long large
language model and if I wanted to I can
also go and attach a document let's
click here and I'm going to add one
image that we've seen before drop it
there
now image is attached let me go back
here and I can
ask tell
me about the image now here's the thing
the reason why it's saying this is
because the model doesn't CU we're using
llama 3.2 it doesn't really know how to
deal with the images however if I change
here go to Lava 7 and ask the same
question tell me about the
image Aha and now because lava is a mul
model multimodal model it will know how
to answer questions about the image
there we go now it's working the image
shows two purple flowers with yellow
centers okay very good so it is
describing it took a little bit of doing
here perhaps it should have reset
everything but it's okay you can see
that now it is actually working because
we're using lava 7 and it will tell me
exactly what it sees now I can change to
Lama 2 or
3.1 and if I say tell me
about the image it's going to give me
some issues of
course okay very good so it can't do
that but the other thing I can do here I
can also add let me just delete this I'm
going to add a document let's say for
instance I'm going to add this document
that we've seen before which is the b o
i PDF there we go I'm when I say
can you tell
me about the PDF file okay so now I have
some issues here the best way to do this
is you go to attach knowledge Stacks so
click here and I'm going to say add your
first knowledge stack so I'm going to
say my
tester and the thing is because we are
now invoking want to create a rag system
we need to have an embedding model so
that it can create those embeddings of
the PDF file of the document that way it
can be saved and then the large language
model is able to Converse that way okay
so we're going to just use whatever they
give us nomic uh embedding test text I'm
going to add that it's very good and at
this point let's go ahead and add an
actual file so I'm going go ahead and
drop it again in this case here just
drop it like that all right
so once you do that you have to hit
compose so that it a it's able to pick
it up do all the things it needs to do
to actually get information from our
file okay embeddings and everything say
compose so it's
composing okay knowledge stack saved and
composed successfully you can now use it
for chatting very good so let me get out
of here and I'm going to go here so
click again here and what I want to do
here is I want to be able to say click
on my tester but this is my knowledge
base I want to include in my chat here
so make sure you click that and that
will what I will do is we'll let this
know that indeed we have included that
knowledge base which includes all the
information about our PDF file so say
Give me a summary of the PDF
file okay let's
uh says there's no PDF file mentioned in
our conversation so far the text we
discussed earlier was from beneficial
ownership and everything so it took a
little bit uh because I'm asking PDF it
doesn't really know about the PDF
because all of that information was
actually transformed into text and
embeddings and everything for the large
language model so I just can say Give me
a summary
of the
document okay there we go so it says the
document provide instruction on how to
fill out certain items including
identifying documents issues a company's
images and everything and it is indeed
beneficial ownership all of that so
we've seen this before I can keep asking
what are
the
penalties for not
filing and and there you go it gives me
exactly the penalty and everything that
will happen if I don't file right I can
just say give me the
penalties maybe that's a better question
perhaps okay that's better because now
it's giving me exactly what I need okay
based on the text from beneficial
ownership information blah blah blah and
everything is there failure to comply
with beneficial ownership reporting
requirements Cil penalty 10,000 fine up
to 500,000 and so forth
okay what are
the deadlines for
filing okay so you can see that this is
working we're able to converse with our
data our own document now the beauty
here if you AR haven't realized yet is
that all of this again is internal it's
our own large language models we can of
course go and spin out a different large
language model OKAY adding new provider
and so forth and we can use
interchangeably as you see here and we
passed in a knowledge base which means
we said here are some documents in this
case which is one document and use that
to create a knowledge base so a rag
system essentially and so that we can
chat and ask questions about that
knowledge base okay and just like chat
GPT we can go ahead and create a new
chat we can do all sort of things as you
see here the point is not for me to go
through Misty thoroughly the point is to
show you what's possible to create a
user interface or to use a user
interface that is guided or is being
fueled by our own AMA model how cool is
that so what I want you to do is to play
around with this and you can see it's
just to show you how amazing this is
because we have our own box that we can
uh pass in sensitive documents and all
those things without worrying about
prices you know having to call an API
anything external put this in a cloud
base or having to pay for usage so it's
all here and we can use as we see
fit so we're able to see all of the ways
in which we can interact with and its
models so we looked at CLI command line
interface and we saw that it's really
easy because it's the fastest access to
our models it's very easy but as you
know it's not scalable in sense that
you're not going to build a full-fledged
application using that and then we went
and looked at the API the rest API which
is essentially the same but now we have
a different back door per se we we using
rest API to pass in certain payloads and
get information so we're doing the same
thing that we can do with CLI hitting
the same functions per se but now we're
using different end points with the rest
API and we just finished looking at the
UI based interface so this is way easier
for us to be able to have a UI based
interface and actual interface such as
chat DBT that way we're able to ask
questions uh change manually quickly
different models that we may have and
start chatting with our models so it's
really really easy to put uh together or
to have that working using the M that
app and of course there are different
flavor different tools out there that
you can use that will do the same thing
right that would give you this UI based
interface so I chose MTI because it's
just easy as you saw to install and get
started so it wasn't about how to use
these tools but I was about to give you
the knowledge and tools that you can
then uh use
on your own and explore more okay so now
we're going to go to the fun part which
is we're going to use now sort of a a
backend combination of the API rest API
through the python Library the AMA
python library because truth be told we
want to be able to create local large
language model applications using AMA
models and so for that we need a way for
us to be able to use python this code in
this case code or any other language but
in this case going to be python to be
able to take advantage of these models
that ama provides and so that's what
we're going to be doing next which is
we're going to get started with AMA
python Library so we can use code to
interact with AMA
Models All right so I have this folder
here we've seen before called AMA we got
a few things here let's get started here
simply by using the AMA library but
actually I'm going to do the hard way
first to show you that we can actually
use code to get to the end point just
like what we saw with the restful API so
real quick so we can see what's going on
so let me go ahead and create a new file
here so we're going to have access to
all of these code so no worry so I'm
going to say start one. P1 or py
okay and so what we'll do here first
actually is I'm going to go ahead and
make sure that we have an actual virtual
environment for our python project so
I'm going to say Python
3 okay like that so we should have a
virtual environment there that's very
good and let's go ahead and activate it
so Source bnv if you're on Windows of
course there's a different way to do
this activate and voila so now we have
our virtual environment AC activate
active I should
say very good so what is the first thing
we're going to do well first of all I'm
going to go and import let's say pip
install I'm going to install
requests real fast here so we have
that okay very good because we're going
to use that to do what we are looking
for it to do all right so I'm going to
go ahead and import requests so we have
that and also I'm going to import Json
so we have that as well okay so I'm
going to create a URL here which is
going to be the URL where is the end
point it's going to be HTTP for our
Local Host here so it's not going to be
that it is 1 4
34 and in this case I want to go to the
generate endpoint like this
okay and for that remember we need to
pass some payload data when we pass
along so I'm going to say data just
create a little uh dictionary
here and for a dictionary I'm going to
pass the model that we need it's not
going to be gpt3 or
gpt2 we're going to call this llama it's
going to be
Lama 3. two and make sure that llama is
running for this to work
always and then I'm going to pass the
[Music]
prompt I'm going to pass something that
says tell me
a short story and make it funny like
that okay so there we go this is our
payload that we're going to pass along
using the rest API so I'm going to send
the request so I'm going to say
response pass in the request. post pass
in url and Json and what I want to do
here
is I want to say stream
to true to say I want this to be
streamed okay as you will see in a
little bit okay maybe this a little bit
smaller so you can see everything all
right okay so then let's go ahead and
check the response first so I'm going to
say check the
response status so it's going to say if
response status is as you see there I
already have all these code so I'm not
going to Bor you with all of the
intricacies
I'm going to put it all there okay so
what we're doing now is that we're going
to check if response code is 200 which
means all is good and then we're going
to go ahead and start generating thing
so we're going to go and iterate over
the streaming response because we said
the streaming to true as it comes the
response that comes in okay and so what
we're doing here we're decoding the line
and parsing everything until we actually
print the generated text as we go
through very good so let's go ahead and
see if this works I'm going to go ahead
and run this make sure that of course
the
AMA icon is running is out there
otherwise this is not going to work so
let's go ahead and run let's run this
looks like I'm having
issues right so I'm need to pass API
like that okay my bad that was my
problem let's go ahead and give it a try
again okay and there we go you can see
that is indeed working and it's really
really fast as you see here generat text
here's a short silly story for you once
upon a time there was a chicken blah
blah blah and there you have it all
right very good so this is indeed
working we're able to use code to
actually interact with the our llama 3.2
model locally right albe it we are not
using the AMA uh python Library but you
can see that this is actually working go
ahead and play around with this and I'll
see you next all right so this was
really nice we're able to use the rest
API in code here the restful API should
sayama restful API in code to interact
with Lama 3.2 this is fine but now I'm
going to show you how to use the AMA
Library the AMA python Library so we can
go straight into code we don't have to
hit these we don't have to explicitly
hit these end
points all right let me go ahead and
create a new file here let's call this
start
to.py remember you will have access to
all this code so do not worry okay so
the first thing we need to do here of
course is to make sure that we import or
we get the right dependency so going to
say pip install we need to install or
Lama as such very
simple and there we go now once we have
a Lama we can start working at right so
first let's go ahead and import
AMA there we go and real fast here I'm
going to show you how simple this is so
the same thing we did before with the
CLI and even with the rest API we can do
the same thing here using
this dependency right using this SDK
perite so now I can say put inside of
response and I'm going to say all Lama
and I can say list so this function here
as in name imply is going to list list
it should list all of
the you guess it all of the things that
we have in this case all of the models
so let's go ahead and save and give it a
quick run run so real
quick okay I ran and you can see that we
have this Json that comes up it says
llama 7 oh name lava 7 and that is the
model and I should have gives all that
information and I should also have the
Lama
3.2 and some other things all right this
is very cool very good all right
very good so you can imagine we can say
AMA list we can also say
AMA let's say chat chat response client
create and delete embed and all these
other different things right so that's
very very exciting I just want to show
you that I'm going to just call comment
that out so we don't have a lot of
things here all right so let's go ahead
and hit the chat endpoint or in this
case the chat API
well as you know it's very simple so
let's go ahead and get started here all
right so I'm going to say res we can put
anywhere orama chat
so. chat and then I'm going to pass a
few things and you can see that if you
have over it says it respects at least
to have the model name so the model I'm
going to say llama
3.2 okay just add the one that you have
and then here because it's chat I'm
going to pass messages okay so this
comes as a list and I'm going to pass
the role as a user and the content is
going to say something I can say
something different that perhaps why is
this sky
blue something like this okay so now I
can actually print
what comes back in this case from our
chat we pass the model and the messages
could be more than one message okay we
can also say uh for our role here
another one that says let me just put
like that I can say the context pass the
context pass all different things that I
may want to pass okay but so far this is
okay let's go ahead and see how this
works I'm going to go ahead and just run
from
here okay to a little bit and you can
see we have the results so you can see
the model and everything but most
importantly we have the content says
content is the sky appears and
everything but to make it even better I
can just go ahead and get the content so
let's go to
result uh let's say I think I have to go
to
messages and they get the content like
that
okay let's go ahead and run again all
not messages
message
okay all right and there you go so this
guy appears blue because of phenomenom
because of phenomenon called scattering
and all of that great stuff so we have
all the information here now you noticed
before when we
just look at showing the entire payload
it comes in with a lot information about
the model when it was created of course
the content the role and all that
information at the bottom we have some
other pieces of data metadata so done
true total duration how long it took and
low duration prompt evaluation and uh
also evaluation count prompt evaluation
duration all these little
things that we may want to keep track
of all right so just like that ladies
and gentle ladies and gentlemen we're
able to use the chat endpoint here using
uh the ol Lama python Library so go
ahead and play around with this and see
what we can do because this is the
beginning of a very exciting thing I
think now one thing you can also do you
should know this is that we can pass
other things in our chat function here
right see we have model messages tool
stream so we can say we want this to be
streamed and format as well very cool so
what you can do I'm going to give
another example here for streaming let
me go ahead and comment this out so we
have that for streaming it's essentially
the same thing notice I passing messages
we're all user now I have a different
question why is the ocean so
salty and I can say stream to true
because I want it to be streamed when I
do that then whenever I want to print
that out I have to put that through a
for Loop or a loop of some sort that way
we can get all these pieces that are
being streamed so we can see them being
flashed out in our command line okay so
let's go ahead and run real quick so you
can
see just run like
this takes a second and there you have
it okay so there we go if you want to
have the this capability of streaming
and showing those messages as they come
in this is how you would do using Code
all right so go ahead and play around
perhap add different messages or
different questions here and see how
this behaves and what you can also do
you can use a different model whichever
model you may have running
okay so one thing to keep in mind is
that all we've seen now this AMA python
Library West DEC I guess I could call
that it's all based or designed around
the AMA rest API so internally when we
call orama that chat internally we're
essentially doing what we did before I
showed you here so it's hitting this
endpoint URL and passing in the API in
this case go to chat or generate or
create and so forth it's just like an AB
it's just an abstracted way of hitting
this end point so we don't have to
explicit L do this so I made sure to put
it here so that you have that okay now
we can hit now we can use another now we
can also use the same now we can go and
use a different function or end point
for instance like generate if want to
generate something right we can do that
what I mean by that is I can say for
instance AMA generate and pass that and
pass that prompt I can even pass all of
that I don't think I can okay I don't
need to pass any of that so we can also
just generate a response instead of
chatting as such so nothing new here I'm
just going to leave it as is so you can
see the other thing we can do is we can
say for
instance we can
show by saying AMA let's just print and
say
AMA and then that
show and we want to show what want to
show anything about
llama three like that that two okay
let's go ahead and
see and there you go after a few moments
you can see it tells everything we need
to know about our model OKAY a lot of
information that's for sure okay there
we
go next thing we can do is we can
actually use the model file as we did
before but all in code to create a
separate model that has Specific
Instructions so it's very simple I have
some parts of the code here so we create
in this case the model file we don't
have the actual model file but we can
put all in line is here as you see here
and the other thing I can do is I can
pass the temperature here let's go back
to this one to see how you do that just
copy this and pass temperature 0.
1 for instance and then all we need to
do now that we have this information is
as we saw before we are going to go
[100:01] ahead and create the actual model
[100:07] so and then we're going to pass the new
[100:09] name know it
[100:11] all and then pass the model file which
[100:15] is this one here so we can make this
[100:17] more robust if we want for instance we
[100:19] can say you are very very suce and
[100:25] informative something like that all
[100:27] right okay so now that we have created
[100:30] our new know it all so we can go ahead
[100:33] and generate something right use the
[100:36] generate message or generate function so
[100:40] I'm going to put into a variable so
[100:44] AMA generate and I'm going to pass the
[100:47] model that I'm going to be using yes
[100:49] going to be know it all this one here
[100:51] same name and The Prompt there so I can
[100:55] just say for instance sure that's fine
[100:59] okay and we can go ahead and print the
[101:06] result need to go into response that's
[101:10] how it comes
[101:12] as okay and there we have it so let's go
[101:15] ahead and give it a run and
[101:18] see so it's generating creating
[101:21] everything and some sometime soon we
[101:24] should have there we go the ocean
[101:25] salinity is primarily due to the Earth's
[101:28] water
[101:30] cycle very good it looks great and it
[101:34] works so now if I come here and just
[101:39] say AMA list we should see that we do
[101:43] have this new know-it-all latest which
[101:46] is two gabes was created about 20
[101:49] seconds
[101:50] ago all right very good so these are are
[101:53] the things that we've done before using
[101:55] the CLI as well as using the rest API
[101:59] but now we're using
[102:01] the AMA we're using the AMA python
[102:05] library osdk to do the same thing
[102:07] because I just want to show you what's
[102:09] possible so this is more flexible as you
[102:11] see because now we actually use encode
[102:13] and the power of the rest API in the
[102:16] back end to create very simple
[102:18] applications of course this is just to
[102:20] show you what's possible and soon enough
[102:22] we're going to be able to create those
[102:23] large language model applications that
[102:25] are based on locally using AMA
[102:30] models so once we done that we can of
[102:33] course
[102:35] delete models okay so can say all delete
[102:40] know it all and if I run this it's going
[102:43] to go ahead and run again and go ahead
[102:45] and delete our friend know it
[102:49] all okay so I have the response again
[102:53] but most
[102:54] importantly if I go back and list we
[102:58] should see that we no longer have no at
[103:00] all okay so go ahead and play around
[103:02] with this and uh I'll see you later okay
[103:06] so now that you have all the tools you
[103:08] understand what Lama is what problem
[103:10] comes to solve and how to actually build
[103:14] a simple uh useful applications using
[103:17] AMA AI based applications using AMA now
[103:20] it's time for us to dive in and start
[103:22] build building real world use cases
[103:26] using and let's go ahead and get started
[103:29] all right so now it's time for us to
[103:30] start building something useful things
[103:33] that are actually useful so the first
[103:35] thing we're going to do here is we're
[103:37] going to build a grocery list organizer
[103:41] okay or categorizer so the idea is you
[103:44] have for instance a grocery list file
[103:46] text file that has all these different
[103:49] items okay and so what we want to do is
[103:51] we're going to use the line 3.2 the
[103:54] model with Allama of course and see if
[103:57] we can recreate this entire grocer list
[104:01] and actually create a new grocery list
[104:03] from this but categorized into
[104:07] categories so for apples and all these
[104:09] things could be produce and then we have
[104:11] snacks and so forth okay that's what
[104:14] we're going to do so you will have
[104:16] access to all this code of course but
[104:17] you notice that I now I have this data
[104:20] folder here which will have this grocery
[104:22] list that here
[104:24] with all of these items and so let's go
[104:26] ahead and create a new file here called
[104:30] categorizer
[104:32] py and this is where we're going to
[104:34] start playing around first thing going
[104:37] ahead and import
[104:40] AMA and then we're going to import OS
[104:42] because we are going to interact with
[104:45] the directories and finding things and
[104:48] at the top we're going to create I'm
[104:49] going to just add this model so we can
[104:52] use it so it's going to be
[104:53] be llama
[104:56] 3.2 okay so let's go ahead and set up
[104:58] the paths and I already have them set up
[105:01] so let me go ahead and get that so
[105:04] essentially for input path it's going to
[105:06] be data grocery list so essentially I'm
[105:09] going to data get this grocery list and
[105:12] output is going to be data and it's
[105:14] going to be called categorize grocery
[105:16] list at txt so it's going to be under
[105:19] data again once all is done
[105:24] all right so the first thing is we're
[105:25] going to go ahead and
[105:28] check paths if the input file Exist by
[105:33] using OS exist and pass this input file
[105:37] path like that and I'm going to print
[105:39] input file if it's not found right and
[105:42] then we're going to get out of there and
[105:44] then we're going to read these ancored
[105:47] and then we're going to read the
[105:49] uncategorized grocery items from the
[105:52] input file
[105:53] to do so we just open the file through
[105:57] the input file the path and read as F
[106:01] and then we take all those items we read
[106:04] everything strip everything into our
[106:07] items so these items now should have all
[106:09] of the items from our grocery list text
[106:12] file and as we know to direct our large
[106:16] language model we need to pass a prompt
[106:18] so to say this is what you need to do
[106:21] this is how you're going to be doing it
[106:22] right right so kind of prompted to give
[106:24] it some directions so it knows not to ve
[106:27] off somewhere else so for that we need
[106:30] to have this prompt that I already
[106:31] created so there is the prompt so you
[106:35] are an assistant that categorizes and
[106:37] sorts grocery items so we specify
[106:39] exactly what this model does okay here's
[106:43] a list of groceries so items going to be
[106:45] all of the items that we received there
[106:48] please one categorize these items into
[106:51] appropriate categories such as produce
[106:53] Dairy meat Bakery beverage beverages Etc
[106:57] and two sort the items alphabetically
[107:00] within each category so in each category
[107:03] you put every every item into make sure
[107:06] you sort um alphabetically and three
[107:09] present the categorized list in a clear
[107:12] and organized manner using bullet points
[107:15] or numbering that's it so we can make
[107:17] this more complicated as we wish but
[107:20] this should be fine all right so now we
[107:23] have the prompt we
[107:25] have of course now we have the file we
[107:28] have all of the items inside of this
[107:29] variable here from the text file and we
[107:32] have the prompt okay we're passing in
[107:35] that information through the prompt so
[107:37] we can use that prompt then to send the
[107:40] prompt and get the response okay so now
[107:43] what we do here put inside of a try
[107:46] catch essentially so we are generating
[107:49] we're using AMA generate we pass in the
[107:52] model and the model remember is what we
[107:55] set up here
[107:57] okay and then we pass in the prompt
[107:59] remember the prompt will have the items
[108:03] and of course the prompt to make sure
[108:05] that the large language models knows
[108:07] what to do and then we have generated a
[108:09] response or text we get that information
[108:12] if we have something or we just get
[108:15] nothing okay return nothing okay so now
[108:19] we are writing the categorized list to
[108:21] the output file because we want to get
[108:23] that what I can do also here is I can
[108:26] print real quick here the generated text
[108:29] so which is going to be the actual
[108:31] response so we can do that here just for
[108:33] fun I can just
[108:36] say generated text or
[108:39] generated or let's say categorized
[108:45] list and so then we write into a file
[108:49] and that's it and we categorize grocery
[108:52] list has has been saved and we show the
[108:54] output file the path that's it all right
[108:58] okay that's that is pretty much it so
[109:00] now we're going to go ahead and run this
[109:02] so I'm going to open this different ways
[109:04] of running just run here or you can
[109:06] write the command for running but that's
[109:09] okay so let's go ahead and what's what's
[109:11] going to happen so things are happening
[109:13] in the background and voila you can see
[109:16] that we see categorize list has been
[109:19] saved data categorize list and there it
[109:22] is but also we wanted to show it here
[109:26] look at this categorize list after
[109:28] categorizing and sorting gral items I
[109:31] present to you the following list so
[109:33] produce apples all of these items that
[109:36] seems correct we have dairy cheese eggs
[109:39] milk and yogurt very nicely done meat
[109:41] Seafood chicken ground beef salmon
[109:44] shrimp and sausages Bakery bread flour
[109:48] pasta rice Pantry almonds and all of
[109:51] these other items beverages that sounds
[109:53] great Frozen ice cream snacks chips
[109:56] chocolate and everything and now if I go
[109:59] to see as I said we have here the
[110:02] categorized grocery list if I click it's
[110:05] going to be exactly the same where we
[110:06] saw printed out there but now it is
[110:09] indeed a txt file now this is very small
[110:14] and very simple but I hope it's showing
[110:17] you the things that you can do and keep
[110:18] in mind that all of this is locally we
[110:21] don't have to pay for chat DBT or open
[110:24] API or anything else this is something
[110:27] that is local to us and keep in mind
[110:29] we're using llama 3.2 but if you after
[110:33] doing some testing and realize that for
[110:35] whatever you trying to do there's
[110:38] another large language model on another
[110:40] a model that will fit with what you're
[110:43] trying to do then obviously it's easy to
[110:45] swap that large language model to use
[110:48] that that's the beauty because remember
[110:51] here I'm saying we're using Lama 3.1 but
[110:53] I can go ahead and pull another large
[110:56] language model another model to use and
[110:59] nothing will change all will change is
[111:01] changing this to a different model that
[111:04] is the beauty of a
[111:06] Lama all right so what I want you to do
[111:09] is perhaps to make this even better or
[111:11] maybe you have some ideas that you may
[111:14] want to implement here because you have
[111:16] the large language model and you have
[111:18] some data maybe you have a list of uh
[111:21] something or
[111:23] a text that has something you want to
[111:25] summarize it you want to see what is the
[111:27] sentiment of that text and so forth
[111:30] there's so many things that you can do
[111:32] all of that because we now have this
[111:35] capability through AMA of picking and
[111:38] choose whichever model that we want to
[111:41] use incredible all right play around
[111:44] with this and I'll see you next so now
[111:46] it's time for us to look at something a
[111:48] little bit more complex something that
[111:50] you will find in the real world we which
[111:52] is building rack systems with AMA so
[111:57] with AMA of course we can build more
[111:59] complex large language model
[112:01] applications as I have mentioned so rag
[112:05] what is this thing rag stands for
[112:08] retrieval augmented generation so
[112:10] essentially is a way for us to converse
[112:13] or to talk or to chat with our own
[112:16] documents our own data
[112:19] internally now the thing to keep in mind
[112:21] is that large language models are
[112:23] actually very limited in terms of
[112:25] knowledge so they're really bombarded
[112:27] with a lot of different information
[112:29] textual audio and everything that's how
[112:32] they were trained on but they don't know
[112:34] what they don't know meaning they only
[112:36] know what they were trained on and so it
[112:39] makes it really difficult for us to
[112:41] build applications that are catered or
[112:43] that are customized to our own needs so
[112:47] rag solves that problem because now
[112:50] instead of the large language models to
[112:52] know only what it knows we can actually
[112:55] use it because it's really good at
[112:57] predicting things at summarizing and
[113:00] kind of brainstorming and getting pieces
[113:02] and pieces and putting together and get
[113:05] a response we can use that to solve this
[113:08] problem by injecting quote unquote
[113:11] injecting our own documents so that is
[113:13] one problem that it solves but also
[113:15] we're solving the issue of hallucination
[113:18] because any large language models you've
[113:21] encountered it makes stuff up because it
[113:24] knows a lot but if you ask questions
[113:26] sometimes just doesn't know but it
[113:27] doesn't say doesn't know it just says
[113:30] things that makes no sense that may seem
[113:32] true but they're false so this is what
[113:34] we're call hallucination that's what rag
[113:36] comes to help large language models and
[113:38] in turn help us when building rag
[113:42] systems now let's go and look have a
[113:45] deep dive into rag I think this is
[113:47] important especially for those of us who
[113:49] don't really know what rag is or or knew
[113:52] new to rag even if you already know what
[113:54] rag is I will just go real fast so you
[113:58] understand what really happens an
[114:00] overview so the idea is very simple is
[114:02] that we have these documents right so
[114:05] PDF files text files whatever and those
[114:08] are actually passed through this
[114:10] processing of parsing and pre-processing
[114:13] them what that means is that all of
[114:15] those are chapped into small tiny chunks
[114:19] documents okay and those in turn are
[114:23] then put through an embedding model what
[114:26] that does is it takes this textual
[114:30] information these documents chopped up
[114:33] and now they're going to be converted
[114:35] into a vector space into Vector into a
[114:39] number representation of those chunked
[114:42] documents that's what embeddings mean
[114:45] essentially and once that happens the
[114:48] vectorization of that that's what
[114:50] happens right it's a different format
[114:52] now of this data and that is saved into
[114:56] a vector database okay so all of this is
[114:59] what we call indexing this whole process
[115:02] we call it indexing what happens then is
[115:04] that a user comes in with a question
[115:06] query so that query say how old is is
[115:11] the universe for instance is also taken
[115:13] in and because of this whole ecosystem
[115:16] now is into um is transformed into a
[115:20] vectorized system that also has to be
[115:23] embedded so the query is also embedded
[115:26] into a model uses embedding model to
[115:30] embed that to vectorize this query and
[115:32] so that is indeed what it is actually
[115:35] either passed onto the vector database
[115:38] or it's used for searching because now
[115:41] you can imagine if we are searching
[115:43] using text and this realm on this other
[115:46] side here it's a vector database or
[115:48] vector store is saving everything mostly
[115:52] in Vector we need to have this
[115:54] correspondence right between we have the
[115:57] question but it has to be transformed
[115:59] into the same format as what we have
[116:01] here so that way it's easier to do this
[116:05] searching and so what happens now is
[116:07] that these systems have what we call
[116:09] retrieval um capabilities so they
[116:13] retrieve the pieces that are
[116:15] needed for answering the question
[116:18] meaning if the query is about how old is
[116:21] universe is going to find all the pieces
[116:24] all the documents inside of the vector
[116:26] store that perhaps are close to that
[116:30] question right and packages are all up
[116:34] and that's what's passed along with the
[116:36] prompt right the documents these pieces
[116:39] that we
[116:41] retrieved we call them relevant
[116:43] documents and the query itself and that
[116:46] is indeed what is passed through the
[116:48] large language model the model so that
[116:51] the large language model model has this
[116:53] contextual idea Sense on how to answer
[116:57] that question so it generates an answer
[116:59] which then we get a response so in
[117:02] nutshell ladies and gentlemen that is
[117:04] what rag is and is very powerful for the
[117:07] reasons that I've just shown you here
[117:09] now what do rag systems need as you saw
[117:12] well they need a few things first of all
[117:14] they need a large language model because
[117:16] that is essentially what drives most of
[117:18] the things essentially because we get
[117:20] all that information the documents
[117:23] pieces of documents related documents
[117:26] for what we are trying to ask for and
[117:29] the prompt and everything that's what it
[117:31] takes and the large language model has
[117:33] so much knowledge but also has these uh
[117:36] reasoning capabilities to know how to
[117:39] package it all up and answer that
[117:41] question coherently and also we need
[117:44] this document Corpus so essentially it's
[117:45] the knowledge base what we passing
[117:47] through the document embeddings is very
[117:50] important for the reasons I saw there
[117:52] because you take these documents you
[117:55] break them down into embeddings small
[117:58] pieces of vectors and if we need
[118:01] somewhere to save those vectors these
[118:03] are not just normal databases this what
[118:06] we call Vector databases these are
[118:07] vector stores so there are many out
[118:09] there uh we have phas Pinacle uh pine
[118:12] cone and chroma DB and then we need to
[118:15] have this retrieval mechanism and most
[118:17] Vector databases have in them some
[118:20] retrieval mechanism to retrieve retrieve
[118:22] what needs to be retrieved to then be
[118:24] passed along Downstream when we want to
[118:27] build large language model applications
[118:30] we can Brute Force everything and do
[118:32] everything our own on our own but it's
[118:35] always nice to have tools that kind of
[118:38] help us and Abstract out all the
[118:41] complexities of building large language
[118:43] model applications and so that's when we
[118:45] use tools such as Lang chain so it's a
[118:48] tool that makes it really really easy to
[118:50] deal with large language models and
[118:52] build robust large language model
[118:55] applications Lang chain is a an amazing
[118:58] framework actually and there's a lot of
[119:00] things that it does but one of the
[119:01] things that it does well is that it's
[119:02] able to load and parse documents easily
[119:05] we can use classes and wrappers to split
[119:08] documents and chunk them up um generate
[119:11] embeddings and many many many other
[119:13] things okay so it provides this unified
[119:16] abstraction for working with large
[119:19] language models and building llm
[119:21] application
[119:22] so tools are always welcome okay so I
[119:25] thought probably would be a good idea to
[119:27] look into Vector store and embeddings as
[119:30] an overview now it says Here length
[119:32] chain but essentially this is just in
[119:35] general how things work when it comes to
[119:37] Vector store embeddings large language
[119:40] models how to build a rack system and
[119:42] how everything actually works in the
[119:43] back end okay so just relax and let's go
[119:47] through this okay all right so this is
[119:50] the idea the main idea really as we saw
[119:52] before is that there's got to be some
[119:53] way of loading documents so we need to
[119:56] load the documents that come in in
[119:58] various forms Sometimes they come in as
[120:01] URL PDF files text file databases and so
[120:04] forth all of that is pulled together
[120:07] through as documents okay this is
[120:09] document loading and then goes through
[120:11] the process of splitting those documents
[120:13] as we saw and then once these are split
[120:15] of course go through the process of
[120:17] embeddings and everything and that is
[120:18] put into storage right this is a vector
[120:21] space or factor store but it has to have
[120:24] some retrieval as we saw because this is
[120:27] a way of getting information related to
[120:30] the question the query that comes in
[120:33] okay so the retriever goes through the
[120:35] vector store finding things are related
[120:38] to the question and then get relevant
[120:42] splits relevant documents and that is
[120:44] all what's put together with the prompt
[120:46] and large language model and pieces of
[120:49] documents and then large language model
[120:51] is able ble to give us the answer the
[120:54] result that's what we just saw so dis
[120:57] spitting what happens we go through the
[120:59] process of splitting and the splitting
[121:03] documents or the chunks are then passed
[121:06] through the embedding model so embedding
[121:08] model embeds creates this factorized or
[121:12] this number system or number format
[121:16] representation of those texts or those
[121:19] pieces of information Okay so so it
[121:22] looks something like this literally the
[121:24] text is converted into vector embeddings
[121:27] and looks something like that okay so
[121:29] embedding Vector keeps the content and
[121:32] the meaning of the text
[121:34] intact now this text with so the ideas
[121:38] that DET Texs with similar content and
[121:40] meaning will have similar vectors and so
[121:43] if you look at this diagram here this di
[121:46] this diagram hopefully puts everything
[121:48] into perspective we said that text with
[121:51] simp similar content and meaning will
[121:53] have similar vectors now remember that
[121:56] in this Vector space High dimensional
[121:58] space there's a lot of things going on
[122:00] lot of high dimension right so this is
[122:03] very simplified so that means then if
[122:05] you look at cat and kitty in this Vector
[122:08] space in this High dimensional space you
[122:09] can see that they are really close in
[122:11] meaning because a cat and a kitty
[122:13] they're essentially the same thing but a
[122:15] cat and run not the same thing right and
[122:20] so that is the representation but if you
[122:21] look at the actual representation in the
[122:25] vector space vectors as vectors numbers
[122:29] you will see something similar like this
[122:30] because cat and kitty are pretty much
[122:32] the same in terms of the semantic
[122:36] demeaning they will have some sort of
[122:38] similar numbers at some somewhere okay
[122:42] that's STA and that is actually what is
[122:45] stored into a vector database these
[122:48] numbers these vectors okay those are
[122:51] embeded things now the great thing is
[122:53] that Vector databases can be used in
[122:56] various cases now you can use all of
[122:59] this in searching for instance If You're
[123:02] Building A system that wants to find
[123:04] relevant uh results to the query string
[123:08] recommendation systems for instance
[123:10] finding items with related text strings
[123:14] uh that are recommended and
[123:16] classification to classify strings uh by
[123:19] most relevant and similar labels okay so
[123:23] this is some these are some use cases
[123:25] that are used when we talk about Vector
[123:27] stores and Vector databases and vectors
[123:30] embeddings and so forth now the whole
[123:33] embedding process works as follows
[123:35] number one of course we have documents
[123:38] that are split into chunks of text we
[123:40] saw this and then those are transformed
[123:44] into embeddings which then are added or
[123:49] saved as embeddings into a vector
[123:52] database now the great thing here is
[123:54] that most Vector databases they not only
[123:57] save the embedding Vector they also
[123:59] embed or save the original text split so
[124:03] on one side we're going to have the
[124:04] actual embedding Vector as you see here
[124:06] and on other side the corresponding
[124:08] original text so that's how most Vector
[124:12] databases store things so this whole
[124:14] process is extremely important because
[124:17] when we then are ready to query the
[124:19] vector store for embeddings in this case
[124:22] this is what happens first thing is we
[124:24] have here as you see let's put it all
[124:28] together we have the question this is
[124:29] the indexing part of thing the question
[124:31] goes through the embed process because
[124:33] it has to be transformed into embedding
[124:35] so that the whole system understands
[124:37] right and then what happens is that we
[124:39] go through the process of indexing here
[124:41] so it knows essentially to look for what
[124:45] is going to be similar to this embed
[124:48] which is part of this question so search
[124:50] and compare
[124:52] all of the entries in the vector store
[124:54] and then it picks out the most similar
[124:56] entry and once those are picked those
[125:00] most similar entries those are the ones
[125:03] that are packaged up with everything
[125:05] through the large language model so the
[125:08] large language model what it will do is
[125:10] going to take the most similar entry
[125:12] found and pass them into the to with the
[125:15] query the question the pieces of
[125:17] documents and everything to then get the
[125:19] answer the result so that how it all
[125:22] works so I wanted to show you this so
[125:24] that you have an idea on how Vector
[125:27] store and the large language model how
[125:30] everything comes together and rack
[125:32] systems so now that we understand rack
[125:34] systems we understand how embeddings
[125:37] large language model how this whole
[125:39] thing comes together this is the simple
[125:42] rack system handson or at least the
[125:44] overview of what we are going to be
[125:46] putting together so the idea here is
[125:47] very simple is that we're going to use
[125:49] Lang chain to pass through the PDF file
[125:52] to load that and then use l chains
[125:56] character text splitter to split that
[125:58] into small
[125:59] chunks as we have discussed and then
[126:02] we're going to use AMA embeddings this
[126:04] is the important part AMA also has a
[126:07] model for embeddings that is going to
[126:10] allow us to embed all of this
[126:12] information and put that into a chroma
[126:14] DB database and then what will happen is
[126:17] that we're going to pass in a query and
[126:18] we're going to use again Lang chains
[126:20] multi-query retrieve
[126:22] that will be able to do all things that
[126:24] it needs to do to transform this query
[126:26] into embeddings of course and search the
[126:29] chroma database with all those
[126:31] embeddings and then pull out the uh most
[126:35] similar pieces of information or data or
[126:38] documents and then put it all together
[126:41] relevant docs query and prompt and pass
[126:44] that through a large language model
[126:45] which in this case again you know is
[126:47] going to be Lama 3.2 and of course
[126:50] generate the resp response that we so
[126:52] desperately need so this is a r system
[126:56] now using
[126:58] AMA embeddings and using Lama 3.2 which
[127:01] is part of AMA now key Point here is
[127:04] that for all these embeddings AMA
[127:07] embeddings models and this large L model
[127:10] we can swap these with whatever else
[127:12] that we might want to use and the whole
[127:15] system will still work again I'm
[127:17] repeating myself I know that is the
[127:19] beauty of AMA is that once we have the
[127:22] system set up we can swap the embeddings
[127:25] we can swap the large language model
[127:28] these models with whatever else we want
[127:30] and it will still work okay so this is
[127:32] what we're going to be putting together
[127:34] next all right so first things here I
[127:37] have this B oi PDF we've seen this
[127:40] before so it should have access to all
[127:42] of this anyway it's just these
[127:45] beneficial ownership information report
[127:47] that I found online and that's what
[127:49] we're going to be using for our RX
[127:51] system okay now one thing I've also
[127:54] added here is this requirements. txt so
[127:57] this has all of the requirements all of
[127:58] the things the dependencies that we need
[128:01] for all these thing for all these
[128:04] project that we're going to be working
[128:05] on so I kept it here so that we can run
[128:09] so Al that you have it here that way
[128:12] it's easier you don't have to watch me
[128:14] always go ahead and type things and
[128:16] importing things and so forth so you
[128:18] have I may still do that but you have
[128:20] this work at txt with all the
[128:23] dependencies so the first thing I want
[128:25] to do of course is go ahead and run that
[128:27] real
[128:29] quick so pip install. R
[128:36] requirements txd like that so it's going
[128:39] to go ahead and get all the dependencies
[128:41] that was pretty fast that's that's very
[128:43] nice and we should be good all right
[128:47] very
[128:48] good and next you can see that I have
[128:50] also created this PDF rag. py this is
[128:54] what we're going to be working in so I
[128:57] have a few things that we're going to do
[128:58] here so I'm going to copy and paste in a
[129:02] few things so first we need to ingest
[129:05] the PDF files and then we need to
[129:07] extract text from that PDF file and
[129:09] split into small chunks so essentially
[129:12] the same things that we have been
[129:14] talking about in our presentation and
[129:16] then we're going to send those chunks to
[129:17] embedding model saved embeddings perform
[129:20] similar research retrieve and do all
[129:22] sort of things okay so that's the
[129:24] overview of what needs to be
[129:26] done all right so let's go ahead and get
[129:28] started now the first thing is I'm going
[129:30] to import a few things here so this is
[129:33] from Lang chain Community now this is
[129:35] working now because I
[129:38] installed a Rand requirements TT which
[129:41] included link chain community and other
[129:44] things so important that this happens
[129:47] all right so we getting the unstructured
[129:49] PDF loader and this online pdf loaded
[129:51] here now we may use one over the other
[129:53] but you have both and next what I'm
[129:56] going to do here I'm going to have the
[129:58] doc path and the actual models so I'm
[130:02] the model is going to be Lama 3.2 and
[130:05] the path through document just have one
[130:08] here is data Boi
[130:12] PDF that's what I'm
[130:15] doing okay so let's load the PDF file
[130:18] okay so I use the unstructured PDF
[130:20] loader and pass the doc path which is
[130:23] this one here and then data is going to
[130:27] get all of the loaded information right
[130:31] okay so now I can go ahead and actually
[130:35] print for preview so we see that things
[130:38] are actually working and you know that
[130:40] things are actually working I'm going to
[130:42] print just about 100 items from our
[130:45] content from data let's go ahead and run
[130:48] this fast
[130:53] so in the back end we should be able to
[130:56] do all the things like getting the
[130:59] information and load the
[131:03] document and I'm going to should be able
[131:05] to see soon here
[131:08] something okay very good so we went
[131:11] ahead and showed a few hundred words or
[131:16] 100 parts of the content that comes in
[131:19] in our content data which is the data
[131:22] that we loaded which is our PDF file
[131:25] very good so we see here P beneficial
[131:27] ownership information and that is good
[131:30] so just to test it out to make sure
[131:32] everything is working okay so that was
[131:34] the end of PDF ingestion so we're able
[131:36] to get that PDF next we're going to
[131:39] extract the PDF and do some chunking
[131:43] essentially okay so for that we need a
[131:47] lot of help of course so I'm going to go
[131:49] ahead and get all of the import so what
[131:51] happens here is that first of all we're
[131:53] getting ama ama embeddings this is a
[131:56] wrapper of AMA through L chain that's
[131:59] the beauty of Lang chain because we have
[132:02] a lot of wrapper classes and methods and
[132:04] so forth and then we have this recursive
[132:06] character text splitter which will help
[132:08] us to recursively split our text and
[132:12] then of course we have the chroma DB
[132:14] through Lang train all right so first of
[132:17] all let's go ahead and split things here
[132:21] keep in mind all we're doing here is
[132:23] independent of large language models
[132:25] we're using this is across the board
[132:28] this is how you would use you would do
[132:30] anytime you want to create a rack system
[132:33] all right so we splitting everything the
[132:36] chunk is 1200 so we're saying for each
[132:39] 1,200 chunks we're going to split so
[132:41] we're saying each chunk will have about
[132:44] 1,200 so we're saying each chunk size
[132:46] will have about 1,200 and the overlap is
[132:49] going to be 300 so the greater the
[132:51] overlap the better because that means we
[132:53] are conserving the context okay very
[132:57] good and then chunks we pass this text
[133:01] splitter which should have all that
[133:03] information actually this is all from
[133:05] the data that we loaded from here and
[133:10] then we use text splitter and split the
[133:11] document so now we have those chunks and
[133:13] if you want to see we can go ahead and
[133:16] print things out real quick here so you
[133:18] can see
[133:22] okay so you can see that we have the
[133:26] number of chunks first says here
[133:28] somewhere number of chunks is
[133:31] 42 and we have the actual
[133:35] chunks table of contents and all of that
[133:38] so this is good that means things are
[133:40] working and the beauty of using lank
[133:43] chain is that it adds metadata to all of
[133:46] these pieces you can see now we have
[133:48] this metadata was added called source is
[133:51] going to be data Boi PDF which is really
[133:54] good okay otherwise we will have to do
[133:56] all of that manually okay so we know
[133:59] this is working let me go ahead and
[134:01] comment this
[134:02] out okay now it's time for us to add
[134:05] into our Vector database because we have
[134:07] those chunks and we need to add them but
[134:09] how do we add to Vector database well
[134:11] first we need to do something first we
[134:14] need to create those embeddings if you
[134:16] go to
[134:18] ama if we go to ama.com and go to
[134:21] models you find many many different
[134:24] models one of the models called
[134:26] embeddings let say embeddings you can
[134:28] see that we have this nomic embedding
[134:31] embed text we also have this MX Bay
[134:35] embed large and many others okay we can
[134:37] use whatever want but this is the one
[134:39] we're going to be using so click here
[134:41] and you can see it's called it's a high
[134:43] performing open embedding model with a
[134:46] large token context window which means
[134:48] that means it allows you to put a large
[134:51] amount of text for embeddings to for it
[134:54] to be able to embed okay so this is what
[134:56] we're going to be using for our example
[134:59] here that means we'll have to pull this
[135:01] in code and so let's go ahead and do
[135:05] that first I'm going to go ahead and
[135:07] import
[135:08] AMA and then I'm going to
[135:11] incode pull in this embedding now I
[135:14] already have this so it's just going to
[135:17] go ahead and update I'm just put this in
[135:19] way so you have and then what we need to
[135:21] do is to actually create the
[135:24] vector database and it's very simple all
[135:27] we do we say chroma from documents and
[135:30] pass the documents chunks remember the
[135:32] chunks are here and the embedding now
[135:36] we're using this AMA embeddings and
[135:38] passing the mo the model nomic embed
[135:40] text to generate the embeddings so all
[135:44] of that is paying p through our cha from
[135:47] documents function and of course we
[135:51] collection name we just pass name simple
[135:53] rag that's
[135:54] fine
[135:56] okay all right so continuing let's go
[135:58] ahead and do the retrieval side of
[136:00] things so how do we retrieve things and
[136:03] so forth so first let's go ahead and
[136:06] pass in a few things here now we need to
[136:10] some help with L chain where we can
[136:12] create prompt templates as you see and
[136:15] also way of parsing whatever comes in or
[136:18] goes through this rack system we'll see
[136:20] in a second here so that's why I'm
[136:23] passing importing all of these and while
[136:25] we're here let's go ahead and also
[136:27] import a few other things such as the
[136:30] chat AMA because that's what we're going
[136:32] to be using that model
[136:34] to as our large language model and then
[136:38] now we have this runable pass through
[136:41] this is part of what we're going to see
[136:43] later part of chaining that happens with
[136:46] AMA with L chain we have this multi-
[136:50] quare retriever as you see it's going to
[136:51] help us with a lot of things because
[136:54] we're going to be able to retrieve
[136:55] documents going to be able to give a
[136:57] query and use a large language model to
[136:59] sort of write a set of queries and pick
[137:02] up the best one to pass through the
[137:05] large language model so this is all
[137:07] optimization under the hood okay let's
[137:09] go ahead and set the model that we're
[137:11] going to be
[137:13] using so this is how set model now this
[137:15] is a wrapper or Lama wrapper and we pass
[137:18] the model which is Lama 3 to all right
[137:21] so we have the large language model
[137:23] already now as you know we're going to
[137:25] go ahead and use the simple technique to
[137:28] allow us again to generate multiple
[137:30] questions from one single question and
[137:32] then retrieve documents based on those
[137:35] questions which allows us to get the
[137:37] best of both worlds so this is the query
[137:41] prompt that I'm going to use here so you
[137:42] can see here I'm using the prompt
[137:43] template the reason why is because we
[137:45] can then pass things like input
[137:46] variables it's going to be the question
[137:49] that internally we'll know how to pass
[137:51] around all these pieces of data what you
[137:53] can see here the templates say you are
[137:55] an AI language model assistant your task
[137:58] is to generate five different versions
[138:00] of the given user question to retrieve
[138:02] relevant documents from a vector
[138:05] database by generating multiple
[138:07] perspectives on the user question your
[138:10] goal is to help the user overcome some
[138:14] of the limitations of distance based
[138:16] similarity search provide these
[138:19] alternative questions separated by new
[138:21] lines original question and it pass the
[138:23] question which is going to be passed
[138:24] along so this prompt template here it's
[138:27] going to be passed along soon through
[138:29] the Retriever and it will know how to
[138:32] deal with all of that this is beautiful
[138:34] because we're using lank chain and it
[138:37] does everything automatically for us all
[138:39] right so now let's go ahead and actually
[138:42] create our retriever so our retriever
[138:44] here we're going to use the multi-query
[138:46] retriever from large langage model and
[138:48] we pass a few things okay we have the
[138:51] vector database so we need of course the
[138:53] vector database that we created here and
[138:55] then we're going to transform it into a
[138:57] retriever so retriever will know how to
[138:59] retrieve things from database it's going
[139:02] to return something and we need to pass
[139:05] the large language model because it
[139:06] needs to know how to take that item all
[139:10] those information retrieved and how to
[139:12] process that and with that we also need
[139:14] the prompt so you remember in the
[139:16] diagram when it comes to once we have
[139:18] everything set up we retrieve
[139:20] information
[139:21] we retrieve pieces in this case we're
[139:23] retrieving pieces that are similar to
[139:25] The query that we're passing along which
[139:27] is inside of a prompt and then we need
[139:29] the large language model which is going
[139:31] to drive everything it's going to use
[139:33] all that information compact everything
[139:35] and be able to answer questions all
[139:38] right very good and next we're going to
[139:40] use the create the rag prompt so let's
[139:43] go ahead and do that real quick so
[139:45] essentially it's another prompt that we
[139:47] can use this is our question that we
[139:49] pass along okay rag prompt this is just
[139:51] another template we're going to pass
[139:53] says answer the question based only on
[139:55] the following context so we're going to
[139:57] pass the context and the actual
[139:59] questions this context and everything is
[140:01] going to passed along as you will see
[140:03] later right internally with blank chain
[140:06] okay so once we have that we need to
[140:08] create an actual chat prompt from all of
[140:10] that so we use chat prompt from template
[140:13] and pass this template here so these
[140:15] wrappers here internally will know how
[140:17] to pass in this context how to pull in
[140:19] the question and so forth cuz these come
[140:21] in as just variables okay and once we
[140:25] done that that's when we then put all of
[140:28] that into a chain which is going to go
[140:30] through the process of pulling in all
[140:33] the all the pieces that we put together
[140:34] so that we can get the result so
[140:36] essentially what it means what is
[140:38] happening here is that we passed the
[140:39] context we know the context is going to
[140:40] be the retriever retriever will know as
[140:43] you saw here how to get things from
[140:47] database knows the large L metal model
[140:50] that's going to be driving everything it
[140:52] also knows the prompt which we created
[140:54] here which has a few input variables
[140:57] we're passing along so we're putting all
[140:58] together and then we have the question
[141:00] runnable pass through let's just say
[141:02] just pass through don't mind much and
[141:04] then we have here a chain to pass in a
[141:07] prompt and then we need the large
[141:09] language model again and then at the end
[141:11] once we have the result we're going to
[141:13] pass that through a string output parsel
[141:16] essentially to parse everything out to
[141:18] make everything clean and nice all that
[141:21] is done we can just start asking
[141:23] questions so I'm going to comment this
[141:25] out let's start with the main question
[141:28] at the top
[141:31] here say what is the document about for
[141:34] instance let's go ahead and run this so
[141:35] this is it this is our fullon rag system
[141:39] let's go ahead and run and see if this
[141:40] works okay so done loading done
[141:43] splitting document done adding to Vector
[141:46] database very
[141:49] good okay
[141:51] it says here the question the answer
[141:53] says the document appears to be a set of
[141:54] instruction for reporting beneficial
[141:57] ownership information especially the
[142:00] Boi filing instructions provide guidance
[142:03] and blah blah blah very good that's
[142:05] exactly right and so now we can pass
[142:08] questions so that's one question let's
[142:11] see if I go and comment this out I want
[142:14] this second on what are the main points
[142:16] as a business owner I should be aware of
[142:19] let's go ahead and run
[142:25] again now this is very inefficient
[142:27] because in a way it goes back and does
[142:30] the splitting and adding everything but
[142:31] that's okay just to show you how things
[142:33] are done okay look at this as a business
[142:36] owner you should be aware of the
[142:37] following main points and gives us all
[142:40] the
[142:41] points and let me do another one here
[142:56] okay based on provided text here's how
[142:58] to report beneficial ownership
[143:00] information Boi to finsen complete
[143:04] filing and there we
[143:08] go all right so it is indeed working we
[143:11] were able to put together a full rag
[143:14] system be it we could improve improve in
[143:17] terms of repetition of you know
[143:20] splitting into all the things that's
[143:21] okay but you can see that it this works
[143:24] we use large language model in this case
[143:26] one large language model from our own in
[143:29] this case ol Lama uh llama 3.2 and then
[143:33] we used embeddings also from all Lama
[143:37] just like that again we can just split
[143:40] around we can change the models that
[143:42] we're using and leave everything as is
[143:45] and it should work very good very good
[143:47] indeed so I hope you're seeing uh the
[143:49] benefits here number one again remember
[143:52] is that we don't pay anything we can run
[143:54] this as many times as we want we're not
[143:56] paying anything for any hits to the API
[144:00] in this case open AI or anywhere else
[144:02] because this is local which is really
[144:04] really awesome if you ask me so I hope
[144:07] this is helpful and uh take this code
[144:10] and change it around make it your own
[144:12] maybe you have a different PDF file that
[144:16] you want to uh converse with or whatever
[144:19] else that you need so go ahead and play
[144:21] around this is your code you have access
[144:23] to it and so just just do it all right
[144:27] so we finished this PDF rag it works
[144:30] fine and so I decided actually to show
[144:33] you here I have the other piece of code
[144:35] which is exactly same that's the same
[144:37] thing but I call this PDF rag clean so
[144:40] essentially it's the same thing as what
[144:41] we've seen it's just that it's a little
[144:43] bit cleaner and looks better okay so
[144:46] essentially at the top here we have some
[144:48] configurations and blah blah blah and
[144:50] have this doc path we have this model
[144:53] name so this created constants at the
[144:56] top here and Vector name as well okay so
[144:59] we have a function here called injest
[145:00] PDF it's essentially that we just pass
[145:03] the doc path and it does all the thing
[145:05] that it needs to do we have the split
[145:07] documents function this is how exactly
[145:10] how you should do it right split
[145:11] everything into modules and then we have
[145:14] this create database and it returns a
[145:17] VOR DB and we create Retriever and
[145:19] everything
[145:20] and we're passing along the templates
[145:24] and all of that so essentially like I
[145:25] said what we've seen before but now in a
[145:28] more cleaner way and then we have this
[145:31] create chain which creates the chain for
[145:33] us as we saw before the code is still
[145:35] the same and then we have this main so
[145:38] now we just call for instance get the
[145:40] data by calling injust PDF and then we
[145:43] get the chunks VOR DB instantiation the
[145:46] llm instantiation and Retriever and then
[145:49] we have to check and then we have our
[145:51] question right so we can add different
[145:53] question here let's go ahead and run
[145:55] this we should see the same thing
[145:57] essentially but just different code
[146:01] which is
[146:02] cleaner okay so it's doing all the
[146:07] things okay so there's a lot of
[146:10] good logging that's happening and voila
[146:13] so at the end response according to
[146:15] instructions this is what we have so
[146:17] nothing out of this world the same thing
[146:19] but this is is way better because it's
[146:21] way organized so you have both and you
[146:24] are set and because it's always nice to
[146:27] have some sort of a user interface I
[146:29] have another version you have access to
[146:32] called PDF Rag streamlit and in this
[146:35] case when we run this we should be
[146:38] having we should be able to have a user
[146:41] interface using streamlet so essentially
[146:43] obviously it's the same code it's just
[146:45] that we have at the bottom here
[146:47] streamlet let look see
[146:50] setting up the title and user input and
[146:54] everything else okay you can look at
[146:55] this code but it's it's not uh out of
[146:59] this world so let's go ahead and run
[147:00] this so I can show you real
[147:05] quick okay there we go so now I
[147:11] can let's see so it's going to go
[147:14] generate response verus running the load
[147:16] Vector DB and in a few moments hopefully
[147:19] we should see an
[147:22] answer okay so to file for beneficial
[147:25] ownership information there we go all
[147:28] that information very good very good
[147:31] indeed and I can
[147:33] say what's the
[147:38] penalty for not
[147:42] filing there we go and it gives us the
[147:46] information all right so there you have
[147:48] it now you have different versions and
[147:51] this is the user interface version that
[147:54] you can see that we are conversing with
[147:57] our rack system with our document so
[148:00] quick break here I just want to let you
[148:02] know that I thank you for being here for
[148:04] learning this awesome tool with me here
[148:08] I do have something for you as a way of
[148:10] me saying thank you for being here with
[148:12] me and I appreciate you watching this
[148:13] video this mini course I have an olama
[148:17] pack that you can download that includes
[148:19] source SCE code code templates and many
[148:22] other things that it's going to be a
[148:24] surprise and you can get that for free
[148:26] obviously you go in the link in the
[148:28] description and you can go and get this
[148:30] AMA starter pack all right okay let's
[148:33] continue all right so the next
[148:34] application that we're going to be
[148:36] building we're going to still use AMA
[148:38] Lama 3.2 but you can always change the
[148:41] model and this application is going to
[148:44] be a full-fledge application that allows
[148:46] us to actually use swarm framework which
[148:49] is is a new open AI framework that
[148:53] allows us to create agents agents that
[148:56] will actually be able to delegate
[148:58] certain tasks to other agents and do
[149:02] different tasks so this is going to be
[149:04] full-fledged application an AI solution
[149:07] application that you can actually use in
[149:10] real world most of the ones that we've
[149:12] been doing you can use in real world of
[149:14] course but this one is something that is
[149:16] actually I would say that it's usable so
[149:19] if you want to get a job usually you
[149:21] have to find somebody who will have that
[149:24] job for you okay and usually you have to
[149:26] go through recruiters and recruiters
[149:28] usually work in the agency and it's a
[149:31] slew of process a lot of people that are
[149:33] involved usually to get you matched with
[149:37] an employer and so in this video I'm
[149:39] going to show you how to take this whole
[149:43] agency with different people working
[149:46] these are recruiters how to actually
[149:48] create AI agents that will do exactly
[149:52] what the real recruiters do these
[149:55] recruitment agencies have a lot of
[149:58] moving parts and one of the things that
[150:00] can be very difficult is that when they
[150:03] need to vet people they actually have to
[150:06] of course read the resumés and they are
[150:08] getting many many of those resumés and
[150:10] have to go through each one of those and
[150:13] contact people and try and trying to
[150:15] figure out their qualifications and how
[150:18] would they match them to an employer
[150:20] what would be a solution well you know
[150:22] this drum roll yes the solution would be
[150:25] AI agents with AI agents they're going
[150:28] to be able to pass along different tasks
[150:31] to other AI agents to do exactly that
[150:34] from extracting information and then
[150:37] analyze the resume and maybe start
[150:40] figure out how to connect or how to
[150:42] match to different employers or
[150:45] different positions and then have a full
[150:48] report with recommendation
[150:50] so that the whole agency knows what is
[150:53] the process that went through to get to
[150:55] that conclusion where X person or X Y
[150:58] and Z people need to be matched to to a
[151:02] certain company and so that is exactly
[151:04] what we're going to be doing in this
[151:06] video I'm going to show you how to use
[151:08] large language model in this case AMA
[151:10] Lama
[151:11] 3.2 to Aid Us in this on top of that
[151:15] we're going to use swarm framework that
[151:17] allows us to create agents AI agents
[151:20] very quickly simply and to the point so
[151:24] let's go ahead and let's go ahead and
[151:27] build a recruitment agency all with AI
[151:30] agents so here's what we're going to be
[151:32] building so here is the user interface
[151:35] so it is indeed a UI recruiter agency so
[151:38] you can go ahead of course and upload
[151:40] things but most importantly you can see
[151:41] we have this about that we click it
[151:44] gives us some about this whole thing now
[151:46] of course here is saying AMA 3.2
[151:49] the framework swarm framework and all
[151:51] the things so it says here our system
[151:53] uses specialized AI agents to do what
[151:57] well to extract information from rums
[152:00] and analyze candidate profiles and also
[152:03] match with suitable positions screen
[152:07] candidates and then provide details
[152:09] recommendations so this is going to be a
[152:12] full-fledged agency that we're going to
[152:15] be building using code and llama 3.2
[152:19] okay of course the front end is going to
[152:21] be stream it which is what you see here
[152:23] so the idea is very simple and let me go
[152:26] back to the upload so here what we can
[152:28] do is we can go ahead and upload files
[152:31] now I already have a resume that I'm
[152:33] going to go ahead and just drop it right
[152:36] there so you can imagine now things are
[152:38] going to be happening in a back end so
[152:40] the analyzing of the resumé is happening
[152:42] well first of all different agents are
[152:45] doing what they need to do to get
[152:47] started so the first agent is going to
[152:49] go ahead and and pull information from
[152:52] this PDF file which is a resume and then
[152:54] it's going to pass on to different
[152:56] agents that will analyze and do all sort
[152:58] of things so this is going to take a
[153:00] little bit and at the end the main agent
[153:02] is going to be able to give us
[153:05] recommendations through analysis and
[153:07] there we go we can see that we have now
[153:09] this analysis complete so we have
[153:11] recommendations so so we have analysis I
[153:14] can also close this tab menu there so
[153:17] scale analysis analysis for Janine
[153:19] Nail's resume and gives us sales
[153:22] engineering industrial engineering
[153:24] project manager a management and all
[153:27] these great great stuff so it pulls in
[153:29] all the information from the resume and
[153:31] at the end here you can see
[153:32] recommendation to enhance J Na's profile
[153:35] consider following add more experience
[153:38] develop technical expertise increase
[153:40] Education and Training confidence is
[153:45] 85% wow it gives us the confidence that
[153:48] this person could potentially do well in
[153:51] a certain industry and of course I'll
[153:54] show you that results also have been
[153:56] saved locally but you can see also have
[153:58] other things here we have this job
[154:00] matches it tells us exactly this person
[154:04] matches well 80% in New York for this
[154:07] data scientist position all right and we
[154:10] have the software engineer position and
[154:12] says it the match is 70% and is remote
[154:16] so this is exciting that's exactly what
[154:18] we're going to be building and also we
[154:20] have the screening here screening
[154:22] results gives us a summary of everything
[154:24] qualifications and the percentage uh
[154:27] experience relevance and all of this
[154:30] information all right and of course we
[154:33] have another tab the final
[154:34] recommendation they give us the
[154:37] recommendation and all of this ladies
[154:39] and gentlemen is built using AI but most
[154:43] importantly using AI agents you can see
[154:46] how helpful how useful this tool can be
[154:49] by the way if you are interested in
[154:51] being a part of a community where you
[154:53] find like-minded people to build AI
[154:56] tools from scratch and you will have
[154:58] templates we have weekly monthly calls
[155:01] that we have for Q&A and many many other
[155:04] things in this community where people
[155:06] are actually building AI Solutions
[155:09] you'll have everything you need to build
[155:12] AI Solutions using code and AI tools is
[155:15] a place where I'm very proud about and
[155:17] there's a lot of people who are enjoying
[155:19] this community and I would love for you
[155:21] to join the community where you'll be
[155:23] able to dive deeper into building AI
[155:27] solutions that actually solve problem
[155:29] okay so let's go ahead and get started
[155:31] all right so I already have the code and
[155:34] because of the nature of this project uh
[155:37] there's a lot of moving parts and uh
[155:39] there's a lot of code as well not too
[155:41] much and so what I will do I will just
[155:43] walk through the code and no worries
[155:45] you're going to have access to all this
[155:47] code but what I want you to keep in mind
[155:49] is that you can take this and apply to
[155:52] something else so that is the beauty of
[155:54] using agents that's the beauty of
[155:56] building these Solutions AI Solutions so
[155:59] I have here inside AI recruiter agency
[156:03] and we have this folder called agents
[156:05] let's focus on that real quick but at
[156:07] the center of everything the way I
[156:09] wanted to organize this code is that
[156:11] because each agent will have to have
[156:15] access to the large language model in
[156:17] this case we're using AMA that's another
[156:19] Point here is that in this case we're
[156:21] using AMA Lama 3.2 which means you don't
[156:24] have to pay for inference or pay for an
[156:27] NPI key which is really great and it's
[156:31] really good because then you can build
[156:33] these systems build these AI Solutions
[156:36] locally and test them out before you can
[156:39] push them to production so that is one
[156:42] of the great things about using AMA
[156:45] models but it is still the same if you
[156:47] want to use different models Okay so at
[156:49] the center of everything the way I
[156:51] organize this code is a little bit more
[156:53] organized in a sense where you would
[156:55] find in a production application now
[156:58] there are a few things here that you
[156:59] probably will have to kind of work
[157:03] around to make it production ready but I
[157:05] put the code so that it is a little bit
[157:07] more organized we're using classes and
[157:10] everything now if you don't know a lot
[157:12] about classes bython it's okay um I have
[157:15] a lot of videos where I talk about
[157:17] python um but this is how you would
[157:19] usually want to put together a
[157:21] potentially production ready application
[157:24] at the center of everything we have this
[157:26] base agent here what is this well it is
[157:29] indeed a class make this smaller so you
[157:30] can see as you can see here it's a class
[157:33] called Base Class that we pass in a few
[157:37] things in this Constructor so when we
[157:39] call Base Class which is going to be
[157:41] that all agents are going to inherit
[157:43] from which means they're going to model
[157:46] after um this is what we have so each
[157:49] agent of course has a name instructions
[157:52] so this is the model uh way that we do
[157:55] with
[157:57] agents using swarm that is what we're
[157:59] using here and you pass in of course the
[158:02] opening eyes now because I have another
[158:05] video where I talk about how to use
[158:08] swarm and opening ey but override
[158:10] opening eye in this case so that we can
[158:12] use l 3.2 or AMA any AMA model okay so
[158:18] this is how you override that I have
[158:19] another video where I talk about that
[158:21] you can find somewhere here or you can
[158:23] just search on my channel okay so what
[158:25] we do here we pass a base URL and
[158:28] instead of having in this case the base
[158:31] URL that would have been for openi we
[158:34] have our own base Ur So this tells us
[158:37] that this is going to be locally using
[158:40] AMA in this case pointing to a llama and
[158:44] AMA model okay and then we pass API key
[158:48] here just AMA it's not requ required but
[158:50] it needs to be passed around so at this
[158:52] point here when we instantiate this base
[158:55] agent we're going to know that we
[158:57] actually going to be using instead of of
[158:59] open AI because we're using swarm
[159:01] framework we're going to be using our
[159:04] local AMA models okay and then we have
[159:07] this function here that is defined
[159:08] called run so this will get of course
[159:12] the message which going to be a list and
[159:15] then we have this query or Lama so at
[159:18] this point we're going to pass a a
[159:19] prompt as a string and then we're going
[159:21] to go through the process of calling our
[159:24] client or Lama client which was created
[159:27] here and then we say hit the chat in
[159:30] completions that create so in this case
[159:33] here you notice we passing the Lama 3.2
[159:36] and we have messages in this case we
[159:38] have system and we pass the self
[159:40] instructions so these instructions will
[159:42] have passed along through the agent okay
[159:46] so this is how you structure it I know
[159:48] for some of you this may be a little bit
[159:49] Advanced but it's a good it's a good
[159:52] exercise and we're going to pass
[159:53] temperature and tokens and all of that
[159:55] all right and then we return the
[159:57] response so whenever we say query orama
[160:01] this is what's going to happen we're
[160:03] going to pass the model and then the
[160:05] messages all of that is going to be
[160:07] passed along also the prompt and the
[160:09] instructions okay okay so here is this
[160:12] helper function here par Jon safely this
[160:15] is just going to bring get the text that
[160:18] is coming in as I said safely pars Json
[160:21] from text handling potential errors
[160:23] that's all we're doing here so that is
[160:25] the base agent so now once we have the
[160:28] space agent what we need is to create
[160:31] the actual agent swarm opening ey agents
[160:35] and if I go back here you can see that I
[160:37] have a bunch of Agents here so these
[160:39] agents we have for instance the
[160:41] extractor agent as name imply is going
[160:43] to be responsible for extracting
[160:46] information from the PDF file and then
[160:49] we have the match agent which is going
[160:51] to match whatever we got from the
[160:54] extraction in this case the PDF and try
[160:57] to match it with the company or the
[160:59] position that the person may be a good
[161:02] fit for we have the orchestrator and we
[161:05] have profile enhancement recommander and
[161:07] Screen agent let's start with screen
[161:09] agent because all of these agents will
[161:12] essentially in terms of code codewise
[161:15] they will look pretty much the same now
[161:17] this is important look how beautiful
[161:18] this is so the screen agent here we are
[161:21] creating a class just to make things
[161:23] simpler and scalable and then it's
[161:26] inheriting inheriting from the base
[161:29] agent right because the base agent as we
[161:31] saw has all the properties all the
[161:33] functions or methods that needs for it
[161:35] to actually become an agent so that's
[161:38] what we're doing here we're initializing
[161:39] our things so we are calling
[161:41] initializing the names to be screener
[161:43] and we have the instructions here to say
[161:46] screen candidates based on
[161:48] qualifications alignment experience and
[161:50] everything so we're passing the actual
[161:52] instructor instructions for this agents
[161:55] we say provide comprehensive screen
[161:57] reports and all that and we have the run
[162:00] so this is just going to go print to say
[162:02] okay screener conducting initial
[162:03] screening that's all that is this uh the
[162:06] Run function is going to do and then we
[162:08] have work workflow context here we are
[162:10] evaluating getting information from our
[162:12] messages that we getting and then get
[162:15] the content so essentially we're going
[162:16] to get what the large language model in
[162:19] conjunction with the agent the results
[162:22] and pass that into the query AMA and if
[162:25] you remember if I command click query
[162:27] AMA what does it do query AMA model with
[162:30] the given prompt so now we get that
[162:32] information we got and then pass through
[162:35] AMA this agent here well it's a call
[162:38] essentially to this endpoint so that it
[162:41] knows what to do and then we get the
[162:43] response and of course we return an
[162:46] object or in this case a dictionary
[162:48] which is going to have screening report
[162:51] and we're passing into date here we
[162:52] could make this more Dynamic but this is
[162:54] okay and screening score 85 in this case
[162:59] right so let's go ahead to another agent
[163:01] let's go to extractor agent so what does
[163:03] it do well this extractor agent as name
[163:06] apply it just extracts the information
[163:09] from a PDF file okay we importing a few
[163:12] things here but the idea is very simple
[163:15] again it's base agent is the base agent
[163:17] that we pass right that's going to to be
[163:19] the model for this extractor agent and
[163:22] we have the name instruction extract and
[163:25] structure information for from reses
[163:27] focus on personal information work
[163:29] experience education skills and
[163:31] certification and all that provide
[163:33] output in a clear structured in format
[163:36] so it's important that we make these
[163:39] instructions very direct and very
[163:41] concise okay we have to run again so
[163:44] this case is going to show us this
[163:46] extractor is processing resume and we do
[163:48] the exact the same thing as we saw
[163:50] before so like like I said earlier most
[163:53] of these are going to be the same in
[163:55] terms of the code right and we extract
[163:58] everything and also we um extract the
[164:01] text in this case from the PDF file
[164:03] because we're going to pass that along
[164:05] because we take of course the raw text
[164:07] that we should have at this point and we
[164:10] pass along to query AMA as you know what
[164:12] it does and then of course we return
[164:14] again the raw text the structure data in
[164:18] this case the exu
[164:19] structured I should say information and
[164:22] the results so we're just returning a
[164:24] dictionary so we can see what's going on
[164:27] all right so it's pretty much the same
[164:28] let's go to matcher the matcher here you
[164:31] know what it's going to do it's going to
[164:33] match whatever it was extracted from
[164:36] resume trying to match with certain
[164:38] positions that they might have so it's a
[164:41] matcher the same thing again that's the
[164:43] beauty and the instructions here is that
[164:45] match candidate profiles with job
[164:48] positions consider skill match
[164:50] experience level location and everything
[164:53] return matches in Json format with title
[164:56] match score and location Fields okay the
[164:59] same thing as you see here this run it's
[165:02] going to print matcher finding suitable
[165:05] jobs and all the great stuff now here is
[165:09] where we just passing sample jobs so
[165:11] this is where you would perhaps hit a an
[165:14] API that might have somewhere with a lot
[165:17] of different jobs or or a PDF file or a
[165:22] CSV file whatever you have that would
[165:24] have different jobs so this case here we
[165:26] just have the samples jobs with with a
[165:29] few titles and everything so that's the
[165:31] beauty you can just plug in the source
[165:34] where you'll have a list of different
[165:37] jobs with requirements locations and
[165:40] everything okay very good so that's it
[165:42] that's what's going to be used for the
[165:44] matcher to be able to match it's going
[165:46] to look at the information that it's
[165:47] received from other people this case
[165:50] other agents and find the match okay so
[165:54] match response so this is where the
[165:56] matching results from a Lama come in so
[165:59] what we do we call the query or Lama and
[166:02] we pass this F string here analyze the
[166:04] following profile and provide jobs
[166:05] matching valid Json okay so we have the
[166:08] profile here it's going to be and passes
[166:10] in okay the analysis because in order to
[166:13] analyze in order to do the matching it
[166:15] needs to have the pieces that it needs
[166:18] in this Cas anal analysis results that
[166:20] were gotten from a different agent okay
[166:24] and then we pass the available jobs is
[166:26] this so here's return only adjacent
[166:29] object with this exact structure so it's
[166:33] very important to usually guide the
[166:35] large language models when it comes to
[166:38] the formats that you want okay and there
[166:40] we go so this is the format that we have
[166:43] here make it smaller so you can see
[166:45] everything and then we can parse the
[166:48] response to make sure that we get
[166:50] everything in this is PA Jon safely
[166:53] function that we saw in the um agent I
[166:57] believe the base agent okay we have some
[167:00] fullbacks here uh for errors and
[167:02] everything and then we just return the
[167:04] par response now before we go to the
[167:07] orchestrator here let's go to the en
[167:10] profile enhancer agent so what does it
[167:12] do well this is going to kind of enhance
[167:15] the profile as the name apply it's it's
[167:18] going to go ahead and pass in the
[167:20] extracted information it's going to copy
[167:22] that information and enhance essentially
[167:25] the user the profile of that candidate
[167:28] and return that enhanced profile
[167:30] essentially to extract what needs to
[167:31] extract and put all nicely okay and then
[167:35] of course uh the profile agent here
[167:38] that's what we pass we call agent
[167:40] profile agent and pass Lama 3 and we say
[167:43] enhance the candidate's profile based on
[167:45] the extracted information so this is
[167:47] where we actually call this is where
[167:49] then we pass this function that we
[167:52] created here
[167:53] okay and pass through the agent because
[167:56] you will know that is the function it
[167:58] needs to run very cool all right let's
[168:01] continue here and let's go to the
[168:03] recommender agent so recommender agent
[168:05] as the name imply it's going to be the
[168:07] same thing it's going to recommend right
[168:09] so we have the instructions here
[168:11] generate final recommendations
[168:13] considering extract profile skills
[168:15] analysis and provide all these
[168:17] information so again nothing new here
[168:20] and of course we're printing everything
[168:23] when we run and in this case here we do
[168:26] the same thing we pass we call the query
[168:29] AMA and pass the workflow context so
[168:32] essentially is the information that
[168:34] we're getting from the evaluation of all
[168:36] the messages come in okay and then we
[168:38] return the recommendation this is going
[168:40] to be sort of a a dictionary with a time
[168:44] stamp and confidence level in this case
[168:47] we just say hi okay of course you can
[168:49] add more things to make it actually um
[168:52] realistic but this is pretty good okay
[168:54] let's continue let's go to screener
[168:56] agent now the screener agent I think we
[168:58] looked at it well that's exactly what it
[169:00] does it screens to make sure everything
[169:02] is good and pass it along all right so
[169:06] that is what it does it's very simple
[169:08] the same thing again and there you have
[169:11] it now the most important thing here
[169:13] again to keep in mind is that everything
[169:15] is base agent so that means all these
[169:17] agents have a parent class that sets
[169:20] everything up so that is the best way
[169:23] for you to think about when you're
[169:24] building real world applications or
[169:26] Solutions all right okay let's go ahead
[169:29] and look at the orchestrator the
[169:31] orchestrator in this case is the one
[169:33] that orchestrates everything that is the
[169:35] the the myestro if you will okay and so
[169:39] it has a lot of things of course we have
[169:41] to import all of those but again it is
[169:44] still an agent so it has to um inherit
[169:47] from base agent so we give it a name we
[169:50] give it instructions Now instructions
[169:51] are very important for agents okay says
[169:54] it coordinate the recruitment workflow
[169:56] and delegate tasks to spe uh specialized
[169:59] agents ensure proper flow of information
[170:02] between extraction extraction analysis
[170:05] matching screening and recommendation
[170:07] phases phases maintain context uh
[170:10] important maintain context and aggregate
[170:13] results from each stage okay so we got
[170:16] self that set up agents so the a
[170:19] function that is been called that goes
[170:22] ahead initializes all the specialized
[170:25] agents all the ones that we' talked
[170:27] about look at that okay very good
[170:29] because as an orchestrator it has to
[170:31] know which agents that it has access to
[170:34] to start
[170:36] delegating we have the run again the
[170:38] same thing we've seen and then we have
[170:40] this process application so essentially
[170:42] it's going to just go ahead and pass
[170:43] some information here along and print
[170:46] out orchestrator starting application
[170:48] process and so forth so we extract the
[170:51] resume so it's going to go ahead and run
[170:54] to go and extract resume update in this
[170:56] case the work context workflow context
[171:00] and then we analyze the candidate
[171:02] profile okay called self analyzer to
[171:05] start the
[171:07] analysis right so this case we're
[171:10] calling all of those agents right you
[171:12] can have over can see it is indeed an
[171:14] analyzer agent okay and then we app
[171:18] upate the workflow context here so each
[171:21] step of the way as we're calling these
[171:23] different or the orchestrator is calling
[171:25] the these different agents is going to
[171:27] do what it needs to do okay delegate and
[171:30] create in this case add to the workflow
[171:33] context okay and does with matching jobs
[171:36] screening candidates and then we
[171:39] generate the recommendation all right so
[171:41] it calls the recommander and that's
[171:43] exactly the same thing and we have some
[171:45] exception just in case things don't work
[171:47] and that's pretty much it now we have
[171:49] other things here logs and everything
[171:51] but that's something that you can take a
[171:53] look yourself but this this is it and so
[171:55] once we have that we also have of course
[171:58] the app that why a app that py should
[172:02] say so this has some code with streamlet
[172:06] which allows us to look at something
[172:09] that is nice to look at as you saw
[172:11] earlier okay so at this point here let's
[172:13] go ahead
[172:15] and do a quick run so we say stream late
[172:18] run
[172:19] app.py and what will happen is going to
[172:22] go ahead and open our agency AI
[172:25] recruiter agency we have a broken link
[172:28] here but that's okay we can go to about
[172:30] for instance and tells us about this
[172:33] agency it's very very cool what it does
[172:36] and everything and all of this was
[172:38] created using streamlet and we go to
[172:40] upload now we can actually browse and
[172:42] upload something a resume so I do have a
[172:45] resume that we can go ahead and upload
[172:47] I'm going to upload here real quick and
[172:49] the moment you upload that drop it and
[172:52] it's going to start the process and the
[172:55] cool thing as this happens I have here
[172:57] logs that are actually being sent out
[173:01] and also I have what's happening in the
[173:03] whole orchestration uh showing here
[173:06] orchestrator starting application
[173:08] process and is analyzing everything and
[173:12] it's working it will take a little bit
[173:14] because there's a lot of things are
[173:16] happening and you can see all of that
[173:19] Happening Here Right tells us
[173:22] orchestrator starting analyzer did that
[173:25] matching screener and recommender and
[173:30] it's happening so we're going to take a
[173:32] little moment here until hopefully we
[173:34] get the final results okay so after a
[173:37] little bit you can see that then we have
[173:40] here analysis complete ah how cool is
[173:43] that so we have different tabs here with
[173:45] different information right so we have
[173:47] skills analysis is it tells us exactly
[173:50] what happened key skills and expertise
[173:52] level and goes over everything about
[173:55] this candidate two years of experience
[173:58] and it gives it so it's extracted
[174:00] everything from the resume and
[174:03] qualifications um and also even some
[174:06] subjections right however the fact that
[174:08] her master degree is not explicitly
[174:10] mentioned in her work experience section
[174:12] raises question about how much of her
[174:14] degree was applied in real world real
[174:16] world scenarios so this is these are
[174:18] agents uh just like you know real world
[174:21] agents real agents who would have to
[174:23] think about um whatever they are
[174:26] thinking whatever they are analyzing so
[174:28] career progression potation red flags
[174:30] look at that all of this okay and the
[174:34] confidence uh score is
[174:36] 85% and here if I go back to next tab
[174:40] look at this job matches data scientist
[174:43] 75% New York Senior software engineer
[174:47] about 60% all right we can look at
[174:50] screening so screening results 85% of uh
[174:54] screening results with confidence based
[174:56] on the provided information I will
[174:58] perform a comprehens comprehensive
[175:00] screening of Janine Nel's resume
[175:03] qualification alignment it goes through
[175:05] all of that so all these all these
[175:08] agents were able to work together to
[175:10] come to a certain uh conclusion okay and
[175:13] we can go through and all that it also
[175:16] says results are under result results
[175:18] analysis so we also create a textual
[175:20] file txt file with all these
[175:23] analysis okay and I can go
[175:26] recommendation and based on this
[175:27] analysis I would recommend that Janine
[175:30] now develop more explicit examples
[175:33] expand her Professional Network and all
[175:35] of these different things all right
[175:37] there you have it so just like that
[175:38] ladies and gentlemen if you not excited
[175:41] about this I don't know why really will
[175:42] make you excited this is something that
[175:45] you can actually use it's usable right
[175:48] you can imagine that if you have in fact
[175:50] here's what I will do if you have a
[175:52] friend of yours who is a recruiter show
[175:55] them this video and see if they could
[175:57] use something like this right don't make
[176:00] them feel like their jobs are in
[176:02] Jeopardy no but just some ask them if
[176:04] they could use such thing how much time
[176:07] they will save them um daily from
[176:10] vetting people and all that all right
[176:13] well thank you so much for going through
[176:15] this course and I really hope that this
[176:17] course was helpful and that you can see
[176:19] the power that we get when use o Lama
[176:22] because now we're able to pick and
[176:25] choose different models and we can build
[176:27] really good AI Solutions locally without
[176:30] having to spend a dime at least um you
[176:33] will have to have a computer of course
[176:35] you don't have to pay for opening ey
[176:37] keys and all of that so I hope this was
[176:40] useful I hope this was helpful to you
[176:42] the thing also I've been hinting
[176:44] throughout this entire course is that I
[176:47] do have a surprise for you which is what
[176:50] I call Ama startup pack which will have
[176:53] everything that you need to get started
[176:55] with AMA from code templates um and
[176:58] other things that are surprise so you
[177:00] should see a link in the descriptions if
[177:03] you don't see it you can you can just
[177:04] leave a comment as well I'll show you
[177:06] the link where you can go and get all of
[177:07] that again thank you so much for your
[177:10] time and I really hope this was helpful
[177:12] if you have any questions at all just
[177:14] leave them below and I'll do my best to
[177:18] um serve you to answer them thank you
[177:20] again till next time be well