[00:00]
In almost all Gen AI engineer job
[00:02]
postings, you will find one common
[00:04]
skill, retrieval augmented generation,
[00:06]
also known as rag. In my company at
[00:09]
Lake, when we build AI projects for our
[00:11]
clients, more than 40% of these projects
[00:14]
have rag component in it. So, what
[00:16]
exactly is rag? What are different
[00:18]
types? Is naive rag dead due to
[00:20]
vectorless rag? We are going to cover
[00:22]
all these rag basics in a very simple
[00:25]
and intuitive language. We will not just
[00:27]
talk theory. I will show you a rag
[00:30]
project which is a customer care chatbot
[00:32]
in telecom domain. In the end, I will
[00:35]
share some useful resources including
[00:38]
rag interview questions. All right,
[00:40]
let's get started. Let's understand rag
[00:42]
using a simple example. When you ask
[00:44]
ChatGPT a policy question for some
[00:47]
private company, it won't be able to
[00:49]
answer it because ChatGPT is trained on
[00:53]
general internet knowledge. It doesn't
[00:55]
know the HR policy details of any
[00:58]
private company. But, if you give HR
[01:02]
policy document to this LLM, which acts
[01:04]
like a brain, it should be able to read
[01:07]
a relevant section and provide you with
[01:09]
the answer. This is similar to having a
[01:13]
very smart student Mira, who is a
[01:15]
computer science student, and you are
[01:17]
asking her to appear in a microbiology
[01:20]
exam, which is an open book exam. Now,
[01:22]
Mira is generally good in terms of
[01:24]
reading writing comprehension
[01:26]
understanding, etc., but she doesn't
[01:28]
know anything about microbiology. But,
[01:30]
in this exam, she has been given a book
[01:33]
on microbiology from which they are
[01:35]
going to ask the questions. Now, Mira
[01:37]
can use her reading, writing,
[01:40]
comprehension skills to uh look at the
[01:42]
book and she can write answers in the
[01:45]
exam. So, here Mira's brain is like LLM,
[01:48]
which has a good understanding of
[01:49]
language, it has reasoning capabilities,
[01:52]
and the book is like an HR policy
[01:54]
document. It is an external knowledge
[01:56]
where LLM can look into it and pull the
[01:59]
answers. Let's now understand the
[02:01]
two-step process of how rag works
[02:04]
underneath. So, here I have HR policy
[02:06]
document from Atliq, which will have a
[02:08]
section on retirement benefits, okay?
[02:12]
So, now if I go to ChatGPT, and if I
[02:14]
copy this particular section, okay, and
[02:17]
I ask my question related to
[02:20]
contribution to employees retirement
[02:22]
fund, in that case, ChatGPT will be able
[02:25]
to answer that question. Because here
[02:27]
you are asking question and providing uh
[02:31]
knowledge as a reference in the context,
[02:33]
and it can pull the answer. But, what if
[02:36]
your HR policy document is 3,000 page
[02:38]
PDF, okay? What if that knowledge is
[02:41]
very big? What's going to happen in that
[02:44]
case is you will run out of your context
[02:46]
window limit. And even if you have a
[02:48]
huge context window,
[02:50]
uh you should still not feed the entire
[02:53]
knowledge because it will be too many
[02:56]
tokens, it will be costly. So, what
[02:58]
people do is they will chunk this
[03:01]
document. So, they will create, let's
[03:02]
say, basic strategy is fixed-size
[03:06]
chunks. And then, for a given question,
[03:09]
you can pull the relevant chunks. So,
[03:11]
for this particular question, let's say
[03:13]
my first chunk is 70% uh probability
[03:17]
that it will it will contain the answer.
[03:19]
Second chunk is 60% match. And you can
[03:21]
have, by the way, I'm showing just
[03:22]
three, but you can have 1,000 chunks,
[03:24]
and some of the chunks might have 5% or
[03:27]
even 10% possibility that it may contain
[03:30]
the answer. Let's say the chunk contains
[03:33]
uh information on uh when Atliq was
[03:36]
founded culture founders etc. then
[03:39]
that doesn't has anything to do with the
[03:41]
retirement question that you are asking,
[03:43]
okay? So, the the relevance of that
[03:46]
chunk will be very, very low. Now, how
[03:48]
do you exactly find this kind of
[03:50]
similarity? So, there is this concept of
[03:53]
embeddings, okay? So, embedding is a
[03:55]
process of converting text into a vector
[03:59]
such that it can represent its meaning,
[04:02]
okay? So, all the chunks, you will
[04:04]
convert them into vector embeddings, and
[04:08]
then you will store them into a vector
[04:10]
database. This is different than your
[04:12]
regular database. Your regular database
[04:15]
can search using exact values, whereas
[04:17]
vector database will be able to search
[04:20]
using the meaning. So, when you search
[04:22]
for, let's say, uh a company that is a
[04:25]
leader in electric vehicle, it will
[04:29]
return Tesla uh from the database. So,
[04:31]
it is searching based on the meaning,
[04:33]
not based on the exact word. To generate
[04:36]
embedding, you can use variety of
[04:38]
models, sentence transformer, text
[04:40]
embedding three small, and so on. And
[04:42]
there are many vector database choices
[04:44]
that you have in market, Milvus,
[04:46]
Quadrant, Chroma DB, and so on. This
[04:48]
step is called indexing. This is the
[04:51]
first step in rag process, where you are
[04:54]
indexing all these vectors of chunks
[04:57]
into a vector database. The second step
[04:59]
is retrieval, where for a given
[05:01]
question, you will generate embedding
[05:03]
using the same embedding model. Then,
[05:05]
you will try to find the relevant chunks
[05:08]
in a vector database. So, here it is
[05:10]
doing the semantic search, giving you a
[05:13]
relevant vectors. You can specify top K
[05:16]
factor, let's say I need two chunks or
[05:18]
five chunks, and so on. And then, you
[05:20]
will generate the
[05:22]
actual text out of those chunks, and you
[05:25]
will put it in your prompt along with
[05:28]
the question. And when the question is
[05:30]
given to LLM, it will give you the
[05:32]
answer. So, here uh below the question,
[05:35]
what you are doing is you are providing
[05:37]
only the relevant chunks. So, this way
[05:39]
LLM will not hallucinate, and it will
[05:42]
give you accurate answer. That takes us
[05:45]
into our next segment, which is the two
[05:47]
major benefits of rag. The first one is
[05:50]
the answers that you get will be highly
[05:53]
accurate, and the chances of
[05:56]
hallucination will reduce because you
[05:59]
are grounding your responses in the
[06:01]
knowledge, in the source of truth.
[06:03]
Second, it is very cost-effective
[06:05]
because if you pass the entire context,
[06:07]
then you are sending too many tokens to
[06:10]
LLM, and these LLM APIs, they charge you
[06:13]
per token. So, if you send less number
[06:15]
of tokens, only the relevant knowledge,
[06:17]
then you will save money on your API
[06:19]
bill. Here is a hands-on customer care
[06:22]
assistant rag project. I have given the
[06:24]
code in the video description below. You
[06:26]
can ask different questions, for
[06:27]
example, why is my mobile internet slow?
[06:30]
And it will find the answer based on the
[06:33]
knowledge that it has. So, the knowledge
[06:35]
is stored in terms of the
[06:37]
troubleshooting PDF file. So, here is
[06:39]
the PDF file, and let's say you have
[06:42]
this question on how do you want to
[06:44]
enable the LTE, then it is pulling that
[06:47]
answer from this particular PDF file.
[06:50]
The other source is the CSV file
[06:53]
containing all the FAQs. And the third
[06:56]
source is a SQLite database containing
[06:59]
all the past ticket. Here we are using
[07:01]
Chroma DB as vector database. So, we are
[07:03]
ingesting FAQs, then PDF, and tickets
[07:07]
into Chroma DB, okay? So, these are the
[07:10]
three files which is ingesting into a
[07:12]
vector database. If you look at ingest
[07:14]
PDF, here we are using the chunk size of
[07:17]
600, overlap of 100. We are using the
[07:19]
recursive character text splitter
[07:22]
strategy. And for embedding, we are
[07:24]
using this particular embedding model
[07:26]
from Hugging Face. As a framework, we
[07:28]
have used LangChain. Now, the retriever
[07:31]
will try to find relevant chunks from
[07:34]
FAQ, tickets, or guides. In terms of
[07:37]
LLM, we are using Quen from Chat Grok.
[07:40]
Please download the project on your
[07:41]
computer, try to run it to enhance your
[07:44]
understanding on rag concepts. Telecom
[07:47]
support chatbot that we just saw is an
[07:49]
example of enterprise QA chatbot. There
[07:52]
are many other industry use cases for
[07:55]
rag. For example, you can build medical
[07:57]
knowledge assistant, which can look at
[08:00]
the vast amount of medical knowledge and
[08:02]
pulls a relevant answer for your query.
[08:05]
The other one is legal and compliance
[08:07]
tools. Once again, here the knowledge
[08:09]
will be your legal documents, and you
[08:12]
want to pull the most relevant and
[08:14]
accurate answer. HR chatbot is another
[08:16]
example. Let's now look at rag
[08:18]
categories. The first one is a vector
[08:20]
rag, and naive rag is the example that
[08:23]
we just saw, where you pull the top K
[08:27]
relevant chunks from a vector database
[08:29]
and answer user's question. The second
[08:32]
category is vectorless rag, in which you
[08:35]
can perform a keyword rag. So, here you
[08:38]
are not generating any vector
[08:40]
embeddings. You don't have a vector
[08:42]
database, but you are using keyword
[08:45]
match, uh techniques like BM25, TF-IDF,
[08:49]
etc., to uh query into the document
[08:52]
using the exact keywords. This method
[08:55]
will work when you have a lot of codes,
[08:58]
jargons, IDs, citations. Let's say you
[09:01]
are doing research, and you are always
[09:03]
searching using some particular ID or a
[09:07]
particular keyword, then this will work.
[09:10]
This is weak for semantic understanding.
[09:12]
When you are not doing exact keyword
[09:14]
matching and searching using meaning,
[09:16]
this is not a good choice. And the key
[09:19]
tools that uses keyword rag concepts
[09:21]
like BM25 is Elasticsearch and Apache
[09:25]
Solr. The next category in vector rag is
[09:28]
hybrid rag, where you're combining
[09:31]
vector search and keyword search, okay?
[09:34]
You do both of these in parallel and
[09:37]
merge the results. This is best for most
[09:39]
of the production systems. The key tools
[09:42]
here are Elasticsearch plus any vector
[09:45]
DB. Now, in Atliq, we worked on one rag
[09:49]
project for our client, where we have
[09:51]
developed our own custom hybrid method
[09:55]
for doing rag, and we have given details
[09:57]
of this approach in a different video.
[10:00]
You can check it out if you are
[10:01]
interested. Also, if you want to learn
[10:03]
AI engineering by building production
[10:05]
grade AI systems similar to the projects
[10:07]
that I just mentioned, then check our AI
[10:10]
engineering cohort where we have live
[10:13]
sessions on weekends and we will teach
[10:16]
you all the concepts plus you will build
[10:18]
eight plus production grade projects.
[10:20]
The next category in vector less rag is
[10:22]
a graph rag. It is also known as KG rag.
[10:27]
So here you will generate a knowledge
[10:29]
graph. So let's say your knowledge is
[10:31]
Elon Musk and all the companies he has
[10:33]
founded. So in that case you will build
[10:35]
this kind of knowledge graph where you
[10:37]
will say Elon Musk founded Tesla,
[10:39]
SpaceX, Neuralink, OpenAI and so on.
[10:42]
And then these companies will be
[10:43]
operating in these different domains. So
[10:45]
these are all the entities and they are
[10:47]
connected through some kind of
[10:49]
relationship. Now when you ask a
[10:51]
question, which companies are founded by
[10:53]
Elon Musk which are working in AI, you
[10:56]
will traverse this particular path,
[10:58]
okay? So you will look at all the
[11:00]
companies and then you will do breadth
[11:02]
first traversal and you will find that
[11:05]
OpenAI is working in AI. The next one is
[11:08]
SQL rag. This is also known as text to
[11:11]
SQL. This method is very simple. Let's
[11:13]
say you have sales database which
[11:15]
contains the sales of
[11:18]
products. Now you are asking this
[11:19]
question, which product sold the most
[11:21]
last month? Using LLM you can first
[11:24]
generate a query for that database. You
[11:26]
will execute the query, get the results
[11:29]
and then give it back to LLM to generate
[11:31]
a comprehensive answer. Very simple
[11:33]
technique. You are taking a sentence in
[11:36]
a natural language, converting it to SQL
[11:38]
using LLM and putting a query in your
[11:41]
database to get the results. And the
[11:43]
last method, which is relatively new, is
[11:47]
called page index. It is reasoning based
[11:50]
rag. So here let's say you have 3,000
[11:53]
page PDF document. First you will
[11:56]
generate the table of content, okay? The
[11:59]
table of content or your information
[12:02]
structure. This is like you are having a
[12:04]
book and you are having all the chapter
[12:06]
and topic layout. Now when somebody asks
[12:08]
this question, what does the contract
[12:09]
say about compensation for breach of
[12:11]
contract, the LLM will use its reasoning
[12:15]
capability and this particular table of
[12:18]
content to traverse this particular
[12:22]
graph and locate the thing that it is
[12:25]
looking for. So for example, in this
[12:28]
case it will first find out that this is
[12:30]
related to performance of contracts
[12:32]
because the contract is already
[12:33]
executed. So it has to be related to
[12:35]
this and then it finds compensation of
[12:38]
breach. So it goes from here to here and
[12:40]
then
[12:41]
you are discussing loss. So due to that
[12:44]
it will out of all these nodes, it will
[12:46]
go to this particular node and it will
[12:49]
pull the relevant document. Now this
[12:53]
might give you an index and using index
[12:55]
you might have to refer back to the
[12:57]
original knowledge. So here is the
[12:59]
GitHub for page index. It is known as
[13:02]
vector less rag but it is one of the
[13:04]
categories of vector less rag, okay? The
[13:07]
right term is reasoning based rag. So
[13:10]
here you can see from document you are
[13:11]
generating a tree, which is your
[13:13]
knowledge tree structure index of
[13:16]
documents and then LLM will do its
[13:18]
reasoning to find the relevant chunk.
[13:21]
Here you are not using any vectors. You
[13:22]
are not using any
[13:24]
embeddings. No vector DB. Just by
[13:26]
looking at the the structure, you know,
[13:30]
the table layout, which looks something
[13:32]
like this,
[13:33]
you will try to find a given node, okay?
[13:36]
And see here there is a summary. So
[13:38]
using the summary, LLM can reason and it
[13:41]
can say, "Okay, maybe the answer is in
[13:44]
this particular node." And then it will
[13:46]
go to that node, refer to the original
[13:48]
document and pull the answer. I have
[13:50]
attached this PDF in the video
[13:51]
description below where you have
[13:53]
categories of rag. You also have a table
[13:56]
comparing when to use what. It is not
[13:59]
that
[14:00]
reasoning rag is here so you should not
[14:02]
use naive rag. You should use it when
[14:05]
you have general text Q&A bots, etc. And
[14:08]
the complexity here is low. The
[14:10]
complexity in case of page index is
[14:12]
high. You should use it when you have
[14:14]
hierarchical tree index LLM traversal.
[14:18]
You know, these are the use cases. So
[14:19]
you can use this table to determine when
[14:21]
to use what kind of rag. And at the end
[14:24]
we have rag interview questions. All
[14:26]
right, folks. So please check it out. If
[14:28]
you have any question, post in the
[14:29]
comment box below.
[14:33]
>> [music]