---
title: 'We Built a RAG System for a Real Client'
source: 'https://youtube.com/watch?v=-TlA_89Djkg'
video_id: '-TlA_89Djkg'
date: 2026-07-31
duration_sec: 1773
---

# We Built a RAG System for a Real Client

> Source: [We Built a RAG System for a Real Client](https://youtube.com/watch?v=-TlA_89Djkg)

## Summary

This video presents a real-world case study of building a retrieval-augmented generation (RAG) system for an immigration law firm. The speakers walk through the problem, the custom hybrid retrieval architecture, the tech stack, and the business challenges, including clause mapping across years and data curation. Interviews with the lead AI engineer and a client-facing team member add practical insights.

### Key Points

- **Manual legal research is slow and error-prone** [00:17] — Immigration law firm attorneys and junior associates had to constantly track changing USCIS rules and thousands of past cases, leading to delays and inaccuracies. The goal was to reduce manual labor and improve retrieval accuracy.
- **Two goals: reduce delay and improve accuracy** [05:18] — The previous manual process suffered from two problems: delay from extensive research and low accuracy due to human error. The RAG chatbot was designed to fix both while citing legal clauses and past cases.
- **Two knowledge sources must be balanced** [08:35] — The system combines USCIS laws (parent sections with child clauses) and individual past cases. A key design requirement was ensuring retrieval returns both laws and cases, not just one type.
- **Single vector DB top-K problem** [09:32] — If laws and cases are stored in one collection, top-K retrieval can return all chunks from one source. The solution uses the same vector database but separate collections for laws and cases.
- **Custom hierarchical chunking** [12:15] — Retrieved child clauses are expanded to their entire parent section to improve context. This parent-child chunking strategy was designed specifically for the structured legal data.
- **Clause-based case retrieval** [14:11] — The system extracts clause numbers referenced in the law section and uses them to search the cases collection, retrieving the most relevant past cases.
- **Context pruning for LLM limits** [16:13] — When the combined context exceeds the LLM window, the team prunes less relevant chunks from the bottom instead of summarizing, to preserve key content.
- **Tech stack and guardrails** [18:48] — The project used GPT-4, a vector database with two collections, PostgreSQL, AWS, React, PII detection models, and Llama Guard with rule-based classification for content moderation.
- **Context engineering across years** [23:33] — Clause numbers change over time (e.g., 5.1 in 2020, 5.2 in 2022, 5.3 in 2026). The system uses a mapping to find and search the correct historical clause numbers in old cases.
- **Curate data into AI-readable formats** [28:19] — KD's advice is to transform human-readable documents into AI-readable formats like markdown, which improves retrieval results significantly.

## Transcript

of mapping, this kind of heuristic custom approaches to retrieve the relevant context. If the context is out of context window, you have to do techniques. How do we reduce the manual labor of
these junior associates so that they could like work on higher level tasks? Today, I'm going to talk about how we build an AI project which was on retrieval augmented generation for one of our clients in our company Atlassian.
I will give you a glimpse of the initial discussions that happened. We'll go over technical architecture. We'll even go over some of the code as well. This way, you will get an understanding on how AI projects are built in real life. If
you're a student, this will help [music] you with your project portfolio, so it >> [music] >> job success. And if you are an AI some projects, >> [music]
>> uh you will get uh some understanding on some of the challenges that we faced and how we overcome it. Let's go over the problem statement for this project, and I'm going to fully respect the non-disclosure agreement that we had
with our client, so we are not going to share the real client name or any the uh domain a little bit, but the fundamental AI engineering principles are still the same. Now, there are many
immigration law firms in US, okay? US has many immigration law companies, and they help their clients with various matters like someone wants to file H-1B
visa, someone wants to uh go there on family migration, you uh go there on family migration, you know, asylum. There are so many uh cases that these law firms fight. Now, Harrison and Kim Law Group is one such
law firm, and they have multiple attorneys. Let's say Bob, Julie, and Mohan are the attorneys working in this particular law firm and they have this particular client. Actually, I used to be the client because I was having H-1B
be the client because I was having H-1B visa status in US at some point and I actually consulted one of these kind of firm to file EAD, which is employment authorization for my wife who was on H-4, which is a dependent visa. So, when
I go to this firm Harrison and Kim Law Firm, they will assign a specific attorney to me. So, let's say Julie is an attorney in this firm. See Julie here? She's fighting my case, okay? She's helping me actually file EAD for
She's helping me actually file EAD for my wife who is on H-4. Now, Julie will have couple of assistants as well. So, Julie will have her own little team. Let's say Mike and Abdul are helping her and the job of this entire team is to
and the job of this entire team is to look into the USCIS policy documents, so which is basically USCIS is the entity which takes care of the immigration matter in US and they have all the rules and regulations. And by the way, these
rules and regulations keep on updating every year or every few months. So, they have to constantly keep an eye on latest rules and regulation and have a complete in-depth knowledge. They also need to look into thousands of cases which were
going on, right? They have dealt with hundreds and thousands of cases. Also, on the website, you will find so many different cases. So, this is their knowledge base, okay? So, this is their knowledge base and this entire team will
help a client like myself using this particular knowledge base. Now, the problem here is that there are millions of such cases. The rules are constantly
changing and for this team it is hard to keep up, okay? It is hard to remember. We are humans of after all. We might forget certain things. We don't have capacity to comprehend thousands and millions of
cases which which are going on or or which have already completed in the past, okay? So, this is a big problem when it comes to knowledge retrieval and we all know that using AI, using retrieval augmented generation, we can
solve this problem and we can build a chatbot which looks something like this. So, whenever you are asking any question, this chatbot has knowledge of
question, this chatbot has knowledge of the latest laws, USCIS laws or policies. Plus, it has knowledge of all the cases, individual cases, you know, which are being fought and based on that, this will generate a comprehensive answer.
Not only that, it will also create a references, see? It will create a reference to the legal case. It will also create references to the old cases which are being fought, okay? So, legal clause as well as the old cases which
are being fought. So, Julie will use this tool, okay, to retrieve the information faster and she can also refer to these cases. In the previous approach, the process was manual and it was leading to two kind of problems,
okay? The first problem is delay. Because you have to do so much research. And the second problem is it is not accurate. It might make mistake, right? Like your human team might make mistake. So, accuracy was a problem. Accuracy was
very low. Their aim of building this AI system is to improve on both the fronts, you know? They want to reduce the delay and they want to improve the accuracy. Now, folks, here is the USCIS website which has
all the rules and regulation. You can search for it. Let's say, for example, and here you will find all the detailed rules and everything, okay? Let's say you want to know something about
changing to F or M student status, they have all the rules here, okay? So, what this form will do is in order to supply the data, they will have sections. This section is for H-4
dependent spouses, then you can have section two for non-immigrant F and M student related information, etc. So, this is a parent section. Each parent section has child clauses, okay? Clause 1.1, 1.2, 1.3, and so on,
So, this is let's say your database. You're This is the the PDF file. So, this is one type of one category of your knowledge. What is the other category? Well, the cases, okay? So, you can go to the the website which contains the
repository of the last cases. So, let's say H-4 EAD, I will find all the cases, everything on this H-4 EAD, okay? So, let's say if I look into it, there will be detailed information on this particular case, okay? So, here,
Okay, there'll be detailed information. Now, of course, see the these cases are so long. Manually searching through it and comprehending the information is pretty difficult. So, we have these two knowledge sources. Before we continue,
let me take a moment to thank today's sponsor, Progress. ChatGPT is good for internal knowledge, but it doesn't know anything about your private enterprise data. You can build a RAG solution, but it has its own limitations, such as no
verification, no orchestration, [music] no enterprise controls. The Progress Agentic Rag solves all these issues. You build a knowledge box by dropping in your PDFs, decks, videos, or syncing it from SharePoint, Google Drive, or S3.
[music] enrich and extract relationship from your content. With 30 plus customized retrieval strategies, it always pulls the most relevant context. Then, when you're ready, you can directly test the
Agentic Rag search functionality and fine-tune the search configuration to match your use case. You can also track a Rag performance metrics so that you're [music] not just shipping something and hoping it works. You can see what's
If trust, accuracy, and traceability matter for your AI use case, check the link in the video description below. All right, back to our video now. >> One is loss. Second one is cases. If you have built a
Rag pipeline using some of the YouTube videos, etc., you would have been doing only one kind of collection or one kind of knowledge source. Here, there are multiple. And what happens is you need to chunk this. You need to
create embeddings for this knowledge and you store it in a vector database, right? So, this is something you will store it in your vector database after creating chunks and also embedding.
This is also something that you will store it in your vector database after chunking, etc. Now, here the normal chunking technique, normal retrieval technique is not going
to work. Let's say you you store everything in a in a let's say single single vector database, okay? And when the user query comes in and And when the user query comes in and when you try to retrieve
using this common knowledge, one problem that will happen is uh say your top K is equal to five. You are selecting first five chunks. And let's say, what if all the five chunks are from the cases? Actually, you want
to look at actual law, also. Case is an individual story, individual Case is an individual story, individual event. Law is a generic law. So, you want to look into law as well as cases, both. But, if you have a single vector
database, and if you're doing top K, it might happen that all the everything let's say, everything you retrieve is from the laws. That is not good, right? You need a balance between the two. So, what if you retrieve,
uh the chunks from laws, as well as you retrieve from cases, and then you do some kind of merge, okay? So, what we will do is this. You
will first store the laws and the cases into vector database by creating this kind of embedding. Now, let's say, somebody ask a question on H-480, okay? What is the process of
filing filing H-480. question. So, we will do the So, we will do the semantic search on this
laws first. All the chunks which are stored in this vector database, we will do semantic search. And let's say, top K is set to two, so you will get two chunks, correct? Or let's say, you top K is set to one. You
are getting just one chunk back, okay? And that one chunk can be in a child section here. Let's say, that chunk is here. Let's say, I'm asking question related to AC-21, okay? What is the process of filing H-480
with, uh let's say, I'm mentioning something related to AC21 here. So, naturally, it is going to find this chunk. Let's say this is the chunk. Now, what we do is, instead of using fixed chunk, etc.,
because fixed chunking and even some of the standard chunking techniques can create problems. And this is the story that we need to learn from the real-life projects that we need to come up with sometimes custom hybrid chunking
strategy, which is what we did for this project. So, here, what we did is to improve the quality of the retrieved answer, took this chunk, and for this chunk, whatever was the relevant clause, we
took the whole thing. Okay? And to improve the context, see, this is how to apply, but then overview of eligibility is also kind of of eligibility is also kind of important. So, what we did is, okay, we
went to a parent section. So, using hierarchical chunking, you can get to a parent section. So, for this clause, we know that this is the parent. Okay? So, in our hybrid technique,
if the retrieved chunk was this, we actually took the entire parent section as the retrieved chunk. So, see, this is a child section. Okay? So, this
child section, let's say, let me put it here side by side. So, this child chunk is nothing but this one. Okay? So, let's say, this child chunk is nothing but this
particular block that you see here is that particular child chunk. Now, what we do is we extend this child chunk to the entire parent. So,
what is the entire parent? Entire parent is section one. This whole thing, you know, whole thing. So, then we expand it. We expand it. And it, of course, contains the child section, too, but it
contains the other clauses and everything. So, this is the entire section one. So, this is our retrieved segment, okay? And then, what we do is we need to look into this another vector database where
into this another vector database where we have cases. So, for that, we will use the technique of Of course, we will search through the text, but we will also search through the sections which are mentioned here. So, see here or the
clauses. So, here, let's say they mention clause So, now, all the clauses, clause 106, 1.1, 1.2,
clause 106B, we search using those clauses into our this second vector database. So, second vector database is our cases, okay? So,
vector database is our cases, okay? So, we amend this section, we improve that section by expanding it to parent section first and then adding all the clauses. So, here,
in this particular section, we will say clause 106A, 106B, search for all these clauses.
text here into this vector database. And here into this vector database. And retrieve me the most top five relevant Okay. So, let me use a different color for this.
So, let's say the retrieved cases, you know, case one, case two, case three, or say chunk one, chunk two, chunk three. Now, this plus this, see this entire
Now, this plus this, see this entire thing plus this becomes your final context. So, this is your final context, okay? And using this final context, I mean, I'm assuming that you have some basic knowledge on rag for this, okay?
basic knowledge on rag for this, okay? So, using this final context, okay, this is the final context. This whole thing is a final context. So, this is my final context. And you will pre-pend your question.
You will say, "Okay, what is the filing process for H-4EAD with some AC-21 So, that is your question, and that you give it to LLM, give it to LLM, and LLM will give you the answer.
Now, see the context can become big. So, in that case, we will measure the context length, and we can do some pruning from the So, these two cases are let's say top most relevant cases, and third one is a
little less relevant. So, then if the the final context goes out of the context window of LLM, we will do some pruning from the bottom. So, bottom pruning means from the bottom, we can remove some of this, and we can
give them that context. You can do context summarization as well. But, in our project, we used the context pruning technique to remove this context and give it to LLM so that it generates the answer. Okay, folks. So, that was the
technical architecture. Now, uh let me go and talk to the AI engineer who worked on this, and we'll discuss some of the challenges that they faced and the technical tech stack. So, folks, we have Srikanth here who is our lead AI
engineer. So, he worked on this project. So, this is the architecture diagram which I just explained. We are going to go over technical stack here. So, Srikanth, I want to know for each of these pieces, like what kind of
technology and tools did you use? >> So, the entire LLM orchestration as a >> Okay. >> And for the vector database, we used databases. >> No, two different databases. There are
>> Okay. So, folks, it's the same vector database, but there are two different collections. One for laws and the other one for cases. Okay? And we used PostgreSQL
>> Yes. >> Okay. And this is the database that client gave you or you did any kind of web >> No, client gave the data because he also did the data sanitization. So, because
this data is available, but again, it is spread across everywhere, right? So, they compiled the data, they cleaned the data so that it is ready to use. >> Okay. So, before it came to you, they also did some technical work. So, folks,
that you don't get clean data all the time. So, our client got the data somehow, they did cleaning, that into PostgreSQL. >> Yes.
>> Any other technical architecture component, like what LLMs did you use? You want to mention anything? >> So, since this is a publicly available data, there is no such thing like data compliance. So, we used our GPT models,
GPT-4 was at the time. But on top of that, we also placed guardrails. Because since there will be some sensitive information as well, information to models. So, we
implemented guardrails as well, how we can protect the PII. And also, since it is about communicating to models, right? We also need to have content moderation. >> So, how we prevent prompt injections and everything. So, these guardrails are the
safety measures that we put in place for this application. >> And did you use LangChain for putting that guardrails or some like any any there? >> Yeah, so for PII, we use open-source
again an open-source. >> find it on Hugging Face, exactly. >> So, we're going to provide a link of that. Yes. >> And for the content moderation, we have
>> So, that model is called Llama Guard, which is by Llama. So, they have different versions of the models as well. And these are the base models. On well. And these are the base models. On top of that, we implemented our ONNX
>> Okay. >> Glyner or Llama Guard will offer reliability to some extent, right? But again, Llama Guard is a LLM >> which has latency. So, to optimize the architecture, we also have this
classification, rule-based, and on top of that, we have Llama Guard to have a >> I see. Okay. And in terms of deployment to anything about cloud, like how you deployed it, how you're monitoring it,
>> So, we used AWS because of the simplicity. And so, entire stack is on >> Okay. [music] And how about front end? >> So, front end is on React. Okay. And
>> AWS. >> And folks, now with Claude Code and AI straightforward, right? >> You can tell Claude Code, okay, create a Next.js uh front end for my personal uh
AI chatbot, and it will just do it. >> Yes. [music] Yes. It has become much simpler. >> Okay. Any challenges that you would like this project? >> So, one thing is obviously, people talk
a lot about RAG. The good side of RAG, but in real world, we have a lot of good. >> Mhm. The data itself may not be good. The [music] chunking may not be good and there will be a lot of hallucinations.
So, we we need to work on prompting as well. So, and when we are dealing with these legal issues, right? The accuracy is of utmost importance. We don't need to have hallucinations. So, every answer
should have a source cited so that we can cross-check. So, getting the chunking correct, what type of retrieval mechanisms we use, and then how did we reduce the hallucinations, these require, you know, our special
>> Okay. And you also mentioned that let's say I'm looking at H4EAD case with AC 21 clause and in the
past, let's say it found one case, okay? But this case was in 2020. And the legal document laws that we are using is from 2026. Yeah, okay. 2026. Yeah, okay. So, what happened was for the specific
clause and specific section, let's say section one, clause 5.1 in 2020 might section one, clause 5.1 in 2020 might have evolved into section one, clause >> Yes. >> I think client you mentioned that client
>> Yeah. >> And when you get folks this kind of mapping, what happens is let's say this case was from 2026, the clause is 5.3. Uh the the clause is this this case is sorry from 2020, clause is 5.1. So, for
sorry from 2020, clause is 5.1. So, for 2026 it is 5.3. And let's say uh in 20, I'm just mentioning here, 2022, this was section one, but clause was 5.2. So, in that case, um I think you did
to talk about it? >> So, here, you know, there is change in clause based on the year. We have a lot of amendments, right? But we are looking at the 2026 data. So in 2026, it is
talking about the 5.3. But when we look at the past, there won't be any cases sub clause. >> Okay. >> Okay. >> We'll see what are cases in 2020 with
>> Okay. >> And we'll go to 2022, then see what are >> We will search. >> Okay. So [music] just to kind of revise this, actually this is a step-by-step process. Here you find the actual
clause, okay? So let's say in the first step, the clause that you found is let's say this is latest, right? 5.3. So here let's say you found 5.3. But you can't just so search for 5.3 because 5.3 is valid only in 2026. So you have a
mapping, so what you'll do is you go to 2020, what is a mapping? 5.1. Using 5.1 2020 mapping, you search. Then for 2022, mapping is 5.2. So using 5.2 clause,
whatever cases are containing clause 5.2, you search for it, and that is how you uh retrieve the relevant context. This is called context engineering, >> So what these guys did is actually context engineering. So from this you
can understand that uh when you're going through any academic grad project, Srikanth, what they learn is okay, you have some nice clean PDF document, you have single vector database, single collection, you retrieve five chunks,
>> Yeah. >> But you saw that in real life, we have to use this kind of mapping, this kind of heuristic custom approaches to retrieve the relevant context. If the context is out of context window, you
those techniques. And also, we literally used a custom I >> Custom retrieval, yeah. >> And you mentioned that you're working on
about it? >> Yes, because it is not the general chunking because in any rag project, chunking is the very first and the most important thing. The way we parse the document and we create chunks as the
So, whatever the chunking mechanisms that we have like overlapping or chunking, that didn't work. If you look at the data, it is very >> Mhm. >> And if you want to prevent all the
hierarchy in laws, we need we need to follow that structure. So, we designed >> Mhm. >> And we are also writing white paper on that so that we can, you know, explain how we approach that how did how exactly
>> Exactly. >> So, folks, that parent-child chunking is something that I explained in the initial section. Srikanth is going to write white paper. We'll provide whenever that paper is out, we are going
to provide a link to it so that we are going to open source it basically so anybody can use it. So far, I have given you the technical details, but the business side of any project is equally important. Karan is a person who was
involved in dealing with clients and exploring business requirements and so on. So, let's go and talk to him as well.
So, KD, you were involved in the interaction with client directly. I just want to know what kind of benefit client is getting or is going to get out >> It is really simple. So, each law project that is going on right like
you have a junior associate, you have a senior associate, and you have somebody generally a partner, a junior or a senior. Now, [music] in many cases, junior associates are just piling through the information that
different public documents, in different literature. What the client was seeking out to achieve from the project was, how do we reduce the manual labor of these like work on higher level tasks. >> Okay. Good. So, that this is going to
improve the efficiency of the overall process. [music] And do you want to mention one challenge that you faced with this project and which probably many other projects are also facing similar challenge?
>> So, one of the like very common and very very grounded challenges, >> when somebody non-technical is coming to us, right? They won't understand how uploading a document to a chat GPT chat window versus rag is going to be
different. So, what used to happen was client was comparing the questions that they asked the system that we built with what they asked chat GPT and the difference was very high. And the reason was simple, like if you're uploading a
document into a chat GPT window, it's in the context window, you're going to get >> If it is a rag project, if it is different, what's going to happen is it's going to look through thousands of documents and then get a answer. So,
this was causing a difference that the client was experiencing all the time. >> Okay. And if there is one advice that you would give to anybody who is building a rag project, okay? Many of the viewers who are watching this video
will be the learners or students who are learning AI and there will be some people who are the business folks who wants to build a rag project in their company. So, is there any advice that you would give to them?
>> Uh one thing that I would always tell people is curate your data if you can. another UK-based project as well and we're also doing it for the US-based law firm project. So, what clients were doing in this case is they were curating
literature. Why? Because one thing is you grab the data as is and you feed it to the AI, to the LLMs, to the embedding models and it would work. But let's see if you can like convert it in a way which is more
going to take some effort. But, if you can make the data from being just human-readable to AI-readable, you're going to see some differences. One of markdown. Humans are not. Humans love to read if
reader, but AI by default can read by a markdown variable. So, smaller changes like these are also like helpful these cases. Okay. Thank you. All right, that's it, folks. I hope you learned how to build
production grade rag project from this video. We are going to provide resources video description, so please check it out. There's a comment box below. If you have any questions, feel free. Thank you very much for watching.