AI Agent Memory: A Full Breakdown & Transcript

[00:07] mistake even after it handled a similar task before? It may have forgotten a project rule, missed the right instruction, or failed to carry a useful lesson into its next session. The reason could be how its memory is designed. In

[00:21] this video, we will explore the four types of agent memory through practical examples. We will begin with the human memory and the Quala framework. Then, we will look at working, semantic, procedural, and episodic memory. We will

[00:34] also see how they work together, whether every agent needs all four, and also we're going to look practical examples on all of them. Now, here's the agenda of our today's session. First, we are going to start with human memory and the

[00:48] Quala framework. Then, we will look into working memory and its limits. Moving semantic memory, knowledge sources, and drag. After then, we will discuss procedural memory, agent skills, and workflows. Moving ahead, we will discuss

[01:03] about episodic memory. Then, we will look into how the four types of memory work together. And finally, we're going to conclude this session with a clear understanding on whether every agent needs all four types of memory or not.

[01:16] info, guys. If you'd like to go beyond the concepts and learn to build AI application, you can explore Simply Learn's Advanced Executive Program in Applied Generative AI in collaboration with IITM Pravartak and Microsoft. This

[01:29] is a 5-month which covers drag, agentic workflows, MCP, multi-agent systems, and LLM Ops with hands-on practice using 29 plus tools and framework. So guys, hurry up now and join the course. The course link is mentioned in the description

[01:43] here is a short quiz to test your knowledge. A company's refund policy is an example of which type of memory? And the options are semantic memory, working memory, episodic memory, and procedural memory. Please mention your answers in

[01:56] the comment section below. So, let's get started. Imagine you hire a new assistant. They are smart, fast, and polite. But, every time you leave the room, they forget everything. Your name, your project, the task you gave them 5

[02:11] And that is why memory is one of the most important parts of building a useful AI agent. But, here's the thing, memory is not one single thing. AI agents remember in different ways, and each way does a different job. So,

[02:26] first, let us try to understand how our human memory works. We all know we have something very common with AI agents, that is memory. The first kind of memory is short-term memory. It is what you are holding in

[02:41] your head right now. Like an OTP you read on your phone and keep in mind just long enough to type it. 10 seconds later, it's gone. And that's also fine. Second one is a factual knowledge. Things you simply know.

[02:56] Python is an interpreted language. Your company gives you 20 days of leave. You don't need to work these out again. You just know them. Third we have is learned skills. For example, like riding a bicycle, typing

[03:10] without looking at a keyboard. You don't think about the steps anymore. You just do them. The fourth one is personal experience. Like the time I pushed a code change on Friday evening and spent my whole

[03:22] weekend fixing it. I don't remember a rule about that, but I remember the weekend. And trust me, I don't deploy on Fridays anymore. Something like this what you can take into consideration. Now, I hope so you have got brief idea

[03:37] regarding human memory. Now, let us explore one of the important part, that is Koala framework. AI agents need something similar. And there is a helpful way to think about it, and that is called as Koala, which

[03:51] stands for cognitive architectures for language agents. It comes from research team at Princeton. Koala describes an agent using three building blocks, that is memory, which is what the agent holds

[04:04] and knows. Next one is actions, which are the things it can do like searching, calling a tool, or writing a reply. And next one is decision making, which how it picks up the next step. Now, inside that memory block, Koala names four

[04:19] types. For example, the working memory, semantic memory, procedural memory, and episodic memory. Now, these names are borrowed from how human thinks about but they are engineering categories. An AI agent does not remember the way your

[04:33] brain does. These are just useful boxes for deciding what to store, where to store it, and when to use it. Here's a simple way to picture all four of them. Think about a new employee in a

[04:46] customer support team. They need what's on their screen right now. That is called as a working memory. Then, they need the company's handbook. That is called as semantic memory. They also need training on the steps to follow.

[05:00] That is the procedural memory. And over time, they learn from the past cases. time, they learn from the past cases. That is episodic memory. So, now let us explore each of these memory one by one. First one we have is working memory.

[05:15] This answers one simple question. What the agent is using right now? Working memory is everything the agent has in it front of it, and also for the current task. That includes the conversation so far, the instructions it was given, and

[05:31] any files it has opened. The results from the tools it has used, and its own notes about where it is in the task. You can think of it like the agent's desk, where everything is actively working on what sits at the desk. Now, let us see

[05:48] what sits at the desk. Now, let us see an action. Say you ask a coding agent to fix a bug. You describe the error, that goes onto the desk. The agent opens a few files to find the problem. Those go onto the desk, too. It

[06:03] runs a test. The terminal shows your result, and that's lands on the desk, as Now, the agent can look all of it together. The error you described, the code it reads, and the test that failed. And that's how it connects the dots and

[06:18] Without working memory, it would forget the error message by the time it opened the file. Now, let us try to understand what is the difference between working memory and RAM. A good comparison is a RAM in your computer. RAM is very fast,

[06:33] but it has limited space. And when you shut down your computer, everything is cleared. Working memory behaves the same way. It is quick to use. It has a size limit, and once the session ends, it's usually

[06:46] somewhere else. For language models, the main part of working memory is a context window. That is a total amount of text the model can see at once. It is measured in tokens. A token is roughly a piece of word.

[07:02] Context windows today are very, very large. You can fit the whole books into some of them. But, they still have a limit. A long chat, a few big files, and a lot of tool output, which can fill them faster than

[07:16] Now, let's also discuss what are the limits of working memory. And here's something that surprises a lot of people. Even when everything lot of people. Even when everything fits, more is not always better.

[07:28] Picture a desk covered in a hundred pages, and the one line you need is on the page 62. It's there, but it's hard to spot. The same thing happens with the When you give them too many files, rules, and old messages, the important

[07:44] detail gets buried. Research has also found that the models often pay more attention to the start and the end of a long input than the middle. So, good agents keep their desks tidy. They bring in what the task needs and they leave

[08:00] out what it doesn't. There is one more limit. Working memory is great for the task right now, but it doesn't carry anything tomorrow. Close the session and the desk is cleared. And remember, a regular chatbot

[08:14] has a context window, too. So, working memory alone isn't that what makes a agent special. What makes the difference is what it keeps and what it can use beyond the desk. Now, with that being said, let us discuss the second

[08:29] type of memory, that is the semantic memory. Semantic memory answers the question, "What should the agent know in general?" Semantic memory holds facts, rules, and general knowledge. Things that stay true, no matter what the

[08:44] current chat is about. It's a company handbook for our new support employee. For a coding agent, this could be how the project is built, which version of the framework the team uses, and how the files and functions should be named.

[08:58] Which commands builds the app and which files it should never touch. For a customer support agent, it could be the product details, the refund policy, what delivered and what delayed actually mean. And also, what a customer

[09:12] must do to qualify for a service. Now, you'd be wondering why this matters. Because without it, the agent has two choices. Guess or ask you again. If it guesses, it might use the wrong framework version or promise a refund

[09:27] policy that doesn't allow. If it asks you, you end up explaining the same Now, you'd be wondering where this semantic knowledge lives. So, it can be

[09:39] in the plain documents like the policy files or the project notes. It can be in a database where the data sits in the neat tables, or it can be in the vector index which lets the agent search by meaning, not exact words. So, for

[09:53] example, you search for a late package, and it can still find a section titled by delayed delivery. Or it can be in a knowledge graph which stores how the things connect. For example, this product comes with

[10:07] warranty which applies in this region. Now, which one is right? It depends on three things. For example, how much information there is, how it's organized, and how often it changes. And that's how the agent needs to search

[10:21] through it. With that being said, let us discuss a very important concept that is RAG. Now, one common way to use this knowledge is called RAG. So, RAG stands for retrieval augmented generation. So, let's go back to our support team

[10:36] example, and let's say a customer asks, "My order is late. Do I get a refund?" So, the company's handbook policy might be a 100 pages long. Putting the whole thing into every chat would be slow, costly, and messy.

[10:51] So, instead, the system searches the handbook, it finds the section on delayed deliveries, and then it hands only the section to the agent. Notice what happened here. The full handbook stays in the semantic memory.

[11:05] The one section that was found gets placed on the desk in working memory or just for this question. One thing to keep in mind is that RAG is only as good as the search. If the system pulls the wrong section, the

[11:19] system pulls the wrong section, the agent will give a confident answer like, "What I still wrong?" Coding tools often use a simpler version of the same idea. For example, a project instruction file. And one example is a file called as

[11:33] cloud.md file, which is present if you are using cloud as an AI agent. Now, the coding assistant reads it at the start, so you don't have to explain your project again every time. But a file like this only helps it if it stays up

[11:49] to date. An old instruction file means old knowledge, and it's just one way to way. So, semantic memory gives the agent the facts, the rules, but knowing the rules is not the same as knowing what to do

[12:04] with them. So, for example, knowing that a refund is allowed doesn't tell you the steps to process one. With that being said, let us discuss our procedural memory.

[12:17] instructions, workflows, reusable skills, rules for using tools, and sometimes code. If semantic memory is knowing the ingredients, procedural memory is the recipe. Now, let us take a case study on

[12:32] handling a lead order and how procedural memory comes into the picture. So, what the step one would be? So, the agent is first going to confirm who the customer is and find the right order. Step two, it is going to check the latest tracking

[12:46] status. Step three, it is going to compare the delivery date with the At step four, it is going to explain the options to the customer or if the case is outside what the agent is allowed to handle or pass it to the human.

[13:01] The order of these steps matter. If the agent skips step one, it might share the order details with the wrong person. And step four is a safety net. The agent knows when to stop and ask for help. Now, let us discuss bit about agent

[13:17] So, how do you give an agent these steps? One way is called as agent skills. A skill is a folder. Inside it, there is a file called skill.md and that explains what the skill does, when to use it and the steps to follow.

[13:33] The folder can also hold extra things. For example, the skill needs like scripts, templates, or reference files. The nice part is that you write the procedure once, the agent can reuse it every time. And if the process changes,

[13:47] you update the file, you don't need to retrain the model again and again. Now, let us discuss one very important question. What if an agent has 50 skills? So, loading all of them at the start would fill the disk before even

[14:01] And that's where the pattern called progressive disclosure helps us. Think of it like a book table's of content. At the start, the agent only sees a short list of skills. For example, just the names and what

[14:16] When the task matches a skill, it opens that skill's full instruction. And if those instructions point to a script or a template, it opens that only and when

[14:28] it actually needs it. Now, this keeps the working memory tidy while still steps. Procedural memory also covers how to use So, guys, the procedural memory answers the question, how should the agent do

[14:42] this? It works hand in hand with the semantic memory. One gives the rule, the other gives the step. Now, let us discuss the final type of memory, that is the episodic memory. It answers the question, for example,

[14:55] remembering. Episodic memory is a record of past events. For example, like what was tried, what was decided, and how it turned out. If semantic memory is the handbook, then episodic memory is the

[15:08] agent's diary. For example, a coding agent once spent time on a login bug. It found the problem was in the authentication middleware, not the user database. Now, a week later, a similar login bug

[15:21] shows up. And if the agent has a short note about the last time, it knows where to look for the first time. And this can actually save a lot of time. So, this is how an agent uses the episodic memory.

[15:34] With that experience, the agent can check the tracking history before, you know, looking for other solutions. Now, you'll be wondering, how do you build episodic memory? And the simple way is to save every conversation and

[15:48] search through them later. But that quickly turns into huge pile of text. Most of it will never matter again, and finding the useful part is So, a better way is to keep a short summary. Keep four things: what the task

[16:02] was, what the agent tried, and what happened. And also, what to remember for the next time. So, this actually a technique helps the agent a lot. And this is one of the very effective techniques when you are

[16:17] actually working for this problem. So, guys, you can uh take this as an example and understand all the uh memory that we have discussed So, let's say if you ask, "My order was due on, let's say, yesterday. Can I get

[16:31] a refund?" So, the working memory is basically the conversation, order number, and the tool results. Semantic would be shipping and refund policies. Procedural will be verifying and check the tracking status. Or it is going to

[16:44] answer or escalate. Episodic memory is going to be the lesson from the past delayed order case. Now, the next thing that is also very lives. So, working memory will be living in the

[16:56] context window, semantic the documents or the search index, procedural will be episodic will be in the database or structured notes. Now, let us discuss a very important question. Does every agent needs all the

[17:11] The answer to this is no. The memory should match the job. A simple reflex agent like the smart thermostat or a bot that routes messages to the right team

[17:23] only needs what's in front of it right now and few simple rules. It doesn't need a diary. For example, a password reset agent needs working memory for the current request. It needs procedural memory to

[17:35] carry out the reset safely. And it may need semantic memory for the company's rule on verifying who someone is. A coding agent can actually use all four of them. So, the current task on its desk, the project rules like present in

[17:51] the procedural memory, and the notes from the past session is in the episodic memory. For adding a memory, you don't need So, when I talk about with respect to the memory, it is more to maintain more than it can go out of date also,

[18:06] and it could be more personal to the data to protect. So, you can start with data to protect. So, you can start with what the job needs and add on to it as what the job needs and add on to it as what is the clear directive of the task.

[18:19] So guys, that was all for today's video on the type of agent memories. I hope so session, and if you have any doubt, please do not forget to mention your thoughts on the comment section below, and our team of experts

[18:33] will get back to you. Till then, keep learning and keep growing.