[00:07] customer support assistant. A customer asks, "Where is my order?" Your AI assistant understands the question, checks the order database, retrieves the delivery status, and responds to the customer. Sounds simple, right? But what [00:23] happens when the assistant gives the wrong answer? Maybe it retrieves an outdated delivery status, maybe the database query fails, or maybe the AI simply generates an incorrect response. Now, here's the problem. Your [00:38] application dashboard might show that everything is working perfectly. The request was successful, the server responded on time, and there were no technical errors. But the customer still received the wrong information. So, how [00:54] [clears throat] went wrong inside your AI application? And more importantly, how do you measure whether your AI system is improving as you make the changes? And this is where MLflow comes [01:07] in. So, before we dive in, let's quickly look at what we are going to cover in our today's video. So, we'll be starting by understanding what MLflow is and why it is important for managing machine learning and AI applications. Next, we [01:21] will explore experiment tracking in MLflow, including how developers compare different models and track their performances. Then we will understand how MLflow works with AI agents. After that, we will look at MLflow [01:35] We will also explore AI evolution techniques. Next, we will cover prompt management and version control, followed by monitoring AI applications in the production. And finally, we will look at real-world MLflow use cases across [01:51] machine learning, AI chatbots, AI agents, and rag applications. So, by the end of this video, you will have a clear understanding of how MLflow helps developers track, evaluate, debug, and improve AI systems. So, now let's start [02:06] improve AI systems. So, now let's start by understanding what is MLflow. So, MLflow is an open-source platform that helps developers and data scientists manage, track, evaluate, and monitor machine learning applications. [02:21] So, you can think MLflow as a control center for your application. When you're building an AI model, you don't simply train it once and deploy, right? You experiment with different algorithms, data set, parameters, and [02:38] version systems. And when you are building an application using large language models or LLMs, you also experiment with different prompts, experiment with different prompts, different models, tools, and workflows. [02:51] So, MLflow helps you keep track of these experiments and understand how your AI system behaves. It also helps you answer important questions like which model [03:03] performed better? Why did my AI agent produce wrong answer? How much time did a particular operation take? And did my latest change actually improve the application? So, instead of relying on guesswork, [03:18] MLflow gives developers recorded information that they can analyze. So, now let's look at the main diagram of MLflow that represents MLflow as an AI management platform. So, this diagram shows how MLflow acts [03:33] as a central management platforms for both machine learning models and AI applications. The main idea is that building an AI system involves more than just training a model. You can need to track experiments, manage versions, [03:48] investigate errors, evaluate results, and monitor performance. And MLflow provide tools to help with all these activities. So now let's understand all of these points in nutshell and then we will look at all these pointers in [04:02] detail in our further tutorial. So first of all, experiment tracking. So it records the different settings you test and their results. Then second is model management. It store, version, and deploy models in a centralized place. [04:17] Then comes AI tracing. In this, it will trace each step of your AI application to understand how it works. Now comes evaluation, where it will measure and compare the performance of models and AI application. Next, we have [04:33] prompt management. So prompt management, what does this prompt management actually does? So the it version, organize, and compare prompts across experiments and application. And last but not the least, we have monitoring. [04:48] It will track real-time performance and detect issue in the production if we have any. So these are the six important So these are the six important capabilities of MLflow. [05:04] right? Let's understand this with a practical example. Let's go. So imagine you're building an AI customer support chatbot. You test three prompts and two models. Experiment tracking records the [05:18] different configuration and results. So prompt management keeps the track of which prompt version you used. One day the chatbot incorrectly approved the refund. AI tracing helps you inspect what [05:32] happened, what information was retrieved, which tools were called, and what the model returned. And then evaluation helps you check whether your revised chatbot gives better answers. Then monitoring helps you observe its [05:45] performance after deployment. So, the key takeaway is very simple. MLflow brings the important part of AI development into one platform. It helps you track what you experimented with, understand what went wrong, compare [06:01] results, and improve your AI application over time. So, this was MLflow in action over time. So, this was MLflow in action with a simple AI chatbot example. Now, I hope it is a little clear to you. So, now let's understand all the [06:16] capabilities we have just gone through in little detail. tracking. So, suppose you're developing a machine learning model that predicts house prices. You can see here. Machine [06:31] learning model that predict house prices. So, first you train a linear regression model, its average prediction error is around $50,000. Then, you try a random forest model, and the error drops to $30,000. And finally, [06:47] you experiment with gradient boosting, where your error decreases to $20,000. where your error decreases to $20,000. So, which model performed the best? So, based on these results, gradient boosting looks promising, right? Here [07:02] boosting looks promising, right? Here you can see, with the least error. But, here is another challenge. Each model might have been trained using different parameters, data sets, or feature combinations. Without recording these [07:16] details, reproducing your best result can become difficult. And that's where can become difficult. And that's where MLflow experiment tracking helps. So, MLflow allows developers to record training runs, including parameters, [07:30] evaluation metrics, and model artifacts. You can then compare experiments through the MLflow interface. Here we can see in linear regression, our MAE was $50,000. In random forest, it was $30,000. [07:46] And in gradient boosting, it was $20,000. So, these are the illustrative mean absolute error values. And lower is always better. So, instead of maintaining separate [07:59] notes or spreadsheets, developers can compare results in one place. And once they have selected a suitable model, MLflow's model registry can help versions. And that's how MLflow supports [08:15] traditional machine learning. But MLflow also helps developers understand also helps developers understand applications powered by LLMs and AI agents. So, now let's see how MLflow works with AI agents. Now, let's [08:29] understand this with our common customer bot example. So, imagine a customer asks, "Where is my order and when will it arrive?" Your application might use multiple AI agents to answer this question. The first agent [08:45] understands what the customer is asking. Second agent will retrieve the order details from the database. Third agent checks the shipment status through the delivery service. And finally, another agent combines information and generates [09:00] a response. So, this is how four agents work simultaneously to solve this problem or to solve this question. "So, your order is expected tomorrow." Now, when everything works correctly, [09:15] the customer receives an accurate answer. But what if something goes wrong? Imagine the delivery service fails to provide the latest shipment information. Instead of receiving an updated delivery date, the third agent [09:30] gets the empty result. However, the final agent might still generate an answer using incomplete information. It tells the customer that your order will arrive tomorrow, but the delivery date was never confirmed. So, how does it say [09:46] that your order will deliver tomorrow? And this is what ML Flow actually solves. And here's where the problem becomes interesting. From the application's perspective, everything might look successful. The customer [09:59] submitted the request, the system processed it, and the response was processed it, and the response was generated. The server might even an HTTP 200 status, which simply means the request was handled successfully. But, [10:14] different. It doesn't mean the information given to the customer was information given to the customer was ever correct. So, how do developers find ever correct. So, how do developers find out what actually went wrong? Was the [10:28] order information incorrect? Did the delivery service fail to return the data, or did the final AI agent generate an answer without checking the available information? And this is where ML Flow becomes [10:43] useful. So, ML Flow doesn't build or coordinate these agents by itself. Instead, it helps developers observe and evaluate behave. With the right tracing setup, developers [10:58] can inspect which operations were performed, what information was returned, and how was the final response was generated. This makes it easier to investigate problems instead of guessing which part of the application actually [11:13] failed. And that's an important part of building reliable agentic AI systems, because creating multi-agents is just the beginning, but making sure they work together correctly, handle missing [11:27] information, and generate reliable responses is equally important. Now, if you're interested in going beyond understanding AI agents and actually learning how to build and deploy them, here's something worth exploring. Simply [11:41] Learn offers professional certificate program in 18 TK AI multi-agent systems program in 18 TK AI multi-agent systems from Vishleshan AI Hub TIH of IIT Patna. This 10-week program takes you through generative AI, large language models, [11:54] rag, multi-agent systems, and workflow automation. You will get hands-on experience with tools and frameworks like LangChain, CrewAI, LangGraph, and Any Ten while working on real-world AI projects. You can find more details [12:09] about the program through the link given in the description box and in the pinned comments. So, go right now. The program is worth checking. Now, coming back to our customer support example, we've seen how multiple AI agents work together. [12:24] But, how do we actually track what happened inside each agent when something goes wrong? And that's where MLflow tracing comes in. So, MLflow tracing helps developer examine the execution of an AI [12:39] request. So, in MLflow, a trace represents the journey of a request through the application. Inside trace, there are small operations called spans. Think of [12:53] trace as a complete journey, but each span represent an individual step. For example, a customer support trace might contain the database lookup, delivery API call, and LLM response generation. Depending on the [13:08] instrumentation, MLflow can record inputs, outputs, execution time, tokens, Okay. It might be overwhelming. No worries, we [13:20] will be understanding this with a simple example flow. Our most common customer support example. Everything will be clear to you with the reference of this example in our entire tutorial. No worries, I will even [13:36] show you the practical working of traces through this example itself in our further tutorial. So, let's understand with this flow. First of all, the customer asked, "When will my order arrive?" The application [13:51] checks the order database and successfully retrieves the customer's order. Next, it calls the delivery tracking API. API, which means application programming interface. [14:04] But this time, the API results an empty value. Here you can see. Despite receiving no verified delivery information, the LLM respond, "Your order will arrive tomorrow." The application returns HTTP 200. [14:21] Technically, the request completed successfully. But the response is incorrect because the delivery date wasn't verified ever, right? It never verified this delivery API which was supposed to verify the [14:36] delivery date, but it resulted with an empty response. Then also, LLM generate the response and said that it will arrive tomorrow, which was never correct. [14:49] So, now imagine opening the request trace in ML flow. Here, ML flow checks that the customer request was successfully recorded. The request was successfully recorded. The AI agent checks the order status. In [15:04] order database, the order was found. Everything was perfect till this point. Then the API work come in and the delivery API resulted empty result instead of giving the particular date. Then also, when the LLM response [15:20] Then also, when the LLM response checked, it result with unverified >> [clears throat] >> result was generated by the delivery API >> result was generated by the delivery API and and LLM, the HTTP status was okay. [15:35] So, instead of seeing only a successful request, the developer can inspect the individual operations in tracing. The trace reveals that the delivery API returned no information before LLM [15:47] generated its answer. This gives developers a specific place to investigate that the problem occurred in this particular step. Tracing can also this particular step. Tracing can also help identify performance bottlenecks. [16:06] seconds and the delivery lookup consumes 2 seconds, developers can investigate that operation. The important distinction is that tracing helps explain how an answer was produced. It doesn't automatically [16:20] determine whether the answer was correct or not. So, this is how we saw the trace will actually work. Let's have a recap for this. So, let's suppose the customer said, "When will my order arrive?" This will go inside ML flow trace and the [16:35] operations will be performed. First of all, customer request was made successful, then the particular order that the customer was asking was found in the order database. Then check the delivery API, but the API resulted with [16:50] empty response. Then also, without passing any verified delivery date, LLM responds with unverified delivery date. And because of its response, the status So, as we have seen, the problem occurred in [17:07] this particular trace. So, this is how the trace actually work inside ML flow. Okay, I understand that this was all theoretical part. We have seen the theory and step-by-step the working of MLflow. Need not worry, we'll take this [17:22] entire example to our actual MLflow website and I will show you in detail that how this MLflow tracing actually works in the database. We will use the same customer support example where application generates a delivery date [17:37] without receiving verified tracking information. So now, let's straight get information. So now, let's straight get into the demo of MLflow tracing. And I'll show you how tracing works step-by-step. So for this, we will be [17:50] using Python, VS Code as our code editor, and the MLflow itself. editor, and the MLflow itself. So let's open our VS Code, Visual Studio Code. This is the most widely used editor code editor. You can even use [18:06] editor code editor. You can even use cursor, but I highly recommend using VS folder. Let's suppose in C drive under projects, Let's suppose in C drive under projects, I'm creating a folder. Name is MLflow [18:21] demo and then I am opening this folder into our VS Code from here, file, open folder, then MLflow demo. This we created right now. So let's open this folder. So this is opened here, MLflow [18:37] demo. Here we will open its terminal. And first of all, drive projects MLflow demo, we will install MLflow first. And the command to [18:53] do the same is this. Python {hyphen} m pip install upgrade MLflow. So this particular command will install MLflow in our VS Code, in our [19:05] like desktop. So the requirement already satisfied, it means like I have already installed flow in my desktop, so this is saying flow in my desktop, so this is saying that it's already satisfied. So, if you [19:18] can do the same and the command for the same is this. same is this. Python {hyphen} m pip show MLflow. And if it is successfully installed, this command will run. [19:32] So, it shows us the location for that MLflow, where the packages are being installed. So, it is in C drive, users, inventor, my section. So, it means MLflow is successfully integrated in our system. Now, let's move forward. So, now [19:49] with this command, let's go inside the MLflow user interface. So, this is how your MLflow interface will look like. Let it run for some time. The [20:02] dependencies are being like adding, application setup has completed. And application setup has completed. And now, our MLflow website is here in front of us. So, here you can see home page, experiments, prompts, the like [20:19] experiments that we have already done, then the prompts here with different then the prompts here with different versions, then MCP registry, AI gateway, and so on. So, here you can even start with the MLflow demo and I highly [20:33] recommend you to check the official documentation of MLflow just to understand the basic idea of how MLflow actually works. Like it is a good habit to read the official documentation provided by any website or any [20:47] particular software the new launched. So, here you can see tracing, evaluation, prompts, AI gateway, model training, and so on. So, now let's create a file inside our MLflow folder, which is [21:03] we can say customer support demo.py and let's press enter. And here let's paste the code and let me explain you this code line by line. So first of all, we have imported MLflow and we have connected our MLflow [21:17] tracking URI with localhost 5000 where our MLflow is actually running. And then this is the customer support AI demo. Let me give it the name as customer support demo. Then the MLflow trace it means the get [21:32] order function will return the order ID and the status as shipped. And the order will be checked in the order database. Now checking the tracking API, here the [21:44] MLflow.trace tracking API will uh by reading the order ID. What is the delivery status of that particular order? But this API will not return any [22:00] information and we are using the same chatbot example that we have covered in our theory section. To make you understand how it actually works inside the system. So this particular order will return [22:13] that your order will arrive tomorrow and here this the error is this that it has not even read the information from the API as API returns no information. And [22:25] finally that the LLM will respond as status okay. So this is how this particular customer support demo code will look like and work like. Now let's run this py file. Python customer support demo.py file. [22:42] And let's see what does this return. So the HTTP status came out to be 200 okay and customer response as your order will arrive tomorrow. So now here notice the problem. The delivery service returned no tracking [22:57] information, but the assistant still generated a confident delivery date. And it says your order will arrive tomorrow. But how does it know? Because the API didn't even return anything. It returned just an [23:12] empty result. This is the exact failure scenario and let's see how MLflow trace will solve this. So now let's open the trace in our MLflow and for this we need to return to our local host. Here we will select our [23:29] experiment that we've like created recently and here we can see the trace. One trace was created. So this is was our trace. Let's open this trace. So [23:41] here is the hierarchy that customer support request performed first the order database. So it returned that order ID is 1001 and the status was shipped. So till now it is right that the particular order was [23:55] shipped already. So here now check the delivery tracking API key and here we can see it returned nothing. That the delivery date was not at all provided [24:07] and the result is empty for the particular order ID. So here we can see that the particular problem was here. So this is how the trace work. It doesn't only say that the particular response was right or wrong but it also helps [24:22] developer to see where the problem occurred and then we will check the AR response and it shows that the particular order was shipped but the particular order was shipped but the tracking came as empty. And then because [24:36] the particular order was shipped the response came out to be your order will arrive tomorrow. So in these three traces the developer can easily figure out that the error happened at this place. So MLflow didn't [24:52] automatically declare the answer as incorrect. Its trace exposed the empty data in the response allowing the developer to identify the exact issue and the step of the execution. Now, let's see how we can fix this problem. [25:06] Let's go back to our So, here we will collect the logic of our program. >> This step three was the incorrect AI response, right? So, here we will replace this with the answer as your order has shipped, but the delivery date [25:22] is not yet available. And if it worked like correctly, so the estimated delivery date will come here. This will return this. If it is not tracking any [25:34] estimated delivery, then this particular code section of the code will be executed. And in case if it tracked the date, then this particular section code will executed. Now, let's save this and let's rerun this. [25:48] So, python customer support demo.py. And now, let's see the difference in the output. So, the response came as your order has been shipped, but the delivery date is not yet available. [26:01] not yet available. Now, we have two traces. One with the incorrect output and the another one that we executed just 54 seconds ago. And here we can see the order database that particular particular order ID the [26:16] status was shipped, then the delivery tracking I API key checked the like order number and it it checked that the particular delivery date was not [26:29] mentioned for this particular order ID. So, according to this, the AI response as as follows. That the order was particular this that the order ID was this and the status was shipped, but the tracking API key didn't [26:46] return any delivery date. So, the output came as your order has been shipped, but delivery date is not yet available. So, now you can see the difference between these two traces clearly. How our first trace responded, and now how the [27:02] corrected code after the correction being made in our code, how this particular trace worked, and how our AI respond in this particular case. So, now you can easily compare which AI response is much better. Of course, the second [27:16] one. So, MLflow tracing helps us understand response. But, there's another important question. How do we automatically check whether the response is accurate, relevant, or [27:30] the response is accurate, relevant, or supported by available information? And that's where MLflow's evaluation capabilities comes in. So, the question here comes, how do we know whether its answer is actually [27:44] good? In traditional machine learning, developers measure performance using matrices such as accuracy, precision, recall, or prediction error. But, evaluating an LLM can be more complicated. Suppose a customer ask, [27:59] "When will my order arrive?" The model respond, "Your package will arrive tomorrow." Another response based on the verified tracking information, "Your estimated delivery date is tomorrow." So, both answers sound helpful, right? [28:14] But, imagine the tracking API never provided a delivery date. In that case, neither response is properly supported. And this is why evaluating AI [28:26] applications require more than checking whether an answer sound reasonable. So, basically, MLflow provides evaluation capabilities including rule-based scorers and LLM as a judge. So, now let's understand this. Rule-based [28:41] scorers check accuracy, validate required facts, enforce business rules, required facts, enforce business rules, and custom evaluation logic. While LLM as a judge helps or uses another language model to access an AI-generated [28:55] response against defined criteria. For example, it can access relevance, groundedness, and whether a response follows application guidelines. Now comparing the two AI responses, response one, that is response A and response B. [29:10] So here the evaluation technique responded that relevance was passed in both the responses, but groundedness and guidelines were not followed by response A. And it was followed by response B. So this is how MLflow will evaluate and [29:27] compare both the AI responses through LLM as a judge technique. And then select the perfect response, and then it will implement that response So response B represents an improved answer that acknowledges available [29:43] tracking data. So let's say customer support assistant has a rule, "Never promise a delivery date without verified tracking information." An evaluator can check whether the [29:56] response follows this particular guideline. If the AI confidently promises delivery without supporting evidence, the evaluation can flag the response. And developers can then compare results across different prompts [30:10] or model versions. However, LLM judges can also make mistakes. That's why teams combine automated evaluation with deterministic checks and human review. So the goal is to measure [30:24] response quality consistently and identify areas for improvements. So the another important part of developing AI applications is managing prompts. That control. Let's continue with our customer support [30:41] assistant. Initially, its system prompt might say you are a helpful customer support assistant. That's a basic instruction. But after discovering incorrect delivery predictions, developers might update it to always use [30:57] verified order information before answering delivery-related questions. And later, they might create another version saying, "If tracking information is unavailable, explain that the delivery date cannot be confirmed." Now, [31:12] which prompt performs better? And what happens if a newer prompt introduces unexpected problems? So, MLflow provides a prompt registry that helps developers [31:24] organize, version, and manage prompts. So, here you can see version one, which has basic instructions, then version two, where it checks verified order data before responding, and then the latest version says verified data, disclose [31:39] uncertainty, and handle the missing information. And then, MLflow will compare versions using evaluation results. So, this becomes especially useful when multiple developers are working on the same AI application. You [31:53] can compare prompt version, renew their evaluation results, and identify which evaluation results, and identify which one performs the best. Instead of changing prompts randomly, you can make improvements based on the evidence. So, [32:07] so far, we have seen how MLflow helps developers track experiments, understand AI workflows, evaluate responses, and manage prompts. But what happens when a AI application is released and real customers start using it? This is where [32:23] monitoring AI applications in production comes in. is now answering thousands of questions every day. Some customers might receive [32:38] answers quickly while others experiences delays. Some requests might use more AI tokens than expected increasing the cost of the running application. And sometimes the AI might give incorrect or incomplete answers. So, how do [32:54] developers identify these problems? And this is where MLflow's monitoring this is where MLflow's monitoring capabilities become useful. MLflow helps developers examine how AI applications perform after deployment. So, let's [33:08] understand this with an example. For example, suppose your AI assistant usually answers a customer question using two LLM calls. But after an update, it starts taking five calls for the same task. This could make the [33:23] application slower and more expensive, right? With MLflow tracing, developers can inspect these calls and understand what's causing the extra work. Similarly, imagine a new prompt causes the AI assistant to give more incorrect [33:38] answers. Developers can use MLflow evaluation capabilities to compare the new responses with earlier results and identify whether the change improved or reduced the answer quality. So, by reviewing traces and evaluation results, [33:53] teams can identify recurring issues and improve the application. It's also important to protect customer information and control what data gets recorded. So, in simple terms, MLflow helps developers understand how their AI [34:09] application behaves in real-world use, so they can identify problems and make improvements. The necessary improvements. So, these were all the capabilities of MLflow. You remember we started our tutorial from [34:24] remember we started our tutorial from here where we have checked all these six capabilities. So, we have covered all these in detail with experiment tracking, model management, AI tracing, evaluation, prompt management, and [34:37] evaluation, prompt management, and finally monitoring. Six capabilities of finally monitoring. Six capabilities of MLflow. And we have even seen the actual or the practical demo of how MLflow tracing works inside the MLflow [34:50] environment. So, now let's understand how developers use it in real-world AI projects. Or we can say where is MLflow used in the real world. So, the common MLflow use cases includes machine learning, AI agents, AI [35:07] includes machine learning, AI agents, AI chatbots, and drag applications. So, in machine learning MLflow helps by tracking experiments and compare model performance. In AI agent, it trace agent workflows and identify failures. While [35:23] in AI chatbots, evaluate response quality, speed, and the token usage. And if we talk about drag applications, so here MLflow helps in tracking document retrieval and check answer accuracy. So, now let's look at the few examples to [35:39] understand this. First, let's start with machine learning. Imagine a bank developing an AI model to detect fraudulent transactions. The data science team test several machine learning models to find which one [35:53] performs the best. MLflow helps them record experiments, compare results, and manage different model versions. So, this is how MLflow is used in machine learning. Now, coming to AI chatbots. Suppose an online shopping company's [36:08] chatbot gives a customer the wrong delivery date. With MLflow tracing, developers can inspect the chatbot's execution and investigate whether the problem came from missing information, a failed tool call, or the AI-generated [36:24] response. So, this is how it is used in AI chatbots. Now, talking about real applications, imagine a company builds an AI assistant that answers employee questions using internal documents. When an employee asks a question, the system [36:37] retrieves relevant document and uses them to generate an answer. MLflow can help developers inspect which documents were retrieved and evaluate whether the answer is supported by that information. So, in simple terms, MLflow helps [36:53] developers understand how their AI system perform, identify problems, and make improvements based on the actual results. So, whether you are building a machine learning model, an AI chatbot, or a complex agent-based application, [37:09] MLflow makes it easier to track, evaluate, and improve your work. So, let's quickly recap what we have covered in our today's video. So, MLflow is an open-source platform that helps developers manage machine learning and [37:24] AI application. Experiment tracking helps compare training runs, model management helps organize model versions, tracing helps investigate what happens inside an AI workflow, evaluation helps measure response [37:39] quality, prompt management help organize and compare prompt version, and production monitoring help teams investigate how applications behave in real-world environments. Remember, the customer support assistant from the [37:54] beginning, the server returned the successful response, but the AI provided an incorrect delivery date. That's the difference between an application that runs successfully and one that produces reliable answers. MLflow helps [38:08] developers investigate the difference, measure improvements, and build more reliable AI systems. So, if you found this video helpful, then give it a like, subscribe to Simply Learn, and let Let know in the Have you ever used MLflow in [38:22] any of your machine learning or generative AI projects? Thanks for generative AI projects? Thanks for watching and happy learning.