[00:08] powerful large language model. It works perfectly in your demo. But the moment thousands of users start using it, responses become slower. Cost increases, hallucination appears and security risk emerge. So how do companies take an LLM [00:21] application from a simple experiment to secure scalable production ready AI system? That is exactly where LLM ops comes in. Hello everyone and welcome to learn. In this course, we will take you through the complete process of [00:36] building, deploying, monitoring and managing large language model. Now here is what we will cover in this course. What is LLMOPS? Why do we need LLM ops? How LLM application works? Component of an LLM application. The complete LLM ops [00:52] life cycle. Data management in LLM ops. prompt engineering and prompt management, retrieval augmented generation or rag, LLM deployment strategies, monitoring and evaluating metrics. So by the end of this course, [01:06] you will have a clear understanding of how LLM powered applications are taken from development to production and managed at scale. Now before we begin, just a quick info guys. If you want to take your learning further and gain [01:20] hands-on experience in generative AI and agentic AI, you can check out Simply Learn's applied generative AI and agentic AI specialization delivered in collaboration with Microsoft. The program covers generative AI, LLM, RAG, [01:35] prompt engineering, agentic AI, multi- aent system, LLM deployment, monitoring and productionready AI applications. We will also work with popular AI tools and build real world project that help you apply these concepts practically. You [01:48] can find the program link in the description below. Now before we move, here's a quick question to test your knowledge. Which of the following best describe LLMs? Your options is option A, a technique used only for training large [02:02] language model. Option B, a process used only for writing better prompts. Option C, a discipline for developing, deploying, monitoring and managing LLM powered application. or option D a database designed specifically for [02:14] storing props. Let me know your answers in the comment section below. Now before understanding and ws let us first understand the problem it solves. Now when organization build application using large language model they are not [02:26] simply deploying a model and finishing the process. There are many moving components involved such as including data prompts, APIs, user interaction, evaluation, system security control and continuous improvement. is the [02:39] discipline of managing all these components throughout the life cycle of an LLM powered application. It focuses on developing, testing, deploying, monitoring and improving application that use large language models. We can [02:52] compare with other two important concept which is DevOps and MLOps. DevOps focuses on improving the software development and deployment process. It testing and release software efficiently. Extend these ideas into [03:06] machine learning by managing machine learning models, training pipeline, data further by addressing the unique challenges of large language models. It manages not only the model itself, but also prompt, context, data, retrieval [03:22] system, user feedback, response, quality, and AI safety. A simple way to understand is that DevOps manage software, MLOp manages machine and LLM ops manage application powered by large language model. Now, why do we need LMO? [03:36] At first glance, building an application with an LLM may seem simple. A developer can connect an API, send a prompt, receive a response, and create a working prototype within a few. But moving from a prototype to a production level [03:49] application introduces many challenges. Now, traditional machine learning system deployment, data management, performance, monitoring, and handling changes in data patterns. Large language [04:02] model application introduce additional complexity. One major challenge is the output quality of an LLM which depends heavily on the prompt. Even a small change in instruction can significantly affect the response generated by the [04:15] model. Therefore, organization needs system to manage, test, compare different prompt versions. Another glance is hallucination. Large language model can sometime generate responses that sound correct but are factually in [04:29] information can create serious problem. LLM ops help organization evaluate and monitor response quality. Cost is another important factor. Every interaction with an LLM consumes computational resource and tokens. When [04:44] an application scales to thousands of user, managing these cost becomes very great. Security and privacy are also major concerns. Organization need to ensure that sensitive information is protected and that user cannot [04:57] manipulate the model through techniques like prompt injection and attacks. Now organization need a structured approach to manage LLM applications. Now how an LLM application works. To understand LLM ops better, we first need to understand [05:12] how a typical LM application works. Now when a user interact with an AI application, the process usually involve multiple layer. The first layer is the user interface where the user enters a question or request. This could be a [05:25] chatbot interface, a mobile application or an enterprise software system. The user request then moves into application layer. This layer manage the business logic and decide how the request should be processed. Now depending on the use [05:38] case, the application may use technologies like prompt engineering, rag or AI agents. The next layer is LM itself. The model receive instruction along with the relevant context and generate a response. This could be a [05:51] generalpurpose model like GPT cloud, Gemini or an open-source model like llama. However, in real world application organization usually do not directly send every question to the model. They often connect the model with [06:03] additional data. For example, an employee asking a question about company policies may require information from internal documents. The system first retrieve relevant information and then provides the context to LLM. Finally, [06:16] system needs monitoring. It tracks important factors like response quality, latency, user feedback and cost. Limops manage every stage of this process ensuring that entire system works reliably from user question to the final [06:31] response. Now, components of an LLM application. An application is not just a large language model. It is an ecosystem made of components working together. The first component is foundational model. These are large [06:43] pre-trained model created by organizations such as OpenAI, Anthropic, Google, Meta and other AI research companies. These model provide the core intelligence required for generating text, understanding information, writing [06:56] code, answering questions and performing reasoning task. The second component is the prompt layer. Prompt act as instructions that guide the behavior of the model. A well-designed prompt can improve accuracy, consistency and [07:09] usefulness in the production system. Organization need to manage prompt carefully track changes and evaluate which prompts provide better results. The third component is data layer. Many enterprise application need access to [07:23] private or domain specific information. This data can include documents, database, customer conversation or internal knowledge sources. Vector data, vector database and embedding system help store and retrieve the information [07:36] efficiently. The fourth component is the application layer. This is where user interact with AI system. Example include customer support, chatbots, AI search applications. And the final component is the monitoring layer. Since [07:50] LLM outputs are not always predictable, organization need system to measure performance, detect issues, monitor cost and continuously improve the application. Now together these component form a complete LLM [08:03] application and LLM ops provide the framework to manage all of them effectively. Now coming to LLM ops life cycle. Now that we understand the components of an NLM application, let us look at the complete life cycle involved [08:16] in building and managing these systems. This life cycle is known as the LLM ops life cycle. The first stage is data preparation. Data is one of the most important parts of any AI application because the quality of information [08:30] directly impact the quality of the output. Organization collect data from different sources from different sources such as documents, databases, customer interaction, website and internal knowledge system. This data needs to be [08:43] cleaned, structured and prepared before it can be used by an alm application. Organization need to choose the right model based on their requirements. Different model have capabilities, cost, response, speed and accuracy levels. For [08:58] example, a simple chatbot may work well with a smaller and fast model. While a complex reasoning application may require now after selecting the model, the next stage is prompt engineering. Since LLM responds based on instruction, [09:11] designing effective prompts becomes an important part of application development. Teams experiment with different outputs. from test output and results. The next stage is testing and evaluation. Now unlike traditional [09:25] software where we can easily define expected outputs, evaluating LLM application is more challenging because responses can vary team use automated evaluation methods, human feedback and quality metrics to measure performance. [09:40] Now once the application passes testing, it moves into deployment. application is made available to users through APIs, website, mobile application or enterprise platform. After deployment, continuous monitoring becomes essential. [09:53] Teams track response, quality, latency, cost, user satisfaction and security continuously improve the application. The important point here is to remember that LLM ops is not a one-time process. It is a continuous cycle where [10:09] application are constantly evaluated, improved and optimized. Check the camera. Now coming to data management in LLM ops. Now data management plays a critical role in LLM ops because large language models are only as useful as [10:22] information they can access. While general purpose LLMs are trained on massive amounts of publicly available data. Many organization needs AI system that can understand their own private and specialized information. For [10:34] example, a banking company may want an AI assistant that understand its internal policies. A healthcare organization may need an assistant that can analyze medical document and a company may want employees to search [10:46] internal knowledge. The first step in data management is data collection. Organization gather information from different s sources such as document, websites, databases, customer support, conversation and enterprise system. The [10:59] collected data usually cannot be directly provided to an LLM. It needs to include cleaning unnecessary information, removing duplicate content, correcting formatting issues and organizing data into meaningful section. [11:14] Now one important technique used in LLM application is document chunking. Large document are divided into smaller section because LLMs have limitation on how much information they can process at once. Now these smaller section are [11:27] easier to search and provide as context to the model. Another important concept is embedding. Embeddings convert text information into numerical representation that capture meaning. This allows system to search based on [11:40] concept rather than only exact keywords. Rings are stored in vector databases Rings are stored in vector databases such as pine cone or chroma DB. When a user ask a question, the system searches the vector database to find the most [11:52] relevant information and provide to the LLM. Now effective data management help improve accuracy, reduce hallucination and make LLM application more useful for engineering and management. Now one of the unique aspect of large language [12:07] model is that the quality of the output depends heavily on the quality of the instruction provided to the model and these instructions are called prompts and designing effective prompt is known as prompt engineering. Now a prompt is [12:20] include context, instruction, example, rule, and expected output format. A understand the task better and generate more accurate responses. For example, instead of asking a model summarize this document, a better prompt could be you [12:36] are a business analyst, analyze this document and provide a summary containing key insights, important numbers, risk, and recommendation. The second prompt provides more context and guidance, which usually leads to a [12:49] better response. Now in production environment prompt engineering becomes a structured process. Teams need to manage different version of prompt test their performance and understand which version produce best result. Now this is again [13:03] similar to managing software code versions. A small change in prompt can improve performance or sometime create unexpected issues. Therefore ops introduce prompt version control and testing practices. No organization also [13:15] perform prompt evaluation through methods such as AB testing. Different prompt versions are tested with real exam and better performing version is selected. Now prompt management becomes specially important when organization [13:28] build customerf facing AI application. A poorly designed prompt can result in incorrect answers, inconsistent responses or inappropriate outputs. Therefore prompt engineering is not just a skill for experimenting with AI model. [13:43] In LLM ops, it becomes an important engineering practice required for building reliable AI system. Now coming to what is rag. One of the biggest challenges with large language models is that they may not always have access to [13:57] latest or private information. A model may have strong language understanding capabilities, but it may not know specific company data, recent update or confidential information. Retrieval argument generation commonly known as [14:11] RAG solves this problem by combining information retrieval with language generation. The basic idea behind RAG is simple. Instead of asking the LLM to answer a question using only its existing knowledge, we first retrieve [14:24] relevant information from an external knowledge source and provide that starts when a user submits a question. The system converts the question into a format that can be used for searching. It then searches a knowledge base, [14:38] usually through a vector database, and identifies the most relevant documents or information. This retrieved information is then added to the prompt and sent to the LLM. The model uses this additional context to generate a more [14:52] accurate and relevant response. For example, imagine an employee asking an AI assistant, what is our company's leave policy? A general LLM may not know the company's internal policy. But with rag, the system retrieves official leave [15:06] policy document from the company's knowledge base and provide it to the model. The company then generate an answer based on the information. So rack provides several benefit. It reduces hallucination, allows organization to [15:19] use private data, keep information updated without retraining the model and make LLM application more suitable for enterprise use case. Now today rag has become one of the most widely used architecture for building reliable [15:33] enterprise AI applications. Deployment. Now let us look at deployment strategies language model there are mainly two common approaches. API based deployment and self-hosted deployment. The right choice depends on factors like cost [15:47] privacy control and scalability. With an API based approach you access model through providers such as G-Cloud. This is usually the fastest way to deploy an LLM because you do manage the underlying infrastructure. It is easy to set up, [16:01] simple to scale and works well when you want to launch an application quickly. How based deployment also comes with some limitation. Your data is sent to an external service which can create privacy or compliance concerns. The cost [16:15] can also increase as the number of users, request or token grows and you have less control over the model infrastructure. The second option is self-hosted deployment where organization run models such as llama or [16:27] their own customized model or private infrastructure. The biggest advantage here is that greater control and privacy. So organization can decide where the model runs, how the data is stored and how the infrastructure is [16:39] configured. It also can provide more predictable cost at larger scale. But self-hosting requires more technical expertise. Teams need to manage servers, GPUs, model updates, monitoring and deployment pipeline. Now as LLM [16:53] application grow, the supporting infrastructure also becomes important. Cloud platforms such as such as Google cloud container orchestration tools such as Kubernetes and model serving technology such as VLLM can help scale [17:05] LLM workload efficiently. So in LLM op there is no single deployment strategy that work for everyone. The best approach depends on how much speed, privacy control, cost, efficiency and scalability your application record. Now [17:18] let us look at key monitoring and evaluation metrics in LM. Now once an LLM application is deployed the work doesn't stop. We need to keep track of the first things to is to monitor is latency and speed. This tells us how [17:31] quickly the model responds to a user request. If response time becomes too high, it can negatively affect the user experience. Next is token uses and cost. Since many LLM service charge based on the number of token processed monitoring [17:45] token consumption help team control expense and optimize prompt. We also track user feedback, rating, comments and user behavior whether the model's response are actually useful or not. Another important area is security [17:58] especially prompt injection attacks. Monitoring suspicious input helps protect the application from misuse. For evaluation one major metrics is accuracy and relevance. We check whether the model is giving the correct and useful [18:11] responses or not. In ragbased system, we also measure context quality to make sure that retrieved information is relevant and helpful. Then comes hallucination rate which tells us how often the model generate incorrect or [18:24] unsupported information. And finally, teams may use an LLM as a judge score evaluates the quality of generated responses based on predefined. So a [18:36] successful LLM application requires multiple technology working together. The LLM ops technology stack includes tool and platform for development, data management, deployment, monitoring, security and optimization. At the [18:49] development level, programming languages and frameworks help developers build AI application. Python is currently one of the most widely used language because of its strong AI ecosystem. Frameworks such as langchen help developers connect [19:02] language model with external data sources, create rack pipeline and build AI system. The next layer is the data layer. Since many enterprise application require access to private information, organization need system to store, [19:14] process and retrieve data efficiently. Vector database play an important role in the layer. Technologies such as pine cone, chromb and vvate store embeddings and enable semantic search. A complete lm op stack bring all these components [19:28] together. It allows organization to move from simple AI experiment to production ready AI application that can operate at enterprise scale. One major difference is version management. In traditional software development teams mainly track [19:41] code versions. In MLOps team track code data sets and machine learning model. In LM ops team need to manage even more components including prompts, embedding, retrieval system, model and evaluation methods. Monitoring is also different. [19:55] DevOps mainly focuses on application performance, availability and infrastructure health. MLOC focuses on model accuracy, data changes and additional factors such as hallucination rates, response quality, token uses, [20:09] latency, safety and user feedback. So this is all for this particular video. the like and subscribe for simply YouTube channel. Till then keep YouTube channel. Till then keep learning. See you in the next video.