Machine Learning with Python: Full Breakdown & Transcript

[00:09] But the real power comes from turning that data into intelligent decision. From recommendation systems to fraud detection and self-driving systems, future of every industry. With that being said, I welcome you all to this

[00:22] session on machine learning with Python full course. Before I begin our session, just a quick info guys. If you want to build a strong career in AI and machine learning, then simply learn offers a advanced professional certificate course

[00:34] in generative AI and machine learning designed in collaboration with EICA consortium of IIT GPU and powered by industry partners like Microsoft Azure. This program includes live online classes, hands-on projects, expert

[00:47] mentorship and career support helping you move from fundamentals to real world AI applications. Now coming back to the course. In this course, we will learn how to use Python for machine learning from scratch to advanced levels. We are

[00:59] going to start with Python refresher and maths and statistics. Then we are going to move into data processing and exploratory data analysis. After that, we are going to explore core machine learning concepts like supervised and

[01:11] supervised, regression, classification, clustering, model evaluation techniques. like deep learning with TensorFlow and Kiras, NLP, reinforcement learning and finally, we are going to apply everything through hands-on industry

[01:26] projects like sales prediction, customer analysis, recommendation systems, and course, you will not just understand machine learning, but you'll also be able to build real intelligent systems using Python. So let's let's begin with

[01:41] this journey and try to understand what is machine learning and why is there so much buzz around it. Let me first uh let me go through what

[01:53] all we would be covering in this particular course and then we begin the discussion. So if we talk about a learning path you know the first uh you know uh topic that we will be covering today is

[02:06] introduction to machine learning which focuses on the basics of machine learning. Second is supervised learning regression and application which focuses on supervised learning with an emphasis on understanding and implementing

[02:19] different types of regression models. Third is supervised learning classification applications like basically we're trying to cover that techniques available in the machine learning. So first technique that we

[02:34] learning. Under supervised learning we would be covering up regression as well as classification. Then we would be moved on to the ensemble learning method which focuses on advanced ensemble methods to enhance the

[02:50] performance and robustness of the models. Then we would be moving on to unsupervised learning and finally recommener system along with the application. What is machine learning and why is this so much a buzz? Why are

[03:03] you here to learn machine learning? Let's let me put this in another way. So basically we want the machine to get trained with our data. We want the machine to learn from the data so that it can predict data. What kind of data

[03:18] that we want to predict? Why do we want to have want these predictions? Right? Because now are we living in the digital technology where data is all around us.

[03:31] Even this you know uh you know session is data right? When all the material that has been sent to you is data, anything on the news which is coming is data. Anything doing for entertainment is data. E-commerce is data.

[03:49] You know your work profile is data because we are living in huge amount of data. Data is all around us. Do you think is there any escape from data now?

[04:01] No, not now. uh when I used to take sessions five years you know and seven years uh before you know uh the scenario was little trying to adapt but I don't

[04:13] was little trying to adapt but I don't think so there is now any survival without data can you survive without this data not moving on social media and you know one day you know as it says you know if the internet stops do you

[04:27] think your life also stops you your mobile phone is lost everything is lost Isn't it the data has become a lifeline you know and now we see

[04:39] several applications which are working or the concepts that are being bas you or the concepts that are being bas you know based on data now it automatic translation translation has not become difficult difficult if you want to

[04:54] convert something from English to German to Spanish any language translation is right there you We speak to the machine and we get the translation. Virtual personal assistants are there. Image recognition. Email spam filtering that's

[05:10] that's actually comes under the domain of machine learning that it you know based on the algorithm or the pattern or the text or the words which are there. It is able to filter out whether the mail is a spam or not. So what do you

[05:25] think would be the criteria? Generally the males with spam are saying that you the males with spam are saying that you have a lottery system other system or there is a bonus. So that becomes your email spam filtering. Then we have the

[05:38] text and the speech recognition. Medical diagnosis, online fraud detection is there you know where we want to detect uh the how is the online fraud happening. Web search uh search and recommendation engines and of course

[05:54] what is going to be the traffic prediction. Not only the traffic prediction on the roads but it also relates to the traffic going onto a particular website whether that website is going to get crashed or not.

[06:07] Data becoming now difficult to handle because it's digital data. You know understanding data everything numerically is difficult. You know be handled. Now it's become like this much and this much and this much and

[06:20] it's increasing. So we need certain algorithm we need technologies which can help to analyze so that we can improve our performance. But on if you if you look at on the overall scenario still there is a lot of

[06:35] confusion about the AI the machine learning and the deep learning. Yeah. learning and the deep learning. Yeah. What is the you know the subset or the is artificial intelligence machine learning and deep learning? Are you able

[06:49] to distinguish between the three? So this is one of the initial chess you know developed by the computers IBM deep blue chess program developed in 1997.

[07:03] When was it developed? It was developed in 1997 in 1997 by IBM and this particular program was by IBM and this particular program was strong enough to defeat the uh world

[07:17] chess champion at that particular point. Uh his name is Gary Kasparov. Okay. He was capable of defeating the world chess champion. Getting my point

[07:30] right. And [snorts] then we have this IBM Watson under machine learning. So AI is the bigger branch. All right. Artificial intelligence is the bigger

[07:43] Artificial intelligence is the bigger branch. Right. under which we have the branch. Right. under which we have the machine learning subset right which we are going to study. In machine learning basically it works on statistical

[07:57] algorithm that is why we say that before learning machine learning it is important to have a good concepts and knowledge of statistics that helps to understand. So based on statistical foundation, statistical algorithm uh you

[08:13] know the machine is capable of doing Google search algorithm, Amazon Google search algorithm, Amazon recommendation and email spam filtering. How like we just saw that it is capable of filtering the email. And then finally

[08:27] of filtering the email. And then finally we have the deep learning which is also under machine learning in which we have the alpho the natural speech recognition and the level four automated driving system.

[08:47] the AI revolution. So the bigger branch is still AI. We are seeing we are the you know witnessing this revolution in front of us. Under AI we have the machine learning. Under machine learning we have the deep learning and under deep

[09:05] gen AI task. Right? So when we talk about the gen AI task generation fine-tuning we have the agents automation and the virtual assistants.

[09:17] automation and the virtual assistants. And this genai is now capable of even doing lot of text generation that we see we ask a lot of things to the chat GPT we ask a lot of things to the chat GPT other uh models like Gemini copilot also

[09:31] we want to image uh generate images video generation all these things are being possible AI that we are talking at the moment is only and only related to the software through software we are able to

[09:46] or give intelligence answers. But what is basically the difference between a traditional programming and a machine learning programming? So let me explain you with this particular con uh example. So in traditional programming again we

[10:02] have this data right that this is my data which is 1 2 3 and 4 right and over here we have machine

[10:14] right and over here we have machine learning algorithm where we have 1 2 3 and four okay the data has not changed I'm giving you a very simple uh you know example now if we talk about the program

[10:29] initially the program when we talk about C, Pascal, Forron even C++ and Java and even Python they are capable of gic. Now if I want to distinguish that what are the numbers what is the logic behind that these uh

[10:44] what is the logic behind that these uh you know numbers are even or odd. So simply we understand the logic that if I if I talk about uh Python I percentage 2

[10:56] if I talk about uh Python I percentage 2 is equal equal to zero that means if the remainder divided by two is zero then the number the number it is odd right

[11:10] it is odd right else it is odd getting my point. So now even if a number 10, it would automatically give me that this is going

[11:26] to be an even number. And if I give the number 57, it is going to be odd. But now machine learning how things happen. I give the output along with it. I say I give the output along with it. I say that one is odd, two is even,

[11:42] that one is odd, two is even, three is odd and four is even. three is odd and four is even. Got it? Now the computer based on certain statistical concepts algorithm will try to detect that when I feed the

[11:56] number 10 over that it is going to be even. Can it predict me as odd also? even. Can it predict me as odd also? Yes, the prediction can be wrong also. Yes, the prediction can be wrong also. Clear? But if the algorithm has to be

[12:11] good enough that it is predicting that 10 is even and 57 is odd. So coming on to the concept the first kind of learning is known as supervised

[12:26] learning. What is it known as? Supervised learning. that if this is my known data, this is already now images that this is my input

[12:40] that this is my input and IO feed the output. When I feed the in as well as output to the machine, it becomes my labeled data, right? That

[12:52] becomes my labeled data, right? That means this is an image of an apple. I feed it into the machine and when I feed this apple it says predicts that is this apple or not and it predicts it's an

[13:06] apple but it can predict wrong also. So might have seen sometimes the chart GPT also predicts gives wrong answer the image uh development jibli and all all

[13:19] give wrong answers. It's the biggest fear you know that these if the if a fear you know that these if the if a wrong or a incapable data is uh set to uh them then it can give false and nonsensical and fabricated information

[13:34] also. So this is the fake fear that we are now moving ahead right AJ Char hardik but if we talk about system if we talk about the traditional systems

[13:47] expert systems expert systems were working that we have a user we are trying to give the query and then try to get the output out of it right if this is my query and this is how I get the output

[14:03] query and this is how I get the output out of it and Then I try to infer what out of it and Then I try to infer what is it? But inference is not coming from the data. But this data it's coming from data. But this data is created from an

[14:18] expert. Now what do I mean by that? Let me explain this to you. Expert can be a cardiologist, a lawyer in different domains, right? a cardiologist, a lawyer, somebody from in the finance domain maybe for 30 years, 30 plus years

[14:34] domain maybe for 30 years, 30 plus years of experience and weated if else knowledge you know inference knowledge that whether you are capable of getting a loan or not and if I have a cardiologist some information has been

[14:49] fed that maybe your BP rating is this much or your um terms if that is matching to the inference engine that's how it will give the output. So what happens for example if you are a user and you enter into the user interface

[15:05] maybe all your uh you know blood reports your BPS and your test reports all your BPS and your test reports all things are given as the user interface right and then we do the inference engine and based on this knowledge base

[15:21] engine and based on this knowledge base we get the output right so what has been replaced now rather than knowledge base it is completely based on the original data and things have become complicated on images on you know nonstructured

[15:34] on images on you know nonstructured data. So uh do we understand supervised learning? Now can I say more the data more my system can I say more the data more my system becomes intelligent

[15:53] triangle and square and if I add a shape of a circle maybe a parallelogram maybe a rectangle it's capable of analyzing that. So that is where you know the systems are getting modeled because the data is becoming huge day by day right.

[16:11] So this is supervised learning. So under supervised learning we have the label data. What is label data? We have the input as well as the output where we try to train the model. After the model has

[16:24] been trained based on the test data we try to do the prediction whether it is a try to do the prediction whether it is a square or a triangle. Clear. We will initially start with supervised learning that we have the input the

[16:39] learning that we have the input the output. We will try the model. So where will we do the model? That means now the data the 70% of the data will be used data the 70% of the data will be used for training and 20 to 30 uh you know

[16:54] percent will be used for testing. Then it will predict the output. Okay. And as I've been telling you, there is a very strong relationship between machine learning as well as

[17:11] between machine learning as well as statistics part of it. mathematics. But when it is combined with computer science, you know, the

[17:23] machine learning, it becomes statistics and machine learning. The idea of statistics is that it helps us to draw inferences, relationship between variables. Whereas machine learning gives optimization, prediction, accuracy

[17:40] etc. Right? And then we have prior assumptions about the data. Some knowledge about the population usually required. This is none. Dimensionality of the data usually applied to the low uh dimensional data. and knowledge

[17:54] overlap. There's no ML knowledge required when we study statistics. But in machine learning, some statistics knowledge is usually needed as it is becomes the foundation for few algorithms. So if you do not know much

[18:10] about statistics, there is nothing to worry. It's not very difficult. Definitely concepts of probability would be required. So I would just require request you the learners to get familiar with the concepts of probability

[18:24] conditional probability basian theorem and probability distribution. So this is and probability distribution. So this is what I expect from you all. Got it? Right. And to make the picture a little more clear, you know, the boundaries are

[18:39] not very crisp now because there is data everywhere. But to make the picture a little more clear that you know when we start with initially Python course you know we are doing visualization exploratory data analysis maths and

[18:54] exploratory data analysis maths and statistics and when we try to overlap do overlap between AI machine learning deep learning we get this data science. So data science becomes the foundation for AI ML and deep learning.

[19:09] If I technically ask you what is learning and now you might have been hearing this word agent you know it could be a human agent it could be a robotic agent it could be an AI agent this

[19:25] so if we talk about a little more technical definition of learning basically we are trying to improve the behavior based on the experience when we experience that means means uh different types of

[19:42] knowledge. The range of behaviors is expanded. The agent can do more. Right? The range of behavior is expanded and agent can do more. The accuracy on the task is improved. The agent can do things better.

[19:59] And the speed is improved. The agent can do things faster. see this means that DS okay so if we talk about

[20:14] learning from the machine as well as from the human point of view this definition is valid that it is the ability to improve our behavior based on our experience right it's not always about acquiring new skills that's one of

[20:28] the thing that range of behavior that you are you know driving you know swimming you know stitching you know cooking of course the range is increased but also So accuracy by doing the thing again and again doing learning increases

[20:42] again and again doing learning increases my accuracy as well as speed. And if we my accuracy as well as speed. And if we now see technically in machine learning the different types of learning techniques are supervised learning,

[20:54] techniques are supervised learning, unsupervised learning, semi-supervised unsupervised learning, semi-supervised learning and reinforcement learning. So learning and reinforcement learning. So all these of actually learning come from

[21:06] all these of actually learning come from the fact that how we as humans also the fact that how we as humans also learn. So do you also agree only agree through training and testing? Training and testing happens when we are

[21:20] going to school we are trained and then we are tested in universities, colleges or even this session. Is there any other way we learn also? Do we learn through observations? Do we learn through our mistakes? Do we do we learn? There are

[21:35] different kinds of learning also possible. And exactly the same thing we possible. And exactly the same thing we also try to replicate in our machines

[21:47] also. So one of the learning method is observation. So one uh basic difference observation. So one uh basic difference between supervised learning and unsupervised learning. What do we mean by supervised and unsupervised

[22:03] learning? Supervised learning says that this is my data right are my apples right and I tell them that are my apples right and I tell them that this is my image these are apples. I

[22:17] this is my image these are apples. I data 70% of my data is used for training data 70% of my data is used for training and 30% of the data is used for testing. Label data is given in absolutely correct mega label data means supervised

[22:34] learning and then the model predicts me that it is an apple and what is unsupervised learning? Have I have do I have a label data? No. I have not told that this is an apple or

[22:49] this is a banana or this is a peach. But the model or the algorithm sorry guys is capable enough to distinguish between the three of them

[23:03] that this is my apple, this is my peach and this is my banana. Right? So initially we will you know build concepts on this supervised as well as

[23:16] unsupervised learning. The machine is given huge sets of data that are not given huge sets of data that are not labeled as inputs to analyze. The machine needs to figure out the output its own where it identifies the

[23:30] patterns. And the two types of algorithm which come under unsupervised learning are association and clustering. And K means for clustering problems and a priority algorithm for association role learning problems.

[23:46] Right? And then we have supervised learning. The input is in the form of raw data that is labeled. The machine is already fed with the required feature set to classify inputs. divided into two types of problems regression and

[24:01] classification and then we will try to understand different regression algorithm. This is the first stage that we are going to work on. Clear? So now let's understand what is

[24:17] selfsupervised learning. You know here we have the input data. Is it labeled? No, this is not labeled data. But we have partial label data that this is an

[24:30] orange and this is a banana and its quantity is less. And here both the types of data are mixed machine learning model and this is my unlabelled data to

[24:42] model and this is my unlabelled data to predict the output. So it's an apple. So this is my input data right. This is my partial data. And when I combine them together, prepare the model, I get the output,

[24:59] model, I get the output, clear effective way of learning from which we as humans learn a lot. That means from

[25:11] our mistakes, from our feedbacks, from our punishments, from our rewards. Right? So if this is the input given to the machine and the machine predicts the machine and the machine predicts that it is a mango and I give a feedback

[25:25] wrong it's an apple it notes it down and now when I feed the apple again to the now when I feed the apple again to the machine it says that it is an apple so take making the overall picture a quick recap that basically you know if we talk

[25:40] recap that basically you know if we talk about machines there are three types of learning where We have the input and output. And this supervised learning is capable of calculating error. That is output minus the input. Uh uh you know

[25:56] and what is the because why are we capable of calculating error in supervised learning? Because we have the actual output over here, right? And

[26:08] based on the actual output, is my machine predicting the correct result that if it is an apple, is it actually telling me an apple or not? Or is it telling me this is a cherry? So I can calculate my error. So the biggest

[26:23] advantage or the simplest way to learn is supervised learning where we are is supervised learning where we are capable of calculating the errors. Right? Here we have unsupervised learning where we do not know the

[26:37] output. It's just trying to do the clustering and association between the objects. And the other type is reinforcement learning that is not going to be part of this journey that's generally taken into deep learning

[26:51] concepts that where the machine learns from its you know uh punishments and rewards right and of course that we are capable of calculating error. Again I'm telling you in this particular course we are going to try to cover supervised and

[27:06] unsupervised learning uh you know concepts algorithms in detail. Now concepts algorithms in detail. Now moving ahead to supervised learning. There are two types of supervised learning. We have regression

[27:22] learning. We have regression and then we have classifification.

[27:41] classification. Now what is the difference between the two? Please try to understand. Under supervised learning, if the output, learning, if the output, it's all about output. If the output is

[27:55] it's all about output. If the output is numerical then this is known as regression and if the output is categorical then it is

[28:08] known as classification. Getting my point learners? What we are trying to achieve that what is going to be the temperature tomorrow. So if it gives me that tomorrow it is

[28:23] So if it gives me that tomorrow it is going to be 84° F or any other value going to be 84° F or any other value then this kind of algorithm is regression. But if I want to predict whether the temperature is going to be

[28:35] whether the temperature is going to be cold or hot this is known as categorical data. Clear? So the regression works on numeric and classification works on variable and

[28:49] categoric. Two types of numerical data. One is Two types of numerical data. One is discrete and other one is discrete and other one is continuous. Do we understand that?

[29:01] And if we talk about categorical data, do we understand nominal and ordinal data? This is what is covered in data science class. So basically you know data types are divided as qualitative and

[29:15] quantitative very very important when you um divide the data as qualitative it's categorical order data is something you know like rating ranking they come under order feedbacks of uh like movies nominal is

[29:33] that there is no order as such the color of the eyes or nationality and quantitative is numerical values continuous which can be divided divided such as distance, salary, price and something which cannot be uh divided is

[29:47] example cats etc. So now I hope the concept of regression and classification is clear to everybody right. So uh making your concept more clear that making your concept more clear that regression is something when the task of

[30:01] predicting a continuous uh you know quantity that I want to predict the price of the house that price of the house in 2014 house in 2014 was this much. Then in 2024 it is this

[30:15] was this much. Then in 2024 it is this much. Right? And then what will happen? What will be the price in 2034? Clear? So since price is a numerical quantity it comes under regression and

[30:30] classification as I told you that if I want to separate whether the male is a spam or not that comes under a classification problem. Clear? So now we

[30:42] can we will begin our journey in this machine learning through supervised learning. First we will try to complete algorithms which are like simple linear regression, multiple linear regression,

[30:55] polomial, support vector, decision tree, random forest. We are not covering neural network. All the others will be covered. Similarly in classification we will cover logistic k nearest neighbor support vector machine nave bias

[31:09] decision tree random forest. Again neural networks are not covered. Yeah. So with this uh we come to the end of the introduction and if you will now the introduction and if you will now look at your uh slide the ebooks that is

[31:24] lesson number two. Now we can begin with lesson number two. So just have prepared the foundation for that. So let me show you the lesson number two. All right. So we are now starting with this. I hope now you would be able to locate this

[31:40] particular file in your LMS in your material. Please look at that. So analyze the distinctions and applications of machine learning, deep through the real world examples of various technical applications.

[31:55] Differentiate among various machine learning models and explore each model learns from data to predict outcomes. Explore Python libraries for effective data manipulation, visualization, implementation and machine learning

[32:09] algorithm. So the business scenario says that ABC is an e-commerce company which is struggling with a surge in fraudulent transactions on its website. The manual review process for transaction has caused delays in order processing and

[32:23] led to negative customer experience. To address this, ABC will use machine learning algorithms to detect realtime fraudinal transactions. So machine learning algorithms are capable for detecting the fraud transaction and

[32:38] these algorithms will be integrated into the company's transaction processing systems to flag suspicious transactions and prevent fraud. Additionally, the company will use these algorithm to predict the customer behavior on the

[32:51] past purchase history thereby improving the recommendation engine's performance. Now, everybody is using mixture. Everybody wants the best. You know, the more you learn, the best output you get. And when I say best output, you want the

[33:06] prediction to be highly accurate. So when your report goes to a machine learning or a AI machine, it it has to give accurate result that whether you have whether you are you know that predicting that you know it should be

[33:22] accurate enough to predict that you know yes you know your you are you know uh cap you have a tendency to have cancer or not you know so no so not only one

[33:34] technique will be used it will try to use mixture of techniques to get the use mixture of techniques to get the best results. Got it? Now, so now are we clear? What is machine learning? So, machine learning is a subset of AI that

[33:48] assist systems to learn and improve automatically from the experience without being explicitly programmed. Arthur Samuel coined the term machine

[34:00] learning in 1959. It enables programs to learn automatically making computers more intelligent without human intervention.

[34:12] But human feedback is extremely extremely important because see ultimately machines are not genius. We have to tell them that this is you know have to tell them that this is you know a an acceptable result or not. So who is

[34:26] known as the father of machine learning? It's Arthur Samuel and he coined the It's Arthur Samuel and he coined the term machine learning in 1959. Father of term machine learning in 1959. Father of AI that's John Mcathi in 1956.

[34:41] AI that's John Mcathi in 1956. Yes, John Mcathi in 1956. Are we now more clear what is the difference between the traditional approach where we had the data and where we were programming explicitly the

[34:54] output right and the machine learning approach based on the algorithm based on approach based on the algorithm based on the data the patterns it understands it predicts the output that is why statistical techniques algorithms are

[35:08] the foundation for machine learning it automatically learns the features and automatically learns the features and reduces the need of manual featuring clear. It handles complex

[35:21] and unstructured data such as images, text, audio without requiring extensive pre-processing. Performance improves with more data and learning iterations

[35:33] with more data and learning iterations enhancing accuracy and generalization. Right? And now do we understand this ven diagram also that machine learning deep learning AI are often used interchangeably. So AI encompass

[35:48] intelligence and machines. So self-driving cars are you know AI all the robotics come under the category of AI but machine learning Amazon Alexa where we are giving it specific instructions and it gives us output. And

[36:03] when I talk about deep learning, deep learning involves neural network which is going to be your next stage after uh you know machine learning to understand how neural network uh you know work, how are they capable of understanding

[36:18] complex pattern recognition such as recognizing patterns in images, speech, recognizing patterns in images, speech, text etc. So where are what are the examples of machine learning in a chess game between a computer and a person?

[36:32] Now why do you think that why is this chess game always coming into the chess game always coming into the picture? Why do you think is the chess game always coming into the picture? Well, when we talk about human beings, a

[36:46] person who plays uh chess well is said to be intelligent. It's an intelligent game. Agreed. It's said to be intelligent. And even if the results go wrong, there's no harm. There are no catastrophic results. You

[36:59] know even if the person is winning and says the machine is winning it is not that harmful. Yeah. So that is why you know that that is was one of the way where AI exploded in you know intelligent gaming systems theorem

[37:14] intelligent gaming systems theorem solving that is why chess alph a little history of AI. So in a chess game

[37:27] uh between a computer and a person, the computer uses AI to analyze the game, computer uses AI to analyze the game, predict moves, decide its decision. AI uses machine learning to figure out the opponent is a beginner, intermediate,

[37:40] opponent is a beginner, intermediate, and an advanced level. How the AI uses machine learning to identify whether you are a beginner, intermediate or an advanced player? By predicting our moves, you know, based on our moves, it

[37:53] will immediately judge, you know, immediately judge that you are a beginner or a intermediate or an advanced level, right? And you might be advanced level, right? And you might be playing a lot of games where there is AI

[38:07] and other, you know, graphics PE games over there, you know, especially the young generation, right? Well, I don't, but you can, you know, how smartly, you know, they are capable of hiding things and they become smarter. The level

[38:20] changes you know as you are also becoming smart the level of the game becomes smarter right we all observe that so AI decides its next move against that so AI decides its next move against the opponent using a complex neural

[38:35] network that learns various features patterns from the data right then of course many applications are there we see machine learning all around us in spam filtering spam filtering

[38:49] actually use Live based classifier, social media analysis, customer service, now a lot of available online sentiment analysis. How does the

[39:02] sentiment analysis happen? Anybody who has an idea based on the words and especially the emojis. Is it a smiley? Is it a angry face? Is it a sad

[39:14] face? Crying face. All these things are taken into account. So these processes allow computers to learn patterns from the data, make predictions, predict outcomes, classify target feature and improve performance. So that's what uh

[39:31] we want machine learning that they help to predict the outcome, what is going to be the price of the house or any other thing after 10 years down the line. classify target features based on the similarities and improve the overall

[39:45] performance of the system. And we have seen is there a very very strong relationship between the data and the output. Yes, the amount of data if it is more of course the output or the quality or the prediction also increases

[40:02] where the red line is uh the quantity and if if it is high quality data we get better results better insights and better predictions of the output. So of course maintaining the quality, authenticity

[40:18] and removing the errors all these points have to be taken into account when we are talking about the data and the machine learning algorithms. Clear? Right. So when we talk about types of

[40:32] machine learning, ML can be divided into four main categories each characterized by its capacity to predict the our conditions or identify the patterns to produce outcomes such as what is supervised learning,

[40:47] unsupervised learning, semi-supervised learning and reinforcement learning. Now I think the distinction is clear. Supervised learning, we've done this. What are the three main points under supervised learning?

[41:02] First is the label data. What do we mean by label data? It will have the input as by label data? It will have the input as well as the output. Second point is splitting of the data into training and testing. Right? Generally training

[41:19] happens on most of the data and testing of on the rest of the data. And third is calculation of the error because in supervised learning we know the actual output also and whether it is predicting it right or wrong. Clear? And some

[41:37] commonly known uh you know uh supervised learning algorithms are linear regression, decision trees, logistic regression, support vector machines etc. So some examples of supervised learning are predicting temperature rise based on

[41:54] the yearly temperature trends. Predicting why supervised learning again predicting temperature rise it comes under regression because it's a numerical problem

[42:07] right numerical output predicting crop yield based on the seasonal crop quality changes. Again regression sorting waste based on the known waste items corresponding to the waste type. This types comes under classification or

[42:21] filtering. So under supervised learning calculation of error is also going to be an important aspect of machine learning because we have to be very clear that the output of machine learning will always not be 100% correct.

[42:37] Clear? Now this examples are these example making the picture more clear. And if we talk about unsupervised learning very talk about unsupervised learning very much used to I you know uh identify

[42:49] different parts of the object image segmentation for object detection. Identific identification of user groups based on commonalities identification of anomalies over geographical landscapes based on the data patterns. The

[43:04] unlabelled data set is provided to an unsupervised learning algorithm to discover hidden patterns and to recognize their relationship. So it's not that unsupervised learning is not important. It is equally important to

[43:19] important. It is equally important to analyze different features, different analyze different features, different relationships in the data. Right? Rather this is the algorithm which helps us to discover the hidden patterns and

[43:31] discover the hidden patterns and discover the relationships. Clear? And now coming on to the unsupervised learning example. It automatically groups the images based on the similarity that this is unlabelled data.

[43:45] Based on that it is capable of distinguishing the middleage people, old age people, the young, the infants, the teenage etc. Got it. And what is semi-supervised learning? As I've already told you, it

[44:00] uses a combination of small amount of label data. Sometimes, you know, the data is that's one of the constraints that we see in data is not completely labeled. A large amount of unlabelled data is used for training. Like

[44:14] supervised learning, it aims to learn from a function that can accurately predict the output variable from the input variable. It uses the unlabelled input to assist the learning process by collecting more information improving

[44:28] model generalization. So it falls between supervised and unsupervised between supervised and unsupervised learning. So suppose this is my raw data and I have partial that this is adults and these are kids then the machine

[44:43] automatically distinguishes between babies, teens and tween and also distinguishes between the senior citizens, youth and adults. Got it? So it automatically learns the correct

[44:58] So it automatically learns the correct uh you know uh groupings of the kids uh or the different people into teens uh tween and babies and the adult ones into into these category clear

[45:13] and another example of semi-supervised learning which we see it practically that was your question Mega that Google photos is popular example of semi-supervised learn learning that when a picture is taken it gets stored in the

[45:25] Google cloud platform and from slowly the Google tracks you know whose picture it is at what place it was taken so in various instances uploaders label images despite Google's lack of knowledge regarding image names its algorithm can

[45:40] identifying by analyzing visual features and shapes and colors and it does that it does a lot of lot for me it is able to it's able to identify my friends my family in which location I was And where and reinforcement learning is a type of

[45:55] machine learning where algorithms learn from the environment by performing actions and receiving either rewards or penalties as feedback. If the pro program finds correct solution, the interpreter rewards the algorithm. If

[46:09] the outcome is incorrect, the algorithm is penalized for incorrect predictions. It must I reiterate until it finds a better result. Right? So ultimately

[46:23] reinforcement learning involves an agent. It interacts with an environment. Learning from the rewards and states to choose from and then based on the output it gives the best action and if it is an error it learns it again. All right. So

[46:41] this is my input raw data based on the environment reward state and action. it's capable of detecting them separately. Okay. So the example is this type of learning is seen in YouTube

[46:56] recommendation where the user searches for a particular song. The program shows the list of available song. So when a user selects a specific song, the system trains itself to remember and deliver similar results for future s searches

[47:11] based on the user's interaction like views, shares etc. So this is the views, shares etc. So this is the concept on which recommendation systems So other examples of reinforcement

[47:24] learning are game where players can play with bots, autocorrect tools, search recommendation income uh engines, self-driving cars and then the Python packages that we would be doing for machine learning. We are aware about

[47:39] NumPy. It's a very powerful tool for numerical Python computing. Mattplot li for drawing data visualization pandas. So I hope you all are aware about numpy mattplot lip pandas but we will be working more on the scikitlearn uh uh

[47:54] you know uh file which consists of different algorithm but the pre-processing the other part is also being taken care before we feed in into being taken care before we feed in into the algorithm. So a quick recap.

[48:08] machine's ability to learn from the data and replicate human behavior. AI includes machine learning, deep learning, each with unique capabilities for simulating intelligence. There are

[48:20] supervised, unsupervised, semi-supervised, and reinforcement learning. Python packages are folders with modules that organize code for easy reuse and maintainance, improving the de development efficiency. Now clear.

[48:38] So now let's go in for a knowledge check. Question number one. Yes learners are you there? Which of the following best describes the machine following best describes the machine learning? A, B, C, and D? Question

[48:53] number two. Which example illustrates the use of machine learning to enhance customer experience in an e-commerce company? What distinguishes between deep learning, machine learning and AI? Yes,

[49:07] it is a subset of ML that uses multiple layers for complex pattern recognition such as recognizing patterns in images, speech and text. Right? So with this we come to the end of the very introduction and basics of

[49:24] machine learning. Yeah. So what I'm looking forward is Yeah. So what I'm looking forward is this is the overall uh picture of statistics that uh we have. The learners who have already done data

[49:39] science are aware about it. The ones who are not aware about it. The different types of statistics that we have is descriptive. Under descriptive we have measures of central tendency and measures of variability. Under measures

[49:51] of central tendency we have the mean, mode and median. Under measure of variability we have the range, variance and dispersion. And here we have the inferential statistics how we infer the results. So this is the important part

[50:07] that we are looking at that that includes confidence interval hypothesis testing. So you can do a lot of search on inferial statistics and uh this is on inferial statistics and uh this is what I am saying that uh statistical

[50:22] inference constructing confidence and intervals on population hypothesis testing that is what I'm looking and what is the advantages of these what is the advantages of these particular uh program because ultimately

[50:35] you know probability in data science and AI play a very very major role in understanding uncertain Certainty predicting outcomes how probable correct the output is even the LLM models the chart GPT is predicting on the

[50:52] probability okay this is the next word so if it is 70% above then let's predict the outward how do we model complex system enhancing AI and it is the system enhancing AI and it is the statistics and the probability together

[51:07] which help in exploratory data analysis and give meaningful full insights and features. So as we are going through this flow we understand we are going to start with supervised learning. What is supervised learning?

[51:23] Before supervised learning you know uh I would like you to cover we would uh you know the cover the basic concepts of machine learning which I have prepared through my PPT and then we will move on to what they have shared. Okay. So the

[51:38] first thing is regression. You know what are the two main algorithms which come are the two main algorithms which come under supervised learning? What is the difference between regression and classification?

[51:54] Regression happens when the output is numerical and classification happens when the output is categorical. All right. So I output is categorical. All right. So I start with my PPT again so that it helps

[52:07] and it gives you strong foundation clear concepts so that we can move along with that. So if we talk about the first knowledge check what is machine

[52:19] learning? Correct answer it is see that it is an autonomous acquisition of knowledge through the use through the use of computer programs. Second question, what is the key difference between supervised and unsupervised

[52:34] learning? A and B are both correct. What's the key A and B are both correct. What's the key benefit of using deep learning for task like rec? How do I u Okay, you've written yes. Okay. What's the key

[52:49] benefit of using deep learning for task like recognizing images? Yeah, they can learn from complex details from data on the own. That's the

[53:02] idea of deep learning. That's the use of neural network. Absolutely correct. All right. So now we start with the concepts of machine learning under supervised

[53:14] learning which are valid for regression and classification algorithms. Whenever you will say you know machine learning these are the basic questions that will be asked right. So if I talk about uh

[53:28] machine learning or supervised learning what are we trying to uh you know do in supervised learning what is the main aim what is our main objective of supervised learning if but prediction of the data. Okay, we would call it prediction of the

[53:45] data future based on given data. Right? We want to predict and is this prediction always correct or it can be wrong? Can be wrong. They can be errors. Right. And so they there should be a limit of accepting the errors or

[54:02] rejecting the result. Right? There should be some way of accepting and rejecting the results. We all understand this very in a conceptual subjective matter. Now let's try to understand it on the basis of mathematical functions.

[54:18] Right? So here we have the plus over here and here we have the minus over here. Right? This is my data set. This is my x-axis or and this is my yaxis.

[54:31] is my x-axis or and this is my yaxis. These are given as my data. Right? And what are we trying to do over here? We are going we are trying to predict the output. So what are we trying to predict the data that what is the value of this

[54:46] question mark? What is the value of this question mark? question mark? All right. And what is the value of this question mark? Right? Is this question mark a plus sign

[54:59] Right? Is this question mark a plus sign or a minus sign? Can you tell me what is this question mark? A plus or a minus sign? What about this? What about this? So this is my first question mark. Second question mark.

[55:12] Third question mark and fourth. So based on your observation, can you tell me what is this first? Do you think it is a plus or a minus sign? What does this particular data point

[55:25] represent? Is it a plus or a minus? Because this this particular data point is more close to the plus. So there are chances that this is going to be plus sign more chances. Yes, it can be negative also but we can say 70 to 75%

[55:41] or 90 to 95% chances are that this is going to be plus. What about the fourth one? This is negative. Now what about the second and the third one? What about the second?

[55:56] What about the second and the third? This is going to be a little difficult to say that this is going this can be plus this can be minus based on the way the neighbors I am selecting what will be my output agreed but but if

[56:12] technically now if I use a straight line function ma mathematically how do we use a straight line function that y is equal to mx + c so All the

[56:28] points which lie on the left side of this line they all are known as plus they all will come under the category of plus and the data points. Now all the

[56:41] points which are lying on the right side of the line they will be all negative because most of the points are minus or the red points. Can I say that all the points which are lying on the left side are plus and all the points which are

[56:57] lying on the right right side are negative. There are very few. Now this is there is only one plus sign and if you see all the points belong to the red you see all the points belong to the red red negative class one way I have to do

[57:11] something. But again the question is why this straight line that how do I decide the straight line? So one point to understand is that when data points are

[57:24] understand is that when data points are given that is known as a hypothesis given that is known as a hypothesis space represented by capital H and the space represented by capital H and the line is represented by small H

[57:41] line which is going to divide the data points into two different classes. So there can be more than one solutions to a problem or infinite

[57:53] solutions. I can draw infinite straight lines. But which one to accept? The line which will be accepted is going to be one with

[58:05] will be accepted is going to be one with gives me the minimum error. Right? The error word will be different for different algorithms. But I will accept that straight line which will give me the minimum error or I can say

[58:21] the maximum accuracy. Clear? We will calculate as I told you in supervised learning we keep the track how we can have maximum accuracy and minimum error.

[58:36] So what is the capital H over here? Hypothesis space is the set of all possible legal hypothesis. This is the set from which the machine learning algorithm would determine the best possible only one which would describe

[58:48] possible only one which would describe the target function and small h is a hypothesis function that best describes the target in supervised learning the target in supervised learning algorithm. So now technically what does

[59:01] supervised learning mean that the hypothesis or the small edge there can be several small edge that an algorithm would come up depends upon the data also depends upon the restrictions bias that we have imposed on the data. So next

[59:16] thing is we need to find out mathematical functions which give mathematical functions which give relationships between the data points. relationships between the data points. Right? So basically now the idea is that

[59:31] Right? So basically now the idea is that we want to find a mathematical function we want to find a mathematical function where you know uh we want the error to where you know uh we want the error to be minimum or the result to have maximum

[59:45] accuracy. Again I'm repeating we want to find a simple mathematical function find a simple mathematical function which gives me minimum error or maximum

[59:57] accuracy. Clear? Thank you at Thank you for understanding. Thank you learners. So now what are the different stages that we are looking at? Now suppose data points can be of any kind. So if this is

[1:00:11] my data points over here and if I want to solve it with with a simple linear to solve it with with a simple linear function see straight line function is a

[1:00:23] simple mathematical form uh you know calculation that y is equal to mx + c. calculation that y is equal to mx + c. Okay. So when y is equal to mx + c

[1:00:37] Okay. So when y is equal to mx + c right and if these are my data points now how do I calculate my error? This is going to be actual minus the predicted going to be actual minus the predicted it will have very high error and this

[1:00:50] particular concept is known as underfitting. Do you think that this straight line is consistent covering all the data points? No. Right? So the simple straight line equation or function is not capable of

[1:01:07] covering all the data points that is known as underfitting. known as underfitting. Other way is that if I draw a function which passes through all the data points so it makes it a very very highly

[1:01:21] complicated mathematical function with high degree. Right? So it makes a complicated function with highderee mathematical function that also we don't

[1:01:33] want. Why we don't want because if the data point is out of all the data points then this is going to give me my maximum error. So we want a mathematical

[1:01:46] function which is simple and which covers all my data points. So something like this exponential right and this gives me a good fit or a good balance

[1:02:00] now clear. So AJ says no. So the model even fails to predict labels what it learned right it will it will predict the labels but with very low accuracy. The error will be high. So we we will not accept models

[1:02:14] be high. So we we will not accept models with high errors or low accuracy. So overfitting is a situation. How do I know the error is high? Overfitting. So know the error is high? Overfitting. So one way is that error is equal to if I

[1:02:30] talk in technical terms. So how do we know that it it is underfitting overfitting. So that's what I'm telling you. Generally the error is I'm telling you. Generally the error is the formula for error is

[1:02:45] the formula for error is bias²ared What is it equal to? It is bias squared plus variance. But the equation is in

[1:03:00] good balance not linear. Yeah it it it's not necessary that it has to be a linear equation. No, we want an equation which is simple and nice. So how do we know?

[1:03:12] is simple and nice. So how do we know? So now if I say that error is equal to So now if I say that error is equal to bias squared plus variance over here right and this is you know so how do I know whether it's overfitting or

[1:03:25] underfitting it's not about the straight line but when I have variance if if the variance of the data goes very high then I know that it is an overfitting model I know that it is an overfitting model if the bias of my data goes very high I

[1:03:40] know it's an underfitting model and to know it's an underfitting model and to keep my error low I want the bias as well as variance to be low. Getting my point

[1:03:58] variance? A very very typical example. A bias means how far are we away from the bias means how far are we away from the original data points. So bias means we are close to the center. Variance what does variance mean? The spread of the

[1:04:14] does variance mean? The spread of the data is also low. So this is the ideal data is also low. So this is the ideal situation we always want the data points situation we always want the data points to be to be in. This is where we will

[1:04:26] say okay a good fit has been achieved where we have the low bias and low variance. Okay. And when we have the low bias and

[1:04:38] when we have the high variance low bias means that the high variance low bias means that the data points are near to my actual target points but spread out. So what is the case happens over here? This is known as

[1:04:54] case happens over here? This is known as the overfitting case. And what is high bias and low variance that over here these are my data points

[1:05:07] and this is high bias. So this is underfitting situation. So in whole of machine learning we don't want underfitting we don't want

[1:05:19] overfitting. Of course this is a worst case where the bias is more variance is case where the bias is more variance is both. The idea is to have low bias as both. The idea is to have low bias as well as low variance. Clear?

[1:05:31] Okay. Now pre I'll explain the previous example with this explanation after this punit I think. So this will make things more clear. So these are my data points over here a linear model is not a good fit. Underfit

[1:05:47] that means over here what is more my bias is more. What is bias? That my data points are very far away from the actual points. Bias means we are far away from the

[1:06:04] Bias means we are far away from the actual points. Right? This is high bias. And what is variance? Variance is when the spread of the data is more. Right? the spread of the data is more. Right? And when spread of the data is more or

[1:06:19] mathematical function is too complex then it is overfitting. When my mathematical function is too simple, it is underfitting and when it is a is underfitting and when it is a balance, it gives me the right data.

[1:06:35] balance, it gives me the right data. So now when I talk about complexity, now when I say overfitting means more complexity that generally linear models give me underfitting, right? We start

[1:06:50] with simple models but generally they do underfitting. Then we will move on to nonlinear models, support vector machines, treebased models, deep learning models. Why are we in doing this? Because it they give me more

[1:07:05] results, better results, more accurate result, less of error. But what is the cost that I my model has become complicated? losing the interpretability

[1:07:19] of the model. What do I mean by interpretability? What do I mean by interpretability? That if this is my input, right? How do I know that this is going to be my output? That's possible in linear model

[1:07:32] output? That's possible in linear model because I know y is equal to mx + c that if this is my input, I will get this output otherwise I will not clear. And to make it more clear you know the overall picture that I was talking about

[1:07:47] please look at this slide. So ultimately where is the trade-off? Where are we fighting in the whole of machine learning models? If life was so easier wouldn't have the problems be solved by now but no there is still a

[1:08:03] trade tradeoff which is going. So how will I know it is underfitting though it is showing in diagram is there any yes yes they are mathematical way but first yes they are mathematical way but first let's get the grasp of the thing okay

[1:08:16] that ultimately the idea is to get highest accuracy right and interpretability is also important why because we want to see why am I getting

[1:08:29] this output so we always start with linear regression or logistic regression regression problems. That is what we do. That is what we are going to do in our That is what we are going to do in our uh whole of this journey. And why uh

[1:08:43] regression linear regression? Because they are linear and smooth, well-defined relationship, easier to compute. And as we move along this journey, decision trees provide good accuracy with high interpretability.

[1:08:57] KN&N clustering. Clustering and KN&N algorithm are mid-range. Interpretability is okay and accuracy is also okay. But if I want really good accurate results, that is why new algorithms, complex algorithms were

[1:09:12] built with kernel based approach for support vector machines onsembled support vector machines onsembled methods and of course neural networks. Now clear is this slide getting clear to

[1:09:26] everybody? That is why complicated neural networks are capable of solving nonlinear relationships, non smooth nonlinear relationships, non smooth relationships and long computation time.

[1:09:40] relationships and long computation time. Clear? Okay. If I want interpretability of the models then I might go in for decision trees but I will have to compromise on my accuracy. But if I am not interested in interpretability that

[1:09:53] we are not we always want best result that is why neural networks deep learning have taken the market. It is we as users who will decide. What is the difference between accuracy and interpretability? Interpretability is

[1:10:08] how am I going to find out the relationship of output given a particular input. So if I know a mathematical equation okay this is mathematical equation okay this is related with beta x1 beta_2 x1 I know

[1:10:21] why am I getting the result but if it is some integraable model like neural networks working in layers in different layers some in some layer differentiation integration addition subtraction is happening I will not be

[1:10:35] able to interpret the result but it is giving me highest result then of course I will use it if I have a very mathematical complication complicated mathematical complication complicated ated system. See,

[1:10:48] now try to understand the whole story again. Let me just uh repeat the story because this is very important in the whole of machine learning and these concepts are valid for deep learning also. Okay, the story says that we want

[1:11:01] to predict the output or the data. You're clear. The first point says we want to predict the output of the data with minimum error or maximum accuracy.

[1:11:14] with minimum error or maximum accuracy. First two points are clear. accuracy and minimum error. That point is clear

[1:11:27] right. And which is the simplest mathematical function that we have? We always talk about a straight line. If that straight line is giving me minimum error, is it so is it that only we will always apply one algorithm or is

[1:11:42] it a different algorithms or a different hit and trial methods that we have to apply? It is different hit and trial methods algorithm that we will try to get the output. It's not just you know oh god regression and decision tree is

[1:11:55] working okay this is the end of it. No, you might go in for advanced algorithms and that's where the research is going. Why do you think that the problems are not solved? Because every time every algorithm will have its own pros and

[1:12:09] cons and the research is going at each NLE level and it's still going, you know, we are just trying to improve on the algorithms every time. Why? Because we still see the LLMs, the chat GPT giving us wrong

[1:12:25] answers, wrong predictions. So there is still a lot of scope of inc improvement you know where we have to work clear. So life was if the thing was so easy I think so by now everything would

[1:12:42] have been solved by AI and machine learning algorithm. No. Why is it not learning algorithm. No. Why is it not simple? First data is variable uncertain. It keeps on changing. Secondly every algorithm has its pros

[1:12:56] and cons. So the errors the accuracy the the the my requirements upon the product you know on the data or the algorithm keeps on changing and that is why this change happens right. So they are pilotra of algorithms

[1:13:13] which have been built. As in machine learning we will always start with linear regression algorithms because of the simple mathematical function and high interpretability but they have minimum accuracy. But as

[1:13:28] we move along with the journey of decision trees, clustering, kernel decision trees, clustering, kernel based, we have we want maximum accuracy but the trade-off is that we decrease on the interpretability of the model.

[1:13:43] the interpretability of the model. Right? So as I have been telling you splitting the data for machine learning in supervised learning, we always and in supervised learning, we always and always want to select the data randomly.

[1:13:56] always want to select the data randomly. So there is a function train test_split that we would be using to split the data. Train underscore test_plit data. Train underscore test_plit which is part of the skarn library. This

[1:14:11] is my data and we will be randomly selecting it and dividing into training and testing data. Now clear and most of you you see this Now clear and most of you you see this is around 70 to 80% of the data

[1:14:33] and this is around 30 to 20% of the data. interpretability is more important as we go through complex problems. Not really.

[1:14:45] Not really. you know ultimately punit uh not really because we are always interested in the final result it's something like as I give my example suppose you know you have a magician and you know the magician turns a pigeon

[1:15:01] into a flower are we interested how does how does it does or are we interested in how does it does or are we interested in the output as a fl okay so we are always interested in the output got it now better yeah so if

[1:15:15] logist IC regression is having minimum accuracy accuracy then it is a no no

[1:15:27] doesn't mean that it's a wrong prediction logistic regression might not give you the best results for the prediction and maybe using a neural network would give you you know it's something like n uh

[1:15:43] you know using logistic regression is giving you 70% of accuracy and neural giving you 70% of accuracy and neural network is giving you 99 97%. So that's the difference but of course neural networks are better that means

[1:15:56] neural networks are better that means right got it. So yeah there are a lot of have to understand it in the correct sense I understand I'm giving you time for doing that. Yeah. Now coming on to the part that you

[1:16:12] all are interested that ma'am what is the mathematical equation you know how do we know the mathematical term which is used to see the output. Now first of all we are going to

[1:16:28] Now first of all we are going to calculate error right in regression the error is known as mean square error. Now let's start understanding this. So we let's start understanding this. So we have this mean square error. Okay the

[1:16:41] error name is that that I'll explain you technically when we move on to regression. Now in supervised learning whether it's classification or regression we we know the data is divided into training and testing. Now

[1:16:54] tell me which error is more important. Now let me put it in this way that you go to a class you know and you are pursuing some course you know you you

[1:17:06] you do errors you are learning and you do errors while training right and still you don't perform well in the test and there is somebody who doesn't attend

[1:17:18] even one single class and performs well in the test. So which error is more important? It is the training error which is important or the testing error among the two of course both are important but among the two which one is

[1:17:32] important. So what is more important to have the less? The training error is less but the testing error is very high. Is it a good thing?

[1:17:47] thing? It's a even big failure. the training error important or the testing error important?

[1:18:02] Of course the final test if you're not performing well on the final day of the performing well on the final day of the test then the whole training is useless. Agreed? So when my training MSE is high, of

[1:18:15] course I can reduce my training by giving in more data, more data and my giving in more data, more data and my testing MSE is less, what is this case? This is the case of underfitting. Of course, the training error should not be

[1:18:30] course, the training error should not be high than the testing MSE. The model is high than the testing MSE. The model is too simple for it to solve and then I will say the model is underfitting. Now clear that was the question I think. So

[1:18:44] so now clear mathematically. So this is how I will I will judge based on my training and uh and testing that my model is underfitting. So I will now not use a straight line to solve my data points. I will not use okay. Other is my

[1:19:02] points. I will not use okay. Other is my training MSE is zero. Excellent. But my testing MSE is so high. absolutely not acceptable case. So this is the case of overfitting which is high variance.

[1:19:17] of overfitting which is high variance. And what do you think the data or the model goes through? Is it more of underfitting problem or an overfitting underfitting problem or an overfitting problem? What do you think most of the

[1:19:29] models and machine learning face which kind of problem? Reducing the training kind of problem? Reducing the training error is easy but failing on the testing is a total failure and that is what we

[1:19:42] will will we will look at most of the algorithms that how to prevent overfitting in the models. That is why I was telling you these concepts are very very common to machine learning deep learning models. So are you getting a

[1:19:57] grasp of it and I don't straight away start with regression. Come on, let's start doing it. Other things you have to have good foundation of machine learning concepts so that you know you understand other materials the indepth knowledge

[1:20:12] how we accept and reject the particular model. Training and testing that we will decide because when the data is there we will split the data into training and testing and then we

[1:20:25] separately and the testing error separately. Okay. we will have the suppose I said 1,000 rows 700 rows will be used for training then we will have

[1:20:37] uh 300 rows used for testing then calculate the errors for training and testing separately and then decide whether it's underfitting or overfitting

[1:20:49] so what will be the best case where my training error is also low testing error training error is also low testing error is also low or almost equal to each is also low or almost equal to each other so Again looking at the concept

[1:21:02] other so Again looking at the concept xaxis we have the predictive error. So over here on the x-axis this is my underfitting

[1:21:16] x-axis this is my underfitting over here and if what is overfitting that when I start with my model simple model my errors are high both the bias and the both the test and as well as the training as my training increases the

[1:21:34] training error or the model complexity increases the training error decreases but the test error will dip and then increase because of high variance. So we want to find out a model. How do I decide which model is best for my data

[1:21:51] decide which model is best for my data where the training and the testing error where the training and the testing error are minimum or close to each other. So this is what I'm trying to explain you. Please try to understand that as the

[1:22:04] model complexity is less here we have the bias more as the complexity increases the bias decreases. When the interpretability is high the variance is low and as the complexity increases the variance increases but we are looking at

[1:22:21] a point where these two inter intersect to find the optimal model complexity. The variance is the spread of the data. Variance is a statistical term which

[1:22:35] means the spread of the data. So as my model complexity is going to increase the the the complexity as the complexity of the model increases the variance or the spread of the data also increases. Got it? That's the relationship. Again I

[1:22:52] am telling you we are trying to create a graph between error or accuracy is going to be opposite of this and this is my model complexity. We always start with minimum complex model. This is the case of underfitting.

[1:23:08] Agreed. When my model is simple but my bias is When my model is simple but my bias is high. As the complexity of the model increases, my bias decreases but my variance

[1:23:23] increases. So I don't want the overfitting situation or the underfitting situation. I am looking at a situation where my error is minimum. a situation where my error is minimum. So the test error is U-shaped in curve

[1:23:37] and where the value is minimum that becomes my optimal model complexity. Now clear to everyone? Okay. Now uh Okay. Now since the uh

[1:23:50] topic has been taken now let's do one thing. If you have downloaded the uh ebooks material I want everybody to make a folder in the desktop and move under lesson number three.

[1:24:09] Something like this. See what I would see is let me share my in work and data break how do I understand that so once you are into now

[1:24:22] we've starting with lesson number three so do you see this 3.1 and 3.2 to learners. So, how do you go about opening it? Open the Jupyita notebook

[1:24:34] opening it? Open the Jupyita notebook environments, install and open the file along with me. So, chapter number three, we have to start with lesson three. Lesson three supervised learning regression and its application.

[1:24:51] Yeah. Are we all able to open this? Let me guide them that how do we go about it? See the prerequisite for the course I've been telling it is Python. So we always we like there are lot of tools like PyCharm, VS Code. You can do it

[1:25:07] through uh other tools but generally we use Anaconda Navigator download. Please go ahead and go to the Anaconda, fill in your uh like

[1:25:21] the details, the email id and I want everybody to download the Anaconda everybody to download the Anaconda right and uh based on your uh system whether it's Mac, Linux or Windows and go in for a full distribution. Don't go

[1:25:37] for mini Google Collab. If you're aware about Google Collab. If you're aware about Google Collab, available you know which helps you to run the code

[1:25:52] you know which helps you to run the code in the Jupy Jupiter kind of environment. Let me show it to you. Anaconda distribution. Go and download it. distribution. Go and download it. Install it. Or the other one is go to

[1:26:05] Google Collab over here. Just login through your Google account. That's the another uh way to go about That's the another uh way to go about it. And from here I will upload the

[1:26:19] file. Which file? The file which I have already downloaded on my ebooks that's already downloaded on my ebooks that's there on my desktop. Machine learning there on my desktop. Machine learning third chapter 3.1

[1:26:38] Yeah. So there are two three methods whichever one you are comfortable with that's not an issue. If you clear you're comfortable anush on VS code that's not an issue. Got it? Now everybody is there with me now?

[1:26:52] So this is one of the very safest tool. You don't need to uh you know download the Anaconda on your desktop through the Google Collab through your good internet connection. You will be able to run the code. Yeah. So this particular file is

[1:27:06] uh you know basically if I talk about 3.1 again I'm repeating go on to your reference material please download the ebooks the Jupiter notebooks

[1:27:18] and then try to open. So if we talk about if we are back to 3.1 let's let's quickly go through the file that file is completely theoretical okay nothing great in that file it talks about what supervised learning is and what are the

[1:27:34] two types of algorithms in supervised learning it's regression and classification what is the difference between regression and classification and supervised learning in regression the output is always

[1:27:47] in regression the output is always numeric IC categorical. So when the target variable is categorical we do classification. So this example we are very clear. So when

[1:28:01] it's predicting numerical value right that is regression. When we are predicting a categorical outcome determining the weather tomorrow it is going to be hot or cold. visualization. The same thermometer scale divided into

[1:28:16] two colored regions cold and red for hot with a threshold separating is classification. And if you look at the applications of supervised learning, do you think HR people use machine learning to um

[1:28:32] uh identify the different job profiles uh shortlisting of the rums? HR is also using machine learning now. Yeah. In finance, what is the use of machine learning and predicting of fraudulent data is are you you know ready for that

[1:28:48] data is are you you know ready for that loan approval or not. Right. And um this is similar to how to how a credit card company determines your credit worthiness issuing before issuing a card. Then emails this is about spam

[1:29:03] filtering whether it's a spam or not. manufacturing. Anybody in manufacturing? So in manufacturing, supervised learning is also used to inspect the quality, classify the products into different grades. For example, a factory might use

[1:29:17] a machine learning model to check for defects in products and ensure they meet quality standards. Much like the quality control inspector, maritime industry,

[1:29:30] it helps in forecasting the historical events, weather condition. So precautionary accident it's capable of predicting any kind of you know weather which might get wrong also which we have seen and supervised learning techniques

[1:29:43] like regression model can be used to predict tidal currents forecast demand and supply reducing inventory losses think of it as how weather forecast predicts rain based on past weather patterns broad predictions we

[1:29:59] use this in agriculture field too I'm into automating parking so under supervised algorithm you know there are lot of algorithms that we will begin tomorrow again as you understand my style we'll start with regression then

[1:30:13] style we'll start with regression then linear regression logistic nave bias k&n under classification they all come under classification so under linear regression we would be working on multiple linear regression and before we

[1:30:27] move on to classification there are lot of concepts that we will study such as excuse Excuse me. Ross validation, regularization,

[1:30:39] me. Ross validation, regularization, hyperparameter tuning, skarn pipelines. So there is lot more to be explored. Tomorrow we will be concentrating on linear regression and its concepts. Tomorrow, right? And linear regression

[1:30:53] is now very clear where the output is numerical. Then like predicting housing numerical. Then like predicting housing pricing, we will use regression. After doing supervised uh regression in supervised learning then we will move on

[1:31:06] to classification algorithm. Under classification pelra of algorithms that need to be done initially we'll start with logistic regression, knive bias, KN&N decision trees, random forest and

[1:31:20] support vector machines. So it's a long journey. Tomorrow we'll do linear regression. Next weekend we'll we will be doing classification and then we will move on to the next stages of ensemble learning. then it is uh

[1:31:36] unsupervised learning right so I think so you all are excited about this journey so it's a long journey I don't want to burden you today with all the concepts so let's go slow and steady that's my rule slow and steady wins the

[1:31:50] race but I hope you got the crux of today's class right so this is 3.1 today's class right so this is 3.1 right and um we'll be beginning with uh 32 we started with the very first session of machine learning where we

[1:32:04] what are the different types of learning techniques, what are the different types of m machine learning techniques, supervised learning, unsupervised learning and reinforcement learning. What is supervised learning? Supervised

[1:32:18] learning means it consists of label data which consists of input and output. We split calculate the errors. The two main algorithms under supervised learning are regression and classification.

[1:32:33] Then we stood unsupervised learning. Unsupervised learning consists of unsupervised learning consist of unlabelled data where based on the similarity of the data the groups and clusters are formed. So two main

[1:32:46] algorithms are clustering and association. And last but not the least is reinforcement learning that based on the feedback positive and negative the machine learns. Are we good to go? Everybody is clear

[1:33:00] with these concepts of machine learning. The basic concepts and then we started with the concepts of overfitting, underfitting and a good fit. Yes, learners. Now you will tell me what do we mean by underfitting

[1:33:16] what do we mean by underfitting fitting in machine learning. Very very important question. It will definitely be asked if you say that you know the concept of underfitting and overfitting. Yes learners.

[1:33:29] Can you tell me what do we mean by underfitting and overfitting? Underfitting is the case where the error is high. What error is high? Bias is high. We are very far from the actual data point. But we talk about simple

[1:33:45] mathematical function. What is the advantage of simple mathematical function? That it gives us more interpretability of the model. What do we mean by more interpretability? that it gives the relationship between the

[1:33:58] output and if we talk about overfitting again the error becomes high because the again the error becomes high because the variance goes very high right and definitely it makes more and more complex so we

[1:34:12] don't want a algorithm to be or model to be underfitted or overfitted we want a good balance fit that is how do we judge that that the training error and the testing error should be similar Right. So today we will start with the

[1:34:28] first uh you know learning technique that is supervised learning technique and the regression model. Are we good to go? So as now you are getting more familiar with my style you know first we will try to cover the concepts and then

[1:34:42] move on to the practical aspect because unless and until you happening how is how are we going to interpret the model there is no point moving on to the code but definitely we'll move on to the code but let's

[1:34:57] start with the concept of regression so here we go with the regression so so here we go with the regression so what is regression now if I ask ask you what is regression? Tell me. So when we're talking about regression learners,

[1:35:13] please be clear with this concept that always the target variable or the always the target variable or the dependent variable are all numeric values or continuous value. Don't say the term data. Right? See

[1:35:30] everything is data. But if you're not very spec technically you go very wrong right? So you have to understand now data we're talking about structured data which cons in tabular form it consists of input as

[1:35:47] in tabular form it consists of input as well as output values right whatever are the input or the output value x over here is referring to your independent here is referring to your independent variables right but whenever we are

[1:36:00] choosing any model or algorithm we always and always take into consideration the dependent depent variable. So it is very important that the dependent variable has to be numerical or continuous right. So we are

[1:36:16] numerical or continuous right. So we are trying to find out so whole so basically what are we trying to find out? We are trying to find out the relationship between the independent variable based on the output. Clear?

[1:36:30] on the output. Clear? So let's try to break the term linear and regression. linear. We understand from simple mathematics that anything of degree one any function which has value one is set to be linear. So this means

[1:36:47] that progressing from one stage to another in a single series of steps sequential extending along a straight line or nearly a straight line. So we

[1:37:00] all understand the term linear right and as very rightly said by Nanda regression is a statistical terminology which is used to find out the relationship between one dependent variable. We always have one output. If

[1:37:18] the output is categorical, we will go in for classification algorithm. And [snorts] if the outputs are continuous or numerical, we will go in for linear

[1:37:30] or not linear but regression algorithms. So moving ahead. Thus a linear rel So moving ahead. Thus a linear rel regression means a linear approach for so when I combine the two terms linear regression it means that the linear

[1:37:45] approach for modeling the relationship between the dependent and independent quantitative variables. it will plot a straight line because that's the simplest linear uh approach as a best fit along the data points to predict the

[1:38:03] fit along the data points to predict the target value. Right? So what are the different types of linear regression available to us? So a simple linear available to us? So a simple linear regression is with one depend obviously

[1:38:16] the dependent variable is always one that's the target or the output but if the independent variable is one it is simple linear regression and it is always represented by a straight line. So beta kn is the intercept and this

[1:38:33] refers to the slope. So this refers to beta kn plus beta 1 x1. getting my point? And then we have multiple linear

[1:38:45] And then we have multiple linear regression that is over here when independent variable then this straight line turns into a hyper plane. So line in higher dimension two dimension becomes a plane and in higher dimension

[1:39:01] becomes a plane and in higher dimension it becomes a hyper plane. Getting my point? Are you all getting the concept mathematical equation along with the graphical view and of course with the

[1:39:14] code try to I'm trying to explain you from all aspects all right and when I from all aspects all right and when I say polomial sure I can do that right over here we are talking about linear regression linear means something of

[1:39:31] degree one so do you see all the variables have all the coefficients have the degree B1 do you see this

[1:39:43] regression. So the term linear regression means we are trying to find out relationship between the independent uh variables with the dependent uh variables with the dependent variable. So a simple linear regression

[1:39:56] variable. So a simple linear regression is beta KN plus beta 1 X1. Clear? That is when we have one input uh variable and one output variable.

[1:40:08] Mathematically it is represented by a straight line where beta kn represents straight line where beta kn represents the intercept of the line and sorry the beta kn represents the intercept and beta 1 represents the slope of the line.

[1:40:24] Clear? Have multiple linear regression. Graphically a straight line now becomes a hyper plane that is a multiple linear regression. Of course you know generally practically we will have more than one

[1:40:39] independent variables with output variables. And then we have polomial variables. And then we have polomial linear regression that is y is equal to linear regression that is y is equal to beta beta 1 x1 beta 1 x² and q. Clear

[1:40:55] beta beta 1 x1 beta 1 x² and q. Clear now? So now let's try to understand it from now the same thing from the machine learning point of view. So in simple terms it is a finding a ba best straight line

[1:41:10] it is a finding a ba best straight line fitting the given data set. So in simple fitting the given data set. So in simple terms find a simple straight line which tries to find the relationship between the independent and the dependent

[1:41:23] variable and best linear relationship. How do I decide the term best? Something with minimum error or maximum accuracy. Right? Something with minimum error or

[1:41:36] Right? Something with minimum error or maximum accuracy is taken in terms of simple terms. And if I talk about technical terms, it is a supervised machine learning algorithm that finds the best fit relationship on the given

[1:41:51] data set between independent and dependent variable. almost the same dependent variable. almost the same thing but the basic concept or the thing but the basic concept or the algorithm it works on it is OS the

[1:42:04] algorithm it works on it is OS the ordinary le square method also known as the sum of the square residuals clear so to explain these terms graphically mathematically I want everybody to

[1:42:18] concentrate here on the graph or the slide share so this is an equation of a slide share so this is an equation of a straight line. What is beta kn? Beta kn

[1:42:30] is the intercept. What do I mean by intercept? What is the value of y when x is = to zero? Are you all getting this point? What does the term intercept

[1:42:42] point? What does the term intercept mean? That what is the value of y when x mean? That what is the value of y when x is equal to zero? Clear? Is this point getting clear to everyone? And if I talk about beta 1, beta 1

[1:42:57] refers to the slope of line. Is this term getting clear to everybody? Since I term getting clear to everybody? Since I have only one input variable or independent variable, it represents a straight line. Clear?

[1:43:17] of the straight line, what do these blue data point uh blue dots represent? the data point uh blue dots represent? the actual data points available right that the first data point has value 1 y is equal to 3 this is value four and value

[1:43:32] equal to 3 this is value four and value is equal to 6 all right now this is where how we are going to calculate the error what is the error the actual value

[1:43:44] error what is the error the actual value this you can consider this y i as my actual value and the red dot on the straight line is the predicted value if straight line is the predicted value if I want to fit a straight line on these

[1:43:58] I want to fit a straight line on these data points. Right? So this is my actual data points. Right? So this is my actual output and this is my predicted output. Clear? The error is clear. How are we calculating the error?

[1:44:13] Now what algorithm are we technically using over here? ordinary le square using over here? ordinary le square method or the residual. Residual means the error sum of square error. So what

[1:44:28] does this mean? It is equal to the sum of the square of error which is actual of the square of error which is actual minus predicted value. Getting my point? So and why are we taking the square of

[1:44:44] the error? There is a story behind that also. Why? Because is + two for example and this error is minus2. So the total error is zero. So

[1:44:56] what I'm trying to say is if the error is plus over here and minus and if I add them it will show me zero error. It does it is not zero error. So I will try to take the square of the value it will become four + 4 that is the error is

[1:45:12] equal to 8. So always the errors are taken as square of errors. And we are looking at minimum square of errors. That is why it is known as the ordinary

[1:45:24] le square method. We want the sum of the squared error to be the sum of the squared error to be the least. This is known as the RSS. Getting least. This is known as the RSS. Getting my point learners?

[1:45:38] So this is the total RSS. Why the total for this particular data point? This is going to be my first error. For this data point, my second error for the third one, fourth one, fifth one. And if they are infinite points, then infinite

[1:45:53] data points are calculated. Clear to everybody? So why y okay now? Clear to everybody? So why y okay now? Because this is my actual y1 and what is

[1:46:06] my predicted y1? This is my predicted value right base because we are using nanda this particular function. This is my predicted value and predicted values are represented as ycap. So for the first one it will become beta minus beta

[1:46:22] first one it will become beta minus beta 1 x1 then y2 beta beta 1 x2 for each data points. Now clear this is how based on this mathematical function I am going to predict my output for each data points. Clear?

[1:46:39] points. Clear? So the formal statement of simple linear regression I'm talking about simple linear regression with one input and one output. Y1 is the value of the response variable. Again, predict, target,

[1:46:55] variable. Again, predict, target, predicted, you know, dependent, they all mean the same thing. Do not get confused with the terminologies in the IAT trial. Beta kn and beta 1 are the parameters. Xi is the value of the predictor

[1:47:09] Xi is the value of the predictor variable in the IAT trial. And epsylent I is the random error with mean E. epsylon i is equal to zero and variance is equal to square. Clear? Okay. Now this random error this is very

[1:47:26] very important to understood that a random error we are talking about are we talking about the whole data set or a part of data set or a sample of data set

[1:47:39] over here. This is my population and if I take a part of it for training and as well as testing do I always consider the whole data set or a part of data set whole data set or a part of data set it's always sample and when I and is my

[1:47:55] sample and we want the sample to be the true representative of the population true representative of the population but since it is a part of the data there will be some error also uh you know associated with it which is irreducible

[1:48:09] that is represented by epsylon right so epsylent I is the random error with mean this is the random error which is always going to be introduced it's

[1:48:22] it's not reducible like bias and variance or overfitting but it's part of variance or overfitting but it's part of the whole linear regression where the mean of that error is equal to zero and the variance is equal to epsylon square

[1:48:37] the variance is equal to epsylon square now clear so The idea of whole uh linear regression or significance model is that it is highly interpretable.

[1:48:49] it is highly interpretable. What is the idea behind this? That it is why the error is zero. We always want the error to be zero. In this case we are considering that uh you know the average of the error not the square the

[1:49:04] the sum of the square or the average of the error is equal to zero not the square. Okay. So now the significance of linear regression lies in the fact that we can

[1:49:18] easily interpret and understand the marginal changes with with with input to marginal changes with with with input to the output. So linear regression is an

[1:49:30] highly interpretable model. How is it interpretable? That if we increase the value of X1 by one unit keeping the other variables constant, then the total increase in the value of Y will be beta

[1:49:46] 1 and the intercept term beta KN is the response when all the predictor terms response when all the predictor terms are set to zero and are not considered. are set to zero and are not considered. Right. So how much is my y value change

[1:50:00] will depend on how much is my beta 1 is it positive or negative beta_2 and beta n. So now what are we trying to find out in linear regression the values of beta kn beta 1 and beta n right where the error is

[1:50:19] minimum. Clear lot of concepts lot of mathematical terms coming so just hold on till we

[1:50:33] and you know you will see them practically just try to grasp as much concepts as you can. So the different assumptions which we follow in linear regression is linearity that is the relationship between the features

[1:50:49] and the target homoidasticity. The error term has this is where you know the linear regression has its assumption or constraint and the the term is known as homosidicity

[1:51:04] that the error term is constant variance throughout the whole data. The error throughout the whole data. The error term will be constant. Multi-olinearity, there is no multi-olinearity between the features.

[1:51:17] And if we talk about independence, observations are independent of each observations are independent of each other. Getting my point? Normality. This is where the uh point I was saying that the error residuals follow normal

[1:51:33] standard distribution where mean is equal to zero and the standard deviation is equal to one. Okay. So multiolinearity means that we can have more than one input. Are they related with each other?

[1:51:49] input. Are they related with each other? No. every in in uh first input should be independent with the second one. Second one should be independent with the third. The input should not be uh dependent on each other. They should all

[1:52:03] be directly dependent on the output. First let's try to understand the concept. So how do we go about in the supervised uh learning process that we have this full data set right and we

[1:52:20] will after doing all the pre-processing so today I told you to revise all your EDA that we will load the data set using PD dot read CSV command do head and tail

[1:52:32] try to find out null values work on them uh do uh encoding if required and then split the data into training and testing. First step is getting clear and

[1:52:44] of course training can be se it's generally 7 it's generally 7 to 80. If you want to keep it 50/50 also nobody's stopping you or 6040 that's completely up to you. Okay. And the function which is used in Python

[1:53:00] in the skarn pre-processing library is train test_plit. X is the first parameter which represents the features of the input data. So this is how we will create a list or array which will contain all the

[1:53:16] input. Y reper refers to the target or the label vector of the input data. Size the label vector of the input data. Size of the test data test size and finally we have random state. What is this random state? It is

[1:53:32] the seed that initializes the pseudo random number generator. What does that mean? That I will give any fixed number. It could be 42, it could be three, it could be zero, it could be 500. Right? I want

[1:53:48] to keep this number constant so that the shuffling or the randomization of the data happens in the same way so that I can check the accuracy and do the Clear? Right? So again remember this point

[1:54:07] whenever data is given to you you have to identify it in terms or split it in terms first in terms in number of inputs and the output. Always separate the and the output. Always separate the outputs from the input and then we will

[1:54:22] use this function and then we will get four outputs. Train test_split

[1:54:34] gives four output that is x train y train x test and y test. Clear? Training test_split is used to split the data. Right? We are

[1:54:46] preparing the data which will be used for training and testing separately. So the parameters of this function are input output how much we want to give input output how much we want to give for testing. It could be 3/4 half

[1:55:00] completely up to you. No rules for that. And random state gives me the seed that initializes or does the shuffling of the data in the same manner. And the output

[1:55:13] of this function is four output. I get input training and output training data and input training and output testing data. Clear? data. Clear? And now if I talk about multiple linear

[1:55:29] regression, right? In threedimensional setting with two predictors, one response, the le squared regression line becomes a plane. So this is a multiple linear regression model. epsylon is that normalized error.

[1:55:46] Okay, which is irreducible. Nothing can be done much about it. And now we will practically deal with data which has three inputs. The budget of TV,

[1:55:59] three inputs. The budget of TV, budget of radio, budget of newspaper and budget of radio, budget of newspaper and which one of them affects the sales most. So this is the first uh you know algorithm that we are going

[1:56:13] uh you know algorithm that we are going to study. Okay. So this is simple this is polomial and these are the errors. So let's start with the practical work and then get back to the different errors and analysis.

[1:56:25] Got it? Have you got an idea what linear regression is all about? So I request you all learners to from the LMS2 open 3.2 to supervised learning

[1:56:39] the LMS2 open 3.2 to supervised learning regression linear regression five. Okay. So now do you understand the term independent independent variables straight line when we have in two dimensional when we have one input one

[1:56:52] output it's a straight line blue dots are my data points and this is a straight line or the line of regression which I want to fit on the data points. everybody. I'm starting with the uh pile now. The

[1:57:07] first 3.2 regression. This graph is clear. And where do you think regression is used? It has a huge huge application in oil and gas uh industry. Various types of data collected in oil and gas

[1:57:23] industry from surface subsurface to understand production sale processes. How much gas is required? How much liquid? How much is the pressure that needs to be created? So linear nonlinear regression model forecast global oil

[1:57:37] production. Oh my god, the whole world is fighting on oil and uh well we can use our regression models to forecast the global oil production right so the

[1:57:50] whole world is on a big fight on oil only right. [laughter] Marketing and marketing linear regression helps to analyze the effectiveness of advertising campaigns. Predict sales based on marketing spend.

[1:58:04] Segment customer based on demographic data. Right? Then we have retail. Then we have linear regression is utilized in retail for demand forecasting, inventory management, pricing optimization and customer analytics. Health care. Linear

[1:58:21] regression is applied in linear healthcare for predicting the patient outcomes, analyzing relationship between the different medical variables and their diseases. Getting my point?

[1:58:35] And finally we have the real estate that in the real estate industry linear regression predicts the property prices based on the factors such as location based on the factors such as location size Mntes uh amenities and economic

[1:58:50] indicators. Clear? So now what are the different types of So now what are the different types of regression? Regression can be classified into two categories linear and nonlinear. Linear regression finds a

[1:59:05] straight line relationship between the dependent variable and one or more independent variable. Nonlinear regression uh finds a relationship between the dependent variable and independent variable using a curve or

[1:59:21] more complex shape. Okay. So getting back to linear regression it is so technically if somebody ask you what is linear regression in in machine learning you will say it's a supervised learning

[1:59:35] algorithm which is used to please try to understand each and every term is used to predict a continuous target variable please be clear it's not continuous data

[1:59:47] as you all were saying that's a wrong statement it helps us to predict continuous output variable by modeling its relationship with one or more independent variables through a linear regression.

[2:00:02] It predicts a continuous dependent variable based on one or more independent variables. So it predicts the continuous dependent variable. Here it says target output dependent completely anything you can say. It uses

[2:00:17] the le square criteria to estimate the coefficients of the regression equation. What do we mean by the le square criteria? That the value between the actual and the predicted value. The square of the error of this error that

[2:00:35] square of the error of this error that is RSS should be the should be minimum is RSS should be the should be minimum that is the residual sum of squares. Getting my point and it can be applied if there is linear

[2:00:48] relationship between the variables. So in case the dependent variable is continuous and independent variables can be continuous or discrete discrete we are not bothered about our input variables. So for predicting the uh you

[2:01:04] know output we are always considered about the output. So the relationship between a dependent variable Y and one more independent variable X is established using a best fit straight line which is also known as the

[2:01:20] regression line. And there are two types of linear regression. One is known as the simple linear regression. Other one is known as the multiple linear is known as the multiple linear regression. All right.

[2:01:35] So simple linear regression is clear to everybody. We have one independent variable and one dependent variable. Do we understand now the terms beta KN and we understand now the terms beta KN and beta 1? This is the intercept and the

[2:01:50] slope. Yes, everybody is understanding now what Yes, everybody is understanding now what is simple linear regression. And as I told you a simple linear relationship is given by one of the

[2:02:05] input that is TV expenses over here as my input my input and in the output we have that sales. That means in this particular data set we are trying to establish the

[2:02:22] relationship that if what is what is going to be the budget of TV and how it is affecting my sales. So what do you conclude? Do you see the red points? These are the predicted values and the blue points are nothing but my actual

[2:02:37] data points and I'm going to calculate or it automatically does the calculation of the error if once I you know apply the algorithm clear. So do you see a kind of positive

[2:02:52] linear relationship? Can I conclude that this shows a little Can I conclude that this shows a little positive linear relationship? positive linear relationship? that is the value of TV

[2:03:04] expenses increase the sales is also increasing. It shows a positive relationship between them. Getting my point and practically if you look at the real world problems they will not be simple

[2:03:18] linear regression problem. They will going to be uh you know multiple linear than one input from how we know positive and negative. See positive and negative button that's that's you know shows it very clearly from the graph. If the

[2:03:34] graph is showing this thing that as my x is increasing y is also increasing it shows a positive relationship. See over here

[2:03:51] decreasing. But if my graph is like this, what does it show? That X increases, Y decreases. And if X decreases, Y increases. Getting

[2:04:06] my point? So this is a negative relationship. inverse relationship between input and output shows negative relationship and

[2:04:18] uh direct relationship between input and output shows positive relationship. So as I was telling you practically if I talk about uh you know real time data uh you know we have more than one inputs and output. So multiple linear

[2:04:34] regression models the relationship between the two and more independent variables predictors features and the dependent variable as a straight line. The equation for multiple linear regression

[2:04:49] equation for multiple linear regression is y is equal to beta kn plus beta 1 x1 is y is equal to beta kn plus beta 1 x1 beta 2 x2 and beta n xn. that x1 x2 xn are the predictor variables and beta 1 beta 2 beta n are the coefficients of

[2:05:03] beta 2 beta n are the coefficients of each predictor. So let's start practically working on this data set that this becomes now if I want to try to find out the relationship between TV

[2:05:17] expenses, radio expenses and sales. So now my straight line turns into this hyper plane. Do you see the difference how the straight line has got converted into hyper plane? And of course there is one constraint that we cannot view more

[2:05:32] one constraint that we cannot view more than 2D or 3D graphs right we cannot than 2D or 3D graphs right we cannot view more than 2D or 3D graphs right view more than 2D or 3D graphs right that correlation co-variance is same as

[2:05:44] that correlation co-variance is same as uh you know um regression but in regression it's a linear relationship but the positive and negative relationship is the concept same as correlation and covariance that

[2:05:57] correlation and covariance is founded by Pearson's you know coefficient of correlation but here we are talking in terms of linear equation but if I talk about one input x1 and y how x1 is related to y then

[2:06:13] x1 and y how x1 is related to y then that is r that is + 1 minus one that's that exactly is the same concept the concept is not very different linear regression is using that particular concept all right so now let's start

[2:06:27] practice practically working on the data set. So what I want everybody to do is on your uh you know go from data set folder open your Jupyter notebooks and

[2:06:40] folder open your Jupyter notebooks and keep the TV marketing CSV data in the keep the TV marketing CSV data in the same folder where you have kept this 3.2 yes learners wherever you have kept this 3.2 going to keep the TV marketing

[2:06:54] 3.2 going to keep the TV marketing dot CSV copy it from the data set folder and start uploading this data clear

[2:07:08] this data set consists of three input variables. This is my x1, x2, x3 and this is my output.

[2:07:22] x3 and this is my output. All right. So once I have done my head, what does the head command do? What does the head command do learners? What does it prints the top five rows by default? And these are my inputs, right?

[2:07:37] And what does the info function do? Good rishi. Good anush. What does the info do? It tells me there are no null values in this data set. And of course unnamed is not required over here. And all other

[2:07:53] my inputs are also in numerical value. And what is my output? My output is sales. And that's a numerical value. And that is why regression will be used.

[2:08:07] Clear? And that is why regression will be used. And that is why regression will be used. Getting my point.

[2:08:21] are going to do? Since this data does not require pre-processing, it's a clean data. So nothing required to clean or check on the null values etc. Right? So

[2:08:33] we are considering the data to be clean. So now when I want to fit any model what is the first step that I will do? I will try to separate the input to the with try to separate the input to the with the output columns. Agreed? The first

[2:08:48] the output columns. Agreed? The first step is to extract the features of the step is to extract the features of the data set and output. What is this ILC in Python code? Index base filtering. So if I see very

[2:09:03] Index base filtering. So if I see very closely that 1 2 3 are my input index. closely that 1 2 3 are my input index. This is of actually no relevance. This is of no relevance. These are my input values

[2:09:28] and this is my output value. Clear? So in this case ILOC the first colon represents it it talks about all the rows but only the columns which are there it says one and two. So how many inputs are we talking about? Only the

[2:09:45] inputs are we talking about? Only the first one only the first one is TV. And first one only the first one is TV. And what is the output? Only the sales. This point is getting clear. What are we taking over here? We are talking about

[2:09:57] TV and sales. So we are trying to implement linear regression, right? We are trying to do simple linear regression. This X and Y is clear. I've

[2:10:09] only taken a part of it. First I've taken one input and then I'll take all taken one input and then I'll take all the three inputs. This point is clear, right? And then we have the from skarn model selection import train test_plit.

[2:10:25] And now I am taking my input that is only the TV input. Y is my sales test only the TV input. Y is my sales test size. 30% of the data is used for size. 30% of the data is used for testing and random state is taken to be

[2:10:40] 42. 42 is generally considered from a fiction that it's cons considered to be fiction that it's cons considered to be a you know a good number for the whole universe. Clear? Okay. So now what are the different

[2:10:55] Okay. So now what are the different steps to now fit the model not much of coding that's why Python is a preferred language beautiful language that is from sklearn.linear model import linear regression that is

[2:11:10] model import linear regression that is first we will import the library that is skarn is the main library and we are importing linear regression from there. So when I imply linear regression it is lin_re

[2:11:25] and this is fitting that is the training input and output training data. input and output training data. Getting my point fit a function is used

[2:11:37] Getting my point fit a function is used to fit the data or train the data. What do we mean by train the data? finding the parameters over here that is beta kn

[2:11:49] the parameters over here that is beta kn beta 1 beta 2 beta 3 etc. Clear? Are you all there with me till this code? Anybody who still facing any difficulty who's not been able to run

[2:12:05] this part of the code till here are we good to go? First step is to import the library which contains the linear regression which contains the linear regression model and then I will create an instance

[2:12:20] model and then I will create an instance of this linear regression function right of this linear regression function right that is lin rig it is a instance what do you understand by instance object of this particular function and then fit

[2:12:33] this particular function and then fit the training input and output data the training input and output data it is divided by using is train it is divided by using is train test_split

[2:12:50] input training and testing output training and testing I explained you train test split where we get the four outputs

[2:13:03] now let's try to find out the linear relationship between the sales and the TV so TV and sales so we generally we want it to be an input.

[2:13:19] So if you visualize the linear regression model X label we don't want it to be that's I think so that's a little opposite of it over here this is little opposite of it over here this is TV change make changes in the code and

[2:13:34] TV change make changes in the code and output is sales constraining factors in model what do we mean by that delineate. What do we mean

[2:13:48] mean by that delineate. What do we mean by that? represent? What does this graph represents? This plot shows a positive linear relationship between sales and TV and where the blue regression line

[2:14:03] indicates the model's prediction. The green data points are generally close to the line suggesting the model fits the data reasonably well through those some variability exist. Right? So does it show a positive

[2:14:19] Right? So does it show a positive relationship basant? Right? Do we see that there is a positive relationship between TV and sales? No, it's not the train data. It is the

[2:14:33] test data. Y test data because I'm plotting input as training and predicting the output. The xaxis is the input. The x is input

[2:14:46] right and the output is prediction of the training. Right? So we are trying to plot over here. Plot over here is the straight here. Plot over here is the straight line.

[2:15:02] data? Because actual data points are my input and output test data. Now clear we are trying to plot uh do a scatter plot over here. Scatter plot is between

[2:15:16] input and output test data. The actual data points are by testing data points. Okay. And the plot the straight line is input

[2:15:28] training and predicting the output. That is why it say it says the linear is why it say it says the linear regression model for test set data set. Now clear. So uh what what do we conclude from this

[2:15:43] graph? We can conclude that the plot shows a positive linear relationship between sales and TV where the blue regression line indicates the model's predictions. The green data points are generally close to the line suggesting

[2:15:57] generally close to the line suggesting the model fits the data reasonably well the model fits the data reasonably well though some variability exists. Okay. So this shows a positive relationship. Now the important point is

[2:16:11] how do I understand is this model underfitting or overfitting. So when we are developing machine model what is the most important point important concept that needs to be taken into account the concept of overfitting

[2:16:26] and underfitting. So when developing machine learning models achieving the right balance between the complexity and simplicity is crucial. What are the points of overfitting? Let's quickly go through that. Overfitting occurs when

[2:16:41] the model learns the noise and details in the training data too well to the extent that it is negatively impacts the performance on unseen data.

[2:16:54] impacts the performance on unseen data. Right? That means when the variance is Right? That means when the variance is high sign high accuracy on the training high sign high accuracy on the training data but poor accuracy on the test data.

[2:17:06] So, how do I know that my model has overfitted? That I get a good score for overfitted? That I get a good score for training data but a bad score for testing data. And why is it what is the reason?

[2:17:21] Because the model is too complex. It has got too many parameters. Underfitting happens when the model is too simple to capture the underlying pattern. Poor accuracy on both training and testing data. Underfitting is more easily to

[2:17:36] data. Underfitting is more easily to detect but overfitting is a constraining problem. So what are we looking? How do we solve this problem? We want a bias we solve this problem? We want a bias and variance tradeoff. Bias error due to

[2:17:49] overly simplistic assumptions in the learning algorithm. And high bias always causes underfitting. Variance is error due to excessive complexity in the learning algorithm. High variance causes overfitting. Clear? So low bias, high

[2:18:07] variance again it is failing uh to be a good fit. This refers to overfitting. High bias low variance it leads to underfitting and optimal tradeoff finding a balance between the model performance on both training and test

[2:18:23] data. So this is what we are actually trying to achieve in the output. Right? So just to understand things you have to understand the machine learning model from the mathematical intuitive behind it. If it is possible to understand it

[2:18:39] graphically well and good because in higher complexity higher dimensional plane it becomes difficult to access uh to understand it visually but mathematical intuition the concept and how do I interpret the results are you

[2:18:54] how do I interpret the results are you understanding and of course how we are implementing those functions in Python that is where you know your concentration should be mathematical concept or or What is the concept of

[2:19:08] that particular algorithm? How are we implementing in Python? And how am I interpreting the results? That should be the option. Clear? Is this point getting clear to everybody?

[2:19:24] And how do I check the error? The checking of the error is going to be checking of the error is going to be through the training and testing error. So if you just look at this, I I'll tell you what these errors are. So if you

[2:19:38] just look at the training error is more than the testing error. So is it underfitting or overfitting? Definitely underfitting. Consider increasing model

[2:19:51] complexity. That means linear regression or a straight line fit is not a good fit or a straight line fit is not a good fit on this particular datas. Agreed? So now getting back to the concepts of

[2:20:06] So now getting back to the concepts of errors, how do I evaluate my regression errors, how do I evaluate my regression model? Okay, so one of the simplest error is known as the mean square error. This is

[2:20:25] known as the mean square error. This is nothing but your RSS, the residual sum of square error. Please try to understand this is nothing so errors. So there do you think one error is easy to define linear regression? No. Right?

[2:20:41] There would be several errors. So now let's try to understand different errors let's try to understand different errors available in the linear regression available in the linear regression model.

[2:20:57] error, this is nothing but my RSS or the residual actual minus predicted value the whole square. So error square of the error and if I take the average of it, error and if I take the average of it, it becomes mean square error. Agreed?

[2:21:13] it becomes mean square error. Agreed? Are you understanding this error? And when I talk about root mean square error that is when I take the under under root square root of this particular MSE. But what is the need of

[2:21:29] taking the root mean? Can anybody tell me we what is the what is the need of taking the root of this mean square error? Anybody who can explain this is exactly the same uh you know

[2:21:43] question which relates to standard deviation and variance. Anybody who understands the relationship between standard deviation and variance

[2:22:01] after square error will increase or square root will get proper value. No nanda that's wrong answer. Yes ma'am. rooting variance is standard deviation like but why do we take that under root that's the question

[2:22:20] variance what is why what is the significance no it doesn't mean that significance no it doesn't mean that squaring will uh get a proper value no square root will yeah because ultimately we have to

[2:22:36] compare with other values such as such as average. So average is a simple value. Simple values are cannot be compared with a squared value. Mean square value cannot be used for comparison with other values. So to make

[2:22:53] it at you know can I it's something like can I compare a cm squared the unit of can I compare a cm squared the unit of area that is equal to cm squared with cm can I compare a cm squared or a me square a unit of area with normal meter?

[2:23:11] No that goes mathematically wrong. So just by taking the square root of MSE now it becomes at at the same level and then the comparison can be done. Yes. And if we talk about mean absolute error that is

[2:23:27] actual minus the predicted value and this is I divided by N I get the mean this is I divided by N I get the mean absolute error. Clear? Now these are all

[2:23:39] absolute error. Clear? Now these are all you know um we can say absolute terms of error and if I want to do comparison between different models now what does it what difference does it make if I'm saying actual minus the predicted the

[2:23:53] whole square and then I say predicted minus actual the whole square is there any difference between that

[2:24:07] mathematically whatever way they write you know the answer will be same. Yeah. Clear? Now now let's start let's understand another factor of comparison understand another factor of comparison that is R². Please try to understand R².

[2:24:21] Okay. Okay. Now to understand R square, it is a relationship between S STO. What is this total sum of square? Yes, these are this total sum of square? Yes, these are my actual points and total sum of square

[2:24:36] is the difference or the total variation between the actual points and the average. Generally, we want to always find out Generally, we want to always find out how far my points are from the average,

[2:24:49] how far my points are from the average, right? How my points are far from the average. Right? So, SSTO is getting clear to everybody. So over here this is my actual

[2:25:04] data point and here I'm doing my comparison. Is the total sum of square clear to everybody? What was what is SSC? Is SSC SSC? Is SSC same as the RSS?

[2:25:18] Here we are trying to find out the difference between actual value and its difference between actual value and its difference with the average yi minus ycap b² again y square because I don't want any no positive and negative values

[2:25:34] cancelling each other clear and if I talk about SSC or the error sum of square this is like RSS actual minus the predicted value

[2:25:46] this is my actual this is the predicted value. Do you see this? So the uncertainty of the data around Y observations lie around the regression line. If SSC all observations fall on regression line, larger the SSC, the

[2:26:01] greater is variation of the Y observation around the regression line. Then we have regression sum of the squares. Right? This is actual uh sorry

[2:26:14] the average predicted value with the difference with the average. So the difference between the predicted or the fitted value on the regression line and the mean of the fitted value that is equal to SSR. The measure of variability

[2:26:31] equal to SSR. The measure of variability of Y associated with the regression line. larger the SSR in the relation to SSTTO, greater the effect. Right? So if we look at now the whole picture that my total sum of the square

[2:26:48] picture that my total sum of the square is equal to SSC plus SSR that is the difference the sum of square values with the actual value minus the average the whole square is equal to actual minus the predicted the whole

[2:27:02] actual minus the predicted the whole square and predicted minus the average. Agreed learners? But what is the use of all these? So now there is this another all these? So now there is this another factor another uh unit which helps in

[2:27:17] factor another uh unit which helps in comparison of value that is R² that is known as coefficient of determination determination which is also known as the R square statistics proportion of

[2:27:33] variance explained the value varies between zero and one and independent of the scale. Y right so R² is equal to S STTO minus S

[2:27:45] Y right so R² is equal to S STTO minus S SSE upon SST SSR upon S STTO so what is the advantage in short let's not get into the mathematics that the R squar into the mathematics that the R squar value will always lie between 0 to 1 so

[2:28:00] the values which are more close to one the values which are more close to one are better fit as compared to zero yeah mean square error and RMSSE relationship is clear but they are not relative

[2:28:14] errors. I cannot do comparison between the different models that this model is better than the other. They are good to tell the errors are less or more comparison between training but R square is a relative error. Even absolute error

[2:28:31] is not a relative error. What do I mean by that? What do I mean by that? Let let me explain you with an example. For example, you scored 70 marks and I have scored 30 marks. Which who scored better? You would say 70 marks. Ma'am,

[2:28:46] you you scored better than me. But I scored 30 out of 30. You scored 70 out of 200. Getting my point? So there how will I find the relative? If I start taking

[2:28:59] percentage of this that 30 out of 30 is 100% and 70 out of 200 is somewhere around 40 50%. then I can do comparisons. Got my point? That is why

[2:29:12] comparisons. Got my point? That is why we are now heading towards ratio between the SSTO the total sum of squares with RSS and SSR taking into all account all the variability of the data points and that's why I now reach on this R squar

[2:29:30] that's why I now reach on this R squar term now better is it better sum of square error ranges between 0 to 1 which measures the amount of variability that is left unexplained after performing the regression. An SST minus SSE measures

[2:29:46] the amount of variability that is explained or removed after performing explained or removed after performing the regression. R² and the proportion of variability in Y can be explained using X. So R square is trying to take into

[2:30:01] account all the errors and variability of the data point. So when R² is equal to 1, SSC is zero. That means it's a perfect fit. It's an ideal situation.

[2:30:13] Will it ever happen? Hardly. I don't think so. You know it's going to be but something which is close to 1.99 is a good and when I get this horizontal line that you all were getting that means r² is equal to zero error is equal to total

[2:30:29] that means there is no linear relationship between x and y. Now clear relationship between x and y. Now clear right but R² suffers from one one

[2:30:44] right but R² suffers from one one limitation right since R² is also known as the if it is greater than one it can never be greater than one that's the assimon that's not possible mathemat can percentage can ever come beyond 100%

[2:30:59] the formula is such now because we're taking a ratio out of 100 right Simon So mathematically it can never be uh true. Got it?

[2:31:11] Okay. So R square also known as the coefficient of determination measures the proportion of the variance variation in your dependent variable X and in your dependent variable X and explained by your independent variable

[2:31:25] X for the linear regression model. But the problem that R² also suffers is that it will always remain or increase as we are adding more number of independent are adding more number of independent variable. So this problem is solved by

[2:31:40] variable. So this problem is solved by adjusted R². What is adjusted R square? That it measures the proportion of the variation explained only those independent variable that really help. So what do we do? We divide the error by

[2:31:57] degree of freedom. Do you understand the term degree of freedom? Learners Degree of freedom is the number of independent

[2:32:11] the number of independent variables. Right? Suppose if there are 10 10 if suppose if there are 10 input features then the degree of freedom is n

[2:32:23] features then the degree of freedom is n -1 that is 9 -1 that is 9 clear how many input features are we depending on a3ri that is referring to my degree

[2:32:38] of freedom it's something like now let's understand it over here now if I want to predict the price of the car. What are the factors that are determining the price of the car? Safety is a very very important factor. How

[2:32:54] many airbags does it have? Does it have a parking sensor or not? The branding. What is the most important factor? The mile, the fuel, consumption, the fuel type in today's world, assistant, infotainment system, is it a luxury car

[2:33:09] or not? the color, the kind of uh you know the design, the seats, the the the the the wheel, the alloy, there could be several factors. Agreed learners. But do you think the

[2:33:24] the kind of wheels or the wheel alloys are as important as the engine of the car or the model of the car to determine the price?

[2:33:36] the price? No. Right. So that is what is taken care No. Right. So that is what is taken care by adjusted R² and when we divide it by the number of degree of freedom getting my

[2:33:49] number of degree of freedom getting my point see normally what will happen is if I keep on adding the factors in R square that I have added the number of alloys the number of uh you know the seat covers the color of the car the the

[2:34:03] R square will keep on improving it will give me an illusion Oh my model is giving so much well answers but that's not true. Maybe you know adding the kind not true. Maybe you know adding the kind of wheel alloy is not that concrete uh

[2:34:18] factor to determine the price of the car. See the these numbers of degree of freedom as I told you this is some example that they are taken degree of freedom depends on the number of input

[2:34:31] features. So that is why adjusted R square is a better better uh you know way of determining the results.

[2:34:45] determining the results. So if I get back to the file over here and now look at how I calculate the results. So what am I trying to do from

[2:34:58] results. So what am I trying to do from skarn metrics import mean square error R skarn metrics import mean square error R square error and then I do the prediction. Prediction is done by the predict function for the training output

[2:35:12] as well as for the testing output. And then the different metrics can be simply calculated in Python by using the mean square error for training data for testing data and R² and then I can do the comparisons.

[2:35:32] So if you do you see the training error is 0.573 and testing error is more than that that's more closer. So they all refer that the model is underfitting.

[2:35:45] Now clear how are we trying to fit the model? How how are we trying to fit the model? How are we trying to uh

[2:36:00] fit it then predict the output and then calculate the error. So if we talk about linear regression, let's let's look at it again that this let's let's look at it again that this is my actual output right and if I talk

[2:36:14] about a single variable as my input and there are two coefficients beta KN and there are two coefficients beta KN and beta 1 this is the epsylent error. Is this first equation getting clear to everybody?

[2:36:32] is getting clear to everybody. Then we have linear regression multiple Then we have linear regression multiple variables right we have x1 x2 x3 variables right we have x1 x2 x3 right absylent over here refers to the

[2:36:46] random error and if we talk about a model evaluation are we clear with these metrics now the mean absolute error the mean square error and the root mean square error right so comparisons between the

[2:37:03] different models is not possible. Therefore, we have R square error which Therefore, we have R square error which talks about the ratio and even better than R square is adjusted R² because it is not affected by the number of inputs

[2:37:19] is not affected by the number of inputs in the data. Clear? So basically we in the data. Clear? So basically we divided by degree of freedom mean model. divided by degree of freedom mean model. Clear? And we have also understood

[2:37:33] Clear? And we have also understood different scikitlearn objects. Please try to understand fit function. Fit function helps us to train the input parameters or train the parameters for

[2:37:48] the data. Transform is transforming the input to output. Fit transform is mixing the function of fit as well as transform. And it is the predict function which is used to predict the output of the training data or of the

[2:38:05] testing data. To understand it better, types of skarn objects are based on transformers. Transformers are nothing which transform

[2:38:17] the data set. Have you learned feature engineering learners? Do you understand feature engineering where we do feature scaling, standardization and normalization, encoding of the data that we need to

[2:38:33] encoding of the data that we need to transform the data before we actually fit into the model. Let me try explaining you over here. So before the data is actually fed into the model, it needs to be transformed.

[2:38:49] Why it needs to be transformed? Because if they are missing values that will not fit into the model. If they are categorical features then we have to in involve uh encoding. Feature scaling is a feature which brings the data into one

[2:39:06] a feature which brings the data into one particular range and outlier detection. particular range and outlier detection. That is why data science is important because these concepts are covered in detail. How do you deal with data? How

[2:39:19] do you handle missing values? How do you deal with categorical data? Scaling of the data is all part of the feature engineering process. So in feature engineering, we are not just building models but we are playing

[2:39:36] matchmakers for data as well as algorithms. algorithms. Getting my point everybody? Okay. So getting back for transforming the data set right. So fit learns what does a fit

[2:39:50] function do? It is an estimator. Please try to understand. Fit is an estimator which estimates the model parameter based on the training data and hyperparameters. And finally we have the predictors that

[2:40:06] is it predicts the data makes the data set as input and does the prediction score method to measure the quality of the predictions. getting my point?

[2:40:18] So here we have the data pre pre-processing training and inference pre-processing training and inference model clear and then we have fit and fit model clear and then we have fit and fit transform right what do we do that when

[2:40:32] we are trying to train the data what I'm trying to explain let me explain it with trying to explain let me explain it with this slide that again please try to understand let me rewind things in supervised learning we have label data

[2:40:47] first of all what is label data that we have the input as well as the output. Now in the story have you understood two types of output in supervised learning

[2:40:59] that we will have the actual output of the label data and one we will predict the label data and one we will predict the data using the X test. This point is getting clear. What this point which I'm trying to explain regarding the output

[2:41:16] is getting clear. Right? So this is output actual as well as predicted. Now we also understand that we divide the training and the testing data. Right? So ultimately we are trying to

[2:41:31] when we are doing training it is all on the training data by the fit parameter. So the model all the training will be done using the fit function in simple done using the fit function in simple terms right and the testing and output

[2:41:46] will be on the predicted but if I want to transform the data so I will fit it and then transform it and can I do the function of fit and transform together yes by using fit

[2:42:03] yes by using fit transform function getting My point can transform function getting My point can I do the function together? Yes, for only training data I can use fit transform and for testing data I will

[2:42:17] transform and for testing data I will only use the transform function. Now only use the transform function. Now clear code. So now are we able to also understand

[2:42:33] how when the model is underfitting and overfitting that point is also clear? Yes. And if we talk about nonlinear regression polomial regression uh

[2:42:45] polomial regression is a subset of linear regression that includes polomial terms the relationship between the independent variable x and dependent independent variable x and dependent variable y is modeled as an nth degree.

[2:42:57] variable y is modeled as an nth degree. So here we are not trying to do it as a straight line but polomial regression which is a subset of linear regression which includes polomial terms. The relationship between an independent

[2:43:10] variable x and dependent variable y is modeled as nth degree polomial.

[2:43:23] called a special case of multiple linear regression. some polomial terms are added to multiple linear regression equation to convert it into polomial regression. So it is a linear model with some modifications made to increase its

[2:43:37] accuracy. The data set used in polomial regression for training is nonlinear. So what do we what do we observe that in

[2:43:49] this particular graph this is a straight linear regression and if I change it to a curve it becomes polomial linear equation. Do we see that exponential curve over here? Okay.

[2:44:04] So I have this uh you know simple data right in this simple data this is the project statement. Okay. project statement. Okay. Wait.

[2:44:22] Yeah. So, a certain spare part manufactured company once a month in lots which vary in sizes. Data on lot and size numbers and man of hours that that means the input is the lot size of the people and how many man hours are

[2:44:36] the people and how many man hours are required. Right? It's numerical. Right? So if I start analyzing the data actually you know it's done through an actually you know it's done through an excel it's simple that I do x - xar y i

[2:44:48] excel it's simple that I do x - xar y i y - y bar and I calculate this what is y - y bar and I calculate this what is this known as rss residual sum of square actual minus the output the whole square x i minus x i the whole square do you

[2:45:03] see this and why am I taking the square value because I simply take the difference between y minus ycap the error is zero which is actually not which is actually not. Therefore we take the square values

[2:45:17] not. Therefore we take the square values and I get the answer as 13 six 16. Okay. So the sample size the number of rows in this data set is 10. Degree of freedom over here is 9. Mean is equal to xbar

[2:45:33] over here is 9. Mean is equal to xbar sigma xi upon n 50. Variance over here is this much. Standard deviation is in this much. So this is where statistics is coming into picture. We have calculated the value. Similarly, we can

[2:45:46] calculated the value. Similarly, we can calculate it for the man R's Yi. Okay. And if I try to see the relationship, it's coming out to be straight line. Right. Now, coming on to the concept of how do we calculate the best fit line.

[2:46:01] how do we calculate the best fit line. Right? So, this is the le square concept where we are trying to find out the RSS. So over here what am I trying to do again? Y I minus

[2:46:19] what am I trying to do again? Y I minus the Y cap the whole square not this one. Okay here it is doing hidden trial method but ultimately the le square estimators can be found out by trial and error method

[2:46:35] but that is not the case. we try to calculate it by the normal equation. This is the equation to for calculating of beta KN and beta 1. So these are

[2:46:47] already predefined formulas which run at the back end when I run the linear regression function. So this example shows that whatever values you are getting by solving mathematically or through linear

[2:47:01] mathematically or through linear regression model I get the same values. Okay. So what does polomial regression on data set work? Again we are using the on data set work? Again we are using the TV marketing CSV features from the skarn

[2:47:16] pre-processing we are including the polomial features. Again first we are polomial features. Again first we are splitting the data. So now I am creating splitting the data. So now I am creating an object of polomial features right

[2:47:29] with degree equal to two. And now I am using the fit transform function. Not the fit function itself but the transform function because I'm transforming the input values as well as fitting but I'm only and only

[2:47:45] transforming the output. Do I want to train the testing data? No. Why I don't I want to train the testing data because so if I give you what is going to be if

[2:47:57] in your test if I tell you the questions is it a actually a test? No. That calls for paper leakage. If I tell you what questions are going to come for an exam that is known as paper leakage and the same concept is known as data leakage

[2:48:15] same concept is known as data leakage over here. Getting my point learners? So are you understanding the difference between fit transform and transform? The fit transform will only and only work for training data. Transformation

[2:48:31] can happen will happen for input as well as output will happen for training as as output will happen for training as well as uh testing. But fit will only happen for training data. When I create an object of linear regression,

[2:48:46] I get this output. So basically the polomial features transformer is configured to generate polomial features up to four degree. Then we are only transforming the input to generate a new feature set that includes polomial

[2:49:02] features and interaction. Then we transform the test data corrected from transform the test data corrected from fit transform. It is used on x test to apply the same transformation and then train the model. The linear regression

[2:49:16] model is trained using the transform training data. Clear. And finally we are going to predict the output

[2:49:33] see the scatter plot is testing data and plot is x range polomial predict the output. Now clear you are getting all the code

[2:49:45] Now clear you are getting all the code how the graph is getting created. So what what is the observation and this is exactly you know why Jupyter notebooks are hit in the market because we are able to see the graphs over here

[2:49:58] directly which VS code lacks right we can see the code as well as I can write can see the code as well as I can write my observations also. So as you can see the regression line is able to fit majority of the data points. You can

[2:50:13] infer from the above implementation that nonlinear inputs require nonlinear model nonlinear inputs require nonlinear model such as polomial model. Clear? So I mean regression linear regression also has capability of dealing with

[2:50:30] nonlinear data up to degree four but not very high nonlinear data. Right? And we have different variations of the algorithm. And if we talk about performance metrics for analysis, why different metrics?

[2:50:47] Because it calculates the average of the squares of error, which is differences between the actual and the predicted. Then we have the root mean square error. Then we have the root mean square error. Oh,

[2:51:04] error R square error zero value indicates model explains none of the indicates model explains none of the variance in the dependent variable. The independent variables have no explanatory power for the changes in Y

[2:51:18] and one represents a perfect fit. The model explains all of the variance in the dependent variable. The change in Y are perfectly captured by the changes in the X. Clear?

[2:51:33] So now we have to start understanding another technique important technique. another technique important technique. Okay. So now let's understand what is cross validation. And before we understand cross validation, we need to

[2:51:50] understand cross validation, we need to understand few more concepts. terms that you need to understand. one is parameters and other one are

[2:52:04] is parameters and other one are hyperparameters. between parameter? A lot of learners are you know get confused because they both think they are the same thing. No again I'm telling you I try to give you

[2:52:21] exactly you know the Chris concepts differences between them. So when I talk about parameters, they are values learned by the model during the training. So during the training, what the model learns is referred to as

[2:52:37] parameters and hyperparameters are defined by the user to control the learning process. Purpose over here is it directly impacts

[2:52:50] the model predictions. So model automatically learns the parameters you know and that is directly impacted in the output whereas hyperparameters

[2:53:03] the output whereas hyperparameters are controlled by us. For example, how much is going to be the training data and the test data is a hyperparameter and the test data is a hyperparameter test size or train_plit.

[2:53:17] Right? They the factors or parameters which are controlled by us during the training process or model fitting process that are known as hyperparameters. Whereas when the model learns the

[2:53:32] parameters on its own using the fit function are parameters. Clear? They are estimated during model training. They are generally uh you know estimated before the training begins. learn from the data using optimization.

[2:53:49] the data using optimization. Whereas the hyperparameters are set methods like grid search, random search or basian optimization. Influence on the

[2:54:06] model affects the output directly affects the speed and the quality of the learning and this is dependent on the data set. These are independent of the data set. These are independent of the data set. So the model parameters are

[2:54:20] nothing but your coefficients beta KN, beta 1, beta_2 or they can also be beta 1, beta_2 or they can also be represented by weights in general. So generally coefficients become weights. That's the general term used in deep

[2:54:33] learning. Whereas the test underscore size, the number of iterations and there are several other factors that will be controlled by us even random state are

[2:54:46] all hyperparameters. Is the difference getting clear? getting clear? Right. So now what is the need of cross validation? Do we cross validate results? Do we want do we like to cross

[2:55:02] validate views reviews of doctors, lawyers? What does that mean? Adinitri and why do we want to do that? What does it mean and why do we want to do that?

[2:55:14] So that you know we are more sure of the answer right? If one of the doctor is say saying that you know you need to be uh you know maybe you know your disease has this and maybe you need to get operated and even the other doctor says

[2:55:28] that means you're more correct that yes or if the other doctor says no no no you it doesn't need to get operated you know you can cure it through these medicines you can cure it through these medicines right so cross validation or also known

[2:55:41] right so cross validation or also known as rotation estimation or out of sample testing refers to the process of rotating or splitting the data into different subsets. So it helps in uh you know in the process of rotating

[2:55:57] know in the process of rotating splitting the data into different sub sets. Okay. So one part we are very very clear that when we take the data set we divide it into training and test and definitely we

[2:56:10] don't want to mix any of the training and testing data sets to avoid any kind of data leakage. That point is also clear that we want to avoid any kind of

[2:56:22] data leakage. That points also makes it clear. clear. Now to make our results better, more confident, more accurate, rather than using only one part of the set, if I use

[2:56:37] only one part or one sample of the set, if I use multiple samples, I can get better results. Right? So this is my mini training data set. So it's always the training data set which

[2:56:54] it's always the training data set which gets divided into further training and validation split right. So what is the idea that we initially the data is getting split into training set and test set train and tune

[2:57:09] tune your models using cross validation. We will try to find out okay we want to make certain changes this is better not better only through the training set the test set is not involved test set will we do not touch this until the very end

[2:57:24] because ultimately that is going to give us whether the model is robust or not is us whether the model is robust or not is this point clear now so how do we go about cross validation so model evaluation is this when we fit

[2:57:40] the model and then we predict the test set. That point is clear that we have one data set that is divided into training and test set. Right? This point

[2:57:52] this is normally that we do we fit it and then we predict the output and do the comparison. But if I want to do model selection between different comparisons, what I will do? Now the training data set is

[2:58:07] going to get uh you know divided into train as well as validation data set. train as well as validation data set. What is the use of this model selection? It helps us to explore the hyperparameters

[2:58:20] grid. Now there are different hyperparameters. How do you how do I know that this is the best hyperparameter for me? I will try to do it through the validation set. fit on the train, evaluate on the

[2:58:34] validation, pick the best hyper parameter. Clear? Why are we doing this? Because we want Why are we doing this? Because we want to validate our training and results

[2:58:47] better. It helps us in finding out the different hyperparameters, right? So, as we move along more uh you know algorithms, you will understand it more. It's not part of regression but part of supervised learning also. So cross

[2:59:02] validation when data set is too small for apply hold out strategy then cross validation can be used for evaluation and model selection. and model selection. So what do we do? Suppose this is my 100

[2:59:17] rows in data set. Okay. And now I divide it into 2020 groups. So I have five it into 2020 groups. So I have five groups. 1 2 3 4 5. So the one of the

[2:59:29] part is going to be used for validation other is going to be used for training. Then the next group is use going to be used for validation next. So I will used for validation next. So I will iterate it for five iteration. Got it?

[2:59:43] Of course it is increasing the complexity but it is giving me a much complexity but it is giving me a much more validated result. Agreed? more validated result. Agreed? So one of the uh you know u uh

[2:59:57] So one of the uh you know u uh techniques or variants of kfold is leave techniques or variants of kfold is leave this is known as leave one out cross validation. Right? So we have 1 2 3 and n over here.

[3:00:15] Right? So if we take one part of the sample and the rest n are taken for training then the next row is taken the rest for training. So how many time the loop will be executed n times. Is it a good idea? No. Taking each sample for

[3:00:32] test sorry for validation and others for training is not a good idea because it will increase the number of computation. But if I have kfold that is I divide it

[3:00:44] into number of groups my number of iterations decrease and I get better results. So how do we go about the performance and the output metrics right? So what will happen it will so the cross

[3:01:00] validation will automatically create an array in the first iteration I will get the first metric when it takes the second kfold I get the second performance metric when I take the third iteration I get the third performance

[3:01:13] iteration I get the third performance metrics fourth the fourth one and last one I get the fourth fifth performance metrics clear so to answer that point everybody please understand the test data is not touched. It is the training

[3:01:29] data and the different folds which I use and final evaluation is done on the test and final evaluation is done on the test data whether I am getting the correct output. So the concept is same we are doing comparison between training and

[3:01:42] results whether it's overfitting underfitting etc. Clear? Yeah. But when we talk about categorical data, what does categorical data means? That the data is in category in terms of males and female.

[3:02:03] cross validated will be taken up for final testing. Final testing will be on final testing. Final testing will be on the test data only. Once the parameters have been found out okay the number of folds okay the number of uh you know u k

[3:02:18] value for this thing is this. So now we will test on this value are we getting the minimum results or not it will help us to calculate the different us to calculate the different hyperparameters. Got it? Okay. So when

[3:02:31] we talk about categorical data dividing it into male and female. So this is my round one. So stratified kfold is being used for categorical class which helps us to keep the ratio of the different

[3:02:48] categories same. Okay. This we will do it when we do uh classification. Okay. So what is the difference between kfold and stratified kfold? Kfold is random. Stratified kfold helps us to maintain proportions may

[3:03:04] vary across folds. maintains class uh distribution across FOS imbalanced data set not ideal preferred use cases balance and imbalance and how do I check the metric that my cross validation is good or wrong first my uh you know

[3:03:20] parameters for cross validations are the estimators input output scoring and CV estimators input output scoring and CV equal to five estimator is equal to the model object X is an array of the features values. Y

[3:03:36] is an array of the target value. CV the number of folds. Scoring the metric and it returns score an array of scores at each split. So you know the beauty of

[3:03:48] this particular function is that it returns at you know score of each uh iteration. So you are able to see which one is better but generally we take the

[3:04:00] average of that. Clear? So now let's get back to the file and do it practically. So this file is quite long. We will be using this file in the next session. Of

[3:04:16] course we will not be able to complete all the concepts today. So we are on cross validation. After cross validation then we have regularization. after regularization then we have uh

[3:04:33] hyperparameter tuning that is model optimization and then the pipeline so it's it's a long journey that definitely we are going to continue okay in this file yeah so cross validation technique everybody is there

[3:04:47] validation technique everybody is there with me 3.7.2 into. cross validation technique? Tell me learners. So cross validation is a

[3:04:59] machine learning technique that evaluates the model performance on evaluates the model performance on unseen data by dividing the data into multiple folds. In each iteration, one fold is used as a validation set and the

[3:05:13] remaining as training. That point is clear. Then the process is repeated. So we have to give the number of iterations also. Yes. Is repeated so that each fold serves as a validation set once and the

[3:05:27] serves as a validation set once and the results from all iterations are averaged to provide a robust estimate of the model performance. Some of the common model performance. Some of the common cross validation techniques

[3:05:49] is kfold cross validation. K refers to the number of equally sized folds that if it is unequal it will make it equal by adding randomly some of the you know randomly the data set only for the last one right and there is no harm also in

[3:06:04] doing it will not affect it much so the model is trained on k minus one folds and tested on the remaining fold that point is clear so if I have five folds

[3:06:16] then one of the fold will become the valid for validation test and four will valid for validation test and four will be tested. This process is repeated K times. So is this K also a hyperparameter?

[3:06:30] Yes, we decide they are going to be fivefold, sixfold, 7fold, 8folds, not the system. They are not parameters to the algorithm. It is decided by us. So it's a hyperparameter. And with each fold exactly once as the

[3:06:47] test data set the results are averaged to produce a single performance estimate we generally average it uh them. What is the advantage of kfold cross validation?

[3:06:59] It provides more accurate estimate of the model performance. Yes, that this is the answer. K is the number of folds. Yes, AJ. And consationally intensive for large data set. Yes, you have to reiterate the training process. But it

[3:07:15] reiterate the training process. But it gives definitely better results. Clear? Then we have stratified kfold. Similar to kfold but ensures that each fold has the same proportion of different classes as the original data

[3:07:29] set. This is especially use useful for imbalanced data set. More reliable performance estimates for imbalanced data set data set and still computationally expensive.

[3:07:43] Third is the hold out method. Simple and fast one. Uh we we divide the data into training and test and the training data is first further divided into training and validation data set. So the model is trained on training set and evaluated on

[3:08:01] the test set. It is simple and fast. The evaluation may be noisy and variability in the training. So the normal one that we do we divide it into training and we do we divide it into training and test is known as the hold out method

[3:08:14] and leave one out cross validation is again a special kind of cross validation technique used in kfold. A special case of k-fold cross validation where k is equal number to the number of data points in the data set. Each observation

[3:08:30] is used once as a test set. The model is trained on remaining data points. trained on remaining data points. Clear? So this maximizes the amount of training data used. Cons, extremely computationally expensive especially for

[3:08:45] the large data set. Clear? Four variants of cross validation techniques. Kfold stratified hold out and leave one out cross validation.

[3:09:01] validation. Okay. Now can we begin with the code? So I hope everybody is familiar with the pandas and the mattplot lab library. We are also familiar with the skarn

[3:09:17] model selection. These are the different uh techniques and metrics linear regression over here and this metrics to compare mean square error absolute R compare mean square error absolute R square etc. Clear?

[3:09:32] square etc. Clear? So here we are talking about housing. CSV. Yeah, take it as housing. So what are the pre-processing steps? What are the steps involved in EDA? Tell me how do you perform EDA? You'll do head tail

[3:09:47] to view the data. Check null values. Do info. Right? So let's look at info. So if you look at info, it has around 20,640 rows and eight columns. Do you

[3:10:02] see this? All are integer values. Do you see this? Yeah. Now the question is can you tell me which is the what is the output over here? Why it is a regression problem? What are we trying to do in this data

[3:10:16] set? Can you analyze it through the number of column based on the latitude, longitude, housing, age, total rooms, bedrooms, they all are inputs? I am trying to

[3:10:30] predict the house value. What will be the value or the price of the house? House price prediction. Now clear all these are my inputs and one output.

[3:10:43] House values are numerical continuous value. Therefore, value. Therefore, this is known as regression. Clear? Okay. Now, let's let's see the observation that price lies between 1.1

[3:10:58] million to 2.6 million. Houses are generally 18 to 37 years of old. Housing generally 18 to 37 years of old. Housing data. Let's check the null values.

[3:11:11] Yeah. Now, let's check the null values. So total number of bedrooms we have 27 So total number of bedrooms we have 27 values. How do we deal with null values? Either we will replace it by mean, median or zero. Since null values make

[3:11:24] median or zero. Since null values make up only 1% of the total data, rows and column features with missing values will be removed. So what are we trying to do? We are going to drop now because 207 is hardly 1% of 20,000 rows. So we can drop

[3:11:40] hardly 1% of 20,000 rows. So we can drop those data and now my data set is clean right and this is the categorical data which we are not using we don't have which we are not using we don't have that and now can you tell me what does

[3:11:53] this mean X and Y now I have dropped the median house value access one because that's my output and now this becomes my output this becomes my input Right?

[3:12:11] So I say print X and then print print Y. Do you see? Yeah. So now do you see

[3:12:24] Y. Do you see? Yeah. So now do you see this? So X basically has longitude, this? So X basically has longitude, latitude, house, median, total bedrooms, etc. And Y is only that. So let's remove Y.

[3:12:36] left. So I keep making changes in the code. Are you understanding? These are my now the input and my output is median house value and then I use the train test_plit.

[3:12:51] then I use the train test_plit. Clear? Okay. Now let's perform kfold cross validation. It implements kfold. Number of splits is 10. Divides the data set into 10 10folds automatically. random state is 42 shuffle is also equal

[3:13:07] to so I have created an object of this kfold now I initialize the model there kfold now I initialize the model there with me

[3:13:22] score based on the model model was linear regression training data and the target variable scoring I'm using negative mean absolute absolute error as

[3:13:34] the performance metric. So this is where the absolute error comes into the the absolute error comes into the picture. CV cross validation is KF. Number of jobs is equal to minus1. What does this mean? It utilizes all the

[3:13:47] available processes for parallel computation. Do we understand the concept of threading or the number of core processes learners? core processes learners? So this parameter takes care of that.

[3:14:02] So this parameter takes care of that. Okay, this parameter takes care of that Okay, this parameter takes care of that particular one. Clear? So from statistics import mean I'll take the mean of the K4 cross validation K4

[3:14:17] scores are not defined. So let me run this. Let me you know show you that since the number of folds is 10. So the output of Kfold crow is an array object. See do you see they all are mean negative

[3:14:32] absolute errors negative does not mean that the error is less it's just the sign okay so we will try to take the average of these right there will be 10 outputs since we have given 10 folds 1

[3:14:48] since we have given 10 folds 1 [clears throat] 2 3 4 5 6 7 8 9 10 clear and [clears throat] then I will try to take the absolute ute average of this

[3:15:01] take the absolute ute average of this clear. So we have trained the model and evaluate on the test set. So are we now understanding how are we going about understanding how are we going about cross validation score learners

[3:15:14] and now finally after cross validation now we can use the test data. So this is how I have fitted the data predicted it through the test and my MSE test MSE is

[3:15:27] here. Similarly, my R square is here, right? So, my test MSE and R square right? So, my test MSE and R square scores are here, right? This the look at the error. It's so huge indicating that and on an average the squared prediction

[3:15:42] interpreted in context of units of dependent variable which are likely in order. Since the numbers are very high that is why the errors is coming out to be large. This can be mitigated by

[3:15:56] scaling the features. Feature scaling is critical in machine learning to ensure critical in machine learning to ensure that all features contribute equally. So this is how and then we demonstrate leave one out also

[3:16:11] loves and we get the absolute mean score. Clear? So the mean absolute error is high to

[3:16:23] improve the model complex more complex model can be considered which will be discussed in further lessons. So the concept of cross validation is clear. Let let's do a quick knowledge check on that. Okay a quick knowledge check.

[3:16:37] First question. First question. Which of the following which of the following cross validation versions may not be suitable for very large data set with hundreds of samples? We just now studied that

[3:16:53] I think so we've all seen it practically you know practical has a major uh impact we all have just now seen the impact and it is leave one out cross validation great which of the following is a disadvantage

[3:17:09] which of the following is a disadvantage of kfold uh cross validation method training algorithm has to return from scratch. Do we understand this? Every time it has to restart and do it again.

[3:17:24] question. Suppose you have picked the parameter Suppose you have picked the parameter for model using 10fold cross validation. Which of the following is the best way to pick a final model to use and

[3:17:38] estimate its error? Train a new model on the full data set using the parameter you found. Use the average cross validation error as its error estimate.

[3:17:50] So you can use average cross validation error as its error. Everybody got this? Why C is not correct? Because we will train the model on the full data set. Okay. So what is the idea? The best way to pick why is the answer B correct? The

[3:18:07] best way to pick a final model is to train a new machine learning model on the full data set using the parameter learn to use the average cross validation error as its error estimate. So please be clear with this particular

[3:18:22] So please be clear with this particular point that is why I have added this MCQ. We can compare different models using cross validation. Cross validation is mainly used for comparison of different models. For each model you may get

[3:18:37] average generalization error on the K validation sets. Then you will be able to choose the model with the lowest average. Clear? And cross validation is also used for

[3:18:49] And cross validation is also used for model checking not model building because it allows to repeatedly train and test on a single set of data set. regression model and a neural network. To select the best one among these we

[3:19:04] can use kfold cross validation. So to compare between the models also we can use cross validation to select a better performing model. Clear?

[3:19:16] performing model. Clear? So when we are now looking at the regression outputs right or the regression analysis some of the outputs are like this. So the output over here that we receive is

[3:19:31] in these terms of coefficients that this is my beta kn this is my beta 1 beta_2 is my beta kn this is my beta 1 beta_2 and this is my beta 3. So this is my

[3:19:43] north south east and constant term and these are the values right. So what is it showing? This is the constant value positive. East has positive relationship. But south has more positive relationship with the heat

[3:19:58] flux. Heat flux is the output and north has a negative relationship with the output. Are you now understanding it better? How do we get the output and how better? How do we get the output and how are we relating it with the uh

[3:20:12] you know the equation of linear regression. So this is nothing but like beta kn plus beta 1 x1 the value of east plus

[3:20:26] beta_2 x2 minus. So a lot of you had question that how do we understand negative it will automatically get this. So this is will automatically get this. So this is multiple linear regression. Yes learners

[3:20:40] multiple linear regression. Yes learners this is multiple linear regression and there was lot of confusion regarding ma'am what does multi-olinearity ma'am what does multi-olinearity mean? Multi means referring to multiple

[3:20:52] independent variables multiple inputs with multiple regression. Call means to uh join or together referencing to the linear movement or correlation as I told you tries to find out correlation in terms of minus1 + one occurring within

[3:21:08] terms of minus1 + one occurring within the linear equation and suffix means the the linear equation and suffix means the idea. So if we look at over here from statistics we understand that p value is a very very important term

[3:21:22] a very very important term right that if I have the value lesser than 000.5 then this is accepted this is accepted this has a strong evidence statistically significant but if this is not less than

[3:21:37] significant but if this is not less than 0.05 05 that means east is 2.12 is not very statistically uh proven or confident that the value is this clear

[3:21:50] and when I talk about vif how do I calculate multicolinearity it is the variance inflation factor which is coming out to be 1.21 21. What

[3:22:02] does that mean? This is the correlation map, a heat map that you understand. So, multicolinearity is the phenomenon of high correlation between the predictor variables can

[3:22:15] create instability and bias in regression model. To identify and address multi-olinearity, we use the variance inflation factor. So the VIF is

[3:22:28] variance inflation factor. So the VIF is equal to 1 - 1 upon R² and if the value is equal to 1 that means all the input values are independent and if it lies between 1 to 5 it suggests moderate correlation. So over here we are getting

[3:22:43] the value between 1 to 5. So there is moderate correlation or we can say independent also and if it is greater than five then it indicates high correlation. That means then linear regression cannot be fit uh

[3:23:00] can be fitted on that input values. So where e vif should should not be used polomial equation dummy variable or a nomial um variable

[3:23:15] reduces the statistical significance of the independent variables. VIF is used to detect these variables. A large variance

[3:23:27] detect these variables. A large variance inflation factor on an independent linear relationship to other variables. that should be considered or adjusted

[3:23:41] for structure because multi-olinearity is one of the assumptions that we uh you know uh you know assume when we are uh trying to

[3:23:53] build the linear regression model. So that means if the vif value is greater than five then linear regression should not be used. Clear?

[3:24:07] not be used. Clear? I hope these points are clear. Now first concept that we will learn data leakage in machine learning. So now we will understand the concepts of pipeline. Today pipeline is a technique

[3:24:22] for automating different processes. What what is the different processes that we do? Whenever the data is loaded, we want to do transformations such as encoding, scaling, right? So we'll try creating a pipeline and

[3:24:37] So we'll try creating a pipeline and what is our main aim that we want to avoid data leakage of course why because if there is data leakage the data gets lost a lot of information is also getting

[3:24:52] lost. So now if we look at the data from the supervised learning perspective it the supervised learning perspective it is divided into two parts. one is known as the training data and the other one is known as the test data. Agreed?

[3:25:09] is known as the test data. Agreed? Okay. So a scenario when the ML model Okay. So a scenario when the ML model already has information of a test datas test data in the training data. Do you think that test data should be present

[3:25:24] in the training data? If I tell you okay in your exam these questions are going to come that going to be very beneficial for you for scoring marks but do you actually learn out of it?

[3:25:38] actually learn out of it? No, that's not a good uh way of learning right that you know it can give you good results but you are not going to become you're not going to be a robust model. That means if any other question is

[3:25:51] asked from that particular topic you will absolutely fail. Right? But this information would be available at the time of prediction called detail leakage. So what is the disadvantage and how can we avoid it?

[3:26:06] That it causes high performance while training set but performs poorly in the training set but performs poorly in the deployment or the production.

[3:26:23] say when there is an overlap of training and test data then it causes data and test data then it causes data leakage right and basically data leakage

[3:26:35] happens due to two reasons. First we understand train and test contamination or the target leakage. What do we mean by that? Target leakage occurs when the

[3:26:47] model is trained on the training data that contains target or the feature information. So we don't want that and that should not be available at the time

[3:26:59] of prediction. So we have to be very very careful when we are doing this. The other one is the train test contamination

[3:27:13] is an event where the test data leaks into the training data and the data prep-processing steps for transformation for example scaling encoding are applied before the splitting the data set. So contamination is when the test data

[3:27:28] leaks into training that and the pre-processing step. So that uh the pre-processing step. So that uh the scaling encoding should not be applied before the splitting of the data set to avoid data leakage. Got it?

[3:27:47] Basically if the there is mixing of training data in the test test data set training data in the test test data set or rather test data in the training data or rather test data in the training data set then this causes data leakage and we

[3:28:01] set then this causes data leakage and we have to avoid it in every case that is we should not apply any transformation that is scaling or encoding before the that is scaling or encoding before the splitting of the data set. Now clear.

[3:28:15] splitting of the data set. Now clear. Now another very important concept of regularization. There are two types of regularization available under regression that [clears throat] is ridge and lasso.

[3:28:30] Preventing overarning. Yes, normalizing the data. Yes, we can say that. So regularizing thing in in a normal way, right? We want to prevent it from

[3:28:42] overfitting overarning. Right? So regularization is a technique in machine learning which prevents overfitting of the model. And how do I know that the

[3:28:55] model has overfitted? How do I know that the model has overfitted? Test error much higher than the training. The training error. And why is it that model is trained so complex than

[3:29:09] requests high variance it covers every possible train outcome making it yeah so the complexity of the model increase it tries to train on each and every data tries to train on each and every data point and fails on the final test data

[3:29:22] due to noise due to learning of noise in the training data so to avoid it so as I've been telling you that in machine learning do we suffer more from

[3:29:34] underfitting or overfitting overfitting right so we need methods so that our model does we need methods so that our model does not overfit and we need certain control

[3:29:48] parameters to achieve that. So there are two you know algorithms under this that two you know algorithms under this that is ridge and lasso. So lasso is known as L1 regularization technique. Can anybody tell me what does

[3:30:05] this term mean? Anybody? It is summation of the actual value or the true output minus the predicted output the whole square. What is this

[3:30:17] term known as? It's an error. I want the typical name of this error. It is the typical name of this error. It is the residual sum of square error. But rather residual sum of square error. But rather than only taking this error actual minus

[3:30:31] the predicted value the whole square I take into account the another hyperparameter lambda which I will use to control or

[3:30:44] regulate the training process. So do we have a regulator? Do we have a regulator to control the speed of the fran? Similarly lambda we will use the Similarly lambda we will use the regulator and similarly summation of

[3:30:58] beta. Can anybody tell me what is this beta over here? What does the term beta mean? It is the coefficients. It could be a single, it could be many multiple variables. So this is known as the

[3:31:13] variables. So this is known as the regression coefficients. is only in twodimensional case like beta kn plus beta 1 x1. I agree both of you

[3:31:26] are clear uh correct on those perspective but in general betas are known as regression coefficients or the weights parameter.

[3:31:41] So what is the advantage that we are getting through this L1 regularization getting through this L1 regularization that in the error term in the RSS term that in the error term in the RSS term now we have added the penalty lambda

[3:31:53] which is controlled by us and which controls the value of these regression coefficients. Getting my point?

[3:32:09] regularization. So this is now also you know known as my error or it is also known as the cost function. function. Try to understand in deep learning the

[3:32:22] error with which we are calculating becomes my cost function. In this case it is represented as L over here. It represents it as my loss function. there

[3:32:35] is slight difference but actually in machine learning they all mean the same thing. Okay.

[3:32:49] this W over here? Is it the same as beta? regression coefficient. Is the lambda same? Yes. The lambda in Python is known

[3:33:05] same? Yes. The lambda in Python is known as alpha and then we have sum of the square of the weights and what is this actual value predicted value. Now are you understanding whether it's w or beta they same mean the same and this is the

[3:33:19] they same mean the same and this is the lambda parameter with the square of the coefficients. Okay. So what are the advantages and disadvantages and now then we will practically jump onto the file and start learning from there. So

[3:33:34] L1 regularization performs feature selection. What do we mean by features independent variables or the inputs? independent variables or the inputs? Yes, the inputs. So now we are trying to

[3:33:49] control. So now what is the use of overfitting? We understand that overfitting? We understand that overfitting generally leads to overfitting generally leads to high variance. Agreed learners

[3:34:02] and complex models. Agreed? Do we understand this concept everybody? So to understand this concept everybody? So to reduce the complexity now I will try to control my coefficients regression coefficients beta kn b1 and beta n or in

[3:34:21] coefficients beta kn b1 and beta n or in general I can also call them as weights. Now getting my point do not get confused with the terminologies it somewhere it would be written as weights somewhere as beta somewhere as independent variable

[3:34:35] somewhere as features. So you should be able to able to understand the concept of it. Clear? So understand the concept of it. Clear? So the beauty of L1 regularization is that

[3:34:49] it performs feature selection by shrinking the less important features weights to zero. So if I have a multivariable data set from beta 1 to beta KN to beta 7 it will shrink few of the features

[3:35:06] that means some of the features or the weights will become zero that means they have no significance relationship with the output. For example, if we want to

[3:35:19] yeah, just try to listen and absorb as much as you can that suppose you know we want to predict the price of the car. Okay. So there are several factors from fuel to design to color of the car to the alloy of the wheels to the security

[3:35:34] systems to the infotainment system to the sunroof. There can be several re features but maybe the alloy of the car or the wheel of the car might not be important. So it can be reduced to zero

[3:35:48] complexity and that will try to reduce the and that will try to reduce the overfitting of the model. So very very overfitting of the model. So very very important concept L1 is used to reduce

[3:36:02] important concept L1 is used to reduce or shrink the less important features weights to zero. So it helps us in feature selection right and it can also be used for highdimensional data set with many number of columns such as 20

[3:36:19] with many number of columns such as 20 30 50 100 with many irrelevant features and the disadvantage is it is not effective for data set with many important features. Now you might say ma'am how can we remove a feature? my

[3:36:34] every feature is important for the output. Then we use L2 regularization. Okay. Where the number of the where the value

[3:36:46] of the coefficients of the weights will not become zero. It provides a smooth not become zero. It provides a smooth solution and improves the generalization solution and improves the generalization performance of the model. So ma'am ridge

[3:36:59] is going to doing the opposite instead of zero it is reducing the weight. of zero it is reducing the weight. Ridge is trying or uh you know it will it is trying to see the first one is making it zero and the L2 will try to

[3:37:16] making it zero and the L2 will try to reduce the value. L1 is lasso and the ridge one is making the value of the coefficient small not huge or big but small. So therefore the biggest advantage is that it can handle data

[3:37:33] advantage is that it can handle data sets with many important features. sets with many important features. Okay. So now moving ahead to great ain. Everybody has the file. Everybody's ready in the Jupiter notebooks. Okay.

[3:37:50] So, regularization in regression in linear regression, regularization encompasses a set of techniques employed to address the issue of overfitting. So,

[3:38:03] what is regularization? Regularization is the method techniques to achieve the is the method techniques to achieve the objective by introducing a penalty term. Please try to understand again it's a very very important question from the

[3:38:18] point of interview. Introducing penalty term lambda to model's objective term lambda to model's objective function to prevent it from overfitting. This objective function typically

[3:38:33] measured by mean square error is minimized during the training process. So generally the error of the mean square error or the RSS to be precise is

[3:38:45] minimized during the training process. What is the advantage of that penalty term or the lambda? The penalty term discourages the model from attaining discourages the model from attaining excessive complexity by penalizing the

[3:38:59] size of the model coefficients. So model coefficients, regression coefficients beta KN, beta 1, beta 2 or the W1, W2, WN thereby mitigating the overfitting

[3:39:12] WN thereby mitigating the overfitting process. Got it learners? All right. So now if we talk about the regularization term alpha, it can be known as alpha as well as lambda. Okay,

[3:39:26] they all mean the same thing. It's written as lambda or sometimes as alpha. So do not get confused. Okay. So there is a little terminology mishap happening. So everyone uses their own technique but try to grasp the concept.

[3:39:42] technique but try to grasp the concept. Okay. scales the penalty term. It controls the strength of regularization.

[3:39:54] Higher the alpha, it imposes stronger penalty on the coefficients. That means they tend to become zero leading to greater regularization. This tends to produce a simpler model that may underfeit the training data but

[3:40:09] that may underfeit the training data but often generalizes better to unseen data. If we talk about lower alpha, it imposes a weaker penalty leading to a model that is less restricted by regularization and more complex potentially capturing more

[3:40:25] details in the data but at the risk of overfitting. So what are we looking at? We are definitely looking at a value which is not very high and very low. So which is not very high and very low. So again minimum mid value to find out.

[3:40:40] Got it? How do we find it? What are the best methods that we will understand today? That is known as hyperparameter tuning. That's part of this today's session also. Okay.

[3:40:55] regularization? It enhance the generalizability of the model by mitigating the overfitting factor. Regularization fosters model

[3:41:07] factor. Regularization fosters model that can perform well on unseen data. reduce model complexity. It promotes interpretability and potentially reduces computational cost associated with training complex

[3:41:21] models. And the two common regularization And the two common regularization techniques are L1 and L2. L1 is lasso. Lasso is a full form of least absolute shrinkage and selection operator. Right?

[3:41:38] shrinkage and selection operator. Right? So absolute uh term is there as the penalty. So Simon is there any standard to label it as high, alpha or low? Yeah, to label it as high, alpha or low? Yeah, we'll understand.

[3:41:54] negative values and high values are in thousands and lakhs. Let's do it practically to understand that point better. Okay. So is the full form of lasso clear to everybody? So which is the penalty term? Of course it is the

[3:42:10] alpha or the l uh you know the lambda. But what are we trying to add? The absolute value of the regression coefficient. So the least absolute shrinkage and selection operator regression relies upon the linear

[3:42:26] regression model but additionally performs a so-called L1 regularization which is a process of introducing additional information in order to

[3:42:38] prevent overfitting. As a consequence we can fit a model containing all possible can fit a model containing all possible predictors. What are predictors?

[3:42:50] feel ma'am is ma'am just reads the data but I'm purposely reading it to make you understand line by line. So what is containing all possible predictors?

[3:43:02] Predicted value is an output. Predictors are input. So we can fit a model containing all possible predictors and use lasso to perform a variable selection by using a technique that regularizes the coefficient. So what is

[3:43:17] that lambda parameter doing? It is having control on the value of the of having control on the value of the of the the lambda or the alpha has it's controlling or regulating the values of the

[3:43:31] coefficients the coefficients the regression coefficients that's up to you how you want to represent it as betas or weights right so it performs variable selection or feature selection it forces some of the

[3:43:48] coefficient estimates to be exactly equal to zero with the help of large tuning parameter. So more the value of the lambda some of the features coefficient value will turn out to be zero. And that is why this is a

[3:44:03] technique which is also used in dimensionality reduction. What is dimensionality reduction? Reducing the number of features in the data set so that it reduces the complexity. So L1 again plays a major major role in that.

[3:44:19] It reduces it helps to reduce learning of more complex data and overfitting. It decreases the variance of the model without increase in the bias. All right.

[3:44:31] So in minimization objective does not include RSS like the OS regression but include RSS like the OS regression but also the absolute value term. So this RSS now is clear. This is the residual sum of square and this is how we can

[3:44:47] expand it and write it. This this point is also clear. What is yi? This is the actual output. And what is this output? This is the predicted output. Are you all understanding it mathematically,

[3:45:03] conceptually? But in lasso, where does the difference But in lasso, where does the difference come? The RSS is the same but we have added the penalty term alpha or lambda along with the absolute value of the

[3:45:16] along with the absolute value of the summation of the coefficients. Now clear if the alpha is equal to zero then there is no regularization that will happen because the error term will be same as the RSS. If it is equal to infinity all

[3:45:33] the coefficients will become zero. And if the alpha is greater than zero lesser than infinity coefficients are between zero that of le square linear

[3:45:46] regression. Got it? Is the theory part clear? Now let's move on to the practical part. Start with a new data set. So let me share it with you. hitters CV ca dot csv okay here we go

[3:46:04] please download this data set and be ready I hope different libraries are also clear the numpy pandas then linear model lasso the metrics mean square error r square all these are there okay

[3:46:22] so basically now let's understand the description of the data set so I I have description of the data set so I I have provided a link over here. So, uh this is for like we have like now IPL matches. So, here we are trying to

[3:46:36] predict the salary of the player based on atbat number of times he batted in 1986 hits the number of hits the number of home runs the number of runs the

[3:46:49] of home runs the number of runs the number of runs batted walks number of times a bat during his career. So lots and lots of um parameters to decide the

[3:47:01] salary of the player that is 1987 annual salary on opening day in thousands of salary on opening day in thousands of dollars. Got it? And then we have another important column that is new league a factor with

[3:47:16] a and n indicating the players league at the beginning of 1997. loaded the libraries. is we load the data set and the view of the data set

[3:47:28] with df do head. Okay, what are the other functions that Okay, what are the other functions that we would perform info? So this data set we would perform info? So this data set is quite huge in the sense that it has

[3:47:41] the number of rows are only 330 uh 322 uh 322 but the number of columns are many. but the number of columns are many. So this is an object uh data type atbat

[3:47:54] everything is integer but we have league division as categorical data and then we have salary over here as the output and new league. So what are the

[3:48:07] different steps that need to be performed? Can we directly apply the model onto this data set? So these are my columns. So I have used df.drop Drop

[3:48:19] unnamed equal to zero. Encoding if any. Do you think encoding is required? Coding is encoding is definitely Coding is encoding is definitely required for league division for all the

[3:48:32] columns which are of object data type. Please remember this point learners encoding is definitely required for object data type. Without that you will

[3:48:44] not load it into the model. duplicate values. If they are then we need to check that. Okay, we are removing unnamed column because that is not required. We are specifying the access. What does in place equal to

[3:48:59] the access. What does in place equal to true mean? Permanent removal of that true mean? Permanent removal of that column from the data set. So now if I see my first column gets removed. So the first processing that I have done. Now

[3:49:14] let me check the null value. Are there any null values over here? Only the null value is in the output in the salary field. Right? Which is my output field. Agreed learners?

[3:49:29] Do I need to separate my x and y also? Yes. So the number of missing value in Yes. So the number of missing value in salary is 59. So 59 out of 322 observation with null values correspond to columns salary. Since we will use the

[3:49:46] lasso algorithm from the scikit, we need to encode our categorical also. Okay. Now, how do we deal with categorical data? What is the use of value counts

[3:50:00] function? It gives the category along with the frequency along with the count. Right? That's incomplete. It gives the category along with the count. So in

[3:50:12] league there is A and N with these categories. In division there is W and categories. In division there is W and E, Western and Eastern and again in new league we have A&N the American League or the National League.

[3:50:27] So if I separate this into a data frame do you see this do you see this data set data frame rather? This code is getting clear to everybody. Are we here till here?

[3:50:44] Okay. So, what are the different ways of encoding data? One is one hot encoding, other one one hot encoding, other one label encoding. What is the difference label encoding. What is the difference between the two? One encoding adds

[3:50:59] column and then gives the binary output. If that column value is there, then it If that column value is there, then it has one. Else all values are zero. Yes, a label and assign integers to each of the category. Yes. And label encoding is

[3:51:14] the category. Yes. And label encoding is used when we have a ordinal data. One hot encoding is used when we have nominal data. That means there is no order. So over here we are using which function learners?

[3:51:28] function learners? Yes, it is now one hot encoding that now it will have instead of three six columns with zero and one value. Right? So do you see this league a league n division e do you see

[3:51:46] league a league n division e do you see now? So wherever we have value of lee get dummies is one hot encoding and label encoder that is the function. So is the output of dummies.head head is also clear that now if these are the six

[3:52:03] the three categorical value get converted into six and wherever the value was there this is one or this is zero this is zero and this is one clear okay so what are we doing we are separated the output x numerical value

[3:52:19] we are dropping the output the league the division and the new league because they all are categorical data and everything gets converted into float type. So the numerical columns are now clear.

[3:52:33] The input x numerical and the output y salary. Good chakra pani. Good. Now do salary. Good chakra pani. Good. Now do we need to concatenate these columns with the original data? So we'll just take one of them since they it's it has

[3:52:47] binary category. Please try to understand. Since all of them have binary category, therefore it can be used for league n, division_w and new league n. Getting my point?

[3:53:04] So now you see all my inputs have become integer values, numerical values and integer values, numerical values and that is what is required before you feed any data into the model. Now once my data is ready now we go in

[3:53:21] for testing and splitting of my input and output. Test size is 25% rest is and output. Test size is 25% rest is training and I get my four outputs. Right? So lasso you know performs best when all

[3:53:36] the numerical features are centered around zero and have variance in the same order. Homocidasticity needs to be maintained. And if a feature v has a variance that uh that is orders of magnitude larger than others, it

[3:53:52] might dominate the objective function and make the estimator unable to learn from others. So it should not be that you know one of the columns have very high values and the other one low. This means it is important to standardize our

[3:54:07] features. We do this by subtracting the mean from our observations and then dividing by standardization. What is this concept known as

[3:54:19] a feature engineering? Okay, I've given you one hint. This is the concept you one hint. This is the concept involved in feature scaling and this refers to standardization that is scaling the factors uh with mean equal

[3:54:35] to zero and standard deviation equal to 1. Clear? To avoid data leakage, standardization of numerical features should always no be performed after data

[3:54:50] should always no be performed after data splitting only for the training data. Please try to understand to avoid data leakage. We will only and only use it after the splitting of the data. We saw that. So what is data leakage? You

[3:55:06] understood what is data leakage that it occurs when the information from the outside the training data set is used to create the model. This can happen if the data would not be available at the time of prediction is included in the

[3:55:22] training process. So data leakage can lead to optimistic performance estimates and models that fail to generalize well to new as well as unseen data. Clear?

[3:55:38] So which is the Python function? Which is the Python function to perform standardization standard scaler which creates an instance of the object fit.

[3:55:50] creates an instance of the object fit. It trains the X-ray numerical data that is training means to find out the parameter to compute the mean and the standard deviation for each feature in X train list numerical. X train is your

[3:56:09] for each uh data set and the list numerical is a list of column names corresponding to the numerical features. Got it? What is the use of transform?

[3:56:25] training data which transforms each feature in the training data set to have mean zero and standard deviation one. Right? And then we only use the Right? And then we only use the transform function on the test data. Now

[3:56:39] transform function on the test data. Now clear function to perform standardization. Standardization is one of the feature

[3:56:52] scaling technique where we take mean equal to zero and standard deviation equal to one. Anush that point is getting clear.

[3:57:04] data that is numerical data. Till here also this point is getting Till here also this point is getting clear numerical data and transform we are doing it on the train data

[3:57:20] doing it on the train data that is how we are going to avoid data leakage now clear got it got it but when we talk about uh you know uh

[3:57:33] testing data are we going to apply fit function on it No, we are only going to transform it. So it applies the same standardization parameters mean and standard deviation computed from the training data to test data and this

[3:57:48] ensures that the test data is scaled in the same data as training now clear. Okay. Right. So practically how do we go about it? So we have imported the

[3:58:04] standard scalar from the skarn library and then we perform this function. So and then we perform this function. So the explanation is given above. are fitting the standard scala to numerical features so that there is no

[3:58:19] numerical features so that there is no biasness in the data set. Now uh you know you'll see positive negative value because the sum of all these values should be zero. The mean of all them should be zero. clear to everybody?

[3:58:39] So the training data has 241 rows and 19 columns. Clear? Similarly, test one will only have 81 rows because of the 25% data and

[3:58:52] 19 columns. Clear? Now, there were missing values in the output data. How do we go about it? output data. How do we go about it? How do we go about it? Let's see. Yeah.

[3:59:07] So, what are we doing over here? Yes. What are we doing? We are taking the training model as well training data as well as the testing data and filling it with the median value. Wherever there is null value, it will fill it up with

[3:59:24] median. Not mode over here. Dr. Henry we see it practically that it's getting filled with median. What is the difference between mean and median? Besides that which one is affected by

[3:59:39] the outliers? Where do we use median when we do not want to get affected by the outliers? So now let's move ahead and understand the

[3:59:51] lasso model. So whenever we have to create how do we begin with the create how do we begin with the hyperparameters? how do we begin with hyperparameters? We will randomly assign any value and

[4:00:08] then start training on it. And to find out the actual meth value we do hyperparameter tuning. So we apply lasso regression on the training set with regularization parameter that is alpha equal to 1. So

[4:00:24] we begin with a very simple value alpha equal to 1. This value is commonly used as default and provides a good balance between maintaining model complexity and

[4:00:37] reducing overfitting. Okay. So by default whenever we have to do thing you can randomly assign any value to the hyperparameter alpha is equal to 1 maximum iteration is equal to 10,000

[4:00:52] and now we are fitting the training data and this is now my new intercept coefficients the regression coefficients beta KN and the other coefficients because since they are how many parameters can you tell me how many

[4:01:08] parameters are there in this particular question. How many are there? 19. All the numerical values, right? So, we would numerical values, right? So, we would have 19 coefficients. And do you do you

[4:01:22] see after applying lasso some of them have become zero? This one, this one, have become zero? This one, this one, this one and this one. Getting my point.

[4:01:36] So if I see the output these are my coefficients and some of my coefficients have become zero. Do you see? Then you might say ma'am what is negative0? It just puts the sign. There is nothing no term in mathematics as0.

[4:01:52] Clear? So what do we conclude? This is the intercept term of your lasso regression model. It represents the expected mean value of the independent variable when all the

[4:02:07] independent variables are set to zero. In practical terms, it is the baseline prediction when no other information from the variables is provided. So the

[4:02:19] lasso coefficients represent the relationship between the variable. Positive coefficients. Now what do we interpret out of it? Let's see that a positive coefficient indicates that as independent variable increases, the

[4:02:33] dependent variable also increases. That's the positive relationship. A negative coefficient indicates that an independent variable increases, the dependent variable decreases. The magnitude of the coefficient shows the

[4:02:48] strength of the impact. Now clear? So the magnitude of the coefficient shows the strength of the impact. A larger absolute value indicates a stronger effect.

[4:03:08] Lasso regression is known for its ability to perform feature selection. So ultimately the values have become zero. So now my 19 features have reduced two. So I have reduced these two. Then I have also reduced

[4:03:30] features get reduced. So 19 gets reduced to 16. Right? When we have taken lambda to 16. Right? When we have taken lambda equal to 1. All right.

[4:03:42] Right. But in this model however it seems that none of the coefficients are exactly zero suggesting that all included variables have some impact on the models through some impacts are those some impacts are

[4:03:58] very very small. So basically four features yeah okay four feature. So now how do I evaluate it? I'll calculate my mean square error for the

[4:04:11] training data and then the for the testing data. This code is also getting clear that helps me to decide whether it's overfitting or underfitting. So now you tell me is it an overfitting model or an underfitting model

[4:04:30] than the training. So is it underfitting overfitting? So is it underfitting overfitting? it's overfitted

[4:04:44] and if I do the R square over here so the value is very very less. So to better understand the role of alpha the regularization regularization parameter. So what is alpha known as the

[4:04:56] parameter. So what is alpha known as the regularization as a function of alpha. Maximum iterate are the maximum number of iterations.

[4:05:15] function. What does the np numpy lindspace function do? that the now the values of alpha will range between 1 to five and with 10 equidistance value. So

[4:05:27] now what are we trying to create lin space function do we understand of space function do we understand of numpy? So now if I say now I will start numpy? So now if I say now I will start understanding the range of the values

[4:05:39] understanding the range of the values range of the values from 0.01 01 right range of the values from 0.01 01 right to 500 and the value is still 100 right

[4:05:51] so over here these are the values or the array created of alpha the maximum iterations are 10,000 and now I will try to run or fit on each of these alphas

[4:06:06] to run or fit on each of these alphas and then look at this beautiful graph Simon specially for 2. So the value of the alpha goes from 0.01 to 100,000 and the alpha goes from 0.01 to 100,000 and we see as the value of the alpha is

[4:06:21] increasing some of the coefficients actually become zero. You see this?

[4:06:34] So that is why L1 regularization is also known as feature selection. Agreed? Okay. So have you understood this graph now? So we create now the best value of

[4:06:49] alpha. How do we find out the value of alpha? That now we will use cross validation to find out the best value for alpha. Lasso regression comes with builtin cross validation. So beauty of lasso is that you can combine lasso with

[4:07:06] cross validation. Alpha again ranges between 0 between 0 to 1,000. CV is equal to 10. Maximum to 1,000. CV is equal to 10. Maximum iteration 10,000 and num n jobs is equal

[4:07:21] to minus1. Can you anybody tell me what is the use of this parameter? Simon, I'll do that. In the meanwhile, tell me what is the use of n jobs

[4:07:36] learners? See, once we have checked on on the training and the testing process and we found out that it is an overfitting model, the test error was too high than the others. So now we have to find out

[4:07:52] the others. So now we have to find out the uh you know the best value of alpha. Okay. Right. Let's reduce the maximum course and which is the you know ma best alpha value which has come out over here six

[4:08:06] value which has come out over here six comes out to be the best uh regularization. So get the best alpha regularization strength selected by the regularization strength selected by the cross validation clear. So it was not

[4:08:20] the value of alpha was not one but it will come out to be six. And now I will create and fit the lasso model taking this value. Now I will fit it and now

[4:08:32] check on my training and testing error. So have they improved? Well, the test error is still higher than the training, right?

[4:08:48] tuned parameters. So zero coefficients such as runs, RBI, CAT and all these new leagues have several coefficients shrunk to zero. This indicates the lasso has deemed these features less important irrelevant

[4:09:03] for the data set and the nonzero coefficient give us the detail relationship. Clear? Moving on to ridge regression. Ridge regression is another technique for

[4:09:17] regression is another technique for handling uh overfitting of the data that's known as L2 regularization. Here none of the features will become zero rather they will values will become less. Right? So where it is used why

[4:09:32] where when to use that and when to use the other one get the best alpha selection validation. Now what is this added below library? Okay. So when is ridge to be used? It is useful for handling multi-olinear data

[4:09:49] where two predictors independent variables are highly correlated to each variables are highly correlated to each other. AJ you are absolutely correct AJ Kana you are absolutely correct but the

[4:10:05] second reason where we use ridge regression is that it is useful for regression is that it is useful for handling multicolinear data where two predictors independent variables are highly correlated to each other. Now

[4:10:20] getting my point. Collinearity refers to a situation where two or more predictor variables in multiple regression model are highly correlated. Meaning they have

[4:10:32] a linear relationship and this makes it difficult to determine the individual effect of each predictor on the target variable. Right? So that's why the penalty term is added. the regularization terms are

[4:10:50] added. the regularization terms are added. Getting my point? ridge CV function which can do so on the same data

[4:11:03] we will first apply the alpha fit function and get the intercept value. Yes, reach ridge needed to use if you want to keep all the important features. So I have not been running the code. So I will get error. So sorry for that. I'm

[4:11:19] not running the whole code. Going back and running it. So now let's move and then evaluate the model. Again testing error is more than the training. So overfitting is there. And do we see the coefficients? Now none of them become

[4:11:34] zero. Now we can use cross validation again to find out the best value. These are the values. I have used rich cross validation. And in this case the value

[4:11:46] comes out to be 204. Right? Then we train on it. And these are my new coefficients. are my new coefficients. These are my new coefficients. Right?

[4:12:01] All values have reduced. If you compare with the original, all values have reduced now. So the negative error relationship is with errors. Division W. relationship is with errors. Division W. they show the negative relationship. So

[4:12:16] the coefficients with highest value suggest cumulative career statistics, total runs, the total hits are most important and showing a positive relationship with salary. Division_W the only feature which is notably high

[4:12:32] negative coefficient which indicates that being in western division is associated with negative effect on the target variable. So do not be in the uh western division ears assist and new

[4:12:45] league and these features have smaller. So are you understanding this is how you're practically supposed to run the code and write down the observations code and write down the observations over here

[4:13:00] this case the values will not become zero but near to zero. Clear?

[4:13:13] regularization? How it helps in preventing overfitting? Whereas L1 helps in feature selection and U ridge helps in uh reducing uh the

[4:13:25] values if all are important. And of course this analysis helps in understanding the relative importance and influence of different aspects of a baseball player statistics on the predicted target. Clear?

[4:13:41] predicted target. Clear? Now moving on to the next concepts Right? Model optimization. Now what does the term optimization mean? What is the

[4:13:56] the term optimization mean? What is the meaning of model optimization? Optimizing resources. We want minimum number of uh code less number of time to

[4:14:08] giving the best. Yes. How the model performs efficiently? Absolutely. So if we talk about the journey of data science or analysis see every thing on

[4:14:21] if you if you are doing a normal Excel maximum minimum drawing graph that's also analysis and even when we are doing machine learning that's also analysis but where is the difference are you able to get this um

[4:14:37] to get this um graph that I'm showing where is the difference can you tell If we already have a data set. So this is the side for that we are going to describe you know about the uh

[4:14:52] historical data that we have that is simple finding out insight in the data. When we try to find out why that happened that becomes diagnostic

[4:15:05] happened that becomes diagnostic analysis that is insight. And if we move further what is going to be predicted what is going to be predicted sorry or what will be the

[4:15:19] output that becomes predictive analytics and similarly why that uh you know we are predicting that output whether that is going to be good or bad that is known as prescriptive analytics. So this is where we start from inside

[4:15:36] and now we want to move to the foresight. How does it happen? When we foresight. How does it happen? When we start optimization of the model. Are you understanding this point? Very very important graph from one analysis

[4:15:52] to another to explain you in more lamean language. Suppose you have cold and cuff right and you go to the doctor. The doctor sees okay you know your nose is running you have cold and cuff. He says you know um

[4:16:06] you take this normal predict uh you know medicine and you know do a lot of steaming and take this medicine you'll be fine. But that was his you know analysis which he done to just by seeing you. So then you come back you are not

[4:16:19] fine. Then you go back after a week to him you say no I have fever also I have other thing. Then he thinks that the fever he's not getting well. So maybe he has some kind of an infection then he prescribes you certain blood test. So

[4:16:33] you certain medicines but you are still not fine your further analysis is needed. Maybe you have a chest infection now or something and maybe you want to go in for an MRI. So that's how the level of analysis increases. All right.

[4:16:49] So basically what do we want to do and what do we want to optimize in this whole machine learning process that if we have this input data that is the training data including the target output. Of course if it is supervised

[4:17:04] learning we would also have the target output. This is my model output. This is my model right and output prediction calculated right and output prediction calculated by model. So when we try to compare the

[4:17:19] predicted output with the actual output therefore we are able to calculate the error and the loss function. Are you all there with me? Then we are able to

[4:17:31] calculate the error and the loss function. And using this error and loss function we can use optimization method to sorry

[4:17:44] further reduce the values. How we can further reduce its value. Getting my point right. So that's how we are trying to

[4:17:57] improve on the error part of it. And then another way to optimize the model is through hyperparameter tuning versus model training. What is

[4:18:11] hyperparameter tuning? That to find the best hyperparameters which give us the most efficient results. results. And this is a very very important slide

[4:18:25] because it gives the whole crux of the data uh of all the points that we have studied till now. Please look here. This is my data set. Feature engineering

[4:18:37] is performed before. What are the different aspects of feature engineering different aspects of feature engineering and why is feature engineering required? Tell me what are the different aspects of

[4:18:52] feature engineering and why is it required to prepare the data to fed into the model not tune we will say to prepare the data prep-process the data to fed into the model and what are the different types of feature engineering

[4:19:09] so what are the different process encoding scaling transformations Right? Dealing with the outliers, they all come under feature engineering. First point, first step is clear. Then what is happening in this particular

[4:19:25] step? Can anybody tell me what is this step known as? Splitting. But what what are we splitting and why are we splitting? It is splitting of the data set. Why are we splitting it into three parts? Normally it is training and

[4:19:40] parts? Normally it is training and testing. This is this step is known as cross validation that it divides the training data into training and training data into training and validation and a separate test data.

[4:19:52] Second step is splitting of the data. Yes, but we generally keep using it for cross validation by taking multiple samples. samples. Got it? By taking multiple samples.

[4:20:06] Got it? By taking multiple samples. Now this is an iterative process. This keeps on getting repeated. So the training data we build models, train the training data we build models, train the results, right? And finally

[4:20:21] once the training is done then on the validation uh thing we do the training results do hyperparameter tuning and keep on repeating this iterative process. Finally, the best model which is

[4:20:36] Finally, the best model which is selected is then used for testing the selected is then used for testing the output and compare the results. Do not get confused. See, every concept

[4:20:48] Do not get confused. See, every concept that we study has its own role and concept. Right? So, where will the regularization fit over here? If I compare the result, my testing result is much higher than the training result.

[4:21:03] much higher than the training result. Then it is an overfitting model. Right? We always start with the basic mega then we will use the model over here as ridge

[4:21:15] we will use the model over here as ridge or lasso and then do the comparison. Now getting my point everybody all right and the two types of hypertuning methods are grid search and random search. Grid search is the one in

[4:21:32] random search. Grid search is the one in which we it's an exhaustive re search technique where we have different hyperparameters one such as a b and c it's not necessary that you know we have understood that regularization as one

[4:21:45] hyperparameter we have other models like decision tree which have many decision tree which have many hyperparameters hyperparameter to xyz hyperparameters hyperparameter to xyz value and then we try to take all the

[4:21:57] combinations and then see which one works best for a model. Disadvantage of this method is it's an exhaustive method uh computationally and timewise it is

[4:22:10] expensive. Clear and random search that randomly we will select any hyperparameters select any hyperparameters and do training on it.

[4:22:24] Clear now and we have seen so cross validation is a very very important technique. Why? Because it helps to split the data into Because it helps to split the data into training, validation and test set.

[4:22:39] Please make this point clear. Cross validation technique will almost be used in supervised learning because it will divide the data set into three parts

[4:22:51] that is training, validation and testing set. Useful if you want to have a metric set. Useful if you want to have a metric on how well your model is performing.

[4:23:03] Clear? So once the grid CV search returns with tuned parameters, build the model using this set with the tuned parameters and test the new model with the test set. Clear? And this is another uh you

[4:23:18] set. Clear? And this is another uh you know um slide which I always uh show it to my learners even in my feature engineering class. Now are you able to interpret this particular slide everybody?

[4:23:32] Yes. Right. So we again have a data set, retrieve the data set, perform pre-processing, wrangling, feature extraction, feature engineering, train the model, then model evaluation,

[4:23:45] hyperparameter tuning and reiterate the process. But this slide gives a much better picture of each step. Got it learners? So getting back to the file. So this file is pretty long. You know

[4:23:58] last week also we have done and I think so this session also we would be using. So model optimization means hyperparameter tuning is the process of finding the best settings for the parameters in the machine learning

[4:24:14] model. Hyperparameters are settings that are not learned during the training but are set before the training process begins. Hyperparameter tuning involves

[4:24:26] trying different combinations of the hyperparameters and evaluating the model's performance using validation techniques. So the two techniques that techniques. So the two techniques that we can use is grid search.

[4:24:45] grid search? It systematically works through the multiple combinations of It performs an exhaustive search on a specified parameter grid. So how do we go about it? Please try to understand

[4:24:59] the different steps. The first step is we define a parameter grid that is defined with the help of dictionaries because they can be more than one hyperparameters in the model. Combination evaluation.

[4:25:14] The algorithm evaluates all possible combinations of these hyperparameters. Model training. For each combination, the model is trained, evaluated using cross validation. And then we have the optimal parameters combining yielding

[4:25:31] the best performance. Highest accuracy is chosen as the optimal set.

[4:25:47] defines the parameter distribution. Specify the distributions and ranges of Random sampling. Random sample combinations of hyperparameters from these distributions. Model training for each sample combination. And optimal

[4:26:03] each sample combination. And optimal parameters. we are running the code. I might get a lot of error. It will

[4:26:19] lot of error. It will let me run the x tree. validation. Here it we are creating a cross validation object with repeated

[4:26:33] kffold function. Number of splits is equal to 10. Number of repeats is three. equal to 10. Number of repeats is three. Random state is equal to 1. Are you now there with me everybody? the other code else code is running then we create a

[4:26:48] grid or a dictionary and then try to find out the value. So still it's giving error. Is it running for everybody else? Oh well I've tried to do a shortcut but shortcuts never work in life. Value y

[4:27:03] input consist of nan. Okay. Yeah. Are you getting it? I'm not running the code. So this point is clear.

[4:27:16] it's coming out to be 0.9. Is it the same with everybody? So then we create and fit the ridge regression model to avoid training data

[4:27:28] with the optimal alpha. So here we have the ridge function results into best parameters alpha. fit them through the training uh input as well as output and

[4:27:40] find the mean square value that comes out to be 341 out to be 341 and R square comes out to be 0.38 which suggests a moderate fit the model captures some of the variability in the

[4:27:55] data but not a large portion and now if I see the coefficient this is how the coefficient values are now let's move on to the next concept last concept by to the next concept last concept by creating pipelines. So skarn pipelines

[4:28:10] are nothing but an automation of the model fitting and data transformation steps for training and data set. So what are we doing at the moment when we upload the data we separately use fit and transform do encoding do scala

[4:28:25] standardization. Now we will try to implement it with the help of a single pipeline. So what do we do? If we have test data, test labels, we would perform

[4:28:37] feature scaling standard scaler initially. Then if feature selection or extraction, dimensionality reduction is required, that can also be done and then the model data can be fed to the different models.

[4:28:53] So how do we go about it? We are very clear in supervised learning. We have a training set and a test set with class label, right? So in the first step we will try to fit the training data that is missing in the test set because we do

[4:29:09] is missing in the test set because we do want to avoid data leakage right the fit will ne fit function will never ever work for the test data

[4:29:24] right and therefore we have the fit transform function over here transform function over here this is the pipeline right so over here this is the pipeline right so over here you know We um do the transformation

[4:29:36] such as for for scaling, dimensionality reduction and then do the predictive model and over here it is fit and transform for the test training data but

[4:29:49] for the test data it is only the transform function. Clear? And the transform function. Clear? And the predict will only be for the test data. Now let's see how do we practically implement it. So why sklearn pipelines?

[4:30:04] Pipelines provide an organized approach to managing your data prep-processing, modeling the code. They combine the pre-processing and the modeling steps pre-processing and the modeling steps into a single streamlined process.

[4:30:19] Cleaner code pipelines eliminate the need to manually manage uh training and validation data at each reprocessing step reducing the clutter and the

[4:30:31] step reducing the clutter and the complexity. Fewer bugs by bundling steps together. So bugs are few. It gives a much more cleaner uh code and easier to productionize. that is they simplify the transition from a prototype to a model

[4:30:48] transition from a prototype to a model to a scalable deployable solution. to a scalable deployable solution. Clear? So we have skarn is a beautiful library. We just need to uh you know create a class or we do import the

[4:31:02] pipeline function and we will create an object of pipeline class whatever steps that we want to perform. Do we want to use any kind of memory or verbose clear? So a pipeline is a sequence of data transformers that can include a final

[4:31:19] predictor also final output also. It lets you apply reprocessing steps to lets you apply reprocessing steps to your data in order and u optionally end with a predictor for modeling. Each intermediate step in the pipeline must

[4:31:35] have fit and transform methods while final step only needs the fit function. Memory stands that you can use cache memory for these transformations to make

[4:31:48] the processing fast using the memory argument. The pipeline main goal is to argument. The pipeline main goal is to combine validated together and have their parameters adjusted. You can set

[4:32:05] parameters for any step by using it name followed by a underscore. So you can set any of the parameters by default followed by a underscore name and the parameter name and you can replace any steps estimator with another estimator

[4:32:22] steps estimator with another estimator or remove transformer by setting it or pass through or none. So now over here they are using a different data set that is the one with ocean proximity that is housing one I think. So I gave you for

[4:32:38] testing also. So it is little different from original because one of the parameter is categorical ocean pro uh you know object. So how do I check that? What are the different uh uh you know categories

[4:32:55] of this particular function? How do I check that? So we have this categorical data. So what are the different encoding techniques? either we use one hot encoding that is better because there is no order in this so we'll not use label

[4:33:10] encoding we'll use get underscore dummies function to perform it value counts function is a function in python which gives the categories along with their frequency count very very important function

[4:33:25] what is happening in this particular code of line number 46 come on learners why is getting the output value and then we are separating the X. Yes, we are

[4:33:38] defining input and out. What is happening in the next step? Line number 48. What is happening in the next step? Splitting of the data. Very very Splitting of the data. Very very important. And then we see the info and

[4:33:52] sum over here. Right? So where are the null values? Where are the null values? null values? Where are the null values? Total bedrooms has Total bedrooms has 62 null values. Data is huge. 14,000

[4:34:08] 448. Right? So definitely cleaning is required. Removing of the null values encoding is required. Right? So rather than now performing each of these steps

[4:34:22] separately can I use the concept of skarn pipelines. First I will feed in my for training data perform feature scaling feature extraction and ML

[4:34:34] scaling feature extraction and ML algorithms and then work on the test data getting my point learners. Okay. So what are the steps that we want to perform before building the model? Feature

[4:34:47] engineering steps. So feature engineering part is used as skarn. Even the other parts can be combined as pipeline. First missing value treatment imputation 162 missing values in the total bedrooms

[4:35:02] uh numeric data column. Then we have the dummy variable creation for categorical data. and finally standardization of the numerical value. So how do we go about

[4:35:14] it? These are my different libraries that I will impute import and most that I will impute import and most importantly is the pipeline library?

[4:35:37] column transformer that importing column transformer class to apply different pre-processing steps to different subsets of the feature. So column transformer has the capability that encoding is applied only on categorical

[4:35:54] data not on numerical data. Getting my point? So it will only be applied on categorical. So let me explain this.

[4:36:11] transformation happens. But if these are my features which is a combination obviously of numerical, categorical and others. So column transformer has the beauty that for numerical columns only scaling will happen. for categorical one

[4:36:29] hot encoding and maybe others can simply pass through. It's not necessary that some operation needs to be performed on it and finally we can get a transform

[4:36:41] data in very one go and it can be combined together. Got it? Column transformer concept is clear to everybody.

[4:36:57] now when you read about column transformer, you will understand that it allows you to apply different pre-processing steps to different subsets of the features in your data set. This is particularly useful when

[4:37:10] you have a mix of numerical and categorical data that require different types of pre-processing. Column transformer ensures that each column or group of columns gets appropriate transformation before combining the

[4:37:25] results for further processing or modeling.

[4:37:38] names of the numeric and object type variables differently. So how are we going to do? And now we have splitted the training and the testing data and we are exclusively selecting the data types include object

[4:37:53] exclude object and result are you understanding this code learners. understanding this code learners. So housing_cat will only have one since this data set has only one categorical data that is ocean uh proximity and the

[4:38:10] data that is ocean uh proximity and the rest are all numerical in nature. Clear. Now the next step says to set up skarn pipelines for numeric variables we need to perform missing value imputation and

[4:38:27] then standardization. So how do we go about it? Now we will create a pipeline about it? Now we will create a pipeline object using this pipeline function num pipeline. of a numerical data. We will import uh you know do the missing value

[4:38:42] import uh you know do the missing value imputation and standard scaling. Is this point getting clear? Simon, are you there? Are you understanding this code there? Are you understanding this code now of creating pipelines?

[4:39:02] What is imputation and what are we trying to achieve over here? Imputation is the concept of missing value and here we are filling the missing value with we are filling the missing value with the median value. Good. Good. Mega very

[4:39:15] clearly it says that right and standardization is that we are trying to create the mean of the data zero and standard deviation equal to 1.

[4:39:27] Then unified data processing with pipelines and column transformer. Now we need to create another pipeline. No. So we've created a numeric pipeline. Now we create a column transformer. The numeric pipeline and the other one categorical.

[4:39:44] For categorical data we only need to perform only and only one function that is known as the one hot encoder function. Clear? So this ensures both numerical and categorical data are pre-processed

[4:40:00] categorical data are pre-processed approximately within single unified work. Now got it. The beauty of column transformer that we have combined the numerical as well as categorical values.

[4:40:15] numerical as well as categorical values. Okay. Then we do pre-processing fit transform x train. We're going to fit and transform. And now check the uh you know data set for training. Clear. Now do you see

[4:40:32] Clear. Now do you see it has already done this? So right. So now we fit and transform. And now check the training data. This is my now check the training data. This is my training data. And now if I check my the

[4:40:46] scaling has already happened. Negative values are there as well as the null values are there as well as the null values have been removed. Got it? Pipelines. Are you understanding? So what is our observation? Good to see

[4:41:00] all missing values are treated. Numeric values are standardized. Auction proximity ocean proximity is converted. Now the other way of creating model is again you can create a final pipeline for

[4:41:15] pre-processing and model ridge over here we are creating ridge regression model straight away and then moving ahead. So have you printed that also the final have you printed that also the final pipeline also? Yeah. So final pipeline

[4:41:29] gives you this that after applying the basic thing we are applying the ridge model right and then we define the grid of hyperparameters to search you can set parameters for any uh step. So over here this is my ridge

[4:41:45] alpha the hyperparameter which will range from 0.1 to 2.1 and then I use my grid search CV along with cross validation estimator is my final

[4:41:58] pipeline these are my grids that I have passed uh onto it scoring I'm using negative mean absolute error number of jobs is equal to minus1 and we fit and

[4:42:10] jobs is equal to minus1 and we fit and get the output clear. So the mean absolute error is 49,000 and the alpha value is coming out to be 0.1. Then I check my test result. The test

[4:42:24] result is slightly higher. So it's almost a good fit of the value now clear almost a good fit of the value now clear how you know and even in classification model we would be using a lot of pipelines to get the result. getting my

[4:42:40] point? So finally today after skarn we come to So finally today after skarn we come to the end of regression analysis which is you know part of supervised learning predicting numerical value. Regression

[4:42:55] analysis is an essential method for estimating examining and predicting variable relationships. In this lesson we type as we had discussed initially types of regression then the different error matrices the cross validation

[4:43:12] technique the regularization technique the pipelines. Are we all now clear with this

[4:43:25] particular file 3.2 finally we are ending with regression finally we are ending with regression today.

[4:43:37] of regularization in uh machine learning model? Why do we want to do regularization? Yes, if we want to prevent overfitting. Which of the following is a characteristic of L2 regular

[4:43:51] regularization? Squaring of the value.

[4:44:09] learners. It is both B and because it adds the squared magnitude its ridge of coefficient and it reduces overfitting by reducing the model complexity. Next question. What effect does increasing the alpha parameter in lasso

[4:44:25] increasing the alpha parameter in lasso regression have on the model lasso? Do we want to increase model complexity or decrease? Obviously when alpha increases the model complexity. What is the primary difference between L1 and L2

[4:44:41] regularization in terms of the effect on the model coefficients? Remember L2 means squaring of the value. Remember L2 means squaring of the value. L1 is absolute. So it's simple. L1

[4:44:54] L1 is absolute. So it's simple. L1 encourages sparity. What is sparity? Having value zero. So it makes most of the features equal to zero. Well, while L2 regularization does not uh encourages sparity. Now

[4:45:10] clear. How does regularization help in improving the generalization ability of a machine learning model? You want to reduce the variance or the bias? Overfitting. Overfitting gets reduced by

[4:45:28] variance. What is the risk with tuning hyperparameters using test data set? What will happen if we tune hyperparameters? What is the risk with tuning hyperparameters using test data set? The

[4:45:44] model will overfit the test set and data leakage can happen. Which of the optimizers? They try to avoid overfitting because

[4:45:57] that's the work of regularization. If searching among large number of hyperparameter, you should try values in a grid rather than random value so that you can carry out the search more systematically and

[4:46:12] out the search more systematically and not rely on chance. True or false? We should carry out more systematic We should carry out more systematic searches or random searches

[4:46:24] when searching a large number of hyperparameters. grid rather than random values. Why? Because then it would become an

[4:46:37] exhaustive search, right? We should always try random values. I hope this knowledge check was helpful for everybody. Start with another part of supervised learning that is classification. Now tell me how will you

[4:46:53] define classification? It is supervised learning used when the output is output is categorical, right? Yeah. So this is supervised learning

[4:47:08] technique when the output is categorical. So these are the independent input variable and the model classifies them as animals and fruits. So are you aware about different classification techniques learners?

[4:47:25] Do you understand different classification techniques? Do we understand the concept of binary classification? What what can you interpret from this particular diagram that the uh you know

[4:47:41] the model will classify the input or not input sorry the outputs into two categories. For example, if a mail is there it will either put it into the spam folder or the inbox. So when there are two categories as output that is

[4:47:59] are two categories as output that is known as binary classification. Getting my point learners? Can we have more than two classes as output? Can we have more than two classes as output? Yes, we can have you know when we want

[4:48:14] to label different vegetables such as capsicum, carrots, tomato, radish, turnip, it could be endless number of classes and that is known as multiclass

[4:48:27] classification. Are you all getting this point learners? That is known as multiclass classification learners.

[4:48:40] classification learners. What is this? Suppose I have a image and the in the image there can be multiple labels. This image consists of a truck, boat, dog as well as a plane. So if we have more than one, it's not multiclass

[4:48:57] have more than one, it's not multiclass but a same uh you know uh image or a same input. For example, a movie. A movie can be uh romantic, thriller, horror also, right? or comedy, thriller, horror. So the same item can have

[4:49:13] multiple labels that is known as multilel classification. What do you observe over here? If the classes are not balanced, right? The the

[4:49:25] images of trucks are around 60% plane only 25% boards 15 then there is a different techniques to deal with imbalance class data set. All right. So

[4:49:38] this basic concept of different classes is clear. Now let's straight away dive is clear. Now let's straight away dive into how we will analyze the classification uh output right or what are the

[4:49:52] different metrics to analyze the classification output. So what do you think? How will I check whether my model is classifying the males and the spam males correctly? The matrix which is used or the major

[4:50:08] role that is being played is done by the confusion matrix. confusion matrix. Please try to understand. For example, for example, I'm taking a very simple example. There is a uh you know bag of

[4:50:22] balls of red and green balls. Now if I tell you to classify them, it is good that in the red ball bag you put all the red balls and in the green ball bag you put all the green balls. Right? So that is true positive and true negative. So

[4:50:37] the green balls which belong to the green bag are in the green bag and the red balls which belong to the red bag are in the red bag. But is there a possibility that a green bag a red ball

[4:50:50] can be in a green bag or a green ball can be in a red bag? Is that possibility? Yes. machine can create those errors and that kind of errors are known as false negative and false positive. Right? When the machine does

[4:51:08] not predict the correct output. So what is our aim as a model? We always want to increase the true negative and true positive to create the best results.

[4:51:20] Right? And we always want to decrease the errors that is the false negative as [clears throat] [clears throat] Getting my point right? So a confusion

[4:51:34] matrix please remember again it is very very important uh point when we always talk about classification that a confusion matrix is a n byn matrix used

[4:51:47] for evaluating the performance and that is why statistics is said to be the strong foundation for machine learning concepts. Bang on mega appreciate this. Yeah. So confusion matrix is an n byn

[4:52:04] matrix used for evaluating the performance of a classification model where n is the number of target classes. So it will always be a square matrix. A good model as we have understood will have high true positive and true

[4:52:18] negative rates and whereas a low or a bad model will have uh you know high uh uh low true positive and true negative length. So what do we do when we have imbalanced data set? That

[4:52:34] means the ratio of the classes is not same or equal. It's always better to use confusion metrics as your evaluation criteria for your machine learning model. So this is how you know you will get the

[4:52:50] So this is how you know you will get the output in terms of uh your Python code. It will create this kind of a matrix uh and display for you. And I hope the terms true positive and true negative, false positive, false negative are

[4:53:04] clear. True positive is predicted positive which are actually positive, false positive, predicted positive but are actually negative. Getting my point? And this is the exactly the same example I always use in

[4:53:20] my hypothesis testing class also I use the same example. interesting example. We have the images of cat and dog right. So we have about

[4:53:37] of cat and dog right. So we have about 20 images of cat and dog right. So for cat they were only seven images and dog they are around 13 images. Right? So I you know ran this particular images on

[4:53:53] the model. So six images which were actually CAD were predicted positive by the model. So that becomes my true positive. 11 images which were not CAD

[4:54:05] were actually predicted that they are not CAD. They come under the true negative category. So very good. But the dis the the errors were that the image the image was of the cat but it says that you are a dog. So one error over

[4:54:22] here of false negative and similarly the image was of the dog but it says that you are a cat. So this is false positive. So the false negative errors are known as type two errors also and the false positive are known as type one

[4:54:39] the false positive are known as type one error or also known as alpha and beta. If you remember from hypothesis testing same concepts same concepts absolutely same concepts are being used.

[4:54:53] Are you understanding is this? So there are different metrics that can be evaluated from this confusion from this confusion metrics accuracy precision recall F1 score FPR and FNR all can be

[4:55:11] calculated. So if I talk about accuracy, accuracy simply measures how often the classifier makes the correct prediction. It's the ratio between the number of correct predictions and the total number of predictions. So what does accuracy

[4:55:27] mean? I'll take my true positive, true negative divided by the total number of output. That gives me accuracy. But what is the illusion over here? The illusion is the illusion over here? The illusion over here is that it is not valid for

[4:55:42] imbalanced data set. Anu says why type one and type two. Why type one and type two? It is there a possibility that the classes might not be uh classified correctly. So type two and type one actually arise because I am

[4:55:59] considering one of the statement as positive. Positive means that the image is of the cat and negative statement means that you are not a cat. Right? So false positive error means

[4:56:14] Right? So false positive error means that actually it is not a that actually it is not a image of a cat but still says you are a cat. That is why false positive that it is not a cat and false negative because

[4:56:28] it is not a dog. the same game we play between null but we'll be uh you know resolving it into different metrics. So one of them is accuracy the other one is precision. So the precision is defined as the ratio of

[4:56:44] the total number of correctly classified positive classes divided by the total number of one predicted. So precision is useful when

[4:56:56] please try to understand precision is a useful metric in cases where the false useful metric in cases where the false positive is a higher concern than false positive is a higher concern than false negative. What do we mean by that? That

[4:57:10] false positive is a higher concern in precision. So what is more detering? whether you know the male is not a spam but the model predicted as spam. Is that more important than a male which

[4:57:26] was spam and model predicted as not spam? male is considered as spam then that is a serious issue. Therefore the precision

[4:57:40] needs to be high. So what is precision? It is true positive upon the true positive plus false positive. Getting my point? All right. So the other metric that we can use is recall. Recall is classified

[4:57:57] can use is recall. Recall is classified as the ratio of the total number of correctly classified positive classes divide by the total number of positive classes or out of all the positive classes how much we predicted correctly.

[4:58:14] classes how much we predicted correctly. So recall should be high and idly one. And again when where is the recall function mostly used? It is useful metric in cases where false negative trumps false positive.

[4:58:30] Getting my point? So what would be the case where false negative is important than false positive? For example, medical cases where it does

[4:58:42] not matter matter if we raise a false alarm but actual positive cases should alarm but actual positive cases should not go undetected. Got my point? So this is the formula for recall the

[4:58:56] true positive divided by the true positive plus false negative. Clear ma'am do we need to calculate all these formulas? No, you just need to understand the mathematical intuition.

[4:59:11] The Python programming will do it all for you. Right? So some might say ma'am u sometimes you know the false negative error as well as the false positive error both are

[4:59:26] important right then how do we go about then the metric which is used to balance it out is known as the f_sub_1 score. So the f1 score is a number between 0 and 1

[4:59:38] and it is the harmonic mean of precision and recall. So the definition already and recall. So the definition already has the formula. Okay. Harmonic mean. We use harmonic mean because it is not sensitive to the extreme values. So F1

[4:59:53] sensitive to the extreme values. So F1 score is valid when we have imbalanced data set because it maintains a balance between the pre

[5:00:07] your classifier. If your precision is low, the F_sub_1 is low and the recall is low again and F_sub_1 score is low. So what do I mean by that?

[5:00:20] This is the harmonic mean and this is the formula. So F_sub_1 score is the harmonic mean and it should be ideally equal to one. So there will be cases where there is no clear distinction whether to use precision or recall then

[5:00:35] whether to use precision or recall then we can use the F1 score. Got it? when true positive and true negatives are more important. Accuracy is better

[5:00:48] metric for balanced data. False positive is much more important. Use precision. Whenever false negative is much more important than use recall. F1 score is used when the false negative and false positives are important.

[5:01:05] Now clear this gives you more clarity when to use this gives you more clarity when to use what and where. And how do we create this confusion matrix in Python? By simply importing

[5:01:21] skarnmet matrix import confusion matrix and running the confusion_matrix function. So this is my y test y predicted and this is how I get the

[5:01:33] predicted and this is how I get the output.

[5:01:46] right but the best part about it that simply by using skarnmetrics import classification report we can get the all the metrics in one particular output this is what we actually use

[5:02:05] not see Simon, it's a very very relative question, right Simon? Obviously 100% is perfect. But depending on the model, the kind of problem that you are facing, if you're not getting accuracy more than that, then this is the best. But if you

[5:02:21] that, then this is the best. But if you can get accuracy up to 92, 95 or 97% can get accuracy up to 92, 95 or 97% then of course 85% is not acceptable. whether to accept or to reject the project

[5:02:36] or you want to make further improvements or optimizations onto it. All right. So this is a very very beautiful uh you know code that you know

[5:02:50] for classification that we will have this classification report in which we get all the output prec precision recall f1 score in one tabular form. All right and this is what we are going to actually use for uh you know comparing

[5:03:06] actually use for uh you know comparing the results getting my point. So again a quick recap accuracy, precision, recall, F1 score, specificity and different metrics can be calculated but as I told you we do not use different metrics. We

[5:03:22] will straight away uh straight away import classification report and get the output. Now another very very important factor Now another very very important factor how graphically we can judge the output.

[5:03:38] how graphically we can judge the output. So in classification problem we have AU So in classification problem we have AU and ROC curve. What does AU stand for? AU stands for area under the curve and ROC stands for receiver operating

[5:03:54] ROC stands for receiver operating characteristics. learn file. This is my material. I'll be sharing it. even the regression

[5:04:08] material, classification material. Right? So another very important factor to understand classification is the graph AU and ROC curve. AU stands for

[5:04:23] graph AU and ROC curve. AU stands for area under the curve and ROC stands for area under the curve and ROC stands for receiver operating characteristics. So AU ROC curve is a performance measurement for the classification

[5:04:37] problems at various threshold settings. Now what do we mean by various threshold Now what do we mean by various threshold settings that we will try to change the you know the that whether the classes have been uh classified correctly. You

[5:04:52] know the threshold value always ranges between 0 to one. So it's like a meter rating. Okay. Now this is a point or this is the boundary. Yes, these two classes have been classified correctly or the other two right. RO is a

[5:05:07] probability curve. Please try to understand. ROC is nothing but a understand. ROC is nothing but a probability curve and AU represents the probability curve and AU represents the degree or the measure of separability

[5:05:20] between the classes. So here we are trying to measure how well the two trying to measure how well the two classes have been classified classes have been classified correctly based on the threshold value.

[5:05:35] It tells how much the model is capable of distinguish between distinguishing between the classes. Higher the area under the curve, the better the better

[5:05:47] under the curve, the better the better the model is at predicting the model is at predicting zero classes as zero and one classes as zero classes as zero and one classes as one. Now getting my point,

[5:05:59] what do we mean by a? It represents the degree or the measure of separability. better the model is at distinguishing between patients with diseases or no

[5:06:15] diseases. So how does it work? Basically, so this graph is drawn between the false positive rate on the x-axis and true positive rate on the x-axis and true positive rate on the right hand axis. Right? And the this

[5:06:31] graph is and this is area under the curve and this is the ROC the receiving operative sorry. So this is receiver operating characteristics and these are the

[5:06:45] different formulas that we have just now seen. So sensitivity and specificity are inversely proportional to each other whereas TPR and FPR are proportional.

[5:06:58] Okay. So sensitivity and specificity. Okay. Recall the name. Recall. Have you understood recall? Recall is nothing but understood recall? Recall is nothing but the ratio between true positive and the

[5:07:11] true positive and false negative. Okay. The recall is also known as the true positive rate and the other name is sensitivity. Now clear

[5:07:29] true negativity uh divided by true negative plus false positive. So ROC curve is the plot between the true positive rate and the false positive rate. That point is clear

[5:07:42] across the all possible threshold. So what is this threshold? The threshold what is this threshold? The threshold value is always between 0 to one and in between you know uh it tells how well the two classes have been classified

[5:07:57] right and AOC is very clear that it is the area uh under the curve. So the more the area under the curve which is equal to one the better is the classification.

[5:08:10] Now clear Anush. So ROC AOC curve is the area under the curve. It sums up how well a model can produce a relative scores to discriminate between the positive and the negative instances across all classification threshold and

[5:08:26] the values will always lie between 0 to 1. So a is desirable for the following that why do we want to understand the area? Why why do we want to use this

[5:08:40] area under the curve? Let me go through it and still you have errors. I'll explain them. First let's get the crux of it. I understand a lot of points are there. First let's get the crux of it. Area under the curve is scale invariant.

[5:08:54] It measures how well predictions are ranked rather than the absolute value. Right? So it's scale invariant. Does it tells how well the predictions were ranked. It's not uh absolute value. It helps in comparison. Secondly, it is

[5:09:10] classification threshold invariant measures the quality of the model's predictions irrespective of what classification threshold is chosen. So this is what I mean by my threshold generally it should be like 50 50%

[5:09:28] generally 5050% that my true negatives have been classified as true negative green balls in green bag red balls in red bag and the total area under the

[5:09:40] curve is one. So this is one of the ideal situations. This is one of the ideal situation that there is no overlap area under the curve is one and the model has ideal measure of separability.

[5:09:56] Now moving on to the next point. If we see it practically if my threshold If we see it practically if my threshold is 0.5 there would be some you know uh you know errors that you know the red ball going into the green bag and the

[5:10:11] green ball going into the red bag but these should be less and if my area under the curve is close to one around.9.8 data 7 then also it is said

[5:10:25] to be a good classification problem. So AU.7 means that there are 70% chances that the model will be able to discriminate between negative and

[5:10:37] positive classes. Now getting my point is everybody now are you understanding the meaning of threshold? Now if I decrease the threshold suppose if I make it to 0.2 what will

[5:10:54] happen? My false negative error will become less but my false positive error will become more. So the threshold we have to try to always create a balance

[5:11:06] have to try to always create a balance to get the minimum errors. to get the minimum errors. Okay. where the model was not able to differentiate between negative and

[5:11:22] positive classes. One of the worst cases that area under the curve is 0.5. So we don't want the area under the curve to be equal to 0.5. [snorts] So this is the

[5:11:34] worst situation where area under the curve is approximately 0.5. The model has no discrimination capacity to distinguish between the positive

[5:11:47] [clears throat] Now better and what could be the even worse situation that the true negatives have been classified as true positive and true negatives uh positives as negative and the area under the curve is

[5:12:03] negative and the area under the curve is zero. So when a c is approximately equal to zero the model is actually reciprocating the classes it means the model is predicting a negative class as positive class and do we want this

[5:12:17] situation never we don't want this situation area under the curve to be zero area under the curve to be close to 0.5 but close to one is a good option

[5:12:30] 0.5 but close to one is a good option that means the classes are able to distinguish each other clearly. What is the role of threshold? Classification threshold in machine learning is a boundary.

[5:12:47] At what boundary? As I told you, we will cut the two classes or cutff point used to assign a specific predicted class for each object. Any machine learning algorithm for classification gives output in probability format. That point

[5:13:04] is also clear. So in the morning also I was teaching So in the morning also I was teaching the LLM. So LLM, chat, GPT is uh you come under the category of classification problems and they all

[5:13:19] give the output in terms of probability. Okay. So a little bit of GI and LLM models. So, LLM, chart, GPT, your co-pilot, all the different models, they

[5:13:31] are all come under the category of classification problem and generate the classification problem and generate the word based on probability. Okay, so that's the tip of the day today. So in order to assign a class to an instance

[5:13:45] for binary classification we compare the probability value to the threshold and if the value is greater than that then the probabil it belongs to class one if it is less than that it belongs to zero. So the story of

[5:14:00] threshold is like this that what is the cutff point that we can achieve to get the best distinguished classes that if you increase the threshold you move left on the curve and if you decrease you move right to the curve.

[5:14:17] Okay let's get back to the file. So now we are on lesson number four. You can go ahead with 4.1. So there are a lot of algorithms that we

[5:14:29] would be covering in this uh classification logistic regression, navebased classifier, KN&N, decision tree, support vector machines. I so today uh we will be doing touching on to the classification algorithms. Let's

[5:14:45] see how many are we able to do. Let's logistic near live bias and KN and definitely we are doing three. Let's see if we if we to begin with decision tree if we if we to begin with decision tree also. Let's see.

[5:15:01] of classification? How will you define it? Tell me what is the definition of classification? Yes learners.

[5:15:15] questions so that you have the you know clarity of the concepts and then you are clarity of the concepts and then you are also well prepared for the interviews. Supervised machine learning algorithm using categorical data always say it it

[5:15:30] is part of which learning technique and where the model is trained to predict where the model is trained to predict the class label the output is important when I say label the output of the given input data. So it is important that the

[5:15:47] output is categorical. Getting my point? Please try to cover all the points in the definition. If you're going to just put three four words, it will never give you a complete answer. Getting my point? Okay. So,

[5:16:03] answer. Getting my point? Okay. So, classification is a supervised machine learning technique used to predict the category of class of n observations category of class of n observations based on the training data. So, I think

[5:16:17] based on the training data. So, I think so this definition is quite correct. So classification algorithms categorize data into categories. So do you think that uh in classification there is a role of this

[5:16:32] regression? Do you think that can we use a straight Do you think that can we use a straight line to uh separate two classes? Let's see since linear regression is a simple function. Let's see can it be

[5:16:47] used. But before that you know classification examples of classification in healthcare whether you are diabetic or not or your medical condition is there all come under the health care category finance banks

[5:17:03] financial institutions our classification algorithm whether you are eligible for loan or not marketing classification aids in customer segmentation and target marketing. It helps business identify potential

[5:17:18] customers and tailor marketing strategies by categorizing customers based on their behavior. Retail in retail classification algorithms are crucial for managing inventory forecasting demand and manufacturing.

[5:17:34] Classification is essential for quality control, fall detection. So there is no domain where classification problems cannot be used. I hope you all are getting this point in retail, marketing, manufacturing, HR, everywhere.

[5:17:50] manufacturing, HR, everywhere. Right learners, classification that we have understood? Can you tell me?

[5:18:03] This is what we are going to try to solve. Rashan, can we use regression in classification problem? This is just we are just about to solve it. We're just about to solve it. Let's just be patient.

[5:18:17] Okay. Okay. So, what are the different types classification? So, when the model is able to classify only two classes for example, whether the male is fraud

[5:18:32] for example, whether the male is fraud or not fraud or spam or not spam comes under binary classification already predefined. Yeah. to classify into predefined multiple classes whether it is a tomato or whether it is a vegetable

[5:18:46] is a tomato or whether it is a vegetable it is a fruit or it is an animal or is it is a fruit or it is an animal or is it a domestic animal or pet so many classes can be used one to many no chakra pani that's not the correct way

[5:19:00] to answer the model is capable of predicting more than one classes of the input that's that's the way to answer It multilel multilel classification

[5:19:15] multilel classification be careful while answering this that each data point can be assigned multiple labels simultaneously rather than just one. Yes, simultaneously like in an image it can it can have an image of a

[5:19:29] image it can it can have an image of a dog, cat, rat, a chair etc. Okay. And the last part is imbalanced classification. What is imbalanced When the classes are there, they can be

[5:19:44] two classes or more than two classes. But they are not equally balanced. Right? one class like 70% of males are there and you know 30% of females when the classes are do not have the same frequency count or we can say when one

[5:20:00] frequency count or we can say when one class has higher significantly more than um other observations right when one becomes a majority class other one becomes a majority class other one becomes a minority class okay

[5:20:13] so binary classification under binary classification some popular algo algorithms are used. So please try to understand binary classification can be achieved by logistic regression, knives bias, KN&N decision trees and support

[5:20:27] vector machine. We are supposed to you know do all of these. So while these methods excel in binary capable of handling multiclass except

[5:20:39] logistic regression or other are capable of handling multiclassification task also this versatility allows them to be used in wider range of applications such

[5:20:51] as recognizing multiple categories of objects in images or predicting several objects in images or predicting several types of behavior. So now we'll start types of behavior. So now we'll start with logistic regression. Before that

[5:21:05] let me also let's do a recap. Now what are the different metrics involved in classification? How do I judge that my

[5:21:17] classification problem is correct or not correct? Confusion matrix. Yes. So what are the different components of a confusion matrix? accuracy, precision, recall. These are the different metrics. But

[5:21:32] components of confusion matrix I said components true positive, true negative, false positive, false negative. Absolutely correct. And based on these components,

[5:21:44] we calculate different metrics which can be used to analyze the classification model. So if this is my data and I divide the data because classification

[5:21:56] also comes under supervised learning technique. So if this is my data and I divide it into training and test set and the training set over here we have right from the training we develop the model. Test set is used for

[5:22:13] model evaluation and we understand for classification problem the different classification problem the different metrics used are accuracy precision and recall right so it's a quick revision again so this is my n to n matrix if I

[5:22:28] have two classes what is what are what are true positive and true negative so when the output is also true the predicted value is also true when the output is false when the predicted value is also false but what are the two types

[5:22:42] of errors. What are the two types of error that we What are the two types of error that we analyze in confusion matrix? Type one error is also known as alpha or the or the false positive

[5:22:59] alpha or the or the false positive error. Right? The type one is also known error. Right? The type one is also known as the false positive or the alpha error. And type two error is also known as the beta errors.

[5:23:15] Clear? Then which metric is to be used when we have accuracy. When out of the prediction model has been made what the percentage is. If it is an imbalance class is accuracy a good metric to be

[5:23:30] class is accuracy a good metric to be used right. If we have more of positives right true positive right out of all the yes how many of them were correct then we use the precision recall and sensitivity are the same when we want to

[5:23:44] answer the question how good the model was predicting at real yes events recall and specificity so there is this always this confusion about recall so

[5:23:56] recall is associated with specificity also and sensitivity also also and sensitivity also The question can accuracy so F1 score so The question can accuracy so F1 score so F1 score is our more matured score which

[5:24:10] can be used uh which is not biased towards the precision or recall and u it is used when deal dealing with imbalanced data set meaning that they are more of one class label than they are of the other. It corresponds to the

[5:24:26] are of the other. It corresponds to the harmonic mean of precision and recall. Right? And what is the graph that we understood which helps us to identify whether the two classes have been uh classified correctly or not. If you

[5:24:41] classified correctly or not. If you remember that graph learners remember that graph learners the AU it's it's AU and ROC curve. ROC stands for receiver operating characteristics. A stands for area under

[5:24:57] the curve, right? So what was that graph telling me? This graph generates probability value instead of binary 0 and one. It should be used when your and one. It should be used when your data is set roughly balanced. The ROC is

[5:25:13] imbalanced data set leads to incorrect interpretation. So ROC curves provide good overview of trade-off between the true positive and the false positive rate for binary classifier using different probability threshold. So the

[5:25:27] value if the value is below 0.5 it's a poor classifier 0.5 random classifier but if it is 7 or greater than one we can set that it is a good classifier.

[5:25:42] can set that it is a good classifier. Now clear. So this is how we understand the ROC and the AOC curve is between the false positive rate and the true false positive rate and the true positive rate. Right? And if you look at

[5:25:55] the more closer look at the A and the ROC curve, this is my X, this is my Yaxis. These this is my optimal threshold. Practically we are going to draw this graph also. and the selected threshold. The ROC curves gives a quick

[5:26:12] visual understanding of the classifier's accuracy. The closer to the right angle curve, the more accurate the model is. So more closer it is to the right. Classification threshold that turns the up upper left corner of the curve

[5:26:28] minimizing the difference between the true positive rate false positive rate true positive rate false positive rate is the optimal threshold. Clear? Okay. So now let's go ahead and

[5:26:41] understand the first algorithm for classification logistic regression that is transform linear regression which gives output linear regression which gives output between zero and one. So there was this

[5:26:54] between zero and one. So there was this uh query of you all can straight line be used for classification problem right that was the query. So when I say you know binary classification, how do you think that binary

[5:27:09] classification would appear? Yeah. So I'm asking you how do you think that binary classification points would actually look

[5:27:21] actually look in the graph? Tell me. Absolutely correct. Right. they can be on the either side of the straight line. Right? So if I say it's a typical good binary classification data, it's a

[5:27:38] typical binary classification data that either the output will be zero or the output this is this is an ideal situation agree the

[5:27:52] will be zero or the y will be one. Do you all agree? good fit on these data points or is it covering is it consistent or is it

[5:28:08] underfitting? No, this is not this is not a good fit. And moreover, the values are also getting lesser than zero. Some of the values and some of the values are greater than one. That we don't want

[5:28:22] because in binary classification the output will either be zero or be one in output will either be zero or be one in idle case. Agreed? So now I need to idle case. Agreed? So now I need to modify this straight line into a

[5:28:37] modify this straight line into a sigmoidal curve or an S shaped curve. Is it helpful for me? First of all, the values will always be within the range between 0 to 1.

[5:28:50] Is this point getting clear to everybody? And suppose if I want to find everybody? And suppose if I want to find out that my threshold value over here is out that my threshold value over here is 0.5. So if my value is 02 it it is very

[5:29:04] clear that it belongs to this class and if the value is 7 it belongs to this class. Uh only the confusion would arise at 0.5. So it is up to me the 0.5 I want

[5:29:16] to keep it on this class or that class. Clear? Yes, linear regression that question that regression are incapable of solving

[5:29:29] that regression are incapable of solving binary classification problem. Why? Because the straight line gives under fit to all data points. The you know the values can exceed greater than one or zero. Right? So are all these points

[5:29:44] zero. Right? So are all these points getting clear to everybody? mathematically? So we understand this sigmoidal function

[5:29:57] So we understand this sigmoidal function equation is written like this right? P equation is written like this right? P is the probability 1 upon 1 + e to the is the probability 1 upon 1 + e to the power minus y where y is nothing but the

[5:30:11] why can you tell me it's like the linear polomial not even polomial multiple linear regression equation. Agreed learners? So this is your linear regression. This is your sigmoidal function.

[5:30:33] So now if I start taking log on both the sides. So now on the left hand side this is the equation that log to the power p is nothing but the probability. We want the answer in terms of probability that the values will lie between 0 to 1 that

[5:30:49] the values will lie between 0 to 1 that is p upon 1 minus p beta kn beta 1 x1 beta_2 x2. So again ma'am do we are we need to solving the coefficients over here. Yes. Logistic regression is a modification of linear regression. But

[5:31:05] here also we are trying to find out these regression coefficients. Got my these regression coefficients. Got my point? Is this point getting clear to everybody?

[5:31:27] So taking this point further log px upon minus px is known as the log it function. Now getting my point where does the term logistic term comes from? does the term logistic term comes from? because of this equation and simple px 1

[5:31:40] because of this equation and simple px 1 minus px is known as odds of p that's the technical term. So now you are clear that the straight line or linear that the straight line or linear regression is not capable of solving the

[5:31:54] logistic regression problem. So the straight line can gets converted into the sigmoidal scurve because the predicted y lies within the range zero predicted y lies within the range zero and one. I hope this is clear to

[5:32:09] and one. I hope this is clear to everybody. success 1 minus P is the failure of the probability right? Same we are using absolutely the concept remains the same. Yes.

[5:32:33] learning sorry what is logistic regression? Please try to understand and try to answer all the points. This is what the interviewer will catch. How what the interviewer will catch. How many concepts you have grasped. Okay. So

[5:32:45] logistic regression is a supervised uh machine learning technique primarily used for binary classification. In this method we apply the sigmoidal function to the linear combination of

[5:33:01] independent variables and predictors. Now clear always say that it is a part of supervised learning classification problem used for binary classification. And here we are applying the sigmoidal function to predict uh you using the

[5:33:17] linear combination of independent uh variables predictors of features. What are what does this mean? What does that mean? Independent variable predictors of features. This refers to the input part of it.

[5:33:33] Right? Absolutely correct. Absolutely correct. Great. between 0 and one. And this probability represents the likelihood of a data

[5:33:48] point belonging to a specific class positive or negative outcome. positive or negative outcome. Clear?

[5:34:02] regression it uses the sigmoidal function. The core is the sigmoidal function. The core is the sigmoidal function sigma zs. So over here logistic regression is also known as sigmoidal function. This is how it will

[5:34:16] always range between 0 to 1 1 + e ^ minus zed where zed is nothing but the minus zed where zed is nothing but the linear combination of input

[5:34:31] multiple variable linear regression and the output is in terms of probability ranging between 0 to 1. Are these points 1 2 3 getting clear to everybody? So now you're getting the crux of machine learning. You have to first

[5:34:46] understand the mathematical intuition of that algorithm or model. Then how do we implement it in Python? Understand it from three perspective as I always tell

[5:34:58] in my data science class also. First try to understand the concept the mathematical intuition behind it. Second, how do we implement it in Python? And third, how are we going to interpret the output? All these things

[5:35:13] interpret the output? All these things need to be taken into account. Got it?

[5:35:30] What is cost function? The error function. How are we calculating the error function in logistic regression? That is calculated with the help of binary cross entropy. Please try to understand the cost function used in

[5:35:48] logistic regression is binary cross entropy which measures the discre dis differences between the predicted probability and the actual class label

[5:36:01] where m is the number of the training samples, y is the true label, ycap is the predicted probability. Clear?

[5:36:23] learners do we need to optimize our models also? See the concept same concepts will be applied. So what is optimization? Why do we need to optimize optimization? Why do we need to optimize the model?

[5:36:45] optimize? What is optimization? Extracting are the best, right? Getting the best. So the goal is to find weights W and bias B that minimize the cost W and bias B that minimize the cost function

[5:37:04] gradient descent. So we'll try doing the gradient descent algorithm as we move ahead with further algorithm mostly in onsemble learning I've covered. Yeah. So basically we want the best output from the model. So now we are

[5:37:21] going to use the breast cancer data set. Do I need to share this with you all?

[5:37:33] So this breast cancer Wisconsin diagnostic data set is widely used data set in the field of machine learning particularly in classification problems related to medical diagnosis. This data sets consist of breast cancer cases

[5:37:49] derived from a group of patients who underwent surgery and had their breast mass tissue sample whether they are cancerous or not. And basically there are two types of cancer. One is malignant and the other one is benign.

[5:38:03] Malignant malignant I don't know how I really pronounce it correct or not and really pronounce it correct or not and the other one is benign. Right? the other one is benign. Right? And this data set contains 569 instances

[5:38:16] each representing an individual sample of the breast tissue. Number of attributes the they are 30 numeric attributes computed from the digitized attributes computed from the digitized images of the tissue sample. These

[5:38:31] of the cell nuclei present in the images. So based on that we are there. So what are the different attributes or inputs in this data set? Radius, mean of the distance from the center to point. So

[5:38:45] it's not an image. Thus they have taken uh different parameters which will help us to uh decide whether it's cancerous or not. Textured standard deviation of or not. Textured standard deviation of the grayscale parimeter area smoothness

[5:38:59] compactness concavity concave point symmetry fractal fractal dimension. And the target variable diagnosis indicates the cancer type diagnosis indicates the cancer type which can be malignant or benign. So how

[5:39:13] many what is the output diagnosis and what are the two outputs of the uh diagnosis whether the cancer is malignant or benign. Now clear is the

[5:39:26] objective. Please always try to give time on the data set without that you will not be able to understand the analysis what you are trying to do and what you are trying to achieve. So we will use this data set to explore and

[5:39:43] compare various binary classification algorithm examining how their performance varies depending on the type implementation of each algorithm

[5:39:55] accompanied by the mathematical explanations. Yes learners are we good to go? Why do we do regularization? First let's understand that is if regularization is required then only we will do anush why

[5:40:10] required then only we will do anush why do we do regularization tell me yes if observed overfitting if the model will overfit then we will okay anush yes

[5:40:22] now clear so now let's start implementing so what are the steps uh in python come on tell me quickly learners what are the steps in python that we what are the steps in python that we need to perform tell me quickly Okay,

[5:40:35] importing of the libraries. I hope the basic libraries are clear to everybody. Then these are for the pre-processing model scaler pipelines. Have we understood the concept of pipelines? Can we implement it in classification?

[5:40:50] Yes. What is the concept of pipelines? EDA. What does EDA mean over here? No, we don't. Yeah, we start with EDA. But first is import of libraries. I'm starting from very basic assignment. So what is the use of pipelines? Tell me

[5:41:05] learners. Yesterday we did skarn pipelines.

[5:41:20] Deployment after package. What do you mean by that? automating workflow for building ML models. Good AJ, that's a better way.

[5:41:36] It's automating because we would do all the pre-processing in one workflow. That's the correct way of answering it. Okay. An skarn linear model logistic regression and accuracy. My skarn version is 1.5.1.

[5:41:53] version is 1.5.1. What about you learners? cancer data set everybody with ID, diagnosis, radius mean, texture mean?

[5:42:08] diagnosis, radius mean, texture mean? Oh, so Subra yours is higher than mine. Oh, so Subra yours is higher than mine. Right. So what do we observe in the info Right. So what do we observe in the info of this data set? Tell me.

[5:42:25] data set? Tell me where is my output? This is my output. And of course it has to be categorical as malignant or benign. And all others

[5:42:38] as malignant or benign. And all others are numerical values.

[5:42:50] So do we need to convert the output categorical value into numerical? categorical value into numerical? Do we need to encode the output? No. AJ, Do we need to encode the output? No. AJ, we don't need to encode the output.

[5:43:09] Yes, we definitely need to because computer will not understand benign or malignant. We all the categorical value need to get converted into numerical need to get converted into numerical value. Okay.

[5:43:40] This is the radius mean, texture mean, the per par meter mean. Do you see this? The count, the average value, standard deviation, minimum and maximum. So you have to uh you know observe whether the values are lying within the range or are

[5:43:55] values are lying within the range or are they too deviated right right so this is if I do df.escribe describe T then I get the values for all

[5:44:07] describe T then I get the values for all other parameters.

[5:44:30] complete null values. So can we drop off this column?

[5:44:46] 1. It is for the column in place equal to true. We understand that we that means it will make the changes original in the data set. So now we will remove

[5:44:58] this column because it is of no relevance. Got it.

[5:45:26] So when I do a df dot shape it has got this particular data set has got 569 this particular data set has got 569 um rows and 32 columns. Agreed? So this particular data set has 569 rows and 32 column. And how and what is

[5:45:44] the best way to check categorical data by using unique function which gives me the different categories. N unique gives me the number of categories but I always me the number of categories but I always prefer value counts function that means

[5:46:00] it gives me the frequency along with the category. So they are 357 benign category. So they are 357 benign cancerous patients and maling 212

[5:46:12] cancerous patients and maling 212 malignant patients. Got it nonas? of it I get this type. Everybody is

[5:46:26] of it I get this type. Everybody is getting this point learners.

[5:46:43] world always try to you know create graphs write down the observation what graphs write down the observation what the graph is doing the graph is doing right then again which which ID has been

[5:46:57] dropped which column has been dropped the ID column has been dropped and this the ID column has been dropped and this is my original data frame got it

[5:47:15] understand the relationship between the features. The heat map visualizes the features. The heat map visualizes the correlation between different in the data set. So what does the heat map do or what is correlation? Yesterday we

[5:47:30] discussed about this. We've discussed this point. What are the values of correlation? What does the correlation do? What is the range of values of correlation?

[5:47:47] uh relationship which will always range between minus1 to + one. Did I share that with you or not? So what are the values of correlation?

[5:48:03] negative correlation and no correlation. Yes. Good. Good. So you all remember other learners. Great. So heat map is the map which gives me So heat map is the map which gives me relationship between the various

[5:48:17] dimensions. So what do we observe that there is high correlation value between radius mean, parameter mean and area mean. Whereas when we talk about other

[5:48:29] observations, feature groups, features related to worst, largest value of these features for being each image, mean and error, standard error calculations are grouped over here. Now tell me what is happening in the

[5:48:44] next stage. What is happening over here? That from skarn pre-processing import label encoder. Right? Label encoder is

[5:48:56] Right? Label encoder is one type categorical encoding. Please try to understand the correct word. This is categorically encoding and we are is categorically encoding and we are using label encoder right and [snorts]

[5:49:11] using label encoder right and [snorts] only one one column is categorical that only one one column is categorical that is diagnosis and here I am trying to fit and transform only the diagnosis column. So now my benign and malignant values

[5:49:26] So now my benign and malignant values change into zero and one clear. Is this point getting clear? So now when I see my DF dotted I see now not M and

[5:49:40] B. Earlier please look at the output the column of uh it is in terms of M and B. column of uh it is in terms of M and B. Now I can see it in terms of

[5:49:52] zero. Can we define what is one and what is zero? How else? Yeah. So when one is malignant and zero is benign. Yes, we malignant and zero is benign. Yes, we can. Now what is happening over here?

[5:50:05] Yeah. So basically we are trying to separate the input and output and then separate the input and output and then split the training and the testing data. Right. And after the splitting only then we move to skarn pipeline uh tot

[5:50:22] pipeline to streamline the pre-processing and the training process. pre-processing and the training process. Why do we do this? Why do we do this after splitting the training and the testing data that the features in the

[5:50:34] training and the test sets are standardized to have a mean of zero and standard deviation of one. A logistic regression model is trained using standardized training data. to avoid data leakage. We don't want our testing

[5:50:49] data leakage. We don't want our testing data to be part of the training data otherwise there would no point uh be of the prediction. Got it? Right. So do we understand this pipeline function? What are the two things that

[5:51:05] we are doing in this pipeline function? Learners, we are creating a pipeline. First step we are doing what is standard_cala do come on learners tell me it's not that difficult we've all done the EDA

[5:51:20] part of it what is standard scaler do feature scaling yes standardization feature scaling yes standardization and then we are applying model selection logistic regression which will be iterated 10,000 time and randomly the

[5:51:34] iterated 10,000 time and randomly the data 42 is getting selected so till here are we clear we have splitted the data right into training and testing. Now after training and testing split and even the categorical encoding has been

[5:51:48] done I am doing standard scalar function. What is standard scalar function. What is standard scalar function that all my data points will have will be standardized with mean equal to standard deviation equal to 1.

[5:52:05] Right? They will all be having that value mean equal to zero and standard deviation equal to 1. And next step we will apply logistic regression on the

[5:52:18] will apply logistic regression on the data. Now better Simon now clear. And finally we go in for the pipeline dofit. What does pipeline dofit do? it will try to find out the parameters or the training data happens on input as well

[5:52:35] training data happens on input as well as output training data till here. So now when I do fit it will automatically first do standardiz standardization of all the input points and then fit the model and then we will predict the

[5:52:51] model and then we will predict the output using training and testing data. Clear? But since it is a binary classification

[5:53:03] problem, we will do not use simple predict function. We use predict probability a. What does that mean? Probable outputs are needed that will

[5:53:15] Probable outputs are needed that will range between 0 to 1. So this is my range between 0 to 1. So this is my output. Obtain the predicted output output. Obtain the predicted output after training the model.

[5:53:35] my actual label. This is my predicted label and this is the probability associated with it. Are you understanding the three aspects Are you understanding the three aspects of it? So when I do the probable A, it

[5:53:49] of it? So when I do the probable A, it gives me the values in binary. Yeah, the values will be 0 and one. The values will be 0 and one. But will it be zero or one will depend on the probability value. Got it AJ? This is what I was

[5:54:05] [clears throat] models, right? The probability decides what is going to be the output and then we can calculate the accuracy

[5:54:17] score and the training score. So in this case both of them are coming out to be quite same. So we can say it's overall coming out to be a good fit. But but which are the metrics which are actually used for classification problem? Which

[5:54:33] are the metrics which are actually used for classification problem? the confusion matrix because it's a little imbalanced not highly imbalanced because if two classes were absolutely equal then it would have

[5:54:49] been balanced data 50/50 but it is somewhere around 6040 I would say okay valueore counts or this graph is telling me it's imbalanced right so the training

[5:55:02] and the testing accuracy is there but ultimately who is the judge the confusion matrix. The components of confusion matrix are true positive, true negative, false positive, false negative. I hope these points are very

[5:55:16] much clear to you. So the provided confusion matrix provides a summary of the effectiveness of COVID 19 tests that identify

[5:55:28] individuals with with the virus and those without it. So we have understood this concepts that the K is correctly classified as positive and predicted also as positive. These terms are very clear. We've discussed it in detail

[5:55:42] yesterday and I had done a recap. So now significance of confusion metrics is in terms of these metrics such as accuracy, precision, recall which is sensitivity, precision, recall which is sensitivity, true positive rate, specificity and the

[5:55:57] true positive rate, specificity and the F1 score. Right learners and do we understand the formula that's also we've gone in detail.

[5:56:10] Then we also understand the ROC curve or the AU curve which is used as a graphical representation of a classification model's performance

[5:56:22] across different classification thresholds. It plots the true positive rate against the false positive rate at various threshold settings. So true positive rate or recall at these settings. Clear? So what is our main aim

[5:56:40] settings. Clear? So what is our main aim in uh you know plotting the AU or the ROC curve that the default threshold for many classification algorithm is 0.5 meaning that if the predicted probability is greater than.5 the

[5:56:56] probability is greater than.5 the instance is classified as positive. However this threshold might not always be optimal especially in the case of skewed classes. Do we understand the term skewed? What do we mean by skewed

[5:57:10] class distribution learners? If the data is not normally balance, it is not known to be skewed data. Okay. So the methods of finding optimal threshold, there are

[5:57:22] two ways. One is maximizing your Jordan's J statistics. In this we will uh find out the value of J by subtracting the true positive rate with the false positive rate. The optimal threshold is where the statistics is

[5:57:38] maximized. This method balances the TPR and the FPR aiming to maximize the true positive while minimizing the false positive. Closest point to 0, 1. Another method is to point on ROC curve that is closest to

[5:57:55] the top left corner representing ideal classifier. Clear? an area under the curve also we are clear. Now getting back to the Python code. So again confusion matrix

[5:58:12] code. So again confusion matrix classification report ROC curve all these are part of which library they are part of which library learners they are part of which library learners learn metrics be very clear skunder

[5:58:27] underscore uh sorry skarnmetric even the RSS MSE they were all under these curves clear so the confusion matrix matrix is

[5:58:39] uh is created simply by calling the confusion matrix by passing the uh confusion matrix by passing the uh actual value predicted value output and actual value predicted value output and based on that I get this display

[5:58:54] what do I observe 70 are my true positive 41 are my true negative and if I talk about type one and type two error it's quite less so logistic regression it's quite less so logistic regression is done well it gives me two and one

[5:59:08] is done well it gives me two and one errors clear and then the best way to go about is creating a logistic regression report. So over here the precision is 97 report. So over here the precision is 97 98 for 0 and 1 recall F1 score is 98 and

[5:59:23] 96 and if I look at the overall accuracy overall accuracy is also coming out to overall accuracy is also coming out to be 97%. So overall this particular logistic regression has come out to be a good classifier for this particular data

[5:59:38] set. Clear? So over here they've given in detail class 0 negative class precision is.97 97% of instances predicted as is.97 97% of instances predicted as class 0 are actually zero. 99% of actual

[5:59:53] class 0 instances are correctly predicted as class 0. And here we move predicted as class 0. And here we move ahead.

[6:00:05] Right? All of the detail is given over here. Okay. And finally, how do I do the judgment that I need to do plotting of the ROC AU curve which shows the trade-off between the positive rate and the false positive rate at various

[6:00:21] threshold settings. The AU value indicates the model's ability to discriminate between the positive and the negative classes. So parameters used in plotting are these many parameters that first of all ROC curve is the

[6:00:39] function which gives me the different values of FPR, DPR and threshold right and based on that I'm going to calculate my Yordan's J and find out my optimal

[6:00:51] threshold value. So what do I observe that optimal threshold value is coming that optimal threshold value is coming out to be 0.4848 48 48 and ROC or the area under the curve as one. So is it a good classifier?

[6:01:03] Area under the curve is nearing it's almost one. It is one not even almost almost one. It is one not even almost one. So it is a very good classifier

[6:01:15] one. So it is a very good classifier getting my point learners. So the ROC curve in the image reaches the top left corner TPR equal to 1, FPR equal to zero which indicates a perfect classification performance. The model

[6:01:30] perfectly distinguishes between positive and negative classes at various And the ROC [clears throat] curve for logistic regression model has AU1. This the model has perfect discriminatory

[6:01:45] power and AU of one means model correctly classifies all positive and negative instances without error and the optimal threshold of 0.4867

[6:01:58] defines the decision boundary for the classifier. Probabilities above this value indicate stronger belief that an instance belongs to the positive class whereas the probability below this value indicate stronger belief that it belongs

[6:02:12] indicate stronger belief that it belongs to the negative class. Clear? You know this practical example has made all the points clear.

[6:02:24] But over here are we doing any kind of sigmoidal function anything that's already part of the implementation. So again I'm repeating this point. First try to understand the mathematical intuition behind the algorithm or the

[6:02:38] model. Secondly how we will implement in Python and thirdly how do we interpret the results? Is it a good model or not? Clear? AJ says ma'am one doubt despite I

[6:02:50] seen that this model is perfect in matrix itself. What is the main purpose matrix itself. What is the main purpose of seeing ROC curve also? See ROC curve tells me whether the two classes have been classifi are they well separated

[6:03:02] from each other or not. That is the idea. Is there more That is the idea. Is there more overlapping or not? So just to reconfirm things. It's not it's not by just doing one test. You know if you are suffering

[6:03:16] from some problem the doctor wants to reconfirm from different test. Maybe the blood test, MRI report and X-rays or other things something like so it's something which is more visually appealing tells us has more impact. Yes,

[6:03:31] appealing tells us has more impact. Yes, this is this is a good model accur the confusion matrix also tells us the different classification report also different classification report also tells us and AOC curves also tells us.

[6:03:43] So there there shouldn't be any doubt left you know when we are doing our analysis all the analysis should give the same result right

[6:03:55] so now do we understand this graph let's see so now this is how is also this is also one of the reason the threshold is used that if I take it as 0.5 any value coming above it as I told you would belong to this class you know so

[6:04:11] distinctly it is there is no overlap mapping it distinctly classifies all the points and any value which is lesser than 0.5 belongs to this class right so can we go in for a quick knowledge check about logistic regression and

[6:04:25] classification so question number one which of the following metrics are used to evaluate classification models confusion metrics

[6:04:37] ROC and F1 score is calculated using confusion what is a classifier. So when we talk about classifier, the output has to be categorical. So which one gives the output as category? It's both A and B.

[6:04:51] No, output as a single discrete value or output as a single discrete value. So both A and B. Input can be anything continuous or discrete. Yes, the answer is C.

[6:05:06] Next, false negative. What does false negative represent? Which option is correct? It's a predictive negatives that are actually positives. Yes. In

[6:05:19] binary logistic regression definitely it is C. The dependent variable consist of is C. The dependent variable consist of category. Next question. Why is linear category. Next question. Why is linear regression model output a poor predictor

[6:05:32] of probability? Why? Because D is the correct answer. It can give you know it correct answer. It can give you know it cannot the range can be beyond zero and one. Next the output in logistic regression

[6:05:46] problem is yes equivalent to one or true what is the pro what is its possible value? So of course it is b it would depend on the threshold value. Okay.

[6:05:59] depend on the threshold value. Okay. So now we have knives base classifier. Okay. Now moving on to the next algorithm. The next next algorithm is algorithm. The next next algorithm is completely based on probability. So what

[6:06:13] is probability learners? Chances of occurring of an event. And what is the formula? Mathematical formula of probability. The number of successful outcomes upon the total number of outcomes. Right? What is the probability

[6:06:26] of getting ahead when I when I throw a coin? It's 1 upon 2.5. When I throw a coin, what is the probability of getting head? What is the

[6:06:38] probability of getting one? When I throw a dice that is one by six. But there is another concept which is associated which is known as the conditional probability. Do we understand what is the probability of event A given

[6:06:55] the probability of event A given probability of event B? Do we understand conditional probability? This particular algorithm is based on that

[6:07:08] basian theorem. That's why data science is important. So learners who have done data science should should be clear about the basian theorem. This conditional probability or the posterior probability is equal to the likelihood

[6:07:26] function that's opposite. What is the probability of event B given A into the probability of event B given A into the prior probability A divided by the marginal probability? Do you remember this point or not?

[6:07:42] this point or not? No.

[6:07:54] is directly proportional to the likelihood function. probability. Okay. So the navebased classifier this

[6:08:08] is the formula that if I have these random figures or you know shapes the classifier is capable of classifying them into different shapes or groups

[6:08:23] them into different shapes or groups right and the basic formula lies in that the posterior probability is directly proportional to the likelihood one function into the class prior probability divided by the predictor

[6:08:37] probability divided by the predictor prior probability. Right? So how is this uh you know posterior probability actually calculated? It is nothing but the product of the likelihood function. Do you say see this X1 my