Deep Learning Fundamentals — Step-by-Step Guide & Transcript

[00:08] recognize images, understand patterns, make prediction and solve real world problems almost like a human brain? That is the power of deep learning. Hello learn. In this video, we will explore the complete beginner to practical

[00:22] journey of deep learning. We will understand how neural networks work, how machines learn from the data, and how frameworks like TensorFlow and PyTorch help us build real deep learning models. This topic is important because deep

[00:34] learning is behind many of today's most powerful AI applications, including image recognitions, speech assistance, recommendation systems, natural language generative AI. If you want to build a strong foundation in AI and machine

[00:49] most important skills to learn. In this session, we will cover the basics of deep learning, artificial neurons, perceptrons, weights, activation function, forward progression, loss function, gradient descent and back

[01:02] propagation. We will also move into practical implementation using TensorFlow, Keras and PyTorch. You will see how neural networks are built, trained, evaluated and applied to real classification problems using data sets

[01:15] like fashion and a capstone style projects. By the end of this video, models learn, how training works, and how to start building your own neural networks using modern deep learning frameworks. Before we move ahead, let me

[01:28] quickly share you something exciting for anyone who wants to build a strong career in generative AI and machine learning. The ENICT Academy IT Kpur generative AI and machine learning is designed to help you learn the complete

[01:42] AI and ML skill set from Python, data science, machine learning, deep learning and NLP to generative AI, prompt engineering, LLMS, computer vision and includes live online classes, master classes designed by IIT Kpur faculty and

[01:59] practical exposure to popular tools like chat GPT, hugging face, DALE, tens of chat GPT, hugging face, DALE, tens of flow, keras, gradio, lang chain, openi hands-on projects covering real world

[02:12] use cases like employee attrition prediction, loan default analysis, AI powered HR assistance, texttoes platform, recommendation engines, sales forecasting and deep learning applications. On completing the program,

[02:25] you'll receive a program completion certificate from ENICD academy IIT Kpur along with official Microsoft course badges for eligible learning path. So if tools and actually understand how generative AI and machine learning

[02:39] solutions are built and applied in real world business problems, this program is worth checking out. The link is given in the description box below and in the pin continue, here's a quick question for you to answer. Which of the following is

[02:51] commonly used as an activation function in hidden layers of neural networks? Is in hidden layers of neural networks? Is it RLU, HTML, SQL, or Excel? Let me know your answers in the comments section below. Let's get started. I I've always

[03:03] found that in in teaching this several times uh in this program getting to the deep learning is um challenging doing the deep learning is challenging but very rewarding and I think a lot of people enjoy the topics here because

[03:19] they're very u modern and very relevant to um very advanced kind of modeling uh techniques. So, a lot of people enjoy it, but it is going to be a little bit

[03:31] more challenging than what we've done so far. So, that's okay. I think we can far. So, that's okay. I think we can handle it. Um, so yeah, let's start with our introduction here. Okay. So, what we're hoping to

[03:45] cover here are just the uh some of the background mostly just background information about deep learning. So some of the um achievements that have really of the um achievements that have really happened in recent times um and we're

[03:58] talking you know less than 10 years old and some of these achievements that are uh have brought deep learning to the forefront and made it a really popular uh field. So we'll understand some of the applications and also some of the

[04:13] deep learning is not perfect. It has some drawbacks and some challenges that make it um difficult at times. So we'll talk about that and then we'll talk already mentioned uh PyTorch and TensorFlow as the two main ones. Just

[04:26] talk about that a little bit. Um and then talk about what are the main steps of any deep learning project that uh we would want to follow. I think what you're going to see there is they mostly mimic the same kind of steps we were

[04:39] doing in machine learning. things like uh data preparation, then kind of building a model, training it, um evaluating it on test data. Those kind the details of those are going to be drastically different. You know, that

[04:55] training process is going to be more involved, I think, with with deep learning. Um building the model is a little more hands-on with deep learning to be architecting and constructing a neural network. It's much different than

[05:09] just instantiating a um a model from scikitlearn which is pretty easy to do. Um there's going to be more hands-on work involved in in building these deep learning uh networks that are going to be our models. Um so we'll talk about

[05:23] that process. Okay. All right. So let's start with just AI in general. So the reason to start here is just to see where deep learning kind of fits into the rest of the stuff that we've studied so far such as machine learning and um

[05:40] historically uh where has deep learning fit in. So, I think we've seen this picture before um early on in our machine learning and even data science uh uh courses, but here it is again um where we have AI as

[05:56] here it is again um where we have AI as this pretty broad umbrella um that is um basically involving machines, so computers um trying to perform tasks that normally humans would have to do. And what we just got finished studying

[06:11] was machine learning, right? So machine learning is primarily having the machine learn from data how to make a decision right how to complete some task whether that's prediction or maybe like an unsupervised case um doing things like

[06:28] compression dimensionality reduction or finding patterns and clusters it's learning from data how to do that right that's machine learning broadly um where that's machine learning broadly um where deep learning fits in is really a subset

[06:41] of machine learning because we're still learning from data. Uh we still are doing that. It's just we're using a specific tool to do that which is going to be a neural network. Um so that's why in this diagram you see deep learning as

[06:56] kind of a circle embedded inside of the machine learning circle because it's it is kind of a sub uh field of machine learning. You're still learning from data. You're just doing it in a very particular way. Um, so it's kind of like

[07:10] a specialization. That's why I said earlier not everybody studies deep learning because you can get pretty far without deep learning. You still have this whole field of machine learning and you can you can

[07:23] still do a lot with that. Um, but this is certainly kind of a specialization is drilling down into working with neural nets in particular as your model. It turns out that working with neural nets is really powerful for many reasons. Um,

[07:39] and what we're going to see is that deep learning really unlocks uh working with specific types of data that we have not worked with so far. And you know, we already have a sense of that. Things like images, things like text, we really

[07:52] haven't worked with too much or at all. Um, but deep learning really unlocks the ability to do that. Um, so you can see that here. Yeah, there there are um we've briefly talked about a couple of them. Uh so inside of machine learning

[08:07] that is um not necessarily deep learning would be something like RL. So and it's not really a small circle but probably another big one is uh and it it honestly it kind of overlaps with deep learning because you can use uh neural networks

[08:22] inside of reinforcement learning. So RL is short for reinforcement learning. I draw it overlap just to say that nowadays a lot of RO is done with deep learning ideas in terms of using neural nets but it wasn't always that way but

[08:37] nets but it wasn't always that way but yeah there's RO and then there's uh certainly like a lot of use cases of unsupervised learning inside of machine learning that does not use neural nets. So unsupervised learning.

[08:51] So things like clustering uh unsupervised learning, you know, thing basically things that do not use neural nets but are but are still you're still learning from data. Yeah, it's basically everything we covered in

[09:05] the last course. Um so even supervised learning that's not using neural nets is still in this in this bucket of machine learning. All right. So, so this is this learning. We're going to be in this course. This is where it kind of fits

[09:20] learning because at the end of the day, we're still learning from data. It's just going to be using a particular type of model which is a neural net. And more so than just a basic neural net, it's often going to be what we will um term

[09:35] as a deep neural net. meaning it will be a pretty complex model that has uh a lot a pretty complex model that has uh a lot of layers to it um as we will see but it of layers to it um as we will see but it is just a subset of machine learning.

[09:52] Okay. So historically how has this evolved to get to that point? Um well we evolved to get to that point? Um well we had originally the the very first model had originally the the very first model of a neuron was proposed in 1943

[10:04] um by mulla and pitts. So they produced a paper kind of um proposing a potential model of the of the brain and the neurons in the brain uh in 1943. So obviously a long time ago. And at the same time some other things going on. We

[10:19] had the Turing test which you may have heard of basically um testing the intelligent ability of a machine. And so we had that going on as well. Obviously other things going on from the 40s to 50. But the big one that's relevant to

[10:35] 50. But the big one that's relevant to deep learning is the very first proposed model of the neuron that was as old as 1943. It was more of an idea then. Obviously they didn't have computing power to do

[10:47] anything with that. Um it was more just the idea. The concept was kind of as old as that of just modeling an actual neuron. Um it wasn't until the 50s that neuron. Um it wasn't until the 50s that we see the actual um term artificial

[11:02] intelligence being adopted. Um as well as the very first very basic neur uh neural network model the perceptron uh that was invented in 1957. At the same time, we have some

[11:17] programming languages being invented uh to take advantage of machines. Um not Python yet obviously, but you know, some older languages that end up inspiring

[11:29] older languages that end up inspiring the uh generation of Python later on, the uh generation of Python later on, but still some some uh highle languages that are invented. But this perceptron is really relevant. Actually, we're

[11:41] going to study perceptron in the next lesson as kind of the first model, the first basic neural network model that uh will propel, you know, uh deeper and

[11:53] deeper networks uh built off of the idea of kind of a perceptron. So, we'll talk about perceptron quite a bit. Okay. So, then we go to the 60s. Are perceptron

[12:05] still being used today? Um perceptron itself is probably not used too much but the idea of the perceptron is very present in so many network. It think of

[12:17] it like the building block of any neural network um was founded on the perceptron network um was founded on the perceptron idea. So the the perceptron itself like to learn about this but the perceptron is basically the model of a single

[12:32] neuron. a single neuron not quite used too much. Uh it's very basic but it forms the building block for nearly all neural networks that are around today.

[12:45] So I would say yeah I would say it's it's inspired pretty much any neural network is inspired by the perceptron. So it's a very very important idea and concept and we're going to spend some time talking

[12:57] about it and studying it uh as kind of our foundation for neural nets and then we'll build on that by kind of stacking uh stacking multiple dural multiple

[13:09] perceptrons essentially together uh till we get a larger and larger network. Yeah. So the thing is the idea has been around a long time. Uh but there hasn't been the you know computing power to really take advantage of it until

[13:23] recently. And when I say recently probably last 15 20 years um hasn't been right there hasn't been the computing power available to really unlock the

[13:35] capabilities of these and make make uh larger and larger networks that can solve harder and harder problems. So we don't see that uh we don't see that until you know recent first job was a key punch operator. I think it's the uh

[13:49] don't know what key punch is. I think it's the um like the binary uh the bit it's the um like the binary uh the bit uh cards that uh fed the original kind of computers or older computers and you had to literally punch out the bits.

[14:06] Also this time in the 60s was the original chatbot Eliza um 1966. Now it's nothing like generative AI right this is this chatbot is all rules driven so

[14:20] nothing datadriven nothing like that certainly no neural net nothing that we see today with these transformerbased models and GPTs and such but it is a chatbot in the sense of you can input text and then it can respond with text

[14:34] basically it's all rule driven though so it had to be pre-programmed uh how to respond to things but that was you know the first kind of AI chatbot you know the first kind of AI chatbot Okay. So then we see kind of a uh lull

[14:48] Okay. So then we see kind of a uh lull in the uh 70s to 70 to 80. Um interest in AI dropped significantly. Um funding from from government sources was insufficient. So there's kind of a lull period here. Now that's not to say that

[15:03] um there wasn't anything going on. You know, there's still statistical research going on and and advancements in the machine learning community, but nothing really relevant to neural networks uh at this period. But still, you know, some

[15:20] in the statistics community are still relevant during this time uh to machine learning, just not necessarily deep learning. But there's a rebound of that in the 80s. Um, and you may be able to guess

[15:38] 80s. Um, and you may be able to guess why. Maybe due to geopolitical things going on in the '8s, like the cold war. Um, we see a revival in uh research and dedication to AI. Um, we also start to see some communities uh develop um that

[15:56] dedicated to the advancement of AI. So the Stanford hosts the inaugural uh conference the AAI um which is kind of the uh inspiration for some of the conferences we see today like nuripss and and things like that.

[16:12] So in the 80s there's kind of a revival and then there's a kind of a um interesting period of time in the late late 90s so

[16:24] kind of early I would say even early 90s to early 2000s in this period um we see to early 2000s in this period um we see a significant investment in AI from the big players like like Facebooks and Netflixes of the world and Google of

[16:40] course Twitter they started investing in AI quite a bit. 1997 we have the IBM deep blue if you've ever heard of that. Um beat the uh world chess champion. Um

[16:52] so that was a really cool kind of AI achievement. But this is this period's achievement. But this is this period's called out here in particular because AI is particular deep learning had not advanced that far even by this point.

[17:06] However, there are some significant investments in this period that lead us to kind of where we are today. Um, I would say two of them that are not necessarily listed here but will be coming up are where it says companies

[17:20] like Facebook, Twitter and Netflix and Google not mentioned but should be there. They really start investing in AI. I want to call out Facebook and AI. I want to call out Facebook and Google in particular because they build

[17:33] the frameworks that we use today. So Facebook invented PyTorch around this Facebook invented PyTorch around this time and Google invented TensorFlow around this time. They did those independently. Now they are mostly open

[17:48] sourced so anybody can use them but they were in-house tools to work with neural neural nets. They had their own in-house tools uh like PyTorch and TensorFlow to kind of build and manage neural nets at the time. And this all started right

[18:03] the time. And this all started right around this period 2006 uh or so you know not that long ago relative to uh history only about 20 years not that long ago and we see kind of those foundational uh libraries being

[18:16] developed and obviously there's been a lot of advancements since then so in uh the period after that um this is kind of where the deep learning golden age is like really the 2010s onward 2012 is called out here.

[18:32] really big event in 2012 that happens which is the Alex net convolutional net which is the Alex net convolutional net is is uh is published and studied. It's is is uh is published and studied. It's a really big computer vision model. So a

[18:46] lot of really amazing achievements from from this period really 2010s onward in deep learning and it kind of coincides with hardware being more available um

[18:59] neural nets being more readily available with those frameworks being more mature and developed like PyTorch and TensorFlow um more people have access to them. They're open source, so more people can work with them and and really

[19:12] get their hands on it, including the research community. So, there's a reason it kind of blows up in the 2010s onward [snorts] and even even today. Uh, it's so I'll tell you why. It's because um a lot of generative AI is powered by the

[19:26] lot of generative AI is powered by the transformer and the transformer was not transformer and the transformer was not developed really until 2017. So, that's less than 10 years ago. Um, but the really the computing power and

[19:40] data availability to work with something like a transformer has not caught up until really like the last five or so years. So that's why it's so recent. The model that they're all built off of is less than 10 years old. Um, the

[19:54] transformer, but it's more like just other things like hardware and data availability. We've caught up to that. uh and and now we can have these really

[20:07] massive language models that power generative AI because compute power is generative AI because compute power is relatively easy to to manage and um data is pretty easy to get your hands on too but even but that's what I'm saying the

[20:21] model is not that old that powers it all right but a really good question um so what are some of our motivations for deep learning so there's a number of they're all listed here. We're going to start with the perceptron. We're going

[20:37] start with the perceptron. We're going to to talk about these topics later on that have to do with like uh the back propagation really having to deal with propagation really having to deal with how to train a neural net. Um so we'll

[20:49] talk about how how they get trained. We'll talk about the earliest model of a perceptron. So we'll certainly cover that. Um then we can stack perceptrons together in what what is known as a multi-layer

[21:03] perceptron. That's kind of the next level. Um that's kind of like a miniature deep neural net. It's it's not very deep, but it does have multiple very deep, but it does have multiple layers to it. Um and then we have the

[21:16] kind of in more recent times these very advanced networks like these that we're going to study. um convolutions mainly being used for images and recurrent neural networks mainly being used for

[21:28] text. So we're talking about the motivation one of the motivations one of the primary motivations for talking about deep learning is really this this idea that deep learning is going to unlock our

[21:43] ability to work with text data and our ability to work with image data. There are so many problems out there that work with this kind of data which in the community is known as unstructured data. Right? So, so far we've really only done

[21:59] problems with structured data. And structured meaning that it's basically a matrix that has rows and columns with really well-defined features, right? That are rows and columns like our like our housing data that has square footage

[22:12] and bedrooms and bathrooms. That's all very well structured. Text is not structured at all. It can be variable length. You can have paragraphs. You can have whole documents. It's unstructured. Um images, same thing. They can be many

[22:25] little bit more structure to them in the sense that they are matrices somewhat, right? They have um rows and columns of pixels, but still they're considered

[22:37] pixels, but still they're considered unstructured data. Um and and uh so neural nets and deep learning will unlock the ability to work with those. There's so many problems that work with images like object detection, image

[22:51] recognition, you know, and that extends to so many applications. And then working with text, obviously generative AI, working with text or even doing language translation, a lot of things work with text. So we need these

[23:05] advanced neural nets to be able to deal with that kind of data. The the models that we've studied so far just are not capable of working with that data very effectively at all. Like logistic regressions or decision trees or even

[23:20] to struggle on that unstructured data. They're not really going to be able to process it. Neural net on the other hand is going to be able to. That's one of our motivations for studying it is it unlocks this whole set of applications

[23:35] for us that we couldn't do otherwise namely working with text and image data. Right? Okay. So some particular tasks there that we were that we are going to study. So

[23:50] things like image recognition. Um now the very basic example that I probably have said before but you know is something like being able to predict if an image is a cat or a dog or if an image is um a cat, dog, a giraffe or a

[24:04] wolf or whatever it is, right? So image recognition tasks and that really extends into so many applications like object detection. So working with images really huge deal. We're going to devote a lot of time to it. Um but we really

[24:20] need neural nets to be able to do that effectively as we are going to find out. Um things with language like understanding and processing human language. We have these transformers that do this so effectively but it

[24:35] wasn't always that way and it was nearly impossible to do um with basic uh models. Just nearly impossible to understand language at the level that we see today. If you think about the the amount of language understanding it

[24:50] takes to have these um things like GPT and cla and stuff it's just immense and they're but they're really huge neural networks that are doing that and they're learning against so much data but we really need deep learning for that um

[25:05] and other applications things like speech recognition you know we haven't talked about it but another type of unstructured data is um audio right so audio data Being able to do things like recognition of audio translation into

[25:21] text, that's a really hard problem. Basically impossible without neural nets uh to be able to recognize speech and audio. Same thing with video. I know it's not listed here, but video certainly unstructured. We need uh

[25:36] certainly unstructured. We need uh neural nets to be able to process those um as well. So basically any type of unstructured data we're going to use neural nets to be able to process them. Yeah. A lot of a lot of uh neural nets.

[25:49] So speech recognition is powered by we're going to cover this later with much later on in this course when we get into RNN's. Um a lot of things like um

[26:01] Alexa devices for instance have been powered by RNN neural networks to uh basically recognize your commands your voice commands. That's all neural network powered, right? To be able to recognize a voice command, translate

[26:14] that into some text and then use that text for creating uh tasks or doing things or or generating new text to be able to respond. Yeah, they're that is yeah

[26:27] they're neural nets. So this speech recognition part is all um neural net driven namely RNN's recurrent neural nets um and and in modern times RNN/transformers

[26:40] we see transformers kind of taking over this space as well a lot of language stuff is being taken over by transformers but traditionally they've transformers but traditionally they've been RNN's okay so um let me give you

[26:55] some breakthroughs so for example You know, we talked about this period of know, we talked about this period of 2010s. One of the uh breakthroughs in 2010s. One of the uh breakthroughs in deep learning was in 2012, the Alex Net

[27:08] paper, which I encourage you to go look up and and read a little bit about. So, so the Alex net is a type of convolutional neural net and it convolutional neural net and it introduced really the convolution as a

[27:22] preeminent operation on image data to be able to classify images into different able to classify images into different categories. Um so it it this network categories. Um so it it this network architecture using convolutions

[27:35] um blew away the competition in the imageet challenge which was a um basically taking a bunch of images. So you can see here like the training data you can see here like the training data has 1.2 million images testing data has

[27:48] more has 1.5 million images. Your goal is to categorize those images into 1,000 different categories. So, so think of it as like 1,000 different animals, not just cat or dog, but a thousand different uh animals. And um the best

[28:05] methods at the time um things like basic neural nets or maybe even like an XG boost were blown away by this convolutional net um by more than this convolutional net um by more than 10% which is astronomical amount in in

[28:19] 10% which is astronomical amount in in research um of this at this kind of research um of this at this kind of level. So this was a huge breakthrough and what we see from there on is convolutions becoming kind of the

[28:32] default building block of anything working with images. So like object detection, it's all going to use convolutions and it was all inspired by this Alex Snap paper. And it was for the first time really like people realizing

[28:48] like hey these neural nets can really work on unstructured data very effectively because the methodologies to work on these images prior to this was to basically try to generate a bunch of structured features which was not very

[29:06] effective. going from like an image into a flattened vector and trying to generate features out of that to pass through like an XG boost um was the through like an XG boost um was the strategy but it got surpassed immensely

[29:19] strategy but it got surpassed immensely by this convolutional net and then we see just an explosion of computer vision progress with people using uh people progress with people using uh people using uh convolutions. So, one of the

[29:33] things we're going to study in this course is convolutions. They're really important for computer vision. If you're ever going to do anything with images, uh, any kind of neural nets, processing images is going to use convolutions in

[29:46] some fashion, right, in the architecture. So, we'll devote some time to studying convolutions for sure. But this was a huge breakthrough at the time. And it again, 2012, that's really not that long ago. It's it seems it's

[30:00] hard to believe that this was 2012. Um and you know that is kind of a long time somewhat but really not really not really in historical terms. Yeah, please really fascinating. Now it when you read that stuff it may not make any sense to

[30:16] you right now. That's okay. It should make a lot more sense by the time we get into uh computer vision and start studying convolutions. I think it'll make a lot more sense. So maybe hold on to those. I encourage you to read them,

[30:29] but they may not make any sense right now because we haven't covered anything about neural nets really um or especially convolutions. So they probably won't make any sense to you, but um I promise they will they'll make

[30:42] a lot more sense by the time we go through and talk about convolutions. more breakthroughs. That's just one of them. Uh however there are challenges uh

[30:54] that were encountered along the way. Um obviously like a lack of research funding was a big challenge. Um probably bigger like bigger than that definitely are these next two which are data and hardware. These are the two things that

[31:09] stalled deep learning progress for a long time. hardware mostly because neural nets and their structure, this is something we're going to learn about, something we're going to learn about, benefit immensely from advanced GPU

[31:22] hardware. Um, it's primarily because the computations that a neural net does benefit a lot from the parallel parallelization that a GPU can provide parallelization that a GPU can provide at the hardware level. And so only until

[31:35] advancements in hardware such that GPUs are readily available. Wasn't always that case. And also it wasn't always the case that that the code the Python code could take advantage of a GPU but it it can today pretty easily um with our

[31:52] frameworks like PyTorch and TensorFlow they they make it easy to take advantage of the GPU hardware and so that's why we've seen an explosion in the last 1015 years because GPUs are more readily available and can be taken advantage of

[32:05] available and can be taken advantage of by the code. Um same thing with data. data is more readily available, right? So there's a lot more of it, a lot more it's it's cheaper to store it, a lot easier to collect it. So it wasn't

[32:17] always that case. And what the cases with neural nets is you need a lot of data to train a neural net more than what we've used so far for the structure what we've used so far for the structure data problems um that we've dealt with.

[32:32] data problems um that we've dealt with. Um so the the the expensive nature of storing data and getting access to it was also prohibitive to a lot of progress in deep learning but that's that's been kind of fixed right it's now

[32:44] that's been kind of fixed right it's now a lot cheaper than ever and consequently we've seen a big explosion right last 1015 years okay all right any questions uh so far just some background right just some background and a little

[32:57] just some background and a little historical context um haven't gotten to this is the point of this first lesson, just more background information in historical context and then we'll start studying the models of the neuron and

[33:11] working our way up to, you know, more complex neural nets. Okay, let's talk about deep learning then uh a little more in particular. So we just covered the fact that um deep learning is really a subset of machine learning focusing on

[33:25] a subset of machine learning focusing on using neural nets and often what are going to be deep neural nets which are going to be neural nets with uh a lot of layers or a lot of um complex structure to it um to be able to uh learn from

[33:42] data. So learning from data is machine learning, right? But using neural nets learning, right? But using neural nets to do that is deep learning. So that's where we're at is using neural nets in particular, moving away from just basic

[33:57] machine learning into more specialization here with neural nets. So, um, what's nice about neural nets is that it can work with structured data, but as I've said, it can unlock the ability to work with unstructured data

[34:12] from many different domains such as imagery, audio, video, text um, to make imagery, audio, video, text um, to make predictions uh, usually um, and the uh, this has unlocked a lot of progress in domains like computer vision, so object

[34:27] detection, self-driving cars, those kind of things. NLP which is dealing with language so language tasks like sentiment analysis, translation, text generation. Um even working with audio data like speech recognition seen a lot

[34:40] data like speech recognition seen a lot of progress in those fields and deep learning is going to be able to surpass a lot of the machine learning techniques that we've seen because they have a very uncanny ability to learn complex

[34:55] uncanny ability to learn complex patterns. Um, one of the things that is that gives that the ability is how complex the model is. But the other data. So it's going to have a lot of examples to go off of. It needs that in

[35:09] order to extract those complex patterns amongst imagery, text, audio, video, right? It's going to require a lot of data. So that's something we'll see when we start doing examples. Okay? So just to compare the two. So we're talking

[35:24] learning. And remember deep learning really being a subset of machine learning. Machine learning is a little bit more broad um where we have you know unsupervised learning supervised. A lot of the problems in deep learning are

[35:37] still supervised. Meaning that there's still a label like if you have an image there's still a label of what that image is. Is it a cat? Is it a dog? Is it a giraffe? Right? Or one of those thousand categories like an imageet uh challenge.

[35:54] categories like an imageet uh challenge. So there's still a supervised nature to It's just the model that we're using is now a deep neural net, right? Instead of a decision tree, instead of a random forest. And also the data is inherently

[36:10] imagery, it's going to be text, potentially audio, video. So, we're potentially audio, video. So, we're moving away from structured data and really focusing on unstructured use cases with things like images and text.

[36:25] One of the things that's true that we've seen so far is especially with machine learning course we just finished, we saw that it's really important to get those features right. you know, we have to do some data prep to get our features

[36:38] correct, get rid of any null values, maybe even engineer some features like scaling it or adding two features together. Um, it's really important to deep learning is it's not that important to do that. It's not that important to

[36:55] do feature engineering at all because then the network's going to be so powerful at picking up on patterns from the original data like the original images. um we're not going to need to do a lot

[37:08] of manual feature engineering with with deep learning. So that's going to be one deep learning. So that's going to be one kind of advantage here is not a lot of manual uh feature engineering. And that's actually the the imageet uh

[37:22] challenge we just talked about with Alexet uh convolutional net winning that challenge. That was the thing everybody was doing was trying to do a lot of manual feature engineering so that they could use like a a classical machine

[37:35] learning model like an XG boost but doing that wasn't as effective as just a really good neural net that can just extract patterns from the raw data very effectively using something like convolution. Okay, so deep learning as

[37:49] I've said really excels in those kind of tasks like image recognition, NLP, speech uh recognition. Um, so we've talked about that. Uh, it NLP, remember, processing. So you'll probably hear me use that word NLP, especially when we

[38:04] get to later on and we're studying things like transformers and RNNs. Those are just language tasks, right? So think of text working with text, text of text working with text, text generation, sentiment analysis,

[38:17] uh speech or language translation I should say. Um so processing natural language which is really text right for for our use cases. Deep learning often

[38:29] for our use cases. Deep learning often requires really large data sets. So much more data than you see in a machine learning technique. Um and also more computational resources. So this is think GPU. Most deep learning is going

[38:43] to benefit from using a GPU. Whereas machine learning doesn't really use that at all. Like if you do a decision tree, an SVM, a logistic regression, you're not using a GPU at all. You don't need to. You it will still be pretty

[38:55] to. You it will still be pretty effective um without that. And uh will take forever. Just to give you some context, like the um GPTs of the world, these transformers that are, you know, neural network based, they train on data

[39:11] that's about the size if you took all of the text from Wikipedia and more. So all the text from Wikipedia, um, plus maybe all the text that would be in an encyclopedia collection. Think of a data set that big as a

[39:28] prototypical training data set for something like a GPT and think of thousands of GPUs working together. Even in that case, it can still take days to

[39:41] in that case, it can still take days to train a model. Days, not minutes, not hours, days. Because the models are so big and there's so much data being big and there's so much data being processed to train it. that is how much

[39:54] time it takes to train those kind of models. So there's substantial amount of resources um which is why there's so many uh news articles written about you know Nvidia and partnering with these uh generative

[40:09] AI companies like OpenAI and Microsoft etc. So a lot of resources are are needed. Now for the networks we're going to build in the in the program not so much. We'll we'll still explore using a GPU and we're actually going to practice

[40:24] using a GPU. Um we will have access to free GPUs in collab. So that's really nice or even the simply learn lab environment has free access GPUs. Um so we will practice using it and it will benefit us for things like image

[40:40] recognition working with imagery or RNN's or transformers even um they'll benefit us but just as a rule of thumb deep learning requires more data more resources to to train computational resources well so if you if you have a

[40:58] Windows machine sure um you can use the Nvidia GPUs if you have a Mac um like a Nvidia GPUs if you have a Mac um like a MacBook um even like one of the M1, M2,

[41:11] MacBook um even like one of the M1, M2, M3, M4 series, those have built-in GPU capabilities that uh we can take advantage of. So, I'll share those resources with us when we get to building our models. But even if you

[41:23] have like one of those M1, M2, M3, M4 Max with the the silicon chips, they can use their internal GPU to train models. Um, and I've done that before and it's really nice. It speeds up the training a

[41:38] lot. We can also use Collab. So Collab offers free GPU access, uh, which is offers free GPU access, uh, which is nice. Um, so we can use that too and we will. So we'll practice that, uh, as when we get to it. Okay. And then

[41:52] finally, deep learning. You know, we're going to be using neural nets that are they have a lot of layers. So, we haven't studied what a layer is yet. That's okay. Just think of a really complex model with a lot going on with

[42:06] complex model with a lot going on with it. That's going to be, you know, way decision tree or SVM or random forest even.

[42:18] here. to train deep learning, we uh often will use GPUs uh because they're going to be really effective. Most traditional machine learning methods that are not deep learning can get away with just using a CPU because they have

[42:31] very simple algorithms that are that are training that do not require using the um underlying GPU resource that can really speed up uh uh deep learning.

[42:43] really speed up uh uh deep learning. Okay. through some of the successes that have been recent, some of the foundational kind of moments. Um, and there's been growth in a lot of areas. So working

[42:59] with audio, computer vision, even reinforcement learning that has been powered by deep learning. Um, most reinforcement learning algorithms prior to deep learning were based on what are called kind of tabular methods. um where

[43:14] called kind of tabular methods. um where they were uh keeping track of um uh they were uh keeping track of um uh rewards and and uh data in in a more rewards and and uh data in in a more structured format. Uh but deep learning

[43:26] has since made a lot of progress in simplifying that and and uh making it much more powerful and and even in reinforcement learning. We'll talk about one of those uh cool moments there. Um so back to the Alexet, you know, we

[43:39] talked about this in 2012. This was the champion in the renowned uh imageet challenge. This is where again they used This is where again they used convolutions to really power

[43:52] learning imagery and being able to classify images. And this this was so classify images. And this this was so groundbreaking that uh a lot of moments in deep learning over the years have been referred to as kind of a imageet

[44:06] moment or Alexet moment um because how big of a breakthrough this really was in big of a breakthrough this really was in 2012. Um these this convolutional model was kind of created. Now we'll study this architecture later on of what this

[44:20] actually means, what these uh layers are doing, what is a convolution. Um we'll study all of that when we get into computer vision. But this was a huge computer vision. But this was a huge breakthrough in working with image data.

[44:38] breakthrough of working with text known as uh the first kind of word embedding model which is really really important for where we are today because word embeddings play a really critical role in working with text. uh because these

[44:52] are um models that map text into numerical vectors which is really critical for um working with language modeling. And every transformer really

[45:05] has a word embedding at the very front of it to take raw text or tokens and essentially um make those into a mathematical vector so the neural net can work with it. The very first uh model there that was a big breakthrough

[45:19] this is 2013 um was a model called wordtovec uh which uh you may have heard of. This was uh used for uh pre-processing text was uh used for uh pre-processing text essentially to to map it from raw word

[45:34] essentially to to map it from raw word or tokens into uh vectors. So really really critical for any kind of language model to do this. And so this this has been around for a little bit longer than the language models, but this is a

[45:46] really important breakthrough for them for for generative AI. So every generative AI like GPT cloud every everybody has a word embedding model everybody has a word embedding model that's at the very front of their uh

[45:59] transformerbased u text generation model. So this is this is a huge breakthrough. Then we have the very first kind of uh language models known as sequence to sequence. Now this is not using a

[46:14] sequence. Now this is not using a transformer. So this is using RNN's to transformer. So this is using RNN's to try to do NLP tasks like a language try to do NLP tasks like a language translation from uh German to English or

[46:27] French to English or or English to French whatever it is. And they tried to French whatever it is. And they tried to to make NLP, so doing things like translation, language generation, sentiment analysis, they tried to make

[46:40] it a lot better and it did. So at the time, RNN's were kind of the best language models around. And this is 2014, but they quickly get surpassed by And that's where we are today. Everything is kind of transformer-based.

[46:56] But at the time RNNs were kind of the best uh for language. We will study best uh for language. We will study RNN's uh later on in this course. Recurrent neural nets RNN's which are just a flavor of deep learning that uses

[47:11] a special architecture. So here it is again just some different variations. There's a lot of different variations on the RNN architecture um that attempts to predict um again what word should be next in the sequence

[47:25] using the RNN. We do that very well today with transformers rather than today with transformers rather than RNN's but at the time these were the best around. Okay. Right around the same time in the image generation uh field we

[47:41] have GANs as a big breakthrough. So what what are GANs? They are uh generative adversarial networks. Um these are models that generate images, fake

[47:53] images. Um and they do it through a competing uh adversarial networks that um one is a generator that builds images, the other is a discriminator that's supposed to predict if that image is real or fake. and they kind of boost

[48:08] each other up so that the generator by the time the training is done is so effective at generating fake images, they kind of mirror real images. This actually led to the explosion in deep fakes. So deep fakes became a real

[48:24] issue. There's still issues today, but the the first issue the first time it was an issue is really when GANs were uh prominent. They were developed in 2014. So again, not too long ago, most models

[48:39] that generate images today are actually not GANs. They're what are known as diffusion. So like Dolly and and the Google banana models that you can um put

[48:51] in some text and generate an image. Um they're not GANs, but GANs were the earliest kind of image generation models that were out there and really effective. And some of them are still used today for sure but majority of

[49:05] image generation models are are what is known as diffusion which we will study later on generative AI but this was a huge breakthrough in using neural nets huge breakthrough in using neural nets to generate images right that idea of

[49:20] generating images through neural nets this was a big breakthrough here 2014 okay and then on the reinforcement learning front um we have the alph developed by Google and Deep Mind team first uh kind of machine program to

[49:37] defeat a professional Go player. So if you've never heard of it, Go is a board game that is thought to be more complex than chess cuz the board is a lot bigger. Um it has more complex rules to it and more uh more states in terms of

[49:53] placed on the board. there's a lot more variability in in the pieces uh in their positions on the board. So, it's thought to be a more complex game and it was

[50:05] mastered by reinforcement learning using deep learning. So, this is the first time where deep learning really benefits a reinforcement learning model so much that it's able to beat uh go professional go players. Um this was a

[50:20] huge breakthrough 2016 about 10 years ago. Huge breakthrough especially in deep you know deep learning being able to be used in RL big deal. It wasn't

[50:33] always that way. Okay. Now to the big one which is 2017. This is what I alluded to earlier with transformers. So this is where um the very first

[50:45] transformer was created. We're going to study transformers later on in the course, but transformers changed everything when it came to language tasks. So, I said RNN's were used prior to transformers around 2014 to do a lot

[51:00] of language tasks. Transformers come along a few years later and really along a few years later and really revolutionize everything. And all of the revolutionize everything. And all of the generative models today like Chad GBT,

[51:13] like Gemini, like Claude, they're all based on Transformers. They are just massive extensions of the Transformer idea. Um these large language models,

[51:25] these LLMs are all based on the transformer idea. This was a huge transformer idea. This was a huge breakthrough. I remember myself um I had just gotten into data science, machine learning in the industry right around

[51:37] learning in the industry right around here like 2016 and I remember reading this paper at the time. Uh and how important it was is just a huge breakthrough. Uh the attention all attention is all you need. Highly

[51:50] encourage you to go and read that if you can. It's it's amazing. Now it might not make sense yet. By the time we get to transformers, our lesson on transformers later on in this course probably make more sense. But um really fascinating

[52:05] breakthrough. This is 2017, not that long ago, right? Really not that long ago. And not too long after this, we see the explosion in generative AI, right? Especially for text.

[52:20] Obviously, there's been other advancements over the years, but those advancements over the years, but those are some of the biggest. Um and uh what we've seen is deep learning has really come a long way. There's so many cool

[52:35] algorithms. There's uh there's also been an explosion in in implementation uh um because we have these um frameworks readily available that are open source that anybody can use. So more it's it's more and more people have access to

[52:52] these libraries and are able to develop their own algorithm in the community. So the research community has benefited a lot and we've seen because of that an lot and we've seen because of that an explosion in AI tools and uh AI models

[53:07] explosion in AI tools and uh AI models because of the ease of use of these deep learning um frameworks and and all the advancements that have come over the years. So a lot of cool uh libraries, a lot of efficient algorithms out there

[53:22] going to study PyTorch so we can build our own networks. But the community has benefited greatly and has been part of the reason there's been an explosion recently is because of how easy it is to get access to those libraries and start

[53:37] building your own neural nets and trying things out really easy. Okay. So that brings us to uh an important question which is why should we study deep learning? I said before that you know it's [snorts] an advanced topic. Not

[53:53] everybody studies it. It's a bit of a specialization within machine learning data science and machine learning and kind of stop there. So what reason do we kind of stop there. So what reason do we have as learners to actually study deep

[54:07] learning? I'm going to talk about that. One of the key reasons is that deep learning as I said before unlocks problems across many different domains that were that are difficult but are necessary in those domains. So I'll give

[54:22] you one example is healthcare. There's so many image related examples in healthcare that are really vital that neural nets help with significantly. So think about analyzing images and image data of the body or of um the brain of

[54:42] um potential diseases. Those can be more readily analyzed with Those can be more readily analyzed with deep learning. Um and so healthcare's seen a lot of advancements in analyzing that kind of data with with deep

[54:55] learning, autonomous vehicles of course, robotics, robotics, um even like fraud detection. There's a lot of advanced methodology there or even like time series forecasting has

[55:07] benefited from uh neural nets. So there's so many advanced use cases that there's so many advanced use cases that neural nets can benefit um because it can work with text, because it can work with image data. Um it just unlocks a

[55:23] lot of possibilities. Um so so that's one reason is it just gives us many more problems we could potentially solve in in different domains. It's worth studying it there. Another thing is we have so much more access, right? So, um,

[55:38] GPUs are more readily available. Um, cloud environments have GPUs readily Collab, like we are going to be able to work with GPUs in Collab for free. But

[55:50] certainly, like if you're paying for it on a on a cloud provider like GCP, AWS, on a on a cloud provider like GCP, AWS, Azure, they all have GPUs that are relatively cheap. Um, considering that we're actually using, you know, advanced

[56:05] we're actually using, you know, advanced hardware there. Um, so because of that, hardware there. Um, so because of that, it is because it's so readily available, we have a better chance at utilizing deep learning more than ever because

[56:19] GPUs are so so much uh easy to get your hands on than ever before. So, it's not to say it's perfect. Like, you know, we were joking earlier about the in Nvidia um 5090. you know, there are some physical hardware elements that are

[56:34] expensive and not easy to get your hands on, but like cloud resources for GPUs um are readily available pretty easily. We're going to see that with collab in We're going to see that with collab in particular. Another thing is uh you know

[56:48] GPUs I I forgot to mention this but let me go back to the slide actually is um one of the things that Nvidia really powered was this uh CUDA framework. This is something that I mentioned a little bit earlier. This allows the Python

[57:05] library to interact with the GPU and be able to offload operations to it instead of having to keep everything on the CPU which is much slower for the neural net operations that that it's doing and its layers. Um and so in uh there was kind

[57:21] of this um framework to interact Python with the hardware this CUDA um library with the hardware this CUDA um library and um because of that the libraries that manage neural nets like PyTorch and TensorFlow have integrated the CUDA

[57:36] TensorFlow have integrated the CUDA framework and have the ability to easily manipulate operations on the GPU um which makes working with uh neural net so much easier. So you can take advantage of a GPU. That's just a long

[57:50] way of saying you can take advantage of GPU really really easy with the modern framework. So with PyTorch, with TensorFlow, they can interact with the GPU so easily because of things like the CUDA framework

[58:04] um makes it so much easier. So the the CUDA is we don't ever see it. It's PyTorch or TensorFlow. We're going to see that we are going to build models in

[58:16] PyTorch and TensorFlow that will take advantage of the GPU. So, we're going to see this directly how easy it is to truly take advantage of that hardware and it's going to speed up our operations a lot. It's it's going to

[58:28] make it so much faster to to do that processing. Okay, so we just said GPUs uh play a critical role um and they're more readily available than ever. So that's another reason to kind of learn deep learning is we have access to these

[58:43] deep learning is we have access to these resources and it's it's really because the GPU can do parallel math operations so efficiently way more than a CPU. So

[58:55] they they really we're going to see the effects of this you know when we do especially working with um computer vision and convolutional networks working with a GPU then becomes almost necessary. It'll speed up operations so

[59:08] much like if we didn't use it, we'd be sitting there for minutes, maybe hour waiting for the model to train, but if we use a GPU, it'll take seconds. It'll really speed it up a lot. Okay, one of the other reasons we we've kind of

[59:21] hinted at this is just a low barrier to entry. So, so this is just to say why not learn it if you're going to learn machine learning. There's so many problems that use neural networks and why not learn deep learning because it's

[59:35] never been easier to do. There's hardware available with GPUs more than hardware available with GPUs more than ever. We talked about that. There's user really extremely userfriendly frameworks like TensorFlow and PyTorch to help us

[59:48] build neural networks and they're open source. So they're easily we can pip install it, get it into our environment and start building neural nets and training them. really really easy and we already have all the fundamentals we

[1:00:02] need to really understand model building, model training, um model evaluation. We have all that fundamentals from working with uh Python, working with uh machine learning that we've already done, data science.

[1:00:16] So it's pretty easy to get into at this point. So why not, right? it it's just going to help us uh expand our the breadth of our skills and problems that we can solve. And like I said, open sourcing these like TensorFlow and

[1:00:32] PyTorch was a big deal. It made it so much more accessible for the community to build their own neural nets. Prior to that, it was really limited to these big companies. You had to be in one of them in order to really start building neural

[1:00:45] nets. But now anybody can because these are free and open source like TensorFlow and PyTorch. Okay. So another key reason and you may be wondering this especially because of generative AI um taking over uh a lot of

[1:01:01] generative AI um taking over uh a lot of development these days but still there's a need for trained deep learning experts and practitioners. So many um companies still work with deep learning models and need deep learning expertise

[1:01:18] um to be able to like tweak the training, set up the training, set up the model, um make the adjustments if the training doesn't go well, really understand the model. Even if you're using generative AI, you still need to

[1:01:32] have an understanding of how the modeling works and how it functions in order to effectively use it. So there's still a need for deep learning expertise regardless of AI and generative AI. So it's a good opportunity. It's a it's a

[1:01:47] good skill to have. Um I can say this like beyond just having data science and machine learning experience going through this program, I think having the deep learning experience is just another

[1:02:03] really um beneficial thing to have if you're trying to get into the industry. you're trying to get into the industry. it it really unlocks more potential for you I think and again not everybody studies it so you're getting a kind of

[1:02:16] advancement above the competition potentially by learning deep learning all right let's talk about some applications so there's many as I said before there's many different uh fields that really benefit from uh deep

[1:02:30] learning and working with neural nets um so here's some of them just a few you know um NLP being a huge huge one in terms of processing text, being able to understand human language. That's a big one. Uh self-driving cars, using

[1:02:46] That's something we'll talk about and study. Um not only like detection of objects, but classification of them. So, think of like, okay, here is an image

[1:02:58] and inside of it, there's a pedestrian over here. There's a there's a a a sign over here. There's a there's a a a sign like a a stop sign. There's a tree over here. There's another car over here. Um here's the road um divider. Um so so

[1:03:16] detect in the environment and really classify what they are. And that's all powered by deep learning. Same thing with audio processing, transforming speech to text. That's how we talked about earlier. That's how things like

[1:03:30] Alexa and those voice assistants really work. Translation of audio into text. And then uh manipulating images as well. So generative AI with images. So much of

[1:03:42] generative AI is all deep learning based transformers um diffusion models that uh generate images, GANs, all of these generate and manipulate images. They're all based on deep learning. So so much of generative

[1:03:57] AI is deep learning based. Okay. Now there are some limitations. Uh one of the things to realize about deep learning, we've talked about it a lot so far is a significant amount of data is usually required. Meaning that deep

[1:04:11] learning may not be the best for tasks that have um little amounts of data. And so what is little? Um usually we're going to be dealing with uh in the tens

[1:04:23] of thousands if not more. That's like a that's on the very low end of maybe that's on the very low end of maybe images or text examples in order to images or text examples in order to train a moderatelysized neural net. Um

[1:04:36] maybe even on the smaller end of a neural net. The bigger the neural net that we have, the more complex of a model we have, the more data we need. Um, so this is this is like for the problems we've dealt with so far,

[1:04:51] throwing a neural net at it would likely be overkill and would not be effective. So it's not like we can just throw a neural net at our housing data and expect it to predict the price very effectively. It doesn't work like that.

[1:05:05] We would need a lot more data for that to be effective. Um, and we would probably need to tweak the neural net. uh a good amount to get an effective

[1:05:17] model there. The neural neural nets have a tendency to overfit. So they they need a lot of data to avoid that overfitting. They need a lot of examples. Um and and that can be sometimes difficult to come by. Um the amount of data that's

[1:05:32] required to be collected. Generative AI like GBT requires so much data. That's why they're using people's chats to continuously build a data set so that it

[1:05:44] can keep training it. Um so much data is required there to train a really effective model especially one that big. effective model especially one that big. Um just requires a huge data set.

[1:05:57] Um let me give you an example. So consider you know a model that's we've we've used this example before right just differentiating or classifying cats and dogs. So this image is a cat, this image is a dog and etc. Um, in order to

[1:06:11] image is a dog and etc. Um, in order to to do this effectively, we would need many different images of cats and dogs and and likely those images would have to be of different size cats and dogs, different breeds,

[1:06:25] um, from different angles potentially from different lighting and brightness of the image. um many different variations on those and they'd also have to be labeled, right? So it's it's still a supervised learning problem. Um so you

[1:06:40] know gathering that data set may be an expensive endeavor may not be trivial. expensive endeavor may not be trivial. So that is kind of a a cost burden of doing deep learning is the data. It's going to be a lot. um you may need a

[1:06:56] really massive data set and that can be difficult to acquire in some instances not all but in some cases that may be difficult to acquire. So it's just a it difficult to acquire. So it's just a it is not a limitation so much as it is a

[1:07:09] learning you generally need a lot of data and so you have to be prepared to collect a lot and process a lot in order to um have an effective model. The other thing is hardware. We've talked about this. Now, it it's again not necessarily

[1:07:25] this. Now, it it's again not necessarily a huge limitation because we do have a huge limitation because we do have access to GPUs, but you may need to pay for it. Um, and you may need advanced hardware. The more complex your model

[1:07:37] is, the more you're going to benefit from, you know, uh, a more complex from, you know, uh, a more complex hardware like a GPU. um it is more readily available than ever, but it doesn't mean that it's always going to

[1:07:50] doesn't mean that it's always going to be free and always going to be easy to uh acquire. So there may be some work there to use that or to acquire that hardware um for a really complex model. Now that being said, we are going to

[1:08:06] work with free GPUs for the examples we will do, but the examples we will do will be relatively moderately complex examples. They won't be massive [clears throat] models that would require a much more significant hardware

[1:08:21] investment to to be able to run. So, I used the example before um something like a Jad GBT is going to be using thousands of GPUs. That's a significant investment. If you're just using one for free from like Collab to do simple

[1:08:35] examples, that's one thing. But for a really big network, um you're going to really big network, um you're going to want to use a significant amount of GPU want to use a significant amount of GPU resources um which may be costly, right?

[1:08:49] resources um which may be costly, right? So it's just something to be aware of. mentioned earlier is that deep learning because there's um it's such a complex

[1:09:01] because there's um it's such a complex neural network it can be susceptible to neural network it can be susceptible to overfitting. Um in fact they are very susceptible to overfitting can easily overfit especially if there's not a lot

[1:09:13] of training data and that's just the nature of the neural net. it it's so complex of a model that it can effectively memorize the training data very easily if you don't have guardrails in place. And that's one of the things

[1:09:27] we're going to study in that lesson on optimization of the training process. certainly things we can do and we're going to study to prevent a neural net from overfitting, but they are very prone to it naturally very prone to it

[1:09:43] Remember, remember we talked about overfitting. The more complex a model is, the more likely it is to overfit. The more basic a model is, the more likely it is to underfit. So, neural nets fall in that category of being

[1:09:58] really complex. So, they are very prone to overfitting. So, something we have to kind of uh be on the lookout for. We'll study some techniques to help us uh overcome overfitting or prevent it. All right. Another one that is very

[1:10:15] think about when it comes to deep learning is the explanability of deep learning is the explanability of deep learning is very very limited. So when we studied machine learning and we did things like logistic regression or a

[1:10:30] things like logistic regression or a decision tree that was super explainable because we like in logistic regression you get coefficients on every feature you get coefficients on every feature right you get those betas. So we know

[1:10:43] exactly which features are important because they're going to have higher coefficients, right? And same thing with like a decision tree, you get an actual tree structure so you know that you know the path that leads to a prediction. And

[1:10:57] so they're really readily explainable. The issue with neural networks is they're often a black box. And and what that means is you put data into it, you get a result out, but it's almost impossible to explain every little thing

[1:11:14] that happened that led to that prediction. Very difficult to explain with neural nets because they're such a complex model. They don't have a very complex model. They don't have a very natural way of um having like a single

[1:11:27] coefficient. Most um to to give you some context, most modern neural nets like context, most modern neural nets like like a uh GPT that's generating text is like a uh GPT that's generating text is going to have in the billions, tens of

[1:11:42] billions, hundreds of billions, if not trillion weights, uh which are, you know, think of those betas. It's going to have like a trillion of those betas. So it's impossible to say what a single

[1:11:57] impossible to say what a single contribution to the prediction is. Now this is an active area of research in the research community is people are trying to figure out good ways to um

[1:12:10] break down how a neural net's making a prediction. But it still is near is pretty much not possible relative to other machine learning techniques. models. You put something in, you get something out. It's hard to say what

[1:12:24] happens in between. You know the architecture, but it's hard to say what exactly contributes to that final result. It's hard to hard to trace that result. It's hard to hard to trace that back. Um, so uh, deep learning can be

[1:12:37] very difficult for explanability and and this actually has an effect in like regulated environments. Let's say you're in the finance industry and you're making um like loan decisions based on a model. Um you probably don't want to use

[1:12:52] a a neural net because it's going to be hard to explain how you predicted your like uh loan or not loan decision. You you probably want to use something that is more explainable like logistic regression or a decision tree which

[1:13:06] would those you can directly see the contribution of their features right um neural net you're not going to be able to see that. It's just too complex of a model. Okay, so that's another kind of drawback of deep learning. Now there on

[1:13:22] cases that we don't care about the explanability at all. So you think about explanability at all. So you think about like a self-driving car, we don't care the explanation of why this is a stop sign, why this is a pedestrian, why this

[1:13:35] really fast. It needs to be real time. It needs to be really fast. We don't right? we just needed to make an accurate accurate prediction. So there are certainly cases where we don't care about that, but in a lot of like

[1:13:51] regulated industries, you may care about that. Uh if you're if you're using a model to make some type of decision um you may care about the explanability of it in which case neural nets are not ideal for that. Okay, let's talk about

[1:14:05] some of the uh the frameworks then that we will use. So these are going to be uh we will use. So these are going to be uh basically our libraries and tools to um basically our libraries and tools to um help us develop neural networks. Um

[1:14:20] help us develop neural networks. Um basically they um they allow us to design neural networks. They allow us to train them. They allow us to debug them like how they're working or what kind of uh evaluation we're doing if they're

[1:14:34] uh evaluation we're doing if they're producing any uh any bad results. Um, and also for deployment. Now, we won't really focus on deployment uh too much. That's not that's going to be out of the scope for us a bit. We're going to focus

[1:14:47] on um how to design and kind of train them. But these Python frameworks will them. But these Python frameworks will allow us to build and train and evaluate

[1:14:59] uh neural network neural networks. And this is huge like like I said earlier, it wasn't always this case, right? you really had to be part of these large corporations that were doing this like a Facebook or a Google in order to have

[1:15:15] access to the tools to build neural networks and now they're readily available like PyTorch and TensorFlow. So the main the main uh frameworks that we're going to study are uh PyTorch and TensorFlow. Now, caris is on this screen

[1:15:31] because what we're going to learn is caris is really um embedded into TensorFlow. It was actually developed by Google, the same people who made TensorFlow. It's really just a part of it. It's it's an interface to core

[1:15:47] TensorFlow. So, when we hear the word caris, we should really be thinking TensorFlow. It's the same thing. Um it's actually technically built on top of TensorFlow and allows us to interface with TensorFlow but so so these these

[1:16:02] with TensorFlow but so so these these were built by Google and then um PyTorch was built by Facebook now Meta um but at the time Facebook now Meta um but at the time Facebook um and they are open source now

[1:16:16] so they're widely available and of course we will learn how to use them to course we will learn how to use them to build neural nets and uh uh train them. Um they're both widely used. So both widely used community. Most people ha

[1:16:31] just have developed a preference for one or the other just based on either what they originally learned to begin with or um what they just naturally like. But um what they just naturally like. But the the truth is there's uh a lot of

[1:16:45] models out there like even like an open-source GPT that exists or Llama, they're developed in both frameworks. They have a version that's developed in PyTorch. They have a version that's developed in TensorFlow. And it's they

[1:17:01] do the same things. They mostly just have syntax differences. That's the only only thing about it is they primarily just have syntax differences. But again, it's worth our time to study both because both are so widely used. Um, and

[1:17:18] honestly, both are still maintained, even though they're open source, both of these companies still maintain uh groups of developers that are maintaining these

[1:17:30] within Google who actively manage TensorFlow and there's still a group TensorFlow and there's still a group within within Meta now that's actively managing PyTorch and they they do work on it all the time but it's also open

[1:17:44] source so contributions can come from anywhere really but there's dedicated kind of people within each of these uh companies that are still maintaining these frameworks because they're so critical right they're so critical to

[1:17:58] doing deep learning um and doing neural networks. So we're going to study them uh pretty deeply. So caris let me talk about caris. It is um going to be a a

[1:18:11] interface in python to the tensorflow library. So tensorflow itself. So tensorflow was created first and then caris came along a couple years later um caris came along a couple years later um to interact with tensorflow. TensorFlow

[1:18:25] is kind of the underlying library that Caris is built on top of and it just it's much much much more user friendly to manipulate and manage neural networks. So most people gravitate towards using the caris library within

[1:18:40] TensorFlow um to building if they're going to use TensorFlow, they're going to be using caris as as the interface to TensorFlow. So it's just built on top of that by the same Google group who made TensorFlow

[1:18:53] and it's it's the advantage of it rather than using the original TensorFlow is so much easier to get started with and build neural nets and so much easier to manage them. It's a really nice interface to TensorFlow. So when we

[1:19:07] build models in TensorFlow, we're really going to be using caris, okay, as that um kind of library on top of uh TensorFlow TensorFlow um to to build to build neural nets.

[1:19:21] So we will study that. Is there an advantage of using caris directly? Yeah, that's what it is. It's it's a really userfriendly interface. So it's designed to make uh building TensorFlow models so much easier because the truth

[1:19:37] of it is like core the original TensorFlow is kind of ugly. It's kind of TensorFlow is kind of ugly. It's kind of ugly and like messy to deal with. So um people when they're building TensorFlow models gravitate towards using caris as

[1:19:52] models gravitate towards using caris as the interface to tensorflow because to like core TensorFlow is really messy. Um, TensorFlow is, as we'll learn later, is is basically a graph library for building like a tensor graph to to uh

[1:20:09] mimic a neural network, a graph of uh computation. But Caris makes it so much easier to to work with TensorFlow. So most people gravitate towards caris then

[1:20:21] just you usually you won't ever use the original tensorflow unless you're maybe doing specific research that would require it. You're mostly always going to use caris to build your models and train them. So that's what we'll do.

[1:20:36] Okay. So underlying caris is TensorFlow which came first um from Google. uh it's you know obviously within Python it's open source now and this was the original library they built inhouse to to manipulate and manage neural networks

[1:20:51] you know caris came along as the interface to tensorflow to make working interface to tensorflow to make working with tensorflow so much easier um so actually tensorflow has some really cool tools uh uh as part of it that we will

[1:21:07] uh work with later on they have tools for like visualizing ing the training process which is really nice. Um so we'll work with those. Okay. So on the other end of it is PyTorch that uh that Facebook developed

[1:21:25] now Meta um who still maintains it but it you know has been open sourced. um same kind of thing they were trying to work with neural networks and needed an in-house you know in-house library to do that and so they developed PyTorch at

[1:21:40] the time you know and since it has been open source to the community and it's open source to the community and it's it's very similar to TensorFlow honestly has the same kind of features it as TensorFlow

[1:21:54] um it's just a different syntax and uh so that we're going to work with both as and with TensorFlow um in order to build models and and uh

[1:22:06] train them. So, as I said in the community, it's kind of a 50/50 split on which one people have a preference for. I personally um like PyTorch. I I I

[1:22:20] actually learned TensorFlow first um but found myself liking PyTorch a little bit more, but it's preference. Honestly, it's preference and I've seen um it it really just depends on which what you like better. And then oftentimes like

[1:22:36] where you're working also determines, you know, maybe everybody has a standard TensorFlow or oh, we're all going to use PyTorch. Um you see things like that. But honestly, if you know one of them,

[1:22:50] you will be well verssed in the other and and we are going to work with both. So we'll get exposure to both. just syntax differences really between the two. Okay, so what I wanted to do was wrap up this introduction. So we've

[1:23:05] covered a lot in the introduction, but I wanted to wrap up specifically with um the life cycle of a deep learning project, which I think we're going to see mostly overlaps with the same steps we were doing for machine learning. It's

[1:23:18] we were doing for machine learning. It's just there's going to be more, you know, are that are involved here when we do training, when we do uh model building, training, when we do uh model building, but mostly the life cycle this is

[1:23:33] learning from a high level, right? It's mostly the same. We have some type of data collection and data prep phase, which is kind of what this is. So, so this is like our data prep that we do for machine learning. We're used to

[1:23:48] for machine learning. We're used to doing that um first usually. Now, prior to that, there may be some planning that has to be done uh to collect data and and in deep learning that may actually be a significant amount of work to um go

[1:24:04] out and gather a bunch of images or process a bunch of images or text data, you know, and label it appropriately. that may actually be non-trivial but uh you know all the examples we will do will assume we already have that data

[1:24:19] but other than that it's mostly the same steps in term at a high level of like once we have our data prepped we go ahead and and um train we build and train our model um obviously the details of doing this with a neural net are

[1:24:34] going to be much different than with a basic model like logistic regression or something um but once we train the model we then evaluate right? We evaluate it on some test data evaluate and see how it performs. And

[1:24:49] that and the reason there's an arrow backwards is based on that evaluation we may need to make make some tweaks. So you know at each one of these it could be very iterative um in the training like if we're not if we're underfitting

[1:25:04] we may need to go back and revise our data. We may need to revise our model even. Um, so it can certainly be iterative and then after you have a model you're very satisfied with in the evaluation phase, that's where you can

[1:25:19] evaluation phase, that's where you can get into kind of um deployment of it and then like monitoring it and m maintaining it. These two things we're because again it's going to be a little bit out of scope for us. um we're mostly

[1:25:35] going to focus our attention on these two phases training and evaluating of neural nets. Okay. But let's talk a little bit more about these steps. So little bit more about these steps. So the planning uh as I said this is where

[1:25:50] um primarily whenever you're doing deep learning this third bullet is is really critical. So I would lean on this bullet as really critical as so planning really

[1:26:02] meaning what resource we going to need in terms of data what kind of data are we going to need like images text you know what are we working with here not only data but hardware like can we get access to like are we going to need a

[1:26:17] access to like are we going to need a GPU um can we get access to that for going to be really critical to know your resources ahead of time um and be ble to have those available especially the hardware um like a GPU. So usually

[1:26:32] there's some thought put into that from a planning perspective um in terms of the resources. Uh then there's the actual collection. So depending on the problem this could be very different right we may be

[1:26:46] physically taking a bunch of images we may be downloading a bunch of images or may be downloading a bunch of images or or grabbing them from some database. Um, we may have IoT sensors that we're collecting sensor data from or like our

[1:26:59] self-driving car is taking a bunch of imagery sensor data. Um, so that could be different. Um, audio, you know, audio or video, we could be collecting for

[1:27:12] examples. We need to collect that and we also need to label it, right? We need to have adequate labels and like think of the cat and dog images, right? we need to take every single image and provide a label with it. Um, if it's a cat or it's

[1:27:27] a dog and we also need to gather those images, right, in the first place. So, there's going to be data prep that goes on here, no different than before. training phase. So one of the things

[1:27:42] really shown here but is definitely a prerequisite is we first need to build prerequisite is we first need to build obviously we need to build I would say obviously we need to build I would say build slasharchitect

[1:28:01] architect uh the model right the neural net neural net so obviously we need to do that and that will feed into the rest of that and that will feed into the rest of the training process. Now, I um we'll

[1:28:14] concerned about these steps. These are just the steps of the training that we're going to learn. Um so, this this will really go backward and iterate through these steps. But training of a neural net is very unique. It basically

[1:28:28] involves sending a bunch of data through the network, figuring out what your error is, and then going back and updating the network. That's essentially what all these steps do. and then you iterate through that on your data over

[1:28:41] and over and over again until your network uh learns how to effectively make predictions. Um so it's a very iterative process which is what makes it take a long time. You're going to iterate through all of your data and

[1:28:56] steps which could be thousands of times that you're doing that. And it's really the same thing over and over again of again sending data through computing how far off you are in the error using that amount of error to go back and update

[1:29:10] amount of error to go back and update all your your model. Um this process is where that gradient descent is going to come into play. So we kind of touched on come into play. So we kind of touched on this in the recommendations with that

[1:29:23] matrix factorization learning. um we are going to study this in detail quite a quite a bit more here in neural nets because that's really what's going to

[1:29:35] power neural net training is um gradient descent. So again these things I don't expect us to know right now just giving you a preview of the training is a lot more involved. There's there's a significant amount of computation

[1:29:50] involved here in like sending data through, generating predictions, computing error, updating all of our weights in our network, which could be billions and billions and billions of weights. So this this back propagation

[1:30:04] step could be significantly computationally intense um in the gradient descent. And you're doing this many many many times in iteration uh many many many times in iteration uh across all your data. So it is

[1:30:19] significant the amount of resources that are spent in training a neural net. training and again we're going to have a lot more to say about these in future lessons. So let's put these terms on hold until we learn more about neural

[1:30:33] nets. I'll we'll cover what forward propagation means, what loss means, what back propagation means. We're going to study all of those. So a lot more to come on those. But for now, just think of the training phase as we have to

[1:30:48] iterate through having our network learn and adjust, learn and adjust, learn and until it [clears throat] learns effectively. Okay. Um so as I learns effectively. Okay. Um so as I said the model is goes through iterative

[1:31:02] uh performance um evaluation essentially in terms of how much error it's generating and so we keep an eye on that and continue to train it over the course of the training phase which may be lots of iterations and you can go backwards

[1:31:16] of iterations and you can go backwards at this point. So based on the training, you may go back and adjust your model. You may go back and adjust your uh parameter like your hyperparameters of the training which is going to be a few.

[1:31:29] which is what this slide's saying is that we may have to go back and collect more data. We may have to double check our labels. Um there could be a lot of

[1:31:42] adjustments. It could be we changed the architecture as well like we actually changed the model itself the neural net. Um so going back to this diagram

[1:31:55] Um so going back to this diagram there's a lot of iteration that happens. So you know based on the training we may go back and alter our data may alter our model based on the evaluation we may go back and um alter our training

[1:32:08] hyperparameters a lot that we you know may be modifying during the course of the training. So the training is going to be more intense I think in neural nets than what we've seen to this point with usually what

[1:32:22] we've seen to this point is we just run a fit and everything happens for us right especially with scikitlearn right we just run ffit and boom we have a model that's trained like a logistic regression or uh decision tree right dot

[1:32:37] fit and it's done this is not going to be as simple there's going to be a little there's going to be more setup more moving parts involved um more computationally intense for sure to do to do the training. Okay. But

[1:32:50] either way we slice it, there's still going to be an evaluation phase, meaning that you know even if we train, we still need to compute metrics to see if our model has performed well enough on the training data and on this testing data.

[1:33:06] So from a supervised learning perspective, that step will basically be we've trained the model, we're going to apply it to a hold out test set of data model's performing well. Is it overfitting? Is it underfitting? We can

[1:33:23] tell that on a test data just like we did with machine learning. That will not change. So what we're going to do is still have training data

[1:33:35] and a chunk of data that's probably smaller but still test data that we will train the neural net on this and then apply it on this to evaluate. Right? So we're still going to do that even we did that in machine learning. We're still

[1:33:49] going to do that here in deep learning. Um and the idea there is to make any um that we may need based on the performance we're seeing. This is what we just said. So still an evaluation phase that's more or less the same.

[1:34:03] Obviously just a different model this time. It's a neural net. Okay. Then if deploy it. And there's many different ways to do that. Not going to get into them in this course. But uh suffice to say that once we have done the

[1:34:17] evaluation, it's kind of ready to go and we would do a lot of um kind of packaging up and infrastructure and things to kind of host the model. Um

[1:34:29] many different ways to do that, but the goal here is to be able to actually use it to make predictions in in some some way, right? So maybe we're using it in like an object detection inside of a self-driving car or something. And of

[1:34:44] there, you're going to keep monitoring it and maybe make adjustments as you see the predictions coming in. So one of the goals that of even building the model in the first place is of course we're going to apply that to new data that has never

[1:34:58] been seen before and that's where things can go off the rails. So you really got to monitor the performance as you are predicting on new data, right? and see how it's performing. So, um you may want to retrain it, make adjustments, maybe

[1:35:14] in the extreme maybe you have to go back and uh change the model up. Um but you performance if you are using [clears throat] that model on new data. [clears throat] that model on new data. Okay, so uh just to wrap up here on this

[1:35:29] introduction lesson. So that's everything we've covered so far is just We have not talked about what a neural net is technically from like a math perspective. We haven't covered um you know how we train it. Um that's all to

[1:35:46] come. This is just background, right? So we've talked about deep learning being a subset of machine learning that particularly focuses on using neural networks. We've talked about that and we've also talked about it's really well

[1:36:01] suited for unstructured data like images, text, audio, video. Um, so it's really really good at that. However, it requires usually a lot of resources to function. So it requires a lot of data, a lot of computational power like a GPU

[1:36:16] to be effective. And then in terms of the life cycle, mostly follows the same steps as machine learning. It's just those steps are going to be different kind of model, right? and neural net. Okay. All right. So, we're going to

[1:36:29] start lesson three. So, if you're following along, you want to go to those following along, you want to go to those uh notes and go to lesson three. So, we're going to start our study of neural nets and particularly with the most

[1:36:43] basic kind starting with a perceptron really like a simple model of a neuron and the idea is to build up from there into uh bigger and bigger neural nets.

[1:36:55] So what we're hoping to learn here is basically get into more of the details of how a neural net functions. So structure functionality um including starting with the very basic perceptron and then expanding that

[1:37:10] a little bit into multi-layer perceptrons and then talk about some of the um advancements beyond there that are out there like a deep neural net that that extends that things like CNN's RNN's we'll mention as well and then

[1:37:25] talk about some of the activation functions that are uh present in neural nets how it it won't make sense this moment when I say activation functions but they'll make more sense when we get into uh what's involved in a typical

[1:37:38] neural neural net especially modeling a neuron um there's things like activation functions that are really critical we'll study some of the most popular ones um study some of the most popular ones um like relu sigmoid softmax

[1:37:51] um and then we'll talk about some of the uh some of the issues that can pop up during training in particular um things like vanishing exploding gradients like vanishing exploding gradients Okay, so a lot to cover. Um, let's start

[1:38:04] with modeling the neuron, right? So, let's start there. Um so obviously the models that we're going to build are networks of neurons and we have to start

[1:38:17] with kind of the most basic model of a neuron and it's very much inspired by neuron and it's very much inspired by the biological neuron that is uh within the brain. And uh so not that we need to know all of these things about it from a

[1:38:33] biology perspective, but just to know that the model of the neuron is obviously based on how a biological neuron kind of functions. Um so from this perspective you know there's inputs that are sent through they go through

[1:38:49] that are sent through they go through the various um nerve cells and then out comes some type of output signal that flows through to the rest of the network. That's going to be very much how the neural networks will function is

[1:39:02] there'll be kind of input data flow through this collection of neurons and then out on the other end comes this output data. So we'll see that as we go. So in a biological neuron there's these various components like the cell synapse

[1:39:17] being a connection between two cells axons transmitting the output. Dendrites receiving input. We're going to have equivalent components in the artificial

[1:39:29] equivalent components in the artificial neuron model that we are going to take a look at such as the perceptron. So in the artificial neuron we have a similar setup. Now this is really really critical to understand is that the

[1:39:46] artificial neuron is very much similar to a biological neuron in which case we to a biological neuron in which case we take all of the inputs and combine them together to generate some type of output. Now the basic model of a neuron

[1:40:00] output. Now the basic model of a neuron is going to do this really important process of taking the inputs to the process of taking the inputs to the neuron. So those um could be let's call

[1:40:12] neuron. So those um could be let's call them x1, x2 and x3 or more generically however many there are xn and then weighting those with different weights. Okay, weighting those with different weights and then basically taking a

[1:40:28] weighted sum. So we have something like w1 w1 uh x1 + w2 x2 uh x1 + w2 x2 plus blah blah blah plus w n xn. Now

[1:40:40] what does this remind us of? Hopefully this reminds us of something like a linear regression, right? It's very similar to that. that that kind of does

[1:40:52] um remind us of a linear regression. The difference is that the neuron is actually going to process this weighted sum through what's called an activation sum through what's called an activation function. So every neuron is going to be

[1:41:07] activated or deactivated based on the this flowing through some activation function. So you take that linear sum, you pass it

[1:41:19] through an activation and often this activation. So this f F is called an activation function

[1:41:32] which gets its name from the biological neuron of you know a neuron activating or not like or some people call that firing right a neuron fires um it activates. So this this f is called an activation function and essentially what

[1:41:47] the activation function is doing mathematically is applying some nonlinear function. So it's typically the activation is going to be nonlinear in nature. It's going to be some type of function that is nonlinear. Okay. and

[1:42:04] function that is nonlinear. Okay. and it's going to um it's going to be applied to that weighted sum of inputs to that neuron and then produce some sort of output which is the activation value. Um it's you know producing some

[1:42:20] value. Um it's you know producing some type of output. So in this case the neuron model this is the this is the model of the neuron is really just applying a activation to some weighted sum of inputs flowing into this neuron.

[1:42:36] Um so in this way it's kind of like the biological neuron where we had input signals flowing input signals flowing in and then some type of output signal coming out. Right? it it's basically

[1:42:50] that and then what the neuron is doing is doing an a weighted sum through an activation. That's all that the neuron is doing is doing the weighted sum

[1:43:02] is doing is doing the weighted sum summation of weights times inputs. So you're have you have your signals X you generate an output which is um the weighted sum through the activation and you get this output signal uh Y here

[1:43:18] training the model with lots of data determines [sighs] yes exactly so during determines [sighs] yes exactly so during the course of um doing the computations the neuron will will be active or not active um it it's really not a matter of

[1:43:32] binary like active or not active it's more of what is this value that comes out of this activation function. That's the output, right? What is that value that comes out of the activation? This is this is this value here. It's that

[1:43:46] weighted sum of inputs through an activation is really the model of the activation is really the model of the neuron, right? So, what is that output is really what we care about. The reason we care about that is because I want you

[1:43:59] to think about where this is headed in a in a network. So in a network of neurons, what we're going to have is a collection of these guys. So we're going to have many more neurons that would fill up a layer of neurons. So this is

[1:44:15] just one. So we'll have a lot of these guys. And each one of these neurons guys. And each one of these neurons could produce a collection of outputs, right? They could produce a collection of outputs

[1:44:29] um from their layer. And so and then what you what happens is these outputs what you what happens is these outputs become the input to the next layer. They

[1:44:41] the signals keep flowing, right? They go into the next set of neurons, right? And then those um it what actually happens is you start to build this network. So not only does this go like this, but there's many more

[1:44:57] connections. So like this will become an input into this. This will be an input into this. This will be an input into this. And this will be an input into this will be an input into this. This will be an input into this. This will be

[1:45:11] an input into this. And this would be an input into this. So everything from the previous layer which is coming out of that neuron, all those neurons will flow that neuron, all those neurons will flow forward into the next layer of neurons

[1:45:26] and on and on and on. however many layers you might have. So you can have a significant amount of these layers of neurons in a really deep network. You might have lots and lots and lots of layers. So we're going to study that.

[1:45:40] We're going to stud study that as we go along. But that's really what we're building up to. Yeah, that's that's what we're building up to in a neural network. It's really laid out in this structure of neurons sending data

[1:45:54] forward into the next layer of neurons and on and on and on until you and on and on and on until you eventually reach a final output. Now I want to give you a basic use case of this neuron. I want to give you a basic

[1:46:06] this neuron. I want to give you a basic use case of this output because it may be um it may be a little vague on what that output is. Um most of the time for that output is. Um most of the time for a neural network this output will

[1:46:19] a neural network this output will generally be a probability. So this will generally be some type of probability especially for a classification. Probability

[1:46:32] will usually be a probability. So especially for a classification. So let's say um let's say we were predicting uh if it if an image like

[1:46:44] predicting uh if it if an image like with with a brain scan is cancer or not cancer this would be like a probability of this would be like a probability of cancer that we generate like in a binary case right or spam or fraud right

[1:46:59] generally represents some type of probability it doesn't have to like if we're predicting a regression type of problem like let's say we were predicting the price for our housing price data. This this could be a price,

[1:47:12] right? This could be like a full value uh some some floatingoint number price, right? That that uh that could be possible, but it's a single output from this neuron that could represent um typically for classifications be a

[1:47:27] probability, but could be a price like in a regression problem. Okay, so again just to recap this because I think this is really really critical to understand moving forward in neural nets is what does a neuron do in the network? Every

[1:47:42] does a neuron do in the network? Every neuron is processing a weighted sum a weighted sum of inputs w2x2 wn xn. It's a weighted sum of inputs

[1:48:00] through an activation function f. That's what every neuron is doing. And that produces this output value from the neuron, which is that activated value,

[1:48:12] right? Some nonlinear function. And by the way, when we train the neural net, what we're actually learning during the training process, what gets updated during the training against real data are these weights.

[1:48:28] are these weights. These weights get learned similar to when we do linear regression, right? We learn all those betas. Same thing here. We learn all of these weights. Now, do you guys see how if we have a really big

[1:48:41] network, there's going to be a lot of weights. So imagine we had a really big network with lots of neurons and they're sending data like there could take take sending data like there could take take an image which is an image could be like

[1:48:57] an image which is an image could be like a th00and by a,000 resolution. to be our input. And then essentially what we're going to do is take those

[1:49:10] pass them through this network and generate some type of classification of like cat or dog some probability of like might be cat it might be dog might be giraffe. Um so maybe the output's actually three. So we have three final

[1:49:23] outputs like cat probability, dog probability or giraffe probability. But there can be a lot of weights because we're going to end up with this really massive network where everything is connected together, right? So there can

[1:49:38] connected together, right? So there can be a lot of neurons um a lot of weights that are that are connecting layers to each other. So these networks can grow massively with the amount of weights that have to be learned. Um now in this

[1:49:53] simple model of a neuron there's only three weights only three weights that we would have to learn. So pretty simple but in like a chat GBT there's 100 billion weights that would be pretty conservative actually. Um their

[1:50:08] opensource model their transformer network open source GPT is around 20 billion weights. Okay, so just to give you an idea, there's 20 billion weights

[1:50:22] that would have to be learned. So you can see why substantial resources would be taken to train a model like that. There's going to be 20 billion of these things that have to be learned. Okay, so let me pause here. Any questions about

[1:50:37] this model of a neuron? So, so mathematically model of a neuron, it's mathematically model of a neuron, it's just taking a weighted sum of previous layer inputs, passing that through an activation to generate an output from a

[1:50:49] neuron. That's all it's doing. Okay, that and that's that idea is is called the perceptron. That's basically the perceptron is taking a weighted sum,

[1:51:01] passing it through an activation. That's essentially the perceptron. We're going to study that next. But this is the most basic form of neural net computation you can possibly have. Just a single neuron that's taking a weighted sum of inputs

[1:51:16] and passing it through an activation. So these inputs could be features. They these inputs could be features. They could be pixels in an image, right? Many different things. Um but you're just taking the weighted sum of those passing

[1:51:30] activation. um and generating an output. One of the questions that we should have at this point is what is the activation? It it the activation is actually something that we can change. We can set what we

[1:51:46] want it to be depending on the problem. So some activations are going to be cases. In other cases, we'll use other activations. We'll learn which activations. We'll actually talk about it later in the lesson. um what this

[1:52:01] function usually looks like um in terms of taking that weighted sum passing it function might change from neuron to neuron in the network but right now

[1:52:13] we're just keeping it generic as some function some nonlinear function that this weighted sum goes through. So in that case you know it's different than a regression. A regression would just take this and be done with it right it would

[1:52:26] just take this weighted sum and that's it. That's the that's the answer. But for a neuron, there's one extra component to that which is taking that and passing it through a function, right? So, so a little bit more involved

[1:52:41] right? So, so a little bit more involved there. Okay. So, uh everything I just there. Okay. So, uh everything I just said um is kind of depicted here where said um is kind of depicted here where we are taking the um the the weighted

[1:52:53] sum. Now, there may also be a bias. I forgot to mention that. um similar to what we had in linear regression, right? There could also be a bias. So what we're really doing here is taking some function. So this this may represent the

[1:53:07] function that we're doing, some step function um and generating an output which kind of mimics the biological idea of an activation of the neuron. Right? So if the signal is big enough, if it reaches some mathematical threshold, um

[1:53:25] then the output signal is generated, meaning that neuron is activated and there's some some output that's generated. It's going to be the same here. We're going to be taking a weighted sum

[1:53:40] and then passing that through some activation F. So basically what we could do is think of this as some intermediate value Z is the weighted weighted sum and then we're taking the activation of that Z and generating uh an output value Y

[1:53:55] right which which the activation is going to be something like this uh maybe a nonlinear step function maybe an exponential um it just depends it depends on the situation we're going to use different

[1:54:07] activations. So that is the artificial neuron model um weighted sum through an activation. That's actually it for the neuron. So some terminology in terms of

[1:54:19] the difference between the artificial model weighted sum through an activation um and the actual biological model or the biological neuron. Um the nucleus is

[1:54:31] known as kind of the node which is the node is just this um the node really is just like a a graphical kind of node here that uh we could imagine as part of

[1:54:43] a network. So a network's going to have lots of nodes uh to it. Um we have our lots of nodes uh to it. Um we have our inputs which are coming from usually a previous layer in the network. Um but you know in this in the case of a simple

[1:54:56] neuron like this think of it like our features right our features like our our bedrooms bathrooms square footage from our housing data things like that um we have weights which are kind of like our synapses and

[1:55:11] the biological sense and then we have axon which is the output so there are axon which is the output so there are kind of mirrors to the biological definitions. Okay. All right. So let's expand on this idea from a single neuron

[1:55:25] expand on this idea from a single neuron into a network of neurons. So a neural network then is just the extension of this idea where now we have interconnected layers of neurons which is basically our

[1:55:42] computation model for the overall brain. Basically a whole network of biological neurons working together. Um each network consists of uh basically an

[1:55:54] input layer and an output layer and then a bunch of potential layers in between. So you can have as many hidden layers the these layers between the input and

[1:56:06] output are known as hidden layers and there might be many of them potentially many of them. It depends on how deep the network is. A really deep network depth is really based on how many hidden layers you have. So if you have very few

[1:56:20] network. If you have a lot of them, that's going to be a deep network. So this each one of these blue circles is a this each one of these blue circles is a neuron. So this is a neuron. And you

[1:56:33] have many neurons connected together, but every neuron's doing the same thing. It's taking this weighted sum of of of inputs and passing it through some activation and generating an output, right? That um goes into the next layer.

[1:56:48] Does the activation of a neuron depend need the inputs to be above a certain need the inputs to be above a certain threshold? Um, biologically yes. But in threshold? Um, biologically yes. But in the in the neural network model, no. Um,

[1:57:00] we just activation really just means taking the function and applying it to taking the function and applying it to the weighted sum.

[1:57:12] not required to be above a certain threshold um to to be activated. We we don't really care um from a biological perspective like activated or not. We care more about what is the output which is this quantity.

[1:57:28] We care about this output right uh the the weighted sum uh through the activation. It's just it's a function. Yeah. The activation is really just a nonlinear function. We care about the output. Exactly. We care about the

[1:57:41] output that's coming out of that neuron because what comes out of does it make sense that what comes out of that neuron is really important for the future layers, right? Cuz like what comes out of that neuron is going to influence in

[1:57:55] the network, right? What we produce because what we produce here flows into the next layer and on and on and on and on, right? So whatever we produce from one layer becomes the input to the next layer. in the network. So it really

[1:58:11] matters what comes out of a neuron because that's going to be an input to the next layer. Really really matters what what the output is of every neuron. Yeah. Okay. So just to go back to this definition again, the network is this

[1:58:25] whole thing. So, it's all of these neurons connected together and a a collection of neurons um that are kind of forming an input

[1:58:38] prior to another layer. Um it this is basically one layer, right? This is a layer of neurons and there and when we build our networks, we can actually control how big that layer is. So, how many neurons are in there? we can

[1:58:54] actually specify that when we architect our network. Um that's something we'll be able to do. And how many outputs do we want? So if we're the outputs are we want? So if we're the outputs are usually going to be relative to how many

[1:59:07] classes are we predicting or how many outputs do we generally want. So in like let's say we're predicting between is this image a cat or a dog. Um we we

[1:59:19] would only have these two outputs. So this would be probability of dog and this would be probability of cat let's say or if it was like uh if it was um if

[1:59:31] we had a giraffe then that would be a third member of the output layer so I giraffe right or in the case of imageet they needed like in the imageet remember they're predicting a thousand different

[1:59:48] categories of items so they need an output layer that has a thousand neurons output layer that has a thousand neurons in it. So they need a thousand uh neurons in the output because they're predicting a thousand potential

[2:00:01] predicting a thousand potential categories of images, right? So that output layer should match what your final prediction is going to be. So if you're predicting 10 categories, you need an output layer that has 10 neurons

[2:00:14] probabilities of each class. If you're predicting a regression, you only need one output like the price or the temperature or whatever it is, right? You only need that one output. Or if it's binary like fraud, not fraud,

[2:00:29] cancer, not cancer, you may only have one output, which is the probability of one output, which is the probability of uh of of the like cancer or fraud or whatever it is. Okay. So, how does a neural network process data and make a

[2:00:44] prediction? So, it's kind of what I've been saying. Inputs go through the very been saying. Inputs go through the very beginning layer and then um every neuron that's in that layer receives inputs, assigns a weight, passes that through an

[2:00:57] activation to generate an output. So, every neuron generates an output and the outputs from that first layer are then forwarded to the second layer for processing in the same exact way. So you take a weighted sum of all those inputs

[2:01:12] from that layer, generate an output, pass that along to the next layer and on and on and on. So this network, you know, could have many many many layers. So there could be another layer here that's another what we would call hidden

[2:01:27] layer, which is anything between the input and output. And what we should realize is there's going to be contributions from every neuron that flow into each one of these neurons, right? So there's a lot of connections

[2:01:41] there. Every connection is weighted, right? There's a weight. Every every connection contributes some weight to that neuron. And we process as usual, we process a weighted sum through an activation of all of these guys. It's

[2:01:57] just that their inputs are going to come from this previous layer. Hopefully that come from the outputs of those neurons from the previous layer. And on and on from the previous layer. And on and on and on. So in this way data we would say

[2:02:11] and on. So in this way data we would say is kind of flowing forward or what we is kind of flowing forward or what we would call forward propagation. We forward propagate because it's it's literally like a propagation like a

[2:02:24] literally like a propagation like a signal propagation. because data is flowing from input all the way to the output going through all

[2:02:37] these neurons. So that that's how we make a prediction is we forward propagate inputs through the network and generate an output. Right? So that's known as forward propagation is sending data through generating a prediction. U

[2:02:52] which could be a prediction could be probabilities. Right? There's 80% probability this image is a cat. 20% probability this image is a dog. So probability this image is a dog. So therefore our final prediction is um cat

[2:03:05] for instance. So this could be 08 this could be 02 and this would be 08 probability it's a cat and 2 probability it's a dog right something like that.

[2:03:17] And we we're generating that by taking our image and flowing it through forward propagating it through this network generating these probabilities. That's just an example. Okay. Okay. So where we're going to start uh with in terms of

[2:03:33] these networks, the very first one that we've actually already been studying is the perceptron. So the perceptron is what is exactly what we've been neuron of a single neuron. um we will graduate

[2:03:49] from that into multi-layers of neurons. So um multi-layer perceptron um so more than one layer, more than one neuron for sure, but potentially more than one layer. And then we'll graduate from that

[2:04:03] into deeper networks that have many layers to them. Uh and then and then position, especially once we learn how to train these guys. um we'll spend a

[2:04:16] lot of effort on training these and optim optimizing that training process and building it within our code like PyTorch or TensorFlow. We will then PyTorch or TensorFlow. We will then start to apply variations on these

[2:04:30] architectures that involve convolutions. So the neurons will go from simple weighted sum through an activation to a completely different type of computation that the neurons are doing which would be like a convolution. So in in the um

[2:04:48] just to go back a step to put this in perspective in this picture we're doing what we would call perceptron style calculations right now meaning we're through an activation and sending that forward through the network. When we get

[2:05:03] forward through the network. When we get into computer vision and working with into computer vision and working with text and RNN's these neurons will now be way more advanced. the neurons will do things like convolve or do convolutions

[2:05:18] which will be um a very sophisticated computational operation that will require a GPU or else will take forever to train any models. So we will graduate from these very basic neurons. So right now we're just talking about a basic

[2:05:33] neuron that's doing this weighted sum through an activation and we'll graduate into convolutions and then we'll graduate into RNN. So doing recurrent

[2:05:45] calculations uh which mimic kind of sequence memory um for for learning how to work with text sequences like sentences. So, a lot to cover obviously

[2:05:57] before we get there, but right now we're starting with the most basic model of a neuron, which is this perceptron kind of style calculation uh just sending a weighted sum through an activation. That's it. But eventually

[2:06:11] these neurons will become more and more complex like doing convolutions for complex like doing convolutions for image data or doing RNN's for uh text image data or doing RNN's for uh text data and eventually transformers. Um so

[2:06:25] so these neurons will do even more advanced things like attention in a advanced things like attention in a transformer.

[2:06:37] about the uh you know the perceptron um being the simplest type of network. That's where we're going to start. It's mainly used for binary prediction. So because like we can just produce that one output which could represent a

[2:06:53] probability. So the perceptron was the simple model of the neuron and it was originally used for binary classification problems. So that's where we'll start is the same thing kind of for binary. Um and the perceptron's

[2:07:09] really only good if your data is linearly separable. We'll talk about what that means and why that's really only the case. Otherwise, we need to go only the case. Otherwise, we need to go to more advanced like more layers. So,

[2:07:22] it the perceptron is going to be kind of limited in the sense that it's a single neuron. It's not a network. So, it's not going to model too many problems that going to model too many problems that well. Um, but it is the foundation for

[2:07:35] building on that and generating many many layers of neurons that that we will see for larger networks. So, and that's where we go with the MLP, which is multi-layer perceptron. Um, basically multiple layers of these uh perceptrons

[2:07:52] multiple layers of these uh perceptrons of these neuron models and uh it can handle a little bit more complexity by learning nonlinear things. Um, because it has multiple layers passing those inputs through these nonlinear

[2:08:05] activations, it can learn more effective. However, the MLP is not necessarily a deep network. So, it's not going to have that many layers of going to have that many layers of perceptrons, maybe only a handful. It's

[2:08:20] not going to be that deep of a network. Um, so itself will be kind of limited and it especially won't work well for like images or text, but for maybe certain problems like our housing data might work okay. But we'll certainly

[2:08:34] might work okay. But we'll certainly graduate from perceptron to multi-layer perceptron as a little bit bigger of a network. Okay. And then we'll work our way up to deep neural networks which are going to have many

[2:08:47] hidden layers. So this we we've had this image before. Here it is again where you know a deep network's going to have um this is the depth here. How many hidden

[2:08:59] layers do you have? A deep network is going to have many many many layers. going to have many many many layers. Okay, so again like modern generative AI neural networks are going to have thousands of layers, thousands of these

[2:09:15] hidden layers here. And and notice how they're all connected because you know they're all connected because you know the inputs the sorry the outputs of one layer become the inputs to the next layer. So you can see connections from

[2:09:27] each of these um contributing to that neuron. This neuron is taking a weighted sum of those and then generating an output and then there's connections from that output to every other uh neuron and on

[2:09:43] and on and on. Right? So this big network you get um a certain number of network you get um a certain number of outputs a certain number of outputs that So again, there's like this could be a thousand for the image net problem. Um

[2:09:58] let's say we're doing cat, dog, or giraffe. This would only be three. We're sending an image through forward propagating it, generating these probabilities, which is our prediction. That's kind of how it works. Is it

[2:10:10] always left to right or can it bounce back? So it is left to right for the networks that we're talking about. The only um time you get the circle like is a recurrent neural net and RNN that is the so the circle you're describing is

[2:10:25] the essence of a recurrent connection. So it has the ability to do to do this So it has the ability to do to do this which is kind of like mimicking a memory of a sequence sort of like going back and and remembering what our previous uh

[2:10:40] inputs were. Um otherwise it's going to flow forward. It's going to be forward propagation left to right. But in an RNN, you would get connections like this like you're thinking about. I think that's what you were talking about those

[2:10:54] recurrent connections. We would get those in an RNN. Yeah. So, not Yeah, we won't have those yet. Not until we get to RNN's later in the in the lessons. So, and then the the the So, we'll we're going to study this.

[2:11:09] We're going to work our way up to this. And then as I said once we learn how to build these guys these deep neural networks and really work with them in PyTorch and TensorFlow we'll graduate to CNN's the convolutional neural nets

[2:11:24] CNN's the convolutional neural nets which was um this was um motivated by mimicking kind of like the visual reception of the of the eyes and the reception of the of the eyes and the cortex. Um using the convolutions to uh

[2:11:38] to accomplish that. Convolution being a um a type of computation will basically replace those neurons with more advanced computation. computation. And the um visual reception is kind of

[2:11:51] mimicked by the convolution allowing us inside of images to pick out things like objects, things like shapes, edges, lines, lines, um human uh features like faces,

[2:12:05] allowing us to learn those features of an image. So the convolutional net again this was that Alex net 2012 really revolutionized being able to learn uh

[2:12:18] how to predict on image data. So we will learn about CNN's learn about CNN's after we cover uh deep neural nets. Uh it stands for I'm not sure what it means in this case actually. Not 100% sure.

[2:12:33] in this case actually. Not 100% sure. Yeah not not 100% sure. I'd have to uh look that up. I think they're just different um sections of the brain. I think they they're short for like different uh sections that process like

[2:12:47] uh visuals. Okay, there we go. Yeah, they were different regions. Inferior temporal cortex. Nice. Okay, that makes sense to me. Okay. Okay, so then talking sense to me. Okay. Okay, so then talking about RNN's really quick. Um, so RNN's

[2:13:03] are a lot, you know, networks that we're going to study later on in one of our lessons after we cover deep networks that will handle sequential data, right? So this is going to be things like language because as I said, that's a

[2:13:17] sequence of words after the other in a in a sequence. Um so we will handle text in a sequence. Um so we will handle text with RNN's also um a a very popular use case for RNN's especially these days is like forecasting so time series data

[2:13:33] like uh sales or revenue day by day by day you can you can build forecast with neural uh RNN's um because they're really effective at sequence learning really effective at sequence learning mainly because they um allow for these

[2:13:48] recurrent connections uh so they kind of have this short-term memory which is effective for some language tasks and for forecasting. Unfortunately, that short-term memory is is too short. It's not able to learn the type of context

[2:14:04] that we see in like a a chat GBT for instance with a very long prompt or a lot of text which is what we see in transformers. So RNN's is kind of limited relative to transformers on language tasks mainly because that

[2:14:18] short-term memory is just not nearly what it needs to be to generate text or do effective translations things like that. So we're going to learn about that when we cover RNN's. Um one of the things that I mentioned that's certainly

[2:14:31] things that I mentioned that's certainly true is when we uh work with neural nets we will be able to customize basically everything about that network. how many layers there are, how many neurons are within each layer, what those activation

[2:14:47] functions are. So you can think of like you know this is going to be completely customizable from the same with input and output. How many

[2:14:59] uh inputs are are there? How many outputs should there be? Like if I'm predicting three or 10 classes or maybe two or maybe just one in a binary case. Um, basically everything about this is going to be customizable when we build

[2:15:14] our models. And and that's just what we would call the architecture of the network, right? How many layers there are, how many neurons there are within each layer. And you can see that like when you have a deeper network with a

[2:15:31] lot of different uh layers here. So like this example on the right, you can see this is going to have a lot more um this has a lot more weights to it that have to be learned. So it's going to take more data. It's going to take longer to

[2:15:48] train a model like this because there's so many more connections, right? there's so many more weights um that have to be learned during this process um than than learned during this process um than than if you had fewer layers. So um that's a

[2:16:03] trade-off, right? Maybe you need more and more layers to learn a really complex problem that may be necessary, but you're going to pay the price when just going to be more weights that you have to keep track of, right? And and

[2:16:20] learn. Um, so this is something we'll see when we get into TensorFlow and PyTorch. We're going to see how do we build this? What's the code that generates this kind of network? That's something we need to learn, right?

[2:16:34] something we're going to learn. It's it's actually going to be relatively easy inside of TensorFlow and PyTorch. They make it easy because those libraries are so good at building it. So not only that like we'll control the

[2:16:49] architecture but we'll also control the training and the evaluation. So all of that we'll be able to set up um inside of our our you know PyTorch or of our our you know PyTorch or TensorFlow when we get to that. Okay. So

[2:17:02] one of the things that's true about neural nets is they typically require neural nets is they typically require more data and um there's uh kind of a graphic here to show that like if you have even a small neural network that's

[2:17:16] very shallow. So small being like number of hidden layers you know it can still outperform traditional machine learning. So think of like logistic regression, decision tree, random forest, xg boost. It can still outperform all of those.

[2:17:32] It's just going to require more data. And the larger the model, the more data kind of see that on this graph, right? The large neural network has a lot of potential to perform really, really well. Um, but it needs the most data.

[2:17:49] it's the larger that the network is, the more data it needs to to function properly. Now, these are relatively close. So, just a shallow network isn't and especially in the small data um down here, it it's not until this point that

[2:18:06] um the the even the shallow neural network kind of surpasses traditional machine learning. So again for small data um it may be desirable to just continue to use traditional techniques like what we've studied but as we get

[2:18:22] larger and larger data size um if we're working especially with unstructured like images text we're going to need neural nets and um you know usually we'll start small and work our way up to larger neural networks if we need them

[2:18:39] you know in terms of the architecture which is how How many layers do we have? How many neurons do we have? Those kind of things. Okay. So the the depth of the of things. Okay. So the the depth of the network like how many layers we have is

[2:18:53] network like how many layers we have is often um relative to the task. So if we're doing, you know, working with images like in an image classification or like object detection or like in a in a transformer for a generative AI use

[2:19:06] a transformer for a generative AI use case, that's going to be a much bigger network than maybe a simpler modeling task like predicting the price on our housing data where it's very structured, has a certain number of features to it,

[2:19:19] we may not need as deep of a network. So this is again where we will uh we will you know practice building networks and seeing when that makes a difference like

[2:19:33] how much we change that architecture uh how big of a difference does that make on our modeling problems. We're going to see that as we get into it. But here's a good you know this is a good picture here on the uh this one has two layers

[2:19:48] and it produces two outputs. Um so it takes four input features produces um two outputs which could be probabilities let's say and you can see like these these symbols are exactly what I've been describing as you take a weighted sum

[2:20:04] and you pass it through an activation. That's what the sigma and the f is. So the sigma is like the weighted sum of the inputs to that neuron. Every one of these circles is a neuron, right? So this is a neuron. This is a neuron. This

[2:20:18] is a neuron. neuron neuron neuron neuron and um technically all these are just output neurons and these are input neurons. So they're not really

[2:20:33] doing much. They're just processing the input or holding the output. But but they're going to be a weighted sum through an activation for all these neurons, right? Weighted sum through an activation. That's exactly what what

[2:20:48] this is meant to represent. Sigma weighted sum through the f through the activation. Um so let's talk about the perceptron uh as a a model that is based

[2:21:01] on that neuron that we've studied so far. Um basically one and the same. So what is what do we mean by perceptron exactly? So I've been using the word perceptron a little bit. Let's actually define what it is. Essentially it is a

[2:21:16] very basic neural network that is designed for binary classification. So think spam not spam right fraud not fraud. So it has only two choices and

[2:21:28] essentially what it is is the neuron such that it has a very simple activation. This is the activation right here which is that um you get a a

[2:21:40] classification of one if your weighted sum times your your inputs plus your sum times your your inputs plus your bias is positive. So this has an activation function like this which is basically just checking is your weighted

[2:21:55] sum uh is your weighted sum times your inputs or weighted sum with your inputs um greater than zero. So if it's greater than zero, we're going to output for than zero, we're going to output for that neuron a one. If it's negative or

[2:22:09] equal to zero, we're going to output zero. So it's in that way, it's just taking that model of the neuron. So, you're taking all of your inputs, your X, X1, X2, X3, X, XN, however many you have,

[2:22:27] and you're feeding all of them into the single neuron, single neuron, which is taking a weighted sum and uh taking a weighted sum of those inputs and then passing it through an

[2:22:39] activation, we're just going to produce a a zero or a one, right? um a y which a a zero or a one, right? um a y which is zero or one depending on this

[2:22:51] activation which is um was it positive or was it zero or negative right is essentially what this activation is. So it's a very simple model used for binary

[2:23:03] classification that's basically just a neuron um just a single neuron taking that weighted sum uh passing it through the activation. This is the activation

[2:23:16] activation function here which is basically just checking is it positive you get a one output uh is it negative or zero you get a zero output right for the for the neuron that's all it is so perceptron a very simple model um that

[2:23:32] is based on this neuron calculation the thing that's unique about the perceptron is it's just using this particular activation um to see you know, do we get a one or a zero for a binary classification? So,

[2:23:46] what's interesting about the perceptron is um you know, it's uh using the basic idea of a neuron in every sense. It's taking a weighted sum through an activation. And what gets learned in this perceptron is the same thing that

[2:24:01] we've talked about learning in any neural network, which are these weights, right? These weights are going to be what gets updated through the training of the perceptron. we need to talk about how does that happen?

[2:24:14] What kind of adjustments do you make? Um but essentially we're the algorithm is going to learn those weights of that neuron um so that it can produce good uh

[2:24:28] outputs for your data, right? You want highquality predictions that line up to your your like spam, not spam, fraud, not fraud kind of uh uh predictions. So it's exactly like this picture. Uh it's just the output again is going to be a

[2:24:44] binary zero or a one and that activation is going to be particularly checking if it's positive or if it's negative or zero. So it's exactly this picture we just talked about from earlier with a neuron. Um, for that reason, you know,

[2:25:00] the weighted sum through an activation is often just referred to as a perceptron style computation because that's essentially what it is. The perceptron is just using that particular activation of seeing if that weighted

[2:25:15] sum is positive. So, it's that very particular activation. Again, I'll draw it here. you know we we do this uh weighted sum and we're passing it through the the activation and the activation is such

[2:25:31] activation and the activation is such that um if this weighted sum that um if this weighted sum is positive then you get um then you get is positive then you get um then you get a a one um if it's uh else basically you

[2:25:46] a a one um if it's uh else basically you get a zero as the output right is that's the activation function there. So the things that need to be updated and learned to have an effective perceptron is obviously these weights. These play a

[2:25:59] really critical role uh in determining what the output is going to be in any prediction. So again what does this look like? Um it's very much similar to

[2:26:11] linear regression. We talked about this earlier and for that reason this model only works really effectively when the data is basically what we would call

[2:26:23] linearly separable. So if our classes like let's say we just had two features X1 and X2 um if we had let's say these red points belong to one class and then these blue points um or let me

[2:26:39] draw them in green belong to a different class over here. If you know if we could draw a straight line through here and say everything on this side belongs to yes and everything on this side belongs to no those binary classes then this

[2:26:56] model would be effective because it's essentially going to find that line and essentially going to find that line and compare it to zero essentially as if we shifted this line to be kind of at the origin and we're checking like does this

[2:27:09] result end up positive meaning it's on this side of the line or does it end up negative, meaning it's kind of on this side of the line, which would be a zero. side of the line, which would be a zero. Um, so the perceptron's really only good

[2:27:22] to be used because it's so simple, just a single neuron. It's really only good in these situations where the data is kind of what we would call linearly separable, which is not always the case, right? That's typically not the case. Um

[2:27:37] which is why this model we need to expand this to have many neurons with expand this to have many neurons with nonlinear activations to model like real world problems. Right? So again the perceptron uh we get a zero or one.

[2:27:52] We're going to adjust the weights during the training process. um they get updated kind of in a gradient descent style uh calculation that happens to adjust those according to the error and the perceptron really only works well uh

[2:28:09] when the when the data is linear separable as we just talked about. Okay, talked about weighted sum through an activation. It's just this prediction is now we know what it is for a perceptron. We know that this is going to be a zero

[2:28:23] or a one. And we know what this activation is going to be. It's it's the you know checking if that weighted sum is positive or negative or zero. Right? So we know what these components are now

[2:28:36] for the perceptron. So everything we've talked about is is here. Nothing really changes except a particular activation for this binary case and it's just a single neuron. So that's the other thing. We've been talking about a

[2:28:51] network of neurons. The perceptron is so basic. just a single neuron. All right, just a single one. It doesn't have a layer of multiple neurons. Um it's just a single one. So very basic. So the components there are again what we've

[2:29:07] been talking about. They we have a set of inputs. We have a set of weights. We of inputs. We have a set of weights. We have a bias using for shifting um uh which is important for checking relative to zero. Um we have a summation function

[2:29:21] to zero. Um we have a summation function that uh just takes the weighted uh sum and then we have an activation which is exactly what's producing our zero or one right the activation everything is there that we typically will have in any

[2:29:35] neuron and certainly in a network of neurons we're going to have lots of neurons we're going to have lots of these guys. Okay, so the perceptron is a good model of a single neuron um in the sense that it takes a weighted sum,

[2:29:51] takes an activation and produces an output. Now what would happen with this output. Now what would happen with this that output in a in a larger network is we know that data would flow forward to the next layer, right? We know that um

[2:30:04] in a larger network, but in the perceptron, we really only just have this, right? we just have some weighted inputs going into that single neuron. inputs going into that single neuron. Um, but typically that data would flow

[2:30:17] forward in what we've said is forward propagation. Another word for this for uh type of network where data only flows forward in a forward propagation manner

[2:30:29] is um sometimes known as a feed forward feed forward network. So, so feed feed forward network. So, so feed forward is basically just a network of perceptron neurons, right? That that perceptron being um their computation is

[2:30:44] just a weighted sum through an activation and it's just flowing forward. The output goes into the next layer. That output goes into the next layer and on and on. So this is sometimes known as just a feed forward

[2:30:56] sometimes known as just a feed forward network because we do later on when we have RNN's we'll have the ability to have a recurrent connection that goes have a recurrent connection that goes backwards but right now you know these

[2:31:08] especially with perceptron style calculations in these neurons um everything here is just what we call feed forward um everything just forward feed forward um everything just forward propagates nothing nothing has a kind of

[2:31:22] feedback back connection yet until we get to RNN's. So, this is another term you may hear is a is what we call a feed forward network. It's really just um whenever you hear the word feed forward

[2:31:35] network, you should think of perceptron style computations that just flow forward weighted sum through an activation going forward. That's all passing their output as the input to the next layer. So, some of the features of

[2:31:49] this is that information only goes one way. um it goes straight it goes forward propagated through never touches a node twice. It has no memory of the input um which is what we would have in an RNN. We would have memory because we care

[2:32:04] about sequences. We care about data that's prior to our current um value that we're processing in the sequence. Um so it h so the the feed forward network which is the most basic type that perceptron style computation data

[2:32:20] that perceptron style computation data just flowing forward basically has no ordering no sequential memory nothing like that because it cannot go backwards just every all the data is um flowing forward. Okay so what we're going to do

[2:32:35] now is just look at the extension of the perceptron. So given that we have a perceptron remember the perceptron is just a single neuron that has inputs. So we have inputs flowing into a single neuron which is generating a single

[2:32:49] output. That's perceptron is just this. It's not even a network really. It's just inputs uh weighted some of those going forward into an output. However,

[2:33:01] going forward into an output. However, if we stack together multiple layers of if we stack together multiple layers of these guys in terms of these neurons, we can generate what's called a multi-layer perceptron, which is starting to be a

[2:33:15] very basic network, right? It's not a full deep network in the sense there's not many many layers. There's not many many neurons. It's very limited. So multi-layer perceptron is designed to be pretty limited in terms of its

[2:33:32] structure. It's not a huge network. It's mainly just a small collection of maybe two, three, maybe four at most perceptron layers. Um and and we

[2:33:46] generate an output which could be one or two or three outputs. Depends on what we're trying to predict. But it's think of just stacking these neurons together in a simple collection of layers and we're starting to build a network that

[2:34:00] is the multi-layer perceptron. Um now every neuron in the MLP sometimes this is known as MLP multilayer perceptron. Um every layer in this multi-layer perceptron is just the simple weighted

[2:34:16] sum through an activation. So this is still every single neuron still functions the same weighted sum through the activation, right? It's it truly is that weighted sum through an activation. So nothing really changes there. Every

[2:34:31] neuron is like that. Okay? And all the data still feeds forward. It's just now we have multiple neurons and multiple layers, maybe two or three or four of them. Okay? So a perceptron um is decent except it fails at the

[2:34:49] separable uh things that are not linearly separable. So a very basic example people like to give of where the neuron model of a perceptron really neuron model of a perceptron really falls apart is like a simple um what's

[2:35:02] called an xor which is uh the like the exclusive or operation between uh um like two binary values like minus1 and one minus1 and minus1. Um if you take

[2:35:15] the exclusive or between the two you can see what their values are. Like the exclusive or between minus1 and minus1 is still minus one. Um the exclusive or is still minus one. Um the exclusive or between minus1 and one is one. And then

[2:35:28] um what's interesting is you know this is not uh separable. You would have to in order to separate these two classes, you would actually need two lines or basically something nonlinear. Uh because you would need um you would

[2:35:45] basically need to draw something uh kind of like this um in order to separate the two uh and say like everything on this side is uh belongs to one class and everything on

[2:36:00] this side belongs to another class. And that's highly nonlinear. So, and this is just one example of a function that would break the the you can think of many more like you could have like let's say um going back to the example we

[2:36:15] could draw circles here and here and then draw um the red ones um like this one could be on this side and this one could be on this side. And again, this

[2:36:27] makes it really difficult to draw a single line. like you can't draw a "Well, I can perfectly separate everything on this side and everything on this side." That's not going to be possible with with a function like this

[2:36:40] possible with with a function like this with data that's like this. Um, so that's just to say the perceptron is not perfect. It's going to struggle on data perfect. It's going to struggle on data that is not linearly separable, which is

[2:36:53] what this means. meaning you can draw a perfect line or a hyper plane and all the other classes are on the other side of it for a binary classification. Um of it for a binary classification. Um you you most likely need a multi-layer

[2:37:08] perceptron because you need nonlinearity, right? You need this this kind of picture here in order to better separate it which is which is nonlinear. Okay, so

[2:37:20] that's just to say a single neuron by itself is kind of limited. That's where you need layers of neurons to more effectively model problems. That's all we're saying here. Okay, so I have a demo for you guys and this is just going

[2:37:33] to use um so it turns out that perceptron a single neuron is actually available within scikitlearn. So we don't even need to use tensorflow yet or pietorrch yet in order to just work with

[2:37:47] perceptron. Um, so there's actually, just to show you that quickly, there is just to show you that quickly, there is a a demo. So, this 3.04 demo inside of a a demo. So, this 3.04 demo inside of um, lesson three here,

[2:38:00] um, lesson three here, which let me go over to that and show you. So, we'll do this demo. There should be a couple in lesson 3. We're going to do this 3.04. I'll give you a moment to

[2:38:13] pull it up. It should be in your uh lesson three materials. And there's also a I believe a data set that you're going to need which is the spam the spam based

[2:38:26] data set which comes in the lesson 3 data sets. I can upload it here. But do you guys have the do you have this notebook from the lesson 3? You might need to download it from your reference materials. Okay, great. Okay, so let's

[2:38:40] um let's practice building just the perceptron. Now again, would we typically use a perceptron? No, we would typically use a a uh neural net that has

[2:38:52] more than just a perceptron, which is a single neuron. So, but the perceptron is single neuron. So, but the perceptron is an important building block to larger and deeper networks because it that fundamental idea of taking a weighted

[2:39:07] sum through an activation is really critical and it's it shows up in a lot of networks. So it it is an important building block, but by itself is not necessarily an effective model, especially in those cases we just saw

[2:39:20] where it's not linearly separable. But let's see how we use the perceptron let's see how we use the perceptron here. Um what you can see is look at where it comes from. It's from scikitlearn in the linear model family.

[2:39:33] Now we've seen other things from there too like the uh logistic regression, right? We've also seen linear regression there. But for classification, we've seen logistic regression from that linear model. Here is perceptron. It's

[2:39:48] considered to be in this linear model family because we're taking that weighted sum which is linear but then passing it through an activation which is uh checking if it's positive or or negative. Right? So let's load let's run

[2:40:03] negative. Right? So let's load let's run that to do our imports. Okay, let's load our data. So, we're going to load that spam based data. I do need to load that spam based data. I do need to put it in here.

[2:40:21] So, that's loaded and we can check the uh head of it to see the first five rows, of course. So, you can see there's some uh some data. Now, this the spam is

[2:40:33] is whether or not the email was spam or not. So it's got a it's all a bunch of not. So it's got a it's all a bunch of data about the uh about the email and then it has a classification category of spam which is the last column which is a

[2:40:49] one or a zero for not spam right so that's that this is a classification problem uh and that and this data is just a bunch of features about different words so for instance this is a word frequency of the word all the word

[2:41:03] address the word make the word power, over, remove. So, it's got a bunch of word frequency features, and then it's got some capital capitalization features, um, characters like the frequency of the

[2:41:18] pound symbol, frequency of the dollar symbol, frequency of exclamation point, symbol, frequency of exclamation point, um, parentheses. So, it's got a bunch of um, parentheses. So, it's got a bunch of features about text, uh, within the, uh,

[2:41:31] email. Okay. All right. So, what we're going to do is um just have a simple check here. If there are any nles, we're just going to fill those with zeros. So, um obviously,

[2:41:47] we could do a more robust check here. Um in fact, let's let's check uh you know, in fact, let's let's check uh you know, we know how to do dataisnull and then dot sum. So, we can see that across all the columns uh if we have any

[2:42:05] across all the columns uh if we have any Um, doesn't look like we have any. have any nles. So, this should be relatively uh this shouldn't do anything essentially cuz this should fail. There

[2:42:18] are none. So, this should be false. It shouldn't have to fill in anything. So, that that's fine. That's just going to produce our data.

[2:42:31] the important part. This is the important part of grabbing our features and our labels, right? So this is grabbing our features and our labels, right? Features and

[2:42:45] labels. So this is grabbing every row and every column but the last one. That's what I look does, right? Do you guys remember that for our data frame? This is grabbing every single row but then every column but the last. So

[2:43:01] everything up until the last which is the minus one index that is all of our features and this is grabbing the very last column all all rows that's what the last column all all rows that's what the colon means all rows and then only the

[2:43:16] last column which is that's the label right this is the lab the spam label uh column is the last and we saw that in the in in

[2:43:29] is the last and we saw that in the in in this uh head of the data frame, right? Spam is the last one. Every other column before it is is a feature. So, we are just separating those out. Pretty typical.

[2:43:48] doing what we usually do with that is train test split, right? Pretty standard where we pass in our features and our labels and we do a train test split. Um, and we can choose whatever test size. I think 04 is a little big. We could lower

[2:44:02] think 04 is a little big. We could lower that to maybe 3 or 0.25 or 0.2. Um, but I'll just leave it for that for now. Um, but this is our usual train test split. So, nothing that interesting here. Just generating our our training

[2:44:16] data and our testing data. It's pretty standard, right? Nothing that uh out of the ordinary there for a supervised problem which this certainly is even though we're using perceptron which is a neuron model to do it still supervised.

[2:44:33] Let's scale our data. So we're going to scale all of our features. Um that's uh still something we're going to do here. Um using this perceptron uh scaling all

[2:44:46] of our features here. So scaling our training features, scaling our test training features, scaling our test features. both of those and then we can fit our perceptron. So

[2:45:02] look how easy it is to initialize a perceptron. Um and and uh basically the perceptron. Um and and uh basically the perceptron is um just a single neuron

[2:45:14] model, right? So that's all it is. And it's going to use um uh it's going to it's going to use um uh it's going to train. So this uh this ffit I should train. So this uh this ffit I should call this out. This ffit is going to

[2:45:29] call this out. This ffit is going to train the neuron which is just the perceptron just a single neuron. It's going to train this by learning the weights through a gradient descent

[2:45:44] uh algorithm. So it's going to run a gradient descent in the background to learn what those optimal weights are. Now we haven't studied gradient descent in detail. We will as part of like when we get to the

[2:45:58] will as part of like when we get to the part on uh coming up shortly on learning how to train neural nets. We're going to learn a lot about this because every single neural net trains through gradient descent algorithm one way or

[2:46:12] the other. It does gradient descent. So it's not going to be any different for the perceptron because it is just a neuron. It's a very basic. It's not a network per se, but it is a neuron. The network is just a huge collection of

[2:46:25] neurons. So um they're all going to train through gradient descent. So we'll we'll learn more about that as we go. But this ffit is going to run gradient But this ffit is going to run gradient descent on our um our our features and

[2:46:39] our labels here for the training data and it's going to the perceptron is going to adjust all of its weights um according to this data. Okay, so we get a perceptron that has been fit um and then we can

[2:46:55] make predictions. So we can pass in our training data to get a collection of then more importantly, we want to evaluate that on our test set. So we

[2:47:07] pass in our test features to our model and dopredict. So so far this is just behaving like any other scikitlearn model, right? Nothing that interesting, but it is cool that the perceptron exists inside of

[2:47:21] scikitlearn as kind of like any other model. Okay. So, we can see the accuracy here on our data and we get about an 89% 88% on the test. So, pretty decent. That's

[2:47:37] on the test. So, pretty decent. That's not too bad. Um, using that perceptron. So, pretty cool. Um, that works out. And that's just a that's just a simple neuron. It's not, you know, this is a very basic model. Um, it's just that

[2:47:50] single neuron. Of course, where we're going with all this is to build up our own neural network that's going to be a huge collection of these neurons, right, that are going to be um organized into layers that are connected into other

[2:48:05] layers and so on and so on and so on. So, we're going to build up to that and one of the things we're going to have to learn is how to construct that in like PyTorch or TensorFlow. Um, so that's going to be more involved. And we're

[2:48:17] also going to learn how to set up this training algorithm, this gradient descent. We need to learn more about that. So that'll be upcoming as well. Okay. And we covered this, but I just wanted to start here with kind of a

[2:48:29] review of what we had talked about in terms of uh neural networks, right? So just to refresh ourselves, we're dealing with uh neural networks in general and

[2:48:42] um we know that those are comprised of kind of a network of neurons, right? And and um these are organized into layers where every layer has a different number of neurons potentially that we could we could specify how many there's going to

[2:48:55] be. But the the idea is that the data that we pass in uh will kind of flow forward through this network and every neuron is really doing something like neuron is really doing something like this. Remember this is and this is the

[2:49:09] equation that we kind of had for a perceptron perceptron which is that uh we take a weighted sum which is that uh we take a weighted sum times the input. Um, and so this W star

[2:49:22] X is really meant to be um kind of a dotproduct like a weighted sum uh times inputs, right? So we have weights times inputs and then we um might add a bias

[2:49:35] in there as well, which is what the B is here. So adding in a bias, which is just another weight really. And but the important thing is passing that weighted sum of inputs through an activation which is this F, right? So we pass that

[2:49:50] through some type of activation which generates the output for that neuron and what happens in the network is these outputs just flow into the next layer. So whatever is output from from this neuron becomes the input into this

[2:50:06] next neuron. It also becomes the input into this neuron and and this one really all the neurons that are in the next layer. And of course those have different weights associated to them. So there'll be a weight here, a weight

[2:50:19] here, a weight here that that dictates how much of that input from that neuron how much of that input from that neuron gets contributed to to this neuron's uh output. And same with this guy, like this guy's going to contribute a certain

[2:50:32] amount to this. This guy's going to contribute a certain amount to this. And there's going to be some type of weighted sum through an activation. And that goes from every layer to every layer. Right? So um that's what we've

[2:50:45] kind of learned so far. The things that we need to study a little bit and we will coming up next are what are these activations um in particular and when do we use certain activations. So in certain situations um you know a certain

[2:51:01] activation will make more sense because they can if you look at the activation from a math perspective they really control what the output of this is allowed to be from a range perspective right because you're taking us an input

[2:51:14] passing it through this function and generating an output. So depending on what that activation function is kind of controls what you can possibly get out of there. If you're predicting things like in a in a probability sense, you

[2:51:29] that's going to limit your range to a probability like between 0ero to one. That makes sense. Um if you're doing a regression, you don't want to do that. You usually want it to be just any real number value, right? To to mimic like a

[2:51:43] price or temperature or something like that you're doing a regression with. So we're going to talk about some very common activation functions that are out there. But this is what we're talking about with neurons, right? Weighted sum

[2:51:57] through an activation and we're doing this in this network of neurons uh that are kind of spread out amongst these layers passing data forward throughout and you finally get to some output. And even the output, remember the output

[2:52:12] when we get into larger and larger complex use cases, the output is going to be different depending on what kind of problem we're solving. So if we're um just predicting like a binary output, that's going to be just kind of a single

[2:52:25] output that's a probability right between 0 to one. Kind of like what we saw for logistic regression for like a you know spam not spam fraud not fraud kind of use case. But if we're predicting maybe we we're predicting

[2:52:39] between three different kind of image categories like this is an image of a cat, this is a dog, this is a giraffe. Um we would we would likely have three outputs, right? We'd have probability of dog, probability of cat, and probability

[2:52:52] of giraffe. Or in you know, if we had 10, then we'd have 10 outputs here. Um how many outputs we're trying to produce. Um that makes sense. And so

[2:53:04] we're going to see that with different use cases want to set up those output layers to contain the right number of outputs and also you know actually be probabilities which is going to be hugely dictated by this uh activation

[2:53:20] right we want the activation to generate probabilities for those neurons. Okay, so that's a little bit of a recap. And if you guys remember um perceptron if you guys remember um perceptron is a a special type of uh neuron. It

[2:53:34] that is just a single neuron that has an activation function that produces a zero or one, right? It's it's like a step function that produces a zero or one. Um depending on if this weighted sum is

[2:53:49] positive, this is going to produce a one. If it's negative, it'll produce uh a zero. So that's what we meant by the perceptron model. It's just a single neuron that does this activation. Hopefully that rings a bell from last

[2:54:03] interested in doing with bigger and bigger neural networks is basically having many perceptrons, which are basically all of these models of neurons here. So you have lots of perceptron style uh calculations and that's where

[2:54:18] style uh calculations and that's where we we graduate from just a single neuron to multiple layers of perceptron which is which is this MLP model multi-layer perceptron model here. So that's that's kind of where we're headed and then this

[2:54:31] is going to keep expanding. So we're going to have even more layers, even more neurons, and then eventually even more complex computations there that are not just weighted sum through an activation, but things like convolution,

[2:54:44] going to work our way up there in the future, but that's kind of where that's where we're headed with all this. So just wanted to recap that a little bit um and pick it up from there, especially the the activations. I think that'll uh

[2:54:59] we'll talk about that next. So we we ended on this demo last time which was um kind of going into scikitlearn if you remember we did this demo went into scikitlearn and scikitlearn actually has the basic perceptron um model um but of

[2:55:15] neural network it's going to be in tensorflow or u pietorch but um scikitlearn itself does have um a perceptron um because it's it's a very basic model of a neuron um but it's not really a

[2:55:30] network. It's just kind of a single neuron. Okay. So, before we go into activations, I really want to pick it up here um and talk more about uh what

[2:55:42] and what are some good examples of activations. Let's talk about activations then. So uh we know what the definition is uh in in terms of it is a function that we're going to apply to that uh inputs that are generated for

[2:55:58] the neuron where we have that weighted sum uh with weights times inputs and that generates this value that we're going to pass through this function calling that function the activation function which gets the name activation

[2:56:12] function really comes from biology and and kind of the actual um workings of a neuron which uh gets activated if there's enough kind of energy there. So same kind of thing here if if the input is enough we run that through a function

[2:56:29] and we kind of generate this uh output of it through this activation function and so that's where it gets kind of its name. Now in terms of the perceptron we know what the activation function is. It's it's actually um a very simple

[2:56:43] function which looks at looks at this weighted sum of inputs. So it takes this um weighted sum. So you can see here's like an input one, input two all the way up to input n. We wait those. So there's a corresponding weight for every one of

[2:56:58] those inputs. We know that. We total that up and maybe even add in a bias. So you can think of an extra term here. there's a bias that it get uh there's a bias that it get uh contributes to the sum typically and

[2:57:14] then this goes through this f right this f ofx um activation which is taking the f ofx um activation which is taking the f of that uh weighted sum uh that is all

[2:57:26] activation function is being applied to those and really in the perceptron it's a simple function the f really says that If so the the f um is such that um it

[2:57:41] If so the the f um is such that um it has a definition such that like if um has a definition such that like if um this weighted sum is positive uh then we this weighted sum is positive uh then we produce um if this sum is positive then

[2:57:54] produce um if this sum is positive then we produce uh one um else we produce uh zero else being you know it's negative or zero then we produce zero right so the activation for the perceptron is really really simple. But that that's

[2:58:09] there. And in fact, when we start building our own neural networks, um activations we can choose from. Some of them are going to be more um popular than others, and some of them have very

[2:58:22] particular use cases. As I said, one thing you want to pay attention to is this activation really helping us kind of narrow down the range of what the output of that neuron could possibly be. So like in the perceptron case, the

[2:58:38] output's only allowed to be a binary one or a zero, which is good for certain problems, right? It's good for like a spam not spam cancer not cancer, but it's not very good for regression at all. Right? It wouldn't be useful at all

[2:58:52] for regression. wouldn't be useful at all for generating a probability even because it's still structured to just zero or one. Uh and it certainly wouldn't be good for multiclass where we have maybe 10 different categories of

[2:59:07] things we're predicting. Um so so the perceptron activation is very limited. Um but we're going to explore some other ones that are found in a lot of lot more perceptron. But but you see what its purpose is really to produce the output

[2:59:24] purpose is really to produce the output of a neuron um passing in those inputs through this activation function. So it has that capability of kind of um mapping those to a particular range which is important in terms of what are

[2:59:37] we trying to get out of the neural net especially as you get towards the the output layers right then you really care about what your final output is. Okay, so if you think about it, if like what would happen if we did not have an

[2:59:50] activation function? So just think about that for a second. If we didn't have that for a second. If we didn't have that um then in theory like you know depending on what those inputs are, we really could get anything from this

[3:00:02] weighted sum with with the weights and inputs, right? They could be any real number. That's all this is saying is like you get kind of an infinite range of possibilities. It's not really limited to anything. um which is fine

[3:00:15] for some problems like a regression that's okay but that is very open-ended like like if we wanted to generate a probability this is really bad because

[3:00:27] we don't want things to be really large or really big actually want it to be confined to be between zero zero or one right coming out of the kind of final right coming out of the kind of final neuron so having an activation helps us

[3:00:41] kind of bound that output coming out of a neuron because if you think about it, a neuron because if you think about it, this weighted sum could be anything. And probably not good that it could be anything negative or positive. Like if

[3:00:55] you're predicting a price, you probably want that to be positive. So even minus infinity infinity is probably not even good enough. You probably want it to be restricted to between zero and infinity at the very least. So that's where the

[3:01:08] activation comes in. we apply a function to this and that helps us kind of map that to a potential range like maybe we map it to 0 to infinity. Um so we so for

[3:01:22] like a regression that probably be good uh for things like price temperature or uh for things like price temperature or whatever. Um or maybe we map that to uh 0 to one for like a probability um that that would be good for that

[3:01:35] case. So it just depends. It just depends on what problem we're solving. It also depends where we are in the network. Usually for things towards the end, we want to make sure we're using an activation that lines up the output to

[3:01:48] we're producing probabilities, want that to be between 0, one, not, you know, all over the place like negatives, positives, which just the weighted sometimes inputs could be anything, right? Could be anything. Um, and again,

[3:02:03] kind of comes into play. It's applied to this this and allows us to kind of map this uh to [clears throat] range at least for that neuron. Okay,

[3:02:17] so hopefully that makes sense. So for that reason, activations play a really that reason, activations play a really critical role in in any neural network. And as I said, especially towards the end of the network, you know, the

[3:02:29] example I always go back to is if we're predicting like three different things like um dog, cat, and giraffe, we don't want those numbers to be all over the probabilities that add up to one actually. So across this kind of out,

[3:02:44] let's imagine this was our output layer. We would want to make sure that these are all probabilities that are between like uh zero. So that are between like uh zero. So probabilities in 0 to one in this range.

[3:03:00] So we'd certainly want that to happen. We'd also want all of these to total up to one. We'd want them to be normalized in some sense, right? So um each one of in some sense, right? So um each one of these outputs should um total

[3:03:13] these outputs should um total across the output [clears throat] should happen if there if there are probabilities. So that's in doing. And what we would hope is there's an activation we can apply

[3:03:29] across this entire layer really such that um all of these uh are probabilities certainly in 01 and total up to one for those kind of cases. So

[3:03:41] we're going to see activations that actually will do that for us be able to normalize that for us which is good. All right. Okay. So, and the other thing about the um activation function as well is the the activation function actually

[3:03:56] also serves a really important part of the model um because it allow it basically allows us to introduce um nonlinearity because if you think about it like if we just had let me go back a couple slides

[3:04:11] to this. If we just had this this is basically a linear model, right? Just we've seen before. It's just a line. This is just a linear regression. Linear regression, right? That's actually all that that is. It's just

[3:04:26] weights times inputs plus a bias. We've seen that before, right? That's all that seen that before, right? That's all that that is. So without activations, we essentially have just a huge collection of linear regressions that

[3:04:38] collection of linear regressions that are um connected together. So important role in introducing nonlinearity which is really really important for a neural network to be able to learn complex patterns. So

[3:04:52] activations are actually really critical. So having this f here this activation allows us to learn more general patterns than just linear ones

[3:05:04] all that we would have is just linear relationships all over the place right throughout the network. Um but having an activation function here that and the activation function will generally be nonlinear. It will gener generally be

[3:05:19] nonlinear. We're going to get into some examples but like some of them are going to be uh like the sigmoid like from logistic regression like a logistic function. Some of them will be hyperbolic tangent is a popular one.

[3:05:32] These are all you know highly nonlinear functions. There's also exponentials that are possible. So different types of functions that are all nonlinear. That's functions that are all nonlinear. That's actually a really big key uh because we

[3:05:46] want to introduce that in the learning process is the ability to kind of learn nonlinear relationships. That's another reason why they're important. They limit our range, but they also introduce nonlinearity so that our network can

[3:05:59] learn those nonlinear patterns, which is really important. Okay. All right. So popular activations. Now, by no means is this going to be um an ex exhaustive [snorts] list. There's definitely more than what's just on this list, but I'm

[3:06:14] going to be the most popular that we deal with in this course for sure. You there that are more advanced. Um but I would say they rarely show up for the

[3:06:26] use cases that we care about. The ones the ones that we care about are mostly going to be what's on this list. Um, and I'm going to explain what all of these are and kind of what they look like. The thing that these all have in common, by

[3:06:38] thing that these all have in common, by the way, is they are all nonlinear functions. All of them. So that's really that's a really important attribute of an activation is generally nonlinear. And again, we're going to have these

[3:06:51] spread out, you know, across all the neurons. And um so so in that way we're kind of like stitching together all these nonlinear relationships which is which is um how we learn complex patterns. But let me go through these.

[3:07:05] So these are some popular ones. We've actually already looked at this one a going to start with that one and then we'll talk about these other three. The sigmoid relu and softmax. We'll talk about all those. So these are some

[3:07:18] already looked at stuff that's the one that's involved in the perceptron. Um let's start there. So the the reason this is called a step function is because it essentially only it has one step up from a zero to a one. So if you

[3:07:35] look at it on a graph, which I'll do on the next slide, it's a function that um goes from zero up to one and it's just a single step up uh one unit from 0 to one. Now this activation is kind of special to the perceptron algorithm

[3:07:50] because um that's just the way it's designed, right? is that uh whenever that weighted sum of inputs that is being activated, right? That that data that's that weighted sum going into this function whenever that is um positive um

[3:08:05] we get a one as our result. Um otherwise we get a zero whenever that weighted sum we get a zero whenever that weighted sum is negative or zero. So we we get zero. So in that way it's it's uh let me show you it's kind of a step function that

[3:08:19] you it's kind of a step function that looks like this. So we this is inspired by activating a neuron, right? So if you have a single neuron, it gets kind of activated when there's enough energy. In other words, it's bigger than zero. And

[3:08:33] this is why it's called a step function is you can see on the graph, it literally looks like a step like like on the stairs, right? It goes up and then over. So what this graph represents is you know this axis is the input

[3:08:48] and the y- axis is the output of course and what you're looking at is for anytime the input is negative so you're on this side of the x- axis um you get

[3:09:00] zero as your output right you're you're down here you always get zero no matter what that is the moment you have something positive meaning you're on this side um you get a one Right? You get a one for all those. No matter no

[3:09:14] matter what that is, as soon as it's positive, you get a one. So you get that step up to one. Um and and that p that neuron is essentially activated, right? And we get a one as a as a result. Um so so this activation function kind of

[3:09:28] looks like this. Now that is nonlinear for sure. Um but what we also see is the range of this is very limited to 0 to one. So the this step function

[3:09:40] activation is really only useful for this perceptron classification that's going to be zero or a one, right? Really only useful for that because we're not able to produce any other value in between, right? We

[3:09:54] only get a zero or one. We also like for regression, we'd be out of luck because we can't produce anything else. Um so definitely wouldn't be a good fit for any of those kind of problems, right? So that's the step function. Again, this

[3:10:07] this one's only going to be used for the traditional perceptron. That's the only time it's really ever used is in that case. So, let's talk about the sigma. Now, this is one that we have seen before and it's actually the same

[3:10:20] function that we have seen with logistic regression. Um, exactly the same function with logistic regression. So the sigmoid produces um a probability

[3:10:33] the sigmoid produces um a probability between 0ero to one which also makes it useful for binary classification. Just like the step function this is helpful for binary classification. The difference is the sigmoid

[3:10:46] um allows us to produce a probability that could be anything actually between 0 to one. Right? So in that way we're actually generating a more direct probability rather than just a just either a zero or a one final result

[3:11:01] we did the same thing right we produced a probability and what we did is we said okay if that probability is big enough meaning it's above.5 we categorize that as a one if it was down here we categorized it as a zero so

[3:11:16] down here we categorized it as a zero so we did that but the sigmoid activation allows us to produce a zero a probability between 0 and one as the output of a neuron. So remember, we're still applying this function to a

[3:11:30] weighted sum of inputs plus a bias maybe. And um now of inputs plus a bias maybe. And um now this function is the sigmoid is what we're saying is called the sigmoid. Um and this is going to map

[3:11:46] sigmoid. Um and this is going to map whatever this is um to between 0 to one. some some probability. We interpret that as kind of a probability between 0 to one. Um so where is this useful? This activation function is mostly used for

[3:12:01] binary classification as the output neuron. So if you're in the network and so let's say let's draw the network. So let's say we have this network and we have maybe we have a hidden layer here. Um and then we kind of produce one final

[3:12:16] output. What we would, if we're doing uh binary classification like spam not spam or cancer not cancer, what we would really like this activation to be is a

[3:12:28] sigmoid. This this guy should probably be a sigmoid because what what we want be a sigmoid because what what we want to do what we want to do is basically produce a probability from this weighted sum of these guys coming into this.

[3:12:42] Right? So we're going to take this weighted sum and uh generate a probability out of that. Um so we want this final result to

[3:12:54] be a sigmoid. Now these these others in the network don't have to be sigmoid activation um to produce their output. Um that's fine. We're going to look at for kind of the middle of the network that produces a general value. But um

[3:13:12] sigmoid is really really often seen at the very end when you want to produce a zero or one. Generally you don't want to produce a zero or one kind of in the middle cuz that limits your range, right? That limits cuz like then this

[3:13:25] this neuron would be very limited to a probability. This neuron would be kind of limited. This would be limited. We don't really want to do that in the want to limit ourselves because then

[3:13:38] we're not going to learn very effectively. Um but as a final output yes, we want that to be a probability between 0ero and one. So sigmoid works great. Remember the the function for a sigmoid is what we saw in logist

[3:13:55] logistic regression. It's this um fraction that looks like this. So it's it's uh definitely nonlinear, right? Definitely nonlinear. You can see it from the graph. But certainly, you know, this

[3:14:08] graph. But certainly, you know, this produces uh this produces um values between 0 and one. When x gets really big negative, um this fraction in the bottom kind of grows exponentially big. When x goes

[3:14:22] this way, it grows the fraction grows exponentially big in the bottom and makes it one over something really huge. Um which is basically zero. So that's why it decreases down this way as the x gets more and more negative. And then

[3:14:36] kind of the other way around is like if x gets really big positive and you go this way um you get closer to one because this term basically decreases down to zero um the larger x gets. So you get one over one. So you

[3:14:51] get something closer to one. So hopefully that graph makes sense like that's what the activation is. And and again this is the same that we this is Sometimes you'll hear it called the logistic function

[3:15:06] logistic function logistic function. that's kind of used in logistic regression. uh in logistic regression remember this it's basically this but we have um we

[3:15:23] have uh we have something like this beta we had our beta iixi we were doing that um so we essentially have that same thing uh for this activating that that

[3:15:35] neuron okay any questions about the sigmoid so when you use it is generally the last neuron in a binary classification if you're building a should be the activation on the final neuron. That's what it should be to

[3:15:51] produce a probability between 0 and one. We generally won't use it in the interior of uh like the earlier neurons because we don't want to limit the output of those earlier neurons prematurely, right? That that will that

[3:16:06] will basically collapse the learning and it won't be it won't be as effective. classification, we do want to produce a probability as kind of that final output of the network. So that's why you'll usually see it as the last activation

[3:16:21] here. Okay. So sigmoid will sometimes see. Now let me show you the most see. Now let me show you the most popular uh activation. And the reason this is the most popular is because it's the one that's used almost exclusively

[3:16:36] in the hidden layers of a network. The reason is is because it does not really reason is is because it does not really restrict the values. In fact, what this restrict the values. In fact, what this this uh activation will just basically

[3:16:51] return back to you the input assuming that it's positive. So as long as it's back what you put in. So see how this function gives you the maximum between zero and x. So what that really means is like as long as x is positive, what's

[3:17:07] the max going to be between zero and anything positive? It's going to be whatever that other what that positive number is, right? So this will always just return to you what you put in as long as it's positive. However, this is

[3:17:21] long as it's positive. However, this is a nonlinear function because it has this piece right here, which is really critical. This piece is saying like whenever we have something negative, it's actually going to chop that off and

[3:17:34] it's actually going to chop that off and just give us zero. So this is actually discouraging negative outputs. It's always going to give us something that's either zero or positive. Right? So this this activation

[3:17:47] function is known as a rectified linear unit. unit. um because this y= x is a perfect linear um because this y= x is a perfect linear line um with a slope of 1, but we're

[3:18:00] rectifying that with um this kind of nonlinear piece which is this uh flattening of anything negative to zero. Um because a true line would continue this way for anything negative. It would still it would just that would be the

[3:18:14] true line, right? Would be um anything negative would just give us that value back in return. But we're actually chopping that off in favor of making it just zero. So we rectify it that way. The the way you pronounce this is called

[3:18:29] The the way you pronounce this is called relu. So relu r e l u. Usually how you'll hear that pronounced relu. This this is this activation is the one that's most often used in the interior of that network, those hidden layers um

[3:18:45] of that network, those hidden layers um because it allows us to basically because it allows us to basically generate the outputs that we care about um as long as they're positive. So it it doesn't limit us down to a specific

[3:18:59] range like zero or one. It does limit us to a bit wider of a range than that, to a bit wider of a range than that, like zero to infinity essentially. But in this case, like these guys would likely be all relu activated. Um, and

[3:19:12] then like this one would probably be a sigmoid for uh a binary classification. Doesn't choosing the max of the range skew the model in terms of prediction? We're not really choosing the max. Uh we're choosing the minimum uh because

[3:19:28] nothing can be negative. So it's always going to be capped at zero. Yeah, it So in theory, yes. I think that's a really good question, but in reality, no,

[3:19:41] because we're going to have many of these neurons um typically in the hundreds of these per layer. And basically what we're going to do is adjust the weights enough to make up for that. So remember, there's a bunch of

[3:19:55] weights here, and we're just going to make enough weight adjustments so that we can always produce something that carries forward into the next layer. So, um, the the weights, hopefully that makes sense, like the weights really

[3:20:10] help us learn because we're going to adjust those weights accordingly to produce the right outputs. That's actually all of what training a neural network is, is making the right weight adjustments. Um, and that's what we're

[3:20:23] going to talk about coming up soon. So, not really. We just have to adjust the weights appropriately uh to to generate the right outputs and that's something that gets learned during the training process. So, we'll talk about that.

[3:20:38] Okay. But this relu is um probably the most popular activation. We're going to use it quite a bit. It's used all over the place uh in most networks. Um relu the place uh in most networks. Um relu is uh very very popular activation. It's

[3:20:52] it's kind of a very simple nonlinear function. If you think about it, it's because it's almost always linear except in this case. This makes it nonlinear, right? Is this piece of it kind of makes it nonlinear, but

[3:21:07] otherwise it's mostly linear. Now, what a good thing about this activation as a good thing about this activation as well is um like if we were producing a regression value for like a price, this would be a good one to have at the end,

[3:21:21] right? because you can generate anything that's not negative. So this would be a good regression candidate to use as an activation sort of towards the end, right? To to basically make sure that you're getting something that's

[3:21:33] positive. So that that would be like realistic. Okay. So this is a popular activation. We're going to use it quite a bit. And of course like in our code, activations. And like I said, where you'll mostly when you would use this

[3:21:48] you'll mostly when you would use this one is pretty much in any hidden layer, your default is going to be relu. There there's very few reasons you should not use a relu in your hidden layers unless you have a good reason not to. Um most

[3:22:03] like 90% of the time you're going to see relu used as as a activation on the hidden layers. The output layer is where you'll see those differences. is you'll you can see a sigmoid you could see that step function in the perceptron and then

[3:22:17] softmax which will which will be used for probabilities as well but most of the time we'll see relu show up in those hidden layers um between the input and

[3:22:29] the output relu is very very popular it's just an it's a very effective activation it's a very effective nonlinear function that kind of just keeps things moving forward um as long as they're positive

[3:22:42] the sigmoid and the relu that we've looked at so far? Well, the sigmoid has a uh it's a little bit more complex of a function. It's that exponential, that logistic. And one of the issues with the sigmoid is it can be susceptible to

[3:22:59] which we'll talk about later. Essentially, this is this is a consequence of the sigmoid always being between 0 to one. So the issue with that

[3:23:11] is it's a fraction, right? A sigmoid is going to produce a fraction that's going to produce a fraction that's between 0 to one. The issue with that is when we're doing the gradient descent learning process, if we're carrying

[3:23:23] you'll never see sigmoids in those interior layers is because if you're carrying a lot of a lot of fractions, they they multiply together with the because they become the inputs to the next layer. they multiply together with

[3:23:38] the weights of the next layer and really decrease that and and really limit your learning ability because you have such small fractional values. Whereas relu can avoid that because it is generally just going to give you what you put in.

[3:23:53] right? It's not going to limit you to 01. So it but it's still nonlinear. So, so it can help you learn those kind of complex relationships still with the help of kind of the weights and the training process. So, this is just to

[3:24:08] say you should generally be using a sigmoid in your output if you're producing a a binary classification output. You should avoid using sigmoids because it's going to make learning more

[3:24:22] the neural net more challenging. And we'll study a little bit more the details of why that's the case, but it's mostly due to its range. It's producing the sigmoid produces fractions which carry through the throughout the network

[3:24:37] and it it kind of compounds like if everything is a fraction, things are going to compound really quickly and and make small values that are very difficult to learn from. Okay, so that's why you typically see ReLU in those uh

[3:24:50] hidden layers um because it makes sense. You don't want to have these small fractions kind of all over the place. Does that make sense? Like if I go I actually let me just illustrate that real quick. Is like you can imagine if

[3:25:03] real quick. Is like you can imagine if we had uh going into a neuron if all of these guys were fractions coming out of a sigmoid from from other neurons in the layer. Um that's just going to compound, right? Like if this is if this is 1/3

[3:25:17] right? Like if this is if this is 1/3 and then this is like uh 1/2 and then you have um maybe another neuron that's connecting to it that has like a 1/4 then you just have a bunch of fractions that are being weighted. Um and uh they

[3:25:33] they're always going to reduce these values and and limit your range and kind values and and limit your range and kind of compound things. Um so so you don't usually see those sigmoids in the hidden layers. You'll see them at the end if

[3:25:46] you want this to particularly be a fraction. You'll see that. Okay. So relu fraction. You'll see that. Okay. So relu nonlinear it's just the maximum between zero and x whatever is going into that activation. And you'll typically see

[3:26:01] these in the hidden layers uh in between the input and output. one more activation because it's also going to be really popular uh especially

[3:26:14] it it's really popular especially as the output um using it in the output layer for multiclass classification. So the softmax uh actually just extends the

[3:26:27] sigmoid. It's like a multi-dimensional sigmoid essentially. So the softmax produces values between 0 and one just like the sigmoid does. it produces probabilities. The difference is that the softmax um is useful for generating

[3:26:43] multiple probabilities that add up to one. So um what you'll typically see the softmax used for is the activation across the uh final layer in the output

[3:26:56] if it has multiple neurons in that final layer. So let's say let's say we had um three. So we we have a neural network that kind of looks like this. And then we just have um maybe a hidden layer of four and then our final output is three

[3:27:13] because we're we're predicting um dog, cat, and giraffe, right? Those three animals. Um what we would what we would use for this activation in this layer is the softmax. We would use the softmax for each one of these. And what that's

[3:27:29] going to do is produce probabilities that such that all of these are between 0 to one and they actually are normalized such that they total up to normalized such that they total up to one. Um so softmax is really really

[3:27:43] great as you'll see it in kind of your final layer if you're if you're doing multiclass classification, right? So so you're predicting amongst a thousand things then you'll have a thousand probabilities that total up to one. In

[3:27:56] this case, we only have three. So we have three probabilities all that are between bounded between 0 and one that total up to one. Um so you'll typically see this in the output layer. Um especially for like image

[3:28:10] classification. Um you'll see that or it could be you know multiclass classification. Uh you'll see you'll see this softmax. Uh you'll see you'll see this softmax. Another place you see softmax is in um

[3:28:23] like an LLM uh like a a large language model will use a softmax as it final layer because it needs to produce probabilities across every token that's that's possible right so and the reason is is because the model like a GBT is

[3:28:38] going to predict what is the most likely next token and so that'll be represented by one of these probabilities right what's the highest probability as kind of the most likely next token. So softmax incredibly useful for

[3:28:53] anything that's multiclass classification and it should be your last it should be in the output layer, right? It's going to be the activation that's used on the output neurons and the final layer to normalize them

[3:29:06] between 01 and and get them to add up to one more importantly. So um you know it's a it's a really uh it's a really critical function. The softmax function,

[3:29:18] if you're curious what it is mathematically, is sort of like the mathematically, is sort of like the sigmoid except it's um more of it's more of an exponential. So it's kind of like across a summation of these. Um so you

[3:29:33] uh you actually apply the softmax to individual components and then you sum that up over all the components. Um and so this normalizes it to be between 0 and 1 and uh make sure that the total of it is one um effectively. So it's a

[3:29:51] and you apply it to a component in the layer. So every neuron in this layer is going to get uh this function applied to it. Okay. So soft so just to recap that it. Okay. So soft so just to recap that sigmoid it should be your activation on

[3:30:06] your final neuron in a binary classification um relu will be in your hidden layers in the middle it's the maximum between zero and x usually just returns x as long as

[3:30:18] it's positive and then softmax is going to be in your final layer for multiclass classification to produce probabilities that total up to one and by the way these are all all these activations are built built into TensorFlow and PyTorch.

[3:30:33] So, we'll be able to use them really easily inside of our code. Um they'll be built into those libraries. So, uh it'll be really straightforward when we declare a layer of neurons. What activations do we want to use? Um it'll

[3:30:47] all up when we start building our own networks. So, all of that will be readily available and there's really good documentation in both of those libraries on kind of what activations are available. Like I said, there's more

[3:31:00] than this. You know, there are other ones like some some popular ones are are like hyperbolic tangent is a popular one. Uh especially with some image problems. You'll see this uh tanch be used hyperbolic tangent. Um that's a

[3:31:16] popular one. Um there's other there's variations on uh on relu. There's there's one called jellu. Um there's there's one called leaky relu.

[3:31:29] them, but um the ones that we've covered on this list are the ones that we'll primarily stick to uh as we go through our examples. Okay, so that's a little bit about activations. So, just saying that there are, you know, definitely

[3:31:43] multiple of them and uh which one we use depends on really the situation. All right, so I wanted to do a little bit of a demo here to show you an example with some activations uh and kind of how the activation is applied. Let me jump over

[3:31:58] to that. Okay, so we're going to look at this notebook. Do you guys have this one? This is the 3.06 uh taking a look at neural networks and activations. Hopefully you have this one. You want to follow along? Okay,

[3:32:11] one. You want to follow along? Okay, great. so what we're going to do the purpose of um Thank you for Yeah, thank you for sharing that. The purpose of doing this demo is uh mainly to see how we can

[3:32:26] build essentially a simple um neural network from scratch. So not using TensorFlow or PyTorch, but see how we can use like a weighted sum through an activation and see that actually work in code, which I think will be really

[3:32:40] interesting to see. Um and and really start to put the pieces together of a what a neural network looks like. Um, so we can kind of get a feel for the different components of it without yet going into our uh frameworks that are

[3:32:57] going to handle that more in depth like TensorFlow and PyTorch. So let's see. So first of all, we're going to import some things from numpy. So some functions we might need like the dotproduct um random value initialization

[3:33:12] to build our own neural net that has those weights. So we're going to start out with random weights. So we need some randomization here and uh we'll have some various like exponential function and dotproduct things that we may need

[3:33:28] and dotproduct things that we may need uh from numpy to do this. Okay. So what we're going to do is um essentially we're going to start by building a neural network uh a fake neural network. I say fake because, you

[3:33:42] frameworks that we're going to learn about, but it's going to show you the the basically from scratch some of the components that you would see in a typical neural network, including the activation, which is really important.

[3:33:55] And what we're going to do is basically build a perceptron neural network. So it's going to it's going to take uh three inputs essentially um an input that's a size three vector basically a 3x1 matrix and uh it's going to map it's

[3:34:11] We're actually going to use the hyperbolic tangent activation and and produce an output um which which would be kind of like a regression almost but just to show you the different components of this network to so you can

[3:34:24] see what's really involved with it. Okay. So to show you what's going on with this fake network. So we're going to build our kind of network from the like a perceptron. What you're going to see is we need some weights and we have

[3:34:40] uh these weights which are generated randomly. Um and this ensures that their range is between minus1 and one because this uh numpy.random.random

[3:34:54] random generates a random value between 0 to one. Okay. So if we multiply something between 0 to one and multiply the by two uh and subtract one kind of the range that we limit ourselves to is between minus1 to one in terms of those

[3:35:09] weights. So these are just going to be randomized weights that that live somewhere between minus1 to one. They're going to be random though. And the out with randomized weights. This is actually how a network usually works is

[3:35:23] when you initialize the network, it has random weights to start with. What needs to happen is you have to train your network. So it learns those weights during the course of the training process. But this is pretty standard. Uh

[3:35:36] this is pretty standard that this is going to be randomized to begin with and in in our frameworks like PyTorch, TensorFlow, um those will also be random to kind of start with and then they get trained, right? How is this plural? Um

[3:35:51] because there's three of them. This random uh 31 generates kind of a a vector of size three. Um technically it's a matrix that's three rows in one column. So there's multiple weights.

[3:36:06] Yeah, that's that's why it's synaptic weights. Um because there's three of them that are just kind of in this like 3x one matrix which is just basically a vector, right? with with three weights.

[3:36:24] All right. So then we have our uh activation function here which is going to be hyperbolic tangent. Now all we're doing is when we give an input to this hyperbolic tangent. We're just going to um apply the numpy hyperbolic tangent to

[3:36:40] that input. Now what this is actually going to be our activation. Now we could use any activation we want. This is just an example. We could use relu. Um we could use uh sigmoid. This is just an example of declaring an activation and

[3:36:55] using it in our network. So I'm going to show you what that looks like. But you know if it was relu then that would be um you know it'd be like max between um you know it'd be like max between zero and x would be the relu. Um but in

[3:37:09] zero and x would be the relu. Um but in this case we're doing uh mp uh tanch which is hyperbolic tangent uh applied to x. Okay, that's just a function, just to x. Okay, that's just a function, just an activation function.

[3:37:26] which is um I'm not too concerned with right now because we haven't learned we're going to in in the later notes in this lesson. We haven't quite learned this lesson. We haven't quite learned it, is the gradient descent process for

[3:37:38] to learn all about that coming up shortly. But one of the things you need in order to uh in order to do gradient descent is the gradient of the activation um which is this uh so the derivative or

[3:37:54] the gradient of uh the hyperbolic tangent uh is this function here that's just a mathematical uh function that involves that hyperbolic tangent. So the reason we need this is the reason we

[3:38:10] reason we need this is the reason we need this is for training uh gradient descent. Um and again I'm going to go into detail on what gradient descent is coming up shortly uh in the notes but um it's just

[3:38:24] here in this example for the an example of how you would train this thing. Okay. So we have the activation derivative which will be useful uh in the training for gradient descent. Okay. And then we have a uh function called train. So this

[3:38:39] have a uh function called train. So this function is called train and takes the training inputs um training outputs which are kind of like the labels and then um a number of training iterations. Again this here is going to be our this

[3:38:55] is going to be gradient descent training uh process which we too concerned with us digesting all of this this moment, but just so we kind of

[3:39:07] this this moment, but just so we kind of preview what's to come. Essentially what we're doing here is an iterative process through our through through our data. So we basically iterate a certain number of times which is configurable this number

[3:39:21] of training iterations. We basically do this uh we iterate this many number of this uh we iterate this many number of times through this descent process which times through this descent process which is to say we basically produce the

[3:39:34] is to say we basically produce the output given our inputs. So we do a forward propagation. Um so that's what this is is we do a forward propagation

[3:39:51] that? It's because we want to see how far off our network currently is. So if we produce outputs, what we can do is actually compare that to our labels, which are these guys, and we can compare our output to that and see how far off

[3:40:08] we truly are. Um, which is what's called our error or sometimes called the loss. our error or sometimes called the loss. So this part here is um computing the So this part here is um computing the loss which is how bad our network is

[3:40:24] currently. So we're we're just gathering up how bad we are currently. Now why do we do that? It's because we use that quantity to make an adjustment to our weights. That's the key thing is we actually want to make an adjustment to

[3:40:39] our weights to basically shift our network to perform better. Our goal is to minimize this error. I should I should write that down. The goal is to minimize this error. So essentially what we're

[3:40:55] doing is measuring our current error and then using that quantity along with the then using that quantity along with the derivative of the uh activation which is here the derivative of the activation. We basically use that amount multiplied

[3:41:11] We basically use that amount multiplied by that derivative. Uh we basically use this amount to figure out how much we should adjust our weight. So this is a numerical calculation that's driven by this gradient. Um so that that's why

[3:41:24] it's called gradient descent is we we essentially want to uh make an essentially want to uh make an adjustment that uh will minim that will lower our error over the course of the iteration. So as we iterate a certain

[3:41:38] better and better and better at making predictions using our network. And you can see here like we make adjustments. So you see how we take our weight. We take whatever our weights currently are and we add in an

[3:41:51] adjustment. So, this makes a tweak. It think of it like turning a knob. So, we essentially turn the knob and the adjustment is telling us how much should we turn that knob. Should we turn it a lot? Should we turn it a little? We're

[3:42:04] that adjustment to get better output of the network. So, this is um turning the knob so the network can predict

[3:42:17] better, right? So it so it can it can make the right adjustment and predict better in the future. So this in a nutshell is how any neural network's nutshell is how any neural network's going to train. Now this process

[3:42:32] will generally be handled by the framework. So by PyTorch or TensorFlow framework. So by PyTorch or TensorFlow will manage this for us because it turns out when you have a pretty big network that computing this adjustment is

[3:42:47] incredibly intensive on a big network computing what that adjustment should be is is actually very computationally intense and it's called back propagation. So com so basically uh doing the weight adjustment and uh

[3:43:03] you should adjust and then actually turning all those knobs. You can imagine going to be an expensive operation. Just to give you some context like modern OM

[3:43:16] uh have hundreds of billions of these guys, hundreds of billions of weights. So you need to compute an adjustment for every single one of them. hundreds of billions of weights, right? And you need and you actually need to do that every

[3:43:31] iteration. So you're imagine training for thousands of iterations and you have hundreds of billions of weights that you need to adjust. It's a that's why they take forever to train. [laughter] That's they're massive networks. They

[3:43:46] take forever. That's that's why uh because of this computation right here can be very expensive. uh because this derivative can be very uh computationally intense to compute. It depends on your activations. It depends

[3:44:00] generally those large networks like an LLM uh it's going to take a while. That's where you benefit from a GPU by the way is to try to make this computation faster. So we'll actually see that like when we build our computer

[3:44:14] vision networks uh we'll do a similar training process and we'll utilize a GPU there to make this faster. Okay. By the way, I forgot to mention uh this forward propagation when we produced output uh we call it think mainly because we are

[3:44:30] uh it's like sending data through our network. So it's like all of our neurons are thinking together in this network. Um so thinking what is thinking? It's really um th this is uh doing our weighted sum through an activation. That

[3:44:45] that's all it is. So it's like our our weighted sum through an activation. That's all it's doing, right? Weighted sum through an activation. So you can

[3:44:57] even see that here. Notice how we apply the activation, the tanch to this weighted sum. Now why is it a weighted sum? It's because you're taking the dotproduct between your inputs and your weights. So that literally now what does

[3:45:11] dotproduct do? Remember dotproduct adds together the product of these guys as vectors, right? So that's that's taking our weighted sum and then does that make

[3:45:23] sense? This is taking our weighted sum and passing it through this activation. and passing it through this activation. So that's this think is just computing that neuron essentially, right? Computing that neuron.

[3:45:36] All right. So just to finish out the demo, um that's the network. So that's our that's kind of our fake network. It's got three weights. Um it it does uh It's got three weights. Um it it does uh it it thinks by doing a weighted sum of

[3:45:49] those inputs through. So it has three inputs, three weights. Take that weighted sum, pass it through a tanch activation. Um and again, the activation have it be whatever we wanted. We it could be a sigmoid if we were producing

[3:46:03] a probability here. It's just producing a tanch which is going to be tant has its um outputs are actually between minus1 and one. So we're always going to get something between there. So then we can run this code here which is going to

[3:46:17] can run this code here which is going to take a random set of inputs um and then the labels for each one of those. So these are all like different labels like sequences. Notice how every input is size three which is what it should be.

[3:46:30] So here's here's one input. Here's one input. Here's one input. Here's one just just to show you how it works. And then here's a label for each one of these. So this guy is labeled zero. This this guy's labeled one. This guy's

[3:46:45] labeled one. And this guy's labeled zero. So we have four labels here corresponding to each one of these uh sets of three inputs. So here's our random weights to start with is [snorts] this vector. So we're

[3:46:59] printing that out just so we see what our random weights are. Now, when we go through the training process, we're just going to have 10,000 steps and use those random inputs and outputs. Look at what the weights are after we do this. The

[3:47:13] weights are now completely adjusted. And we can actually put in a new uh input and see what the output would be. The output, which is just running think

[3:47:26] on this input, um it produces a label of one. Okay, so the network functions as we would expect. Um, and the big thing is to to take away here is to see how those weights actually get adjusted. You see how far off they are from what they

[3:47:40] see how far off they are from what they started as. So here they started as these really random numbers between minus1 and one and uh here they get updated through the training process. So those weights actually get you know

[3:47:54] changed which is the objective of the training. Okay. Hopefully that makes sense. I think this is a good preview of what's to come in terms of the training process and kind of uh how we um think about how

[3:48:09] and kind of uh how we um think about how this stuff actually learns. But you see all of the pieces that we've talked about so far, things like an activation, things like a weighted sum going through the activation. It it's mostly just the

[3:48:22] supposed to gleam anything from these outputs. uh these outputs aren't very meaningful cuz it's kind of a random input and it you know random activation. Uh so it's not a realistic example but the the point of it is to see kind of

[3:48:37] the inner workings of this network to see the different pieces of it. Like typically what's going to be involved in building a network is you know setting up an activations doing forward propagation backward

[3:48:49] propagation train it like there's training you kind of see what goes on in the training. It's mostly so we see that inner working prior to us building our own using using the frameworks mostly just so we get exposed to that. Uh okay.

[3:49:04] So let's talk about more let's talk more about those uh terms we just used mainly forward and backward propagation uh in our neural nets particularly um you know

[3:49:16] just thinking about it from a neuron perspective data going in um and then making those weight adjustments. We'll talk more about that. So training [snorts] any neural net really involves two

[3:49:30] particular phases and we just saw those in the code really uh involved in that training function, right? And um those two phases have a name which is known as forward propagation and backward propagation. We're going to break down

[3:49:44] what each of those really means. Um, but it's kind of what the name suggests, uh, generally is that forward propagation is going to be all about sending data through to produce outputs. That's mainly what you want to do. And in fact,

[3:49:58] once you're done training and you're just using the neural net to make predictions, that's actually all you're doing is just putting data forward through it to produce an output, which is sometimes why that is called

[3:50:10] inference. Uh, because you're just making a prediction at that point. Now the backward propagation is what makes those adjustments to all the weights across the whole network. And so when you have a very um deep network, you

[3:50:26] need to be able to adjust all the weights across the whole network. And algorithm that kind of does that and manages that weight update. And it's all going to be driven by this gradient descent that's that's dictating how much

[3:50:42] of an adjustment should we make to these weights. And then we basically need to the network, which is going to be what back backwards propagation is actually going to do. And so you really only do backward propagation in the course of

[3:50:56] backward propagation in the course of training. But it is an iteration of doing forward propagation to generate predictions to see ultimately to see how far off you are in those predictions from an error perspective. Uh and then

[3:51:10] you basically do backward propagation to go back and uh adjust all your weights these over and over and over until you make enough adjustments to kind of uh

[3:51:22] have a solid network that makes predictions really well. So training is things. So we're going to we're going to dive deeper into what these two things are, but that's that's basically the summary of it. So I think forward

[3:51:39] propagation is the easiest one to understand of the two. And I think it's because it's really straightforward. It's essentially um it it's called forward propagation because when we have this network of neurons um data flows

[3:51:54] kind of forward through all of the layers, right? It goes uh we have inputs that go into our first layer that generates some output which becomes the input to the next layer and on and on and on and eventually we reach that

[3:52:08] output. So in a perceptron in a which is a very basic obviously a very basic network basically just one neuron it's not even really a network but uh just a

[3:52:20] single neuron um we think of that as as forward propagation really as just producing the output and we know what does that what produces the output is the weighted sum through an activation right that's really what produces the

[3:52:34] output is this typical uh what we've been calling weighted sum uh through an activation and so this produces the output data flows forward in this process. So forward propagation is just generating the output through weighted

[3:52:51] when we have a more sophisticated network so if we were to draw multiple neurons in the input and then maybe some hidden layers um and maybe multiple

[3:53:03] hidden layers and then like a final output let's say and let's say that's a probability you know the the forward propagation is going to be a little bit more involved because we're going to have I can even draw this in a different

[3:53:15] contributions from all of these that are going to be weighted and then contributions here, right? Contributions here, contributions here. So, it's this much more involved network. And then data is going to flow

[3:53:31] like this. And then data is going to flow like And then data is going to flow like this. So, much more sophisticated um flow,

[3:53:48] moving this way, right? We're sending data in generating an output. That's the that's the forward propagation really producing that output. Okay. So for the perceptron, we know that the output is either zero or one, right? Because the

[3:54:01] activation for the for the perceptron is this is basically this, right? So if this is basically this, right? So if that weighted sum is positive, we get an output of one. That's the activation that that gives us the final output. If

[3:54:15] that weighted sum is negative or zero, then we get zero, right? That is the that's just the activation function. Now, what needs to be learned are these

[3:54:27] weights and that's true of any neural net. All of those weights are really what need to be actually learned during the training process. So that's kind of what we saw in that in that demo is we need to make adjustments to those

[3:54:41] weights to basically have the best network structure that we can have. We want to make adjustments to those weights so that we accurately predict one or a zero or accurately predict the probability

[3:54:56] um for our labeled data. So we can't forget that training a neural net is still like a supervised learning problem, right? We still have labels like on our like think about images. We have all these images of cats, dogs,

[3:55:09] have all these images of cats, dogs, giraffes. Every one of those uh images has a label associated with it so that we know if we're getting an error or not. And that can inform us making an adjustment to all of our weights. Um so

[3:55:24] weights which is what the training process will do. It will make adjustments to those based on the error that we're producing. We can only really calculate that error if we're doing forward propagation, right? Because we

[3:55:38] need to know what our network thinks is the right answer. That's again kind of in that demo we we saw it was called think is really the forward propagation. The weighted sum through an activation. That's what forward propagation does. It

[3:55:52] produces those outputs. Okay. So we have forward propagation just the simple act of sending data through our network producing an output which we know is producing an output which we know is really just this series of weighted sum

[3:56:05] big network it's still going to be this collection of weighted sum through activations over and over and over again to produce some type of output. Okay so that's forward propagation. Now let's talk about the error function that we

[3:56:19] saw like in that demo, right? So we need some way of evaluating where our network some way of evaluating where our network is currently. If we're producing results in order to make the right weight adjustments, we need to know how far off

[3:56:35] our network is from the true labels. And that's where something like a loss function comes into play. We've actually seen loss functions in machine learning quite a bit. Remember we had mean squared error, we had uh cross entropy

[3:56:50] um for classification like with logistic regression. Um so loss functions are not regression. Um so loss functions are not new to us but in the context of neural we're going to break down kind of how they're used. But this idea of a loss

[3:57:06] us. Right? We've seen it before in machine learning especially with things like linear regression. You guys remember we had the mean squared error loss function that told us how far off our regression was from from the the

[3:57:20] true value. It was kind of that distance, right? So in terms of neural networks, it's actually the same kind of concept. We have um we need some way to measure how far off our output that we get from forward propagation is from the

[3:57:37] true value. And this is really really important because this is going to inform how much of a weight adjustment we should make. Should we make a really big one? Should we make a tiny one? That's that uh that's that notion of

[3:57:51] turning the knob. Should we turn the knob a lot? Maybe just a little bit. That's all informed by how much error we have when we do forward propagation have when we do forward propagation compared to the true label. Okay. So, um

[3:58:04] that's just the general idea of the law. So uh in in neural network terminology um we usually think of this error we usually call it a loss function which is some type of function that computes what

[3:58:18] that error is between the label and what our network is producing. We usually call that a loss function and the goal of the training is to actually minimize that error minimize that loss function over the course of the training. But how

[3:58:33] over the course of the training. But how much loss we have influences how much of a weight adjustment we make to the network. That's a really key idea in the training that that we should uh kind of understand intuitively that you know if

[3:58:46] we're really far off we need to make more of adjustment to our weights. If we're doing pretty good and we're not that far off, then it's a smaller weights. So it's it is proportional to that loss or that error. and we'll see

[3:59:01] exactly how proportional in in the gradient descent formula. Okay, so just gradient descent formula. Okay, so just to give you an example, you know, um it you know we've seen the mean squared error loss function before from

[3:59:15] something like linear regression, right? So we know like this value here would come from the prediction of the model uh in a linear regression that was our typical like beta times our times our features, right? beta 1 times feature 1,

[3:59:29] beta 2, feature two and we added those up and we could we generate a regression value and we see how far off we are from this in terms of the distance, right? That's the mean squared error. We've seen this loss function before. This is

[3:59:43] just an example of one uh that we see in in machine learning generally when it comes to neural networks. We're basically going to have the same kind of setup. We're going to have some type of loss function we use that is supposed to

[3:59:57] quantify this amount of error. So kind of what is our average error across our data? That's what this is getting at, right? Because this is averaging right? Because this is averaging this is averaging our distances from all

[4:00:11] of our predictions to the labels. Um, and we want the same thing in the neural nets is to be able to average this amount of error. So we again can know amount of error. So we again can know how badly we're doing and use that to

[4:00:25] influence what type of update do I need to make to my network in terms of the weights. So the loss function by the way the loss function sometimes goes by different names. You'll mostly hear me call it a loss function but in in like

[4:00:39] literature or in other places you may hear it called uh cost function or objective function. um or sometimes we just call it the error. It's kind of all meaning the same thing. Um but I think

[4:00:52] the proper terminology is truly a loss something the loss function is going to be something that's quantifying that error. Uh and and there's different functions out there for different

[4:01:05] problems. So like this is a really good loss function for regression, right? Because we're measuring the distance between our prediction and what the true value is. that makes sense to do for regression. For something like a

[4:01:19] classification, we have different loss functions like in our logistic regression, right? We had the entropy, the binary cross entropy loss function which was uh a different different loss function that but it's meant to do the

[4:01:32] same thing is kind of penalize those mistakes, right? When you're making a mistake, you want that to show up in your error. So this function is a way of quantifying that. Okay, so we have a loss function

[4:01:44] loss function uh and so you know we we see we've seen this picture visually for the mean squared error before uh you know the the difference between what our uh prediction and actual value uh is

[4:02:00] captured in that mean squared error metric. And obviously the greater that is that means on average the farther away our prediction is from the uh from the true value. So like in a linear regression the

[4:02:14] right? So the predictions are like here and here here they're always on this line and you can kind of compute this distance between this line and here. Okay. So why

[4:02:27] do we care about the cost function? um it's mainly so that we can do uh gradient descent. So I've used this word before but the great we're going to talk

[4:02:39] about that coming up in the next few slides. We mainly need that to to again compute how much of an adjustment we should make to our weights to make our model better. We can only do that if we know how bad we currently are. So if we

[4:02:54] go back to linear regression where we had a model that kind of looked like this to make our predictions that's actually how under the hood like in scikitlearn when we call model.fit fit. These betas were learned. Remember these

[4:03:09] were betas times all of our features. These betas were learned in the bias were learned um actually through a gradient descent process which would uh find out what those are based on this error and uh kind of a a formula that

[4:03:25] takes into account that error to make an adjustment to these betas. um it's actually going to be the same exact thing for neural networks. It's just a a little bit more complicated of a formula to compute that adjustment because we

[4:03:39] have a more complicated model in the neural net than just a basic linear regression. Um but that's going to be the idea is we want to adjust our weights of our network and we're going to use the loss function to help

[4:03:52] facilitate that like how much of an adjustment we should make. So it's really important the loss function is really really important in the training process of finding those weights of the network. Really important. Okay. So

[4:04:05] again like the gradient descent process in a linear regression sense would would basically find the best fit parameters those betas in order to minimize that with the neural net. We're going to find the weights that minimize that error uh

[4:04:22] through a gradient descent algorithm. Any questions? uh on this picture are kind of uh let me slow down and say are there any questions about what the loss function is and kind of what its purpose is. Hopefully that makes sense. We're

[4:04:36] going to use it quite a bit as we go through setting up the training and executing the training of our own neural nets when we build them and and truly like the loss function will actually be something built into the framework. So

[4:04:48] they'll be built into uh TensorFlow and PyTorch. So you know we don't need to do it manually but it's good to have that intuition understanding of what they are doing and why we need it. So again it's capturing that error so that we can make

[4:05:03] adjustments to our weights and that's how we that's how the neural network how we that's how the neural network learns and makes itself better uh is by doing those weight adjustments. Okay. So let's talk about that weight adjustment.

[4:05:16] So so far what we have in the training process is forward propagation which sends data through and generates an output. What we can do with that output output. What we can do with that output is then measure through a loss function

[4:05:30] how much error are we producing because we have generated those outputs. So actually let me go back and and mention that like this um this prediction here

[4:05:43] uh I should call out is what we're producing from the forward prop right this is what we are actually producing in our network uh when we put data through is we're generating a prediction and so that's why forward propagation is

[4:05:59] is first it generates those outputs then we compare that to the true label through the loss function, right? And so this kind of tallies up and keeps track this kind of tallies up and keeps track of how bad are we doing or what is our

[4:06:14] error in terms of our current network um setup. And again, the point is to use this quantity that we're measuring to influence weight adjustments to get a

[4:06:26] better network the next time in the next iteration. So let's talk about that weight adjustment. That's going to be the back propagation part of this. The forward prop is producing the outputs. We have a loss function measuring how

[4:06:38] far off those outputs are from the true labels. Now we need something to actually make the adjustments which is going to be the back propagation. So let's talk about that. So essentially what the uh back propagation is going to

[4:06:53] do is systematically make adjustments to all the weights in the network. So of weights that you have. So you can kind of take a look at this picture here. This is going to have a lot more weights involved in it than just a single

[4:07:08] perceptron which you know is just a single neuron and it only has a handful of weights depending on the inputs. Um but essentially what we need to do is but essentially what we need to do is make adjustments to these weights um

[4:07:21] that can uh make our network better. That's really how it learns. It makes those weight adjustments. It's it's almost like it's turning all the knobs to configure itself so that it can effectively predict on on the data,

[4:07:36] right? It can effectively predict if this image is a cat, a dog or a giraffe. And so just to summarize how this is going to play out, the forward propagation produces the uh so the forward prop, let me just label that

[4:07:51] here. forward prop produces the uh output or the I know [clears throat] the output of the network. So it produces the prediction. The prediction goes into the loss function

[4:08:06] and we do that we generate predictions against all of the data in our training set and and we can take our prediction we can take our labels as well and inject those into our loss function to figure out how how far off we are.

[4:08:21] That's the whole point. Now this loss function produces basically a weight function produces basically a weight adjustment

[4:08:36] that demo. Right? We have we're generating some type of weight adjustment through this error. And the weight adjustment is is uh a formula. It's a formula that's given by gradient descent. So it's uh it's actually a

[4:08:52] formula that's computed based on the gradient of this loss function and it's uh directly influences how much that adjustment should be. We're going to learn about that. But the back where back propagation comes into play is now

[4:09:07] that we know the adjustment, we need to propagate that out to all of the weights. We need to systematically update all the weights. And it turns out that it's not trivial to do that bec and let me tell you why. It's because to

[4:09:23] update a weight that's earlier in the network. So let's say let's say we want to update this weight on this connection. This weight here. So I'm just singling out that weight. Let's call it W. If I want to update

[4:09:38] that weight, the update there actually that weight, the update there actually depends on the updates in the further layers down in the network. So the the adjustment here actually is dependent on

[4:09:52] the adjustments to all of these guys. So to everything after it, it's dependent on um in order to update this guy that's earlier. So for that reason, we actually

[4:10:04] work backwards. We work back to front which is why this is called backward propagation. We need to start here and figure out what are all the weight adjustments I need to make from here and then we can use that to then update all

[4:10:19] the weight adjustments in the next layer and then in the next layer and then in the next layer and then in the next layer. So this is why it's called back layer. So this is why it's called back propagation or backwards prop.

[4:10:33] propagation or backwards prop. back prop back propagation and the reason there's a dependency there is all about calculus it's it's a very I don't want to get into the math of it too much but it's all about

[4:10:47] calculus essentially it's because the gradient of a weight earlier on in the network is dependent on gradients that follow it later on in the network this follow it later on in the network this way so um in order to compute earlier

[4:11:03] updates, we need to compute updates that are further down the network. So algorithmically, it makes sense to start here on the end and then once you know those updates, you just work your way backwards and you can use those to

[4:11:18] compute the updates here. Then you use these guys to compute the updates here and it works its way backward. Okay, so this is this is actually the essence of training. You do forward propagation produce a prediction use that prediction

[4:11:34] to generate some loss. That loss is comput is uh used to compute a weight adjustment for every weight. So the weight adjustment is a formula that's that's utilizing the derivative or the gradient of the loss with respect to

[4:11:49] every weight and you start with weights towards the end of the network and then you back prop towards the front of the Ultimately you're going to adjust every weight in the network. Uh but you're

[4:12:04] to front backward [snorts] propagation. That's what that does. Okay, it it sounds complicated but in Okay, it it sounds complicated but in reality like we will not manage this

[4:12:18] process manually of course. So this is the beauty of TensorFlow and PyTorch. the beauty of TensorFlow and PyTorch. This is why they exist because they This is why they exist because they manage this network propagation for us.

[4:12:32] They manage all of this for us. So that's really why they were created was to make this so much easier of a process to manage the flow of data through this network whether it's forward or backward with the gradients and all that. It's

[4:12:46] complicated. I agree it is it is complex like training a neural net is not like training a neural net is not trivial. Uh it is a complex process but the um frameworks that we're going to learn with in Python the TensorFlow and

[4:12:59] PyTorch make this relatively easy. They they make this relatively easy but that's the beauty of them is they do make it easier on on us but I think it helps to understand these steps forward propagation

[4:13:14] figure out how bad you're doing with the loss function compute your weight adjustment which is based on the derivative of that loss that's the derivative of that loss that's the gradient and then you propagate that

[4:13:26] gradient and then you propagate that back uh from back to front because earlier weight adjustments ments depend on later weight adjustments. So you have it makes sense. You cannot do earlier ones without knowing the later ones. So

[4:13:39] it you really have to go back to front. You have no other choice because you can't just jump straight to the front and update these guys. You would in this. In order to know this, you need to know this. In order to know this, you

[4:13:52] need to know this. So you have to start from the back and go to the front. You really have no other choice to do the adjustments. Okay? And again, that's why I said this stuff is not trivial. It's definitely advanced. Um, but really glad

[4:14:07] you guys are uh, you know, trying to learn it because I think it's I think it's well worth it to understand how to work with neural nets. As I said, one of the preeminent examples of neural nets are like generative AI. Neural nets are

[4:14:20] all over the place. So, it'll help to understand, you know, that how those cover those later on, things like transformers. Okay. So to recap that

[4:14:32] process, um you know what we're doing here is everything I just said. We're uh basically setting up our network with the initial weights. So when you first build your network, you're going to have random weights and we saw that in our

[4:14:47] code in that demo. Everything has random weights. The goal is to learn what those weights should be over the course of training. So in order to um in order to do that we're going to do forward propagation which is what uh this first

[4:15:01] propagation which is what uh this first step is. So this is forward prop step is. So this is forward prop because we are producing the output. Um calculate the output just means forward propagation right send your data

[4:15:13] through generate an output. Um then you do your update on the weights which is through uh back propagation right. So back prop because uh we're going to compute a weight adjustment. Now this is saying

[4:15:28] that the weight adjustment that you should make. We're going to learn this a little more. The weight adjustment that you should make is based on the that you should make is based on the previous weight plus this amount here.

[4:15:41] And this this is just saying it's proportional to your error or your loss proportional to your error or your loss cuz this is your loss in a simple uh in a simple simplified way, right? It's saying here's what your label is, which

[4:15:56] is D. Here's what your predicted output is. Uh here's what the predicted output is. Uh here's what the predicted output is Y. And uh the gradient of this is typically proportional to the input X. And then there's also some learning rate

[4:16:10] that's involved. And this is this is just the gradient descent formula. That's all that this is. So we'll learn a little bit more about that coming up uh especially in lesson four, maybe in less the end of lesson three. Um but uh

[4:16:25] this is the gradient descent formula. All it's saying is this is how much I what it's proportional to. It's the error. It's really proportional to that error. Not only the error but the derivative of that error which again we

[4:16:39] don't have to compute manually. Pytor or tensorflow would compute that for us. They will keep track of that and compute what that adjustment should be. But under the hood we should realize it's coming from this gradient descent

[4:16:51] algorithm which is basically saying in order to make an adjustment I need to take some proportion of my error in particular the derivative of the error. Okay. And then we iterate. So essentially we iterate these steps over

[4:17:04] and over again. So iterate these steps over and over again until you've completed all your steps which is basically convergence. Maybe you do 10 steps, maybe do 100, maybe do a thousand. Um that's actually

[4:17:18] configurable is how many steps do you want to take and um we'll we'll um see sometimes we need to take less. Depends on the situation. Um but we continue to do that and every time we do it we're adjusting these weights be and the

[4:17:33] these weights should be a little bit better. So our error should hopefully decrease. One of the things we should look for is the error should be steadily decreasing. That's what should happen, right? Because we're making an

[4:17:48] adjustment. As long as we're making a a solid adjustment, which we should be because we're doing gradient descent, we should be getting better. So the next time we do this and do forward prop and then measure our error, it should be

[4:18:01] better and we should have less error and therefore we need to make less of an adjustment. So over the course of the training, we should be getting better, we've done all of our steps, we're really good. Our network is performing

[4:18:16] even better than it did at the beginning, obviously. What is T? Uh so T is not really a true variable. it t is just uh it's like a step. So it's like an iteration count. So all this is saying is like if I want to update the i

[4:18:31] saying is like if I want to update the i weight for the next step. So t+1 is just saying like on the next iteration um you just take the the weight at the previous just take the the weight at the previous iteration. So what it currently is t. So

[4:18:44] like time t plus this adjustment. That's all it's saying is you take you take all it's saying is you take you take what it currently is at this time and just saying in order to update the weight you take what it currently is and

[4:18:59] add this adjustment. So think think of t like a step an iteration. So the weight at iteration. So if you were to plug in like the beginning like t equals 0. This is saying the weight on the first iter. So like on the first iteration t equals

[4:19:14] z. If I want to make an adjustment, um, the weight will now become what it was at when I started. So, the weight at at t0 plus this adjustment. And this this

[4:19:27] formula is going to be different. By the way, it's not always going to be that, but it depends on how complicated your network is, but that this generally is the formula. Yes. T is iteration. Yes. So the the by the way the back

[4:19:40] So the the by the way the back propagation is this step here um that I I wrote down. This is the back propagation. Um and and again that's it's called that because this formula cannot be applied for earlier weights

[4:19:55] cannot be applied for earlier weights unless I know the the value of um later weights because generally this is this formula will involve weight updates of

[4:20:07] of weights that are further down in the network. So for anything earlier, it's going to require knowing what the weights are in the in particular the derivatives of the weights for um later adjustments. That's just it's just a

[4:20:21] calculus fact that is it basically known as the chain rule. So if you're curious more about that, you could look up um back propagation and chain rule and see uh if you want some more like math intuition behind

[4:20:35] that. But that that's the reason the chain rule from calculus why we have to do back propagation. So I wanted to you know take a look at what the error like a typical error function might look like in terms of uh the some of the square

[4:20:52] differences. So basically like the mean squared error uh over the course different weight adjustments. So obviously what we're looking for is to obviously what we're looking for is to minimize that that loss. And the goal is

[4:21:06] to make weight adjustments enough so we end up down here at these kind of minimum loss values. Um we would love to end up here or here. We would love to make enough weight adjustments to end up there. What makes this tricky is that um

[4:21:23] in terms of like a minimization, you can that the gradient descent can can kind of get tricked in some cases to believing it has found a minimum in believing it has found a minimum in these kind of valley spots. So like here

[4:21:38] it kind of looks like a minimum, right? Even here and certainly even here it looks like we've bottomed out in terms of the error. And so what can happen is

[4:21:50] you can think that you've bottomed out and you can basically get stuck. You can get stuck here in these error positions where your error you believe you've you've hit a minimum. And so um your weight adjustment is not going to be

[4:22:04] that much. you're basically going to be flat and and at like an equilibrium and you're not going to be able to make a weight adjustment and climb out of that uh to to climb out of it and hopefully find a better minimum. So like ideally

[4:22:18] you're finding a really low minimum of of the error. Um and that's what you want, but you can get unlucky in certain um situations where you can kind of get stuck. And we're actually going to talk about things that help prevent this. So

[4:22:33] there are um certain algorithms that are out there that have been around for a while that that basically propel you beyond these uh valleys and and help you search for the lowest minimum you can in the error. Um because what you don't

[4:22:49] want to happen is again you don't want to get stuck here. Um in which case you haven't truly minimized your error. you think you have because when you look around, this is the classic like local versus global problem is when you look

[4:23:03] around from this perspective, it looks like you've hit a valley. It looks like you've hit a minimum. Um because everything around you is increasing. So everything around you is increasing. So it looks like you've hit a a valley and

[4:23:15] you can't get any better. Um but truly there's, you know, more of a minimum down here or down here. So where you really want to end up is one of these guys. Um, and so the training, luckily the training algorithm that we're going

[4:23:29] to use is going to help propel us to these minimums of the of the loss and these minimums of the of the loss and and help us avoid getting stuck in these and help us avoid getting stuck in these uh kind of fake minimums. Okay. So from

[4:23:42] a perceptron perspective uh computing the error is actually really easy if if we just take the difference because what you can see is that like if you imagine what our prediction is our prediction is going to be 0 or a one and uh the the

[4:23:57] label is going to be a zero or a one in terms of a binary classification. So we could always compare the uh we can compute the error relatively easily with just a simple difference. So we could say like, okay, here's our prediction um

[4:24:11] which is one and here's the actual label which is one. So we had no error. It's kind of like a true positive, right? We got that one right. So there's no error. Um but we here we predicted one and it was actually a zero. So that's an error.

[4:24:25] was actually a zero. So that's an error. So that's an error of one. Uh so so that contributes an error of one if we take the difference there. Now that's a very simple error function. In a lot of cases, um, we actually want something

[4:24:37] better than that, a more sophisticated error function because in a lot of problems, we're not going to produce just a 01 will actually produce a probability. So that's where things like the entropy will come into play um, in a

[4:24:51] little bit more advanced of an error error calculation. But you know you could think like you could think of this as one simple way to tally up that error predicting uh this is kind of like a false uh

[4:25:04] negative. So actually sorry this is like a false positive. We predicted one but actually was zero. That's an error. Okay. So this is just going through everything I said and once we know that error so that's an example error

[4:25:17] calculation. Of course, the true like loss functions that we we will use will typically be more advanced than that. They'll typically be like uh maybe a mean squared error and a regression, but in a classification, it typically will

[4:25:30] be something else. The reason being is because those lend themselves better to working with probabilities, which is generally what we'll produce for most neural networks for classification is a probability rather than just a zero or a

[4:25:43] probability rather than just a zero or a one. But once we compute that error, now this is talking all about back propagation, which is just the term we're giving to going back and traversing back to front and making all

[4:25:58] of our weight adjustments to all of all the weights throughout the network. And and ideally we um get better at this as we go along. That's what I was saying is we make adjustments and then we iterate, make

[4:26:12] iterate and we should be getting better each time. Our loss should be reducing over the course of these iterations, right? So everything you know everything we've been saying just kind of summarized here. Okay, so the weight

[4:26:27] adjustments are according to formulas like this. Now these uh the these um formulas are coming from gradient descent. So they're they're not out of thin air. Um these formulas are relative to your loss function and really like

[4:26:43] the derivative of the loss function. Um so they're not always going to be this so they're not always going to be this simple. In fact, they're they're usually simple. In fact, they're they're usually um more complicated, but um generally

[4:26:56] the formula is something along the lines of your new weight is equal to your old weight um plus some learning rate times the times the um gradient

[4:27:18] function. So this is this is more generally the formula. It's just when you're using a simple loss function um it it it turns out to just be this is the gradient just this quantity. But

[4:27:31] we're going to learn more generally that it's more of an update like this uh it's more of an update like this uh generally um is is the gradient descent. generally um is is the gradient descent. One thing to to realize is instead of

[4:27:44] making an adjustment 100% related to the gradient, there's actually a what's called a learning rate control over. It's actually a hyperparameter. A and there what this

[4:27:59] hyperparameter. A and there what this thing is doing is preventing us from thing is doing is preventing us from taking way too large of a step. So when we make this adjustment, we don't want to make an adjustment that's completely

[4:28:13] governed by the gradient of the loss. I'll I'll tell you why. Like the intuition behind that is mainly due to this picture. Let me draw a picture to kind of explain why. And this actually basically explains

[4:28:26] gradient descent is this picture. So imagine imagine um this is our loss function, right? And we're trying to reach the minimum of the loss. So we're trying to we're trying to get down here and what we can evaluate on every

[4:28:41] this loss function. Right? We know from forward propagation. We can compare that to our labels and we can compute where we currently are. Let's say it's right We know we're somewhere on this loss function graph. Let's say it's right

[4:28:55] function graph. Let's say it's right here. Our goal is to move towards the minimum. We want to make weight adjustments that move us towards the adjustments that move us towards the bottom. Right now imagine now let me

[4:29:07] turn this to an analogy of hiking. Imagine we were working our way down a mountain. We want to reach the bottom of that mountain. One of the things we want that mountain. One of the things we want to avoid is taking a gigantic step,

[4:29:19] right? And we risk kind of like tripping and um maybe traveling way too far. you know, if we if we go too far in any one direction, what can happen? And if going

[4:29:31] back to this graph, um if we take too big of a step, we can actually um end up big of a step, we can actually um end up over here, which would be bad. So, the idea is imagine working our way down a mountain kind of in the dark. Let's take

[4:29:47] mountain kind of in the dark. Let's take small steps so that we guarantee we don't overshoot. Cuz if we take huge steps, we might overshoot. You might come all the way over here and and overshoot. So let's take small

[4:30:01] proportions of that step. So that's what this that's what this learning rate is. It's a fraction that controls taking a small step towards a minimum. And by the way, this is why it's called descent is we want to make an adjustment to our

[4:30:18] weights so that we descend down this loss function landscape and end up at the minimum of the loss function. But we don't want to take such a huge step that

[4:30:30] we just skip over the minimum entirely. So the learning rate controls how big of a step do we take and usually you want that to be not incredibly tiny

[4:30:42] but not too big cuz it the tin does it make sense that like the tinier it is down the mountain. It's going to take us forever if it's a really tiny uh proportion, right? We're going to take baby steps and barely make any progress.

[4:30:58] We don't want to do that. But at the same time, we don't want to risk taking way too big of a step and then jumping over the minimum and then and then we're we haven't minimized the the loss. So gradient descent a basically is

[4:31:13] governing we should take a step towards the bottom. And by the way, what's the the bottom. And by the way, what's the bottom? It's in a decreasing slope. So um and that's the gradient. The gradient

[4:31:25] loss function. So, we want to take a step in the direction of the gradient of this loss. That's why it's called gradient descent. Okay? And again, we gradient descent. Okay? And again, we don't want to take way too big of a step

[4:31:38] cuz then we'll jump over that minimum. So, we're going to take gradual kind of steps governed by this uh learning rate. Um, by the way, a typical value of the Um, by the way, a typical value of the learning rate is something like 0.01 or

[4:31:51] 0.001. But, and that seems small, but it's really not. Um that's this has been proven to be a pretty good value uh for the learning rate for most most examples. This is these have been proven

[4:32:05] to be pretty good um fractions of the of the error um so that your your step is a meaningful step but it doesn't risk going way too big. Okay. So this is this is gradient descent in a nutshell is it's it's a gradient descent is an

[4:32:20] optimization procedure. It's an optimization algorithm that says in order to find our way towards the bottom of a loss function, which is what we towards the bottom of the error. In order to do that, let's let's take a

[4:32:34] step in the direction of the gradient of the loss. That's because the gradient is the slope of this guy. So, no matter what side we're on, we want to take a that's going to bring us towards the bottom of the mountain, right? And we

[4:32:51] want to control that step by some amount so we don't overshoot it. We want to take a controlled step in that direction. All right, I know this might be a little complex, but hopefully um that helps a little bit on gradient

[4:33:04] descent and kind of what it's doing and why the formula is like this. The formula is like this again because we're taking a step. We're we're updating our weight in the direction of the gradient. This here is just a it's um the gradient

[4:33:21] of the loss. That's all it is. More generally, it's the gradient of the loss function. So, we're taking how far off we are and using kind of the rate of change of that to figure out what direction we should adjust our our

[4:33:37] weight or how much we should adjust our weight. Okay. So I have a an example of back propagation um that I will go through. It's kind of an in-depth through. It's kind of an in-depth example. So what I think we'll do is

[4:33:51] take uh a little bit longer of a break now and then we'll come back and go through an example of back propagation just so you can see kind of the flow of the data going back through and see truly why we have to go back to front um

[4:34:06] adjustments uh according to that gradient descent formula. Um so we'll do that. Okay. So I just wanted to walk through a back propagation example to through a back propagation example to mainly show you how um the weights get

[4:34:21] adjusted uh back to front um through a simple example. There's now as a disclaimer there's going to be quite a bit of math involved with it and I'm not too concerned with the math but more the

[4:34:34] process just so you can see how there's kind of a gradient descent update that's relies on weights later on in the network to update weights that are earlier in the network which really is what the back propagation is all about

[4:34:48] computing those later ones then using that to update those earlier ones. Uh but the goal of back propagation is to have all of our weights be updated have all of our weights be updated according to our uh adjustment formula

[4:35:00] and so that uh that's how we learn right that's how the neural network learns and trains is it it makes those adjustments uh each iteration. So, as an example, uh we just have some kind of fake values in here just just to uh put some numbers to

[4:35:17] here just just to uh put some numbers to things so we can see uh kind of a a typical like network setup. So, this is a this is a very small network. Of course, it only has a couple of inputs and then it has uh a few a couple of uh

[4:35:33] hidden layer neurons. So it just has these two neurons right here as in the hidden layer and then it produces two outputs um which are uh basically like probabilities essentially uh in the

[4:35:47] output layer. So this would be a good example of something that's kind of like softmax here in this output layer to produce um kind of two probabilities that add up to one. But what you can see in this network structure is there are

[4:36:00] weights that have these green values associated with them. And obviously in the beginning those are random. So there's really no um special reason why

[4:36:12] starts out with kind of random values there for those weights. And as we keep saying, the goal is to adjust those weights over and over until we really learn and make uh predictions effectively with with this network

[4:36:27] effectively with with this network setup. And so the initial uh input setup. And so the initial uh input values that we have are uh 0.05 for values that we have are uh 0.05 for input one and 10 for input two. So th

[4:36:40] those are the two inputs that we have to start with. And of course we have all these other weights initialized to kind of random values that are going to be adjusted in the uh back propagation process. Um so this is this the setting

[4:36:56] just for this is just a random example. Okay. And the purpose of the example is to kind of walk through over the course of the next several slides how we would adjust all these weights. And and as I said before, one thing that I really

[4:37:09] want us to pay attention to is the fact that we cannot update this weight or this weight or this weight or this weight or this bias. We can't update any

[4:37:21] weight or this bias. We can't update any of these guys without knowing the update this, without knowing the update for this, and without knowing the update for this, and without knowing the update for this. These are required in order to

[4:37:34] this. These are required in order to update these based on the gradient calculation. They are it's required to know ones that are later down the line know ones that are later down the line in the network. So that's why back

[4:37:46] propagation goes this way because you need to know these in order to update these earlier ones and you're going to see that directly in the formulas that we're going to show in in the upcoming slides. Okay? So, that's something to be

[4:38:00] on the lookout for. Uh, again, there's going to be quite a bit of math. I wouldn't get too caught up in the math, but it's important to see the back propagation kind of in action in this example. Okay, so let's see uh let's see

[4:38:16] how we can um use back propagation to kind of make adjustments to these kind of make adjustments to these weights. Okay, so uh we have those two neurons we just said. We have biases, we have weights. They're all initialized

[4:38:28] here. And that's fine. So nothing that that uh And that's fine. So nothing that that uh special. And then we have uh two outputs uh which we see their current values of 0.01 and 0.99. So we have those two

[4:38:41] outputs. Suppose right suppose we generate those as the forward generate those as the forward propagation outputs. Okay. So the forward propagation the the forward propagation would generate um would

[4:38:54] would generate the output of each neuron throughout the layers. So for hidden layer 1, let's take a look at hidden layers one's uh input. So if we go back to the screen, H1, let's just focus our attention on

[4:39:08] this. Let's let's look at forward propagating through the network, right? H1 has contributions from uh this from both inputs. It has

[4:39:20] from uh this from both inputs. It has weight one times input one, right? and weight one times input one, right? and it has um it has weight three times input two and it has a contribution from bias one. So what we should be doing in

[4:39:35] order to compute this neuron uh is we should be doing that weighted sum through an activation. Okay, weighted sum through an activation. Now let's go back. Let's take a look at that. So going forward

[4:39:50] this is our uh this is our computation, right? It's the weighted sum. Um the the weighted sum looks like this. It's weight one time input one, weight three, input two, and then bias uh technically times one, but really it's just plus the

[4:40:03] bias, right? So if we were to compute that, we get this value as the input. that, we get this value as the input. And what we should be doing for H1 is really doing an activation to this quantity, right? This is the quantity we

[4:40:17] would be passing into an activation. um whatever activation we had there. So um for example, if that activation was relu, it would actually just leave this

[4:40:30] as it is because remember the relu is just the maximum between zero and this just the maximum between zero and this input. So uh even if we applied f as the relu, it's still just going to equal this quantity.

[4:40:45] So, so do we see how the rel would kind of function there on that weighted sum? We get 3775. Even if we applied rel to it, it would still be 3775 cuz that's that's what the maximum is. It's just that value. Okay.

[4:41:02] So, that's for H1. And uh actually the so I guess it wasn't mentioned there but the the activation that they are actually using it looks like is the sigmoid. So um just as an example so we're not actually using the

[4:41:18] rel looks like we're using the sigmoid. I spoke too soon. So it's using the I spoke too soon. So it's using the sigmoid as the activation. So uh here we get 59326 and um carrying out the same process for

[4:41:32] and um carrying out the same process for H2. So if I go back to the picture, H2, what contributes to H2? What contributes to H2 is weight 2 * input 1 and then

[4:41:47] to H2 is weight 2 * input 1 and then weight 4 * input 2 and then plus the bias, right? So those things should contribute to this. We should be doing through the activation which is the sigmoid. And that's how we produce those

[4:42:01] uh those quantities. So that's how we produce these two values. Okay. So so far just forward prop. Nothing that interesting going on besides just a weighted sum through an activation. So we're just doing forward

[4:42:16] activation. So we're just doing forward propagation at this point. Okay. So then what we're going to do is the same exact process to compute the output neuron. So if you look at the output neuron, let me go I have to keep going back to this.

[4:42:30] We're going to compute the output one. So essentially what the uh output one is So essentially what the uh output one is going to be is um taking the weighted sum, right? So we do um weighted sum of this guy from H1 input and then a

[4:42:47] weighted sum of this guy times H2's input and then plus this bias should all contribute to output one. And we should be passing that through the activation which I think is going to be another uh sigmoid to generate output one. So let's

[4:43:02] see. Yep. So there it's another sigmoid. So Yep. So there it's another sigmoid. So we have uh we have this here. So we we we have uh we have this here. So we we generate uh weight five plus weight 7 *

[4:43:14] generate uh weight five plus weight 7 * this plus the bias and we generate uh through the activation and we produce this value for output 01. Now we do the

[4:43:26] exact same thing for output two which would be again going back to the picture. So for output two what contributes to that would be should be contributes to that would be should be this amount here times H2 and then it

[4:43:41] should be um this amount here times H1 and then plus bias 2 and then uh we pass that through the activation right so those are the contributions for H2 or so sorry for output 2 and we generate a similar we generate

[4:43:57] and we generate a similar we generate this output uh here. That's all we're doing so far is just showing you an explicit example of makes sense. The weighted sum through the activations and these are what we

[4:44:12] get for those two neurons in the output. So that's what we have so far, right? That's what we have so far uh in our in our diagram. Um we have not made any forward propagation with the given inputs and uh just taking the weighted

[4:44:28] sums uh through through the activations. So we get these two values currently. So we get these two values currently. Right? Now let's compute the uh mean squared error. So we have labels on these guys that was the original values

[4:44:43] these guys that was the original values here. So these blue numbers here for 01 and O2 are the labels for what these two should be. Okay? These are the labels for what those two should be. Let's compute the error so that we can do back

[4:44:57] weights. So we should be able to do that and we're going to use a mean squared error to compute the error. So you know not

[4:45:09] before with like uh linear regression for example. So what we can do is compute the two errors individually. So we have basically uh we have error one

[4:45:22] we have basically uh we have error one is target01 minus uh output 01 squared. So we have this quantity which was the 0.01 minus this squared and we take a half of that which is this. And then error 2 is um you know we take we have

[4:45:40] um basically the same thing but for target O2 and output O2 we generate target O2 and output O2 we generate error O2. Um so we have both of those errors and we should be able to total those together to get the total error.

[4:45:54] Okay so this is just computing our loss value so far. So if we take the total of value so far. So if we take the total of that um we have a total error of 3038. So that's our total error. Okay. So error one was this amount. Error two was

[4:46:09] this amount as the mean squared error between this and 001 and then this Right? We were comparing those two values generating our error. Okay. So

[4:46:21] just computing the error. What we need to do is use this error with our back propagation and uh gradient descent formula in order to make adjustments to

[4:46:33] all these weights now that we know what that error quantity is. We're going to use it to make all the adjustments according to the gradient descent formula. Okay. So, so again, this is where we do forward and now we're going

[4:46:46] to do backward to adjust all our weights now that we know what the error is. Okay. So we're we're going through this process step by step just so you can see process step by step just so you can see it. Okay. So let's do our adjustment to

[4:46:59] it. Okay. So let's do our adjustment to do back propagation. So this is where uh we want to do our gradient descent formula. Oh there was a question sorry how it calculated error 2. Oh so error 2 is from it is doing this formula again.

[4:47:16] is from it is doing this formula again. So it's um 1/2 of the uh target So it's um 1/2 of the uh target of output 2 minus the actual output of

[4:47:28] of output 2 minus the actual output of um uh neuron O2 uh squared. So it's the mean squared error between those two guys. So remember um this had a value a label of 0.99 from the beginning and then this had a value of

[4:47:45] then this had a value of uh whatever it was 77 or I think it was 79 something like that and so we're computing this this uh mean squared error between the two and then squaring that does that make sense so we're gener

[4:47:58] it's the same formula yeah it's the same mean squared error it's just from like we it was 76 six. Sorry, it was 76 because we computed these from forward propagation. We know what those outputs should be and they originally had labels

[4:48:14] from the data. They originally had a like output one had 001, output 2 at like output one had 001, output 2 at 0.99. So we're we're updating those or going to use those to do back propagation with. Okay, so this is where

[4:48:29] kind of the math kicks in that I've talked about before is um in order to do talked about before is um in order to do the gradient descent of updating weight number five. Yeah, there was there was quite there was a lot of error. Yeah.

[4:48:43] No, definitely there was a lot of error. So, what what should make sense is we're We're going to have to make quite a bit significant error there. Um, but what I

[4:48:56] want to zoom in on now is let's let's update this weight. Okay, let's take a look at updating this weight. And when we're going back propagation, we do have we're going back propagation, we do have to start with these back uh weights. We

[4:49:10] do again we cannot start with these because mathematically we would not be able to compute those until we know what these updates are or these adjustments in terms of the gradient. So what we want to do is make an adjustment to this

[4:49:27] according to the gradient of the loss with respect to this weight. So something we need to do is compute the gradient gradient of uh the loss

[4:49:40] um with respect to um weight five. If we want to update that we need to do it with respect to weight five. So the formula for doing weight five. So the formula for doing that is here. So the the gradient which

[4:49:56] is uh the the derivative essentially which is shown by this uh quantity this which is shown by this uh quantity this delta um delta e with respect to del to delta um delta e with respect to del to to um delta 5 is um mathematically the

[4:50:14] derivative is this by what's called a chain rule of calculus. Again, I I don't think it's that important that we know all that specifically, but in order to get to it, we take the we take the derivative of the error with respect to

[4:50:27] output one times the derivative of output one with respect to uh the the um H1 essentially the hidden layer 01 and then the the H1 with respect to weight

[4:50:39] five. So, this is all like a chain rule kind of update. But the the thing to kind of update. But the the thing to notice here is that the the this

[4:50:51] quantity is what drives this is the quantity we use in gradient descent uh to to figure out how much of an adjustment should we make. It's totally adjustment should we make. It's totally related to this uh quantity here because

[4:51:06] um basically weight five is going to be adjusted um the new weight five is going to be adjusted by the old one minus the the um

[4:51:19] minus the uh learning rate times this quantity quantity the this gradient here.

[4:51:33] is. Um which is uh we're going to do that now. We're going to compute what that is. Okay. So this is all about computing the gradient uh so that we can computing the gradient uh so that we can do the back propagation. Now the in

[4:51:47] order to do that we're just going to work our way kind of backwards uh through this chain rule. So that's why it is kind of a backward propagation. We know the total error is error 1 plus error 2. Error 1 is exactly this amount

[4:52:03] error 2. Error 1 is exactly this amount and so uh the output 01 is exactly given and so uh the output 01 is exactly given by this uh um activation. So that's that by this uh um activation. So that's that and then the this is this is computed

[4:52:16] this this is computed by this weighted sum here. So hopefully it makes sense that like taking the update with respect to five actually involves an update with respect to another weight because every weight contributes to computing the

[4:52:33] outputs of those neurons. So that's why you cannot do earlier ones without doing later ones because they all depend on each other as inputs flowing throughout from layer to layer. Right? So this is just unraveling the formula um a bit on

[4:52:49] what this error actually is so we can take its gradient. Um, again, not too concerned with us knowing all that math. Um, but mainly just wanted us to see how th those different weights enter into the picture, right? Because those we

[4:53:05] know from we just know from the way the network is structured. Um, in order to compute like in order to compute anything that involves um this guy, it sorry,

[4:53:21] mouse is just going crazy. Um, anything that involves this guy um is going to involve contributions from this and contributions from this. So you know in

[4:53:33] order to in order to figure out what the gradient is with respect to this um there's going to be contributions from multiple areas of the network and that actually flows like if I want to know what the gradient is with respect to

[4:53:47] this um there's you know contributions from uh uh contributions at this level from this weight um and ultimately contributions that I'm going to need

[4:53:59] from these and these later later on in the network to kind of um figure out what is coming out of H1 or H2. So that's why back propagation has to go

[4:54:11] this way. It has to mathematically it has to cuz as I keep saying you cannot update these without knowing what the rate of change of these are. Just mathematically you can't do it and it's because of these uh gradients.

[4:54:27] Okay. So unraveling all of this, we just take we just take uh we're just filling out this formula for our gradient descent update. Now something I should keep in mind is will we have to do this manually each time? Absolutely not. This

[4:54:43] is this is something that um TensorFlow and PyTorch will manage for us. So that's one of the beautiful things about those packages is it knows how to compute these gradient descent updates based on the graph based on the network

[4:54:57] graph. It can do all these calculations. But this is just to show you everything that's involved so we can gain an appreciation for basically everything that goes into doing uh doing a back propagation update which is a lot.

[4:55:12] There's a lot that's involved there which is why it usually takes time to train a neural net because these calculations are not trivial but the the network framework like PyTorch or TensorFlow manages all that for us.

[4:55:26] TensorFlow manages all that for us. Okay. So plugging in everything um our error is this which makes sense. It's 1/2 of this distance plus another half of this distance. So it's just the sum of both of those. Um, so in order to

[4:55:39] calculate our derivative, um, we're going to do piece by piece by piece, we're going to compute this derivative, this derivative, and then this derivative. So we're just doing all of those pieces. So if you take this error

[4:55:53] and do with respect to the output one, it comes out to be this quantity. Um so we can plug in all those values and compute this first then we can compute this then we can compute this which is going to have those contributions.

[4:56:09] going to have those contributions. So uh we have um so then we can compute this piece. So we've done this we can compute this piece next which is uh the

[4:56:21] output is this quantity um from the softmax or sorry from the sigmoid. So the derivative of it with respect to the h1 uh we can easily compute which is

[4:56:33] going to be uh this with the exponential. So it just equals this. So it's this amount. So this is all derivative calculations. Again do we need to know these derivatives? No. Tensorflow and pytorch will manage it

[4:56:46] for us. But just showing you how that's broken down. So this is this quantity. from the contributions from both of those weights um from from that part of

[4:57:01] those weights um from from that part of the network. So this uh quantity um we put everything together. So like this this uh is how we get H1. We know it's that weighted sum that goes through activation. how we uh take the rate of

[4:57:16] change of that with respect to weight five is that's actually just this coefficient um which is just this this amount. We know that we've computed that earlier. We have that value and so we can multiply all of those together and

[4:57:30] can multiply all of those together and we get this as our total rate of change for the error with respect to weight five is this amount. So what that means

[4:57:42] five is this amount. So what that means is this should be used as our uh in our is this should be used as our uh in our gradient descent update

[4:57:55] of weight five. So that's going to be an important part of our gradient descent important part of our gradient descent adjustment for uh weight five. So we can compute that if we and now this is the formula I just wrote down a little bit

[4:58:09] earlier is we take the current value of weight five and we adjust it by doing the learning rate times that gradient we just computed. So the original value of

[4:58:22] the weight was 04 and we subtract off this amount. Our learning rate is a little bit high. It's 0.5. Typically it won't be that high. Usually be a fraction like 0.01. 1. But so that's a pretty significant um

[4:58:36] learning rate. But just as an example in the calculation um it just takes half of this. And so our new weight is down here. Now that's so this is the gradient

[4:58:48] descent formula. This is a gradient descent formula for adjusting that weight. It'll be a similar formula for weight six, weight seven, weight 8, which would be um weight six would be whatever it used to be minus the

[4:59:04] learning rate times the gradient of the of the loss. You see why the loss is so important? It's the gradient of the loss with respect to weight six this time. be slightly different calculations, but um we could go through all of those and

[4:59:20] figure out what the adjustments are for all of those weights. So in reality, the all of those weights. So in reality, the back propagation is going to start with the network. So it's going to start with all of these guys being adjusted

[4:59:34] according to that gradient descent formula. Okay, all of those guys getting adjusted. And um once we do that now we can start propagating forward or I should say backwards and updating all of

[4:59:50] these guys. Should be able to do that. So we can compute the um we can compute the weight adjustment for weight one weight two uh weight three weight four. We should be able to do all of those. You can see

[5:00:05] that it's going to be a similar formula. That's going to be the gradient of the error with respect to weight one. Now what's really interesting about this is again these are actually going to utilize the the weight adjustments that

[5:00:20] utilize the the weight adjustments that we made in the later layers. So like the this formula is going to involve an adjustment we already computed according adjustment we already computed according to according to um output h1. we've

[5:00:32] already actually done that computation. Um so we can kind of reuse it which under the hood when we use these frameworks they are going to as much as they can efficiently reuse calculations in the back propagation. They're going

[5:00:47] to reuse those adjustments that they've already been computing in order to work our way back and make all of these weight updates. We ultimately want to update every single weight. So again, we're going to start towards the end and

[5:01:01] work our way backward to adjust all these weights because those adjustments are going to reuse calculations we've already done. Okay. Okay. So um we are

[5:01:14] gradient of the error with respect to output one, it's actually the gradient output one, it's actually the gradient of error 01 um uh times the gradient of O2 um with respect to output one. And uh we

[5:01:29] can see that there's um we're going to be reusing uh we're going to be reusing weight five. It turns out that this calculation um here actually just equals weight five, which we've already computed. This

[5:01:47] we've already computed this as our weight five adjustment. This is why it's weight five adjustment. This is why it's so critical to, you know, update um it's so critical to to figure out what those updates are and then use them to update

[5:02:02] earlier weights. So if you do the math on this um gradient, it actually just evens out to weight five by itself because that's this coefficient attached to this. So the derivative of it is just this. Um,

[5:02:18] and so we're just reusing stuff we've already calculated, which is great. back propagation. It tries to reuse stuff if it's already it's already been computed um later down in the network. So we've already we just did weight

[5:02:32] five. So we already know what that should be. Okay. So plugging everything should be. Okay. So plugging everything in uh we get um the the weight five in uh we get um the the weight five value and then we uh compute just a

[5:02:44] bunch of gradients. uh continuing along like the gradient of error O2 with respect to output h1. And this gives us our total which is just this quantity uh here. So that's just one piece of the puzzle. In order to update weight one is

[5:02:59] In order to update weight one, we need this quantity here. Okay. So we just computed that guy. So this this part is done. And then we just need to go through and compute these two um which shouldn't be too bad. So

[5:03:14] they're going to look similar to before. Um we can do the gradient. So this this guy is particularly the sigmoid which is this. So it shouldn't be too bad to compute its gradient. Um and then this is the weighted sum which shouldn't be

[5:03:29] too bad to compute its gradient. Um and then we can make therefore our total then we can make therefore our total adjustment for weight one is this. Um which is actually very very small. It's a very very small adjustment or our loss

[5:03:44] for weight one or I should say derivative of loss with respect to weight one is very very small. So it's going to be a relatively small adjustment that's going to be made to update weight one. Okay. So weight one

[5:03:57] can be updated. So uh we just go through and now the hopefully you're seeing some patterns emerge here in the sense of look at this formula. It's our usual gradient descent formula we've been using this whole time. It's the weight

[5:04:10] one minus the learning rate times this loss function gradient with respect to that weight. So it turns out to be a that weight. So it turns out to be a very very small adjustment because this

[5:04:23] is a very small number. So 0.15 minus this really small number is still pretty close to 0.15. So weight one actually didn't change that much. Um weight five did. Weight 5 6 7 8 actually did change a good amount. weight one actually

[5:04:39] doesn't change too much. It didn't get adjusted by that much. And similarly, so gradient calculations and update weight two, weight three, weight four all in the same way. Okay. So we can make all those adjustments all in the same way.

[5:04:56] All right. So what we have so the takeaway from this is not the math. I about the gradient and all that because framework that they're going to manage all that math for us and we don't have

[5:05:09] to worry about that at all. What I really wanted us to see is this aspect really wanted us to see is this aspect this aspect of back propagation where in order to update these guys we really rely on updates that we've made

[5:05:22] rely on updates that we've made previously right from the from the later on layers down in the network. We try to reuse as much of those calculations as reuse as much of those calculations as possible and we will mainly because of

[5:05:34] possible and we will mainly because of how the the calculus chain rule works. It requires us to reuse those calculations in order to update these calculations in order to update these guys. So um that's why we start

[5:05:46] backwards and work our way forward. It's so much more efficient because in order to calculate this, we would need to know this calculation anyway. But we're going to do that calculation as part of the updates later down in the network. So

[5:06:02] what's great about the frameworks is they kind of store that information and reuse it in all these gradient calculations. That's one of the beautiful things about PyTorch and TensorFlow is they are so effective at

[5:06:16] these gradient calc that's what makes that's actually one of the fundamental aspects of those libraries is being able to manage back propagation. they do it really really well all these weight adjustments of these networks. Okay.

[5:06:30] Okay. So let me talk about a couple of things that can go wrong during the back propagation. So one of the things we saw is that we're computing these is that we're computing these derivatives right and what can happen is

[5:06:45] you can get these derivatives that can be very very small and close to zero. So and that can be an issue. So like when you have let me go back to

[5:06:57] this like this is a good example of a derivative a gradient of the error or the loss with respect to a weight that is pretty small. This is a very small is pretty small. This is a very small amount. And so what that's saying is I'm

[5:07:12] going to adjust my weight by a very tiny minuscule amount. And that can make minuscule amount. And that can make training slow down significantly because you're not making that big of an adjustment. And so the next time around,

[5:07:26] you're not you're probably not going to make that different of a prediction and you're just going to get stuck in making very small progress because your gradient's really small. Now, one of the things that contributed to this really

[5:07:39] small gradient in this example, I'll I'll tell you the main culprit. The main culprit of producing these really small values is this. The fact that we were

[5:07:51] using a sigmoid as our activation, that's the sigmoid is purposely going to squash things into fractions, which is going to make our update values pretty

[5:08:03] small when we do these because look at what we're multiplying together. We're multiplying this times this times this and we just get a tiny fraction. So again, this um sigmoid being there in kind of our hidden layer is not this was

[5:08:19] just a dummy example, but it really causes this issue with these small causes this issue with these small gradients and um small gradients can be a a big problem because they can slow down our training quite a bit. So

[5:08:35] whenever you're using gradient descent, which every neural net training uh does, that's something we have to be on the lookout for is really tiny uh gradients. And one of the things you can, you know, like we'll be able to visualize this

[5:08:50] during our training progress is that you would you would see your loss basically stall out. So, one of the things you would see is in an equivalent graph, if you were to graph your loss over the course of your iterations,

[5:09:05] um you would see like a stalling out essentially because your loss isn't going to change that much if your gradient is really tiny, right? Because you're you're not changing your weights by that much. So, you're pretty much

[5:09:18] going to get stuck there. So, this is very um indicative of vanishing very um indicative of vanishing gradients. Um, so, so vanishing gradient just means really tiny descent update and it it causes you to get stuck. Now,

[5:09:32] we're going to have ways to combat this. Um, I'll talk about those later as we architect our network. One of the ways to combat this is to stay away from sigmoid and thing and like softmax in the hidden layers of your network. You

[5:09:47] don't want to make your values smaller in the in a small range on purpose. Um, so you so that can be a huge contribution to that. So that'll be one way to combat against it. Um, we'll see some others as we go along, but that's

[5:10:02] just one issue that can happen during the course of training is those gradients can can become tiny which lead to very little if any weight updates and you're kind of stuck. So we want to avoid that. Now on the reverse side, you

[5:10:16] can have gradients that blow up and get out of hand. And when that happens, you basically have weight updates that are humongous. And you'll notice that too with your loss. Because what'll happen with the loss is maybe initially it

[5:10:32] starts decreasing, which is a good sign. That means it's learning. It's getting better. But then if your gradients are getting really huge and you're making bad updates to your weights, it can start to explode.

[5:10:44] So exploding gradients is also a bad uh a bad thing. And usually what we see a bad thing. And usually what we see there is uh the the uh you get really

[5:10:56] large weight updates which creates a lot of instability in the training. So you'll usually see that in in your loss just becoming exponentially big because your weight adjustments are just so big. You're you're getting a really huge

[5:11:12] gradient update. You're adjusting your weight so much and you're just you start weight so much and you're just you start to explode in your in your error. Um so that can be bad um obviously and so uh we want a way to control that as well.

[5:11:26] Luckily we have ways to mitigate against both of these um primarily in the use to train our network. We'll actually have a really effective way to kind of manage both of these. Uh I'll talk about that probably in lesson four when we get

[5:11:42] to it. Okay. So let's talk about gradient descent. Um so this is the gradient descent. Um so this is the algorithm that is used to minimize our algorithm that is used to minimize our loss function particularly to to tell us

[5:11:55] how much we should adjust our weights in order to minimize that loss function and and it works by deriving this kind of formula right which says I should be taking a step in the direction of the gradient um of the loss function in

[5:12:11] order to minimize that loss function. Um, so we've kind of seen that and the picture that I think makes the most sense for descent is thinking about our loss function as kind of a parabola like this. And we're trying to work our way

[5:12:26] towards the minimum of it. And we do that by taking steps in the direction of the slope, right? The descent of the slope. So we're taking steps this way. um that slope is represented by the gradient and we're taking our steps down

[5:12:41] towards the minimum hopefully reaching that minimum. Now one of the issues that can happen with gradient descent is that it the loss function is not a perfect parabola. We would love it to be that way um but it's not always that way. um

[5:12:58] it can be jagged and kind of have multiple um multiple peaks and valleys talked about earlier where you can kind of get stuck thinking that you've reached a minimum but you haven't reached the true minimum of the loss. So

[5:13:14] your model is not as good as it could be. That's something that can happen with um gradient descent especially if you have a complex loss function. You going to we'll talk about how to kind of mitigate against that as well. Okay, so

[5:13:29] let's talk about gradiented descent um in the context of linear regression. So It's just finding a simple model like this. Maybe it only has one feature. So it only has one weight and a bias. This would be most basic linear regression

[5:13:42] there is. Um and we're trying to find the best fit model, right? The best we're trying to find out what these parameters should be, the the M and the C. uh in order to get that best fit line through our data points, right? That was

[5:13:57] linear regression. Um so linear regression, you know, is um used to to train linear regression actually behind the scenes because there you can

[5:14:11] iteratively go through and find out what those weights should be through a gradient descent update on those weights in the exact same way we just did gradient descent for the neural net. It's just not as complicated because

[5:14:25] there's only a couple of parameters, the the slope and the intercept in this case. And um I don't need to do back propagation to update them because it's propagation to update them because it's just a simple derivative. Um so it's

[5:14:38] actually really easy to do that. Um, so if our loss function is MSE, the mean squared error, it's actually really easy to find the the minimum of that loss

[5:14:50] function by taking steps down this parabola. So exactly the picture I had before, we are going to descend and take steps towards the minimum until we reach the minimum and find the best parameters

[5:15:05] that minimize this loss function. Okay, the mean squared error. Okay, so that's that's what it's doing, right? It's fitting it's trying to find the best betas or the C and the M. Uh, and that's actually exactly the same thing that

[5:15:21] we're doing with neural nets. We're trying to find the best weights that minimize this loss function. And we're going to take steps [snorts] towards the bottom by taking the gradient of this loss function. Uh,

[5:15:35] because the gradient is is the slope. We want to move in the direction of that decreasing slope here to get towards the bottom. We want to take steps towards that. So this so gradient descent in a nutshell is an algorithm. It's an

[5:15:48] optimization algorithm that iteratively updates weights to minimize the loss. we should see it decreasing over the course of the training. Um and we want to move in that direction of that slope,

[5:16:00] that gradient. Um so we update the weight. We know there's a formula there that updates the weight by taking a step proportional to the gradient, which is which is really nice. That allows us to kind of make our way in steps towards

[5:16:16] the minimum of the loss. And along the way of doing that, we're making these getting better and better and better and producing less loss each time. That's producing less loss each time. That's the goal. So like I said before, the

[5:16:31] the goal. So like I said before, the issue that can happen is the loss can be a complex function um and not as straightforward as like a mean squared error. Um and so it can be difficult to find our way towards the best optimum

[5:16:46] minimum, which is a challenge with gradient descent in general is you're not necessarily guaranteed to find the lowest minimum. So you may find what looks like a good minimum um but it may not always be the optimal minimum which

[5:17:03] can be a risk. But again we're going to have ways to kind of combat that and try to find the optimal minimum as best as we can. So now think about it from the opposite perspective of finding a maximum. So if we were looking for a

[5:17:17] maximum, we would actually do what's called gradient ascent. meaning we would work our way up. Like if we were trying to reach the top of the mountain, we would take steps that slope us upward, right? We would want to go upward. So

[5:17:31] it's kind of the same like if you want to maximize a function, you actually do gradient ascent, which should go upwards towards a peak. That's what you would want to f find, right? Is a peak. If you're trying to do gradient descent,

[5:17:46] you're trying to go down because you're minimizing something, which is usually what we're doing with a loss function is minimizing that loss. But this just to call out like gradient descent actually could be used to maximize something by

[5:18:00] taking a step towards the in kind of towards the the slope um going upward. We could do that. Okay. So you can see we're kind of now the question is like we're kind of now the question is like when do we converge essentially um the

[5:18:15] convergence is once we've done enough steps. So uh now we'll be able to steps. So uh now we'll be able to actually have other um ending or stopping criteria which will say like okay yes we want to we want to converge

[5:18:28] if we reach a certain number of steps in our descent but we could also converge our descent but we could also converge if we start to see that our loss is is diverging away from uh where we want it to be. So, like if we're if our loss is

[5:18:45] decreasing at a good rate and then it starts to turn upward, maybe we want to say, "Okay, let's cut it off here cuz it looks like it's getting worse." Um, so see how we can set that up as kind of like a convergence criteria. So maybe we

[5:19:01] want to keep going, keep going, keep going, keep going, and maybe it starts to get a little worse. That's where we can kind of cut it off and stop. Or we could just reach a maximum number of steps and stop then. So, but we'll we'll

[5:19:13] which will be really nice during the training process um using TensorFlow or PyTorch. They they have uh built-in ways of kind of evaluating your progress

[5:19:25] during the training and controlling that uh optimization. Okay. So, again, in the uh in the neural network land, what we're trying to do is change our weights. Uh and usually we're using gradient descent because we want to find

[5:19:41] the minimum of a loss function. So we're we're evaluating our loss taking a step that is going to decrease it, right? Taking a step in the direction of that slope that decreases the loss function. So it's this picture. I've been using

[5:19:55] this picture quite a bit, but uh here it is again that the learning the learning rate enters the picture in the gradient descent to kind of control how much of a step should we take in the direction of the gradient. So the gradient gives us

[5:20:10] generally we should be going downwards in this direction of the gradient. But the um learning rate says well we should take a proportion of that gradient value and go you know this amount down and then take another step this amount down

[5:20:27] and keep doing it that way so that we don't risk overshooting the minimum and jumping all the way over here and then if we take too big of a step we could jump back and forth and just keep jumping back and forth over and over and

[5:20:40] over again and we'll never actually reach the minimum. So uh the learning rate is a really important parameter in gradient descent. Um it's typically chosen to be something small like 0.01 is a very uh industry standard value or

[5:20:55] even something smaller like uh 0.01. Another thing to realize about learning rate is it doesn't have to be constant during the whole training. It can actually be changed. So there are some strategies and we'll learn about this

[5:21:09] later on. There are some strategies that um basically give us a schedule over the maybe the learning rate starts off high in the initial first handful of

[5:21:22] iterations and then later on it actually slows down and the learning rate um becomes smaller and smaller and smaller. And the idea behind that is maybe initially like when your error is bound to be the highest because you have

[5:21:37] random weights um you're willing to take a larger step right so initially you're willing to take a pretty big step and you're willing to take big steps initially but then as you do enough steps you start to slow down and take

[5:21:52] much smaller steps. So there are things like that that we'll study um that it it like that that we'll study um that it it can be variable. It can be like uh um you can do like a decay an exponential decay essentially to say like okay if

[5:22:07] start decaying this so I encourage smaller steps as I've as I've taken more and more and more. We'll study that later on. All right. So just to wrap up questions on gradient descent. So, so just to recap that gradient descent is

[5:22:22] just to recap that gradient descent is our algorithm for training a neural net. It's what it's what is going to optimize our weights. The thing that does the weight adjustment is the back propagation. The thing that tells us how

[5:22:34] much we should adjust is gradient descent, right? It's what's telling us, it's giving us that formula so we know how much we should adjust that uh adjust those weights. Okay, so just to recap on the perceptron before we move on to

[5:22:49] lesson four. The perceptron is a really simple algorithm, right? It's it has a single neuron that takes a weighted sum through an activation and produces a zero or a one based on that activation. Um, so it's limited. It can only produce

[5:23:05] a zero or one. It cannot produce a probability, which is bad for a lot of situations, right? It can only produce a zero or one. So it's not going to be useful in multiclass classification for sure. Um but uh it it is kind of the

[5:23:21] most basic neural net that there is in a sense of it's modeling a single neuron taking a weighted sum through an activation. The other thing is we saw this before the perceptron only works with linear data. So as as a model the

[5:23:35] perceptron is very limited in what it can actually model. Um so whenever the classes are kind of arranged in a nonlinear separable way um you tend to

[5:23:47] produce a lot of error. Um so you really what this is signaling is you really need a more complex network of neurons right you need that network so you can learn nonlinear relationships. That's what you really need uh beyond just a

[5:24:02] basic perceptron. So like not every situation is going to line up perfectly like this which is you know where okay everything can be separated by this uh line and so everything on this side of it one class everything on this side's

[5:24:18] the other class um and essentially what this line is is this dotproduct and seeing if you're bigger than zero you're on this side if you're less than zero exactly what the perceptron is doing in terms of its activation. So, but not

[5:24:35] everything is like this, right? In fact, most data is not. And so, that's where you need a neural network with more layers and more neurons to pick up on those nonlinear patterns. So basic perceptron very limited but it what I'm

[5:24:51] hoping we got out of this lesson is it serves as the inspiration for bigger through activations and stitch together many many neurons and learn nonlinear relationships very effectively. So again, the perceptron fails to um be

[5:25:09] optimized, fails to converge. Usually, basically, you end up with an endless cycle of weight updates because you're never going to converge uh unless you have a fixed set of steps, but even then your model is not going to be very great

[5:25:22] whenever the data is not linearly separable, which is most of the time. Okay, so for this lesson um what we studied was the perceptron which is our basic building block for neural nets because it models a neuron. It takes it

[5:25:36] takes inputs, takes a weighted sum, passes it through an activation. We are going to build off of that with larger and larger and larger networks. What we're seeing is we really need that power of those big networks in order to

[5:25:50] perceptron's just not going to cut it for most problems because it's so limited. Um but it's really good inspiration uh because we want that style of

[5:26:03] computation to persist in a larger network but just with more activations, right? More nonlinear relationships. Okay. So what are we hoping to get out of lesson four? Well, we want to just expand our knowledge about perceptrons

[5:26:17] to larger and larger networks. So, we're going to talk more about the um things that we need to do to get there, including uh setting up our loss functions. Um so, see what kind of different loss functions that we'll use.

[5:26:32] squared error we know is there, but just want to review like for classification, what kind of loss functions would we use, which are normally going to be along the lines of those cross entropy

[5:26:44] ones that we've used uh in logistic regression. So, very similar. And then we'll talk about the forward and backward propagation that occurs in deep neural networks and see kind of what the difference is to uh what we've been

[5:26:58] talking about. Really no difference except now the network is bigger and so there's more more neurons at play, more activations, more gradients that have to be computed, those kind of things. So just expanding everything we've done to

[5:27:13] uh deeper networks here. And this the point of this lesson two is to prepare us for putting all this together and building our own neural nets in which will be in lesson five and six. So lesson five and six will be all about uh

[5:27:29] TensorFlow and PyTorch. So probably on Wednesday uh we'll finish up this lesson we'll build our own neural network inside of TensorFlow. We'll get a good feel for that um and and do a lot of examples there. So Wednesday will be

[5:27:44] pretty good for building our own. Okay. So talking about the uh deep networks So talking about the uh deep networks here. Um now this is just a referral to

[5:27:56] the how big the network is in terms of its hidden layers. So if it's got a lot its hidden layers. So if it's got a lot of hidden layers um that means it's a deeper network than than having very few which would kind of be a shallow

[5:28:10] which would kind of be a shallow network. Um, so the the deep neural networks um are going to have a lot more more layers. They're going to have a lot more weights. And one of the

[5:28:22] consequences of that is they're going to be more intensive to train because there's going to be a lot more weights to learn the more hidden layers that we have. Um, where things are connected together in this network structure, we

[5:28:36] have a lot more weights to to deal with and a lot more weights to update. Okay. So on the flip side of that though, the deeper the network, the more powerful it can potentially learn patterns. So um you know we we want things to be

[5:28:50] reasonably deep for the problem at hand. So like when we get into computer vision for example um those are going to require somewhat sophisticated deep networks because learning how to do things like object detection or image

[5:29:05] recognition are difficult problems. Um, same thing with text and things like same thing with text and things like text generation and generative AI um, in transformers. Those have to be deep networks because they're representing

[5:29:18] networks because they're representing really complex uh, relationships. Um, kind of like what we see with language understanding in a GPT, right? It's they're very deep networks have lots of layers, lots of weights. As I said,

[5:29:33] modern transformers um are pushing into the hundreds of billions of weights. So, it's a really massive network uh really deep network. So, going back to computer vision, you know, um we may have we may be looking

[5:29:50] at images of different dog breeds and we may want to predict which dog it is. So if there were three different breeds that we were choosing from there may be you know three probabilities that are produced and that's an example where you

[5:30:04] know we may have a really sophisticated deep network but in the final layer of it um there's going to be three neurons and it's likely going to be a soft max output uh in terms of the activation because we want to produce three

[5:30:19] probabilities for three different breeds and we want to produce probabilities um uh generally internally that add up to one. So this this would be this is probably a typo and this this one here should be something like 0.2

[5:30:33] um sorry no.3 which is which is what it is. Sorry I read that as 2. This is a.3. is. Sorry I read that as 2. This is a.3. So this this should total up to one um which it does. So this should total up to one and they should be

[5:30:47] probabilities between 01 if we were doing that kind of example. So again, it could be a really sophisticated network, but the final layer is something like a softmax activation that's producing these probabilities for our classes. Um,

[5:31:02] and we'll see that when we get into computer vision. Um, we may have audio. So we may have a deep network that's um analyzing sound data. So like uh uh signals like waves and those could be categorized into different uh like music

[5:31:19] could be put into different genres. It could be put into different sound types. Um and again this would probably be a very sophisticated deep network and it something like this off without a deep network. So we've talked about this

[5:31:34] before. Neural nets give us this they unlock the capability to work with this unlock the capability to work with this you know advanced data like uh audio you know advanced data like uh audio images text video things like that. Um

[5:31:46] so deep networks are present in in so many areas like self-driving cars image voice recognition chat bots voice assistance online translation. I just gave an example you know of the chat bots uh where we're going to learn about

[5:32:01] that later on. the transformer network. Those are very deep networks that have enabled, you know, generative AI in the most recent years a and uh really amazing networks, but they're they're very complex they're very complex. They

[5:32:16] have a lot of layers to them. And the other thing is with some of these deep networks is they involve different types of computation than what we've studied so far. So what we see in like computer vision is convolutional

[5:32:30] layers which are advanced types of computations that are beyond just doing that is in a lot of ways that's the most basic type of computation a neural through an activation which is what we've studied to this point right but we

[5:32:47] there's going to be a lot more advanced types of calculations like in a in a types of calculations like in a in a transformer there's attention attention layers which are doing you know an advanced calculation to determine how

[5:32:59] words should be related to each other so we can understand context and meaning. we can understand context and meaning. Um, so a lot to come, but uh these are all powered by kind of deep neural nets. All right. So when we start to put

[5:33:13] All right. So when we start to put together our own deep nets, um, one of the things we're going to need to do is define our loss function so that we can train it, right? So we can, of course, we'll build the architecture of them.

[5:33:26] We'll learn how to do that in TensorFlow and PyTorch. Um, but in order to train it, we will need to specify a loss function. So, we'll need to understand how to do that. And so, we're going to go through um some examples of loss

[5:33:41] functions here. Now, we just we just wrapped up talking about loss functions. of time on what it is. We kind of know that it's measuring the error. What we want to do is find the minimum of that through gradient descent, right? uh and

[5:33:58] minimize that error um during the training process. So we've talked about that and what we're going to do is talk about different types of loss functions that we typically see in uh training of deep networks. Now one example we've

[5:34:14] deep networks. Now one example we've used before is you know you could one very simple loss function is just take the difference between um you know take probability let's say and what the label is. So in this in this example um that's

[5:34:31] just a simple difference but this is not a very good loss function for training. So we're going to learn like what a what a better loss function would be um for doing classification. So just subtracting like zero or one not the

[5:34:46] most effective. Um we're going to use something more advanced than that that lends itself to a better like gradient descent update than just taking a difference. Uh so we'll see that in uh coming up shortly. But generally we know

[5:34:59] the loss function is capturing the error uh between our model our network's output and what the true label is right that's what the loss function is doing. So generally you know the loss function can be broken down into uh the loss that

[5:35:15] we use for a regression problem versus the loss that we use for classification. looked at the regression loss for the most part. Uh but we what we're going to most part. Uh but we what we're going to focus our attention on um is mostly on

[5:35:28] classification loss because a lot of uh a lot of deep learning problems actually are more classification than they are regression. Um like object detection or predicting what word should be next when we're generating text from a LLM or

[5:35:44] something like that. um most of it is class mean squared error as a really effective regression loss function. Um so we we know all about that. We've been studying that before. So nothing really new to say there. Um again it's

[5:36:00] measuring most of the time it's measuring that distance, right? It's measuring the distance from the label to our predicted point. That's what these regression loss functions tend to do. And the ones that are most commonly used

[5:36:13] for regression um are mean squared error and mean absolute error. Now we've looked at both of these before and again they show up here in deep learning because we need some type of loss function to measure

[5:36:27] our error use that to make a gradient descent adjustment to our weights. Um but these are the two options that are primarily used in uh deep learning as well. We've seen these before with things like linear regression. Not much

[5:36:44] mean squed error or mean absolute error. So the mean absolute error as a function kind of looks like this where it's taking the average of these distances. Now the distances are computed a little bit different than um taking the square

[5:37:00] absolute value difference. So that's that's a little bit um different than doing the squared which is more of a natural kind of distance calculation there. So uh this that's one of the that's the primary difference really is

[5:37:16] doing the absolute value of those rather than the square. So we get that for our than the square. So we get that for our loss function kind of that average um distance in terms of absolute value. And so if we were doing an example, we could

[5:37:32] compute the uh average error by just taking the difference between the uh what our network says the value should be um which would be like these predicted values and what the actual value is for each of these examples.

[5:37:46] We take the absolute value of their difference and then we average those together and we get like a mean absolute error of 088 which is pretty good. That's pretty low, but you know that that'll be up to us if we were doing a

[5:37:58] regression which loss function we would want to use. We could use this. We could use mean squared error. In code, it would kind of look like this. Now, when when we um when we use this in TensorFlow or in PyTorch, it's going to

[5:38:14] be much different. We'll be able to specify that loss function as part of our training code uh when we get to that point. So, but this is just to show you like if we were to do this in numpy, um it would just be taking the absolute

[5:38:27] value and then taking the average of those uh those absolute errors. Now, the mean squared error, and this is a typo, this squared error, and this is a typo, this should be parentheses here.

[5:38:39] Not really a big deal because it's, you know, going to pretty much be the same, Now, this is the mean squared error we've been using all along. Um, so we take the squared distance between our our actual target and our predicted

[5:38:54] value from our network and this gives us our loss function which we average this across all of our data. Um, and so that's a this is a very standard loss function for regression. So this one is more widely used. Probably 99% of the

[5:39:09] time mean square error is going to be used for for a regression problem. Uh, used for for a regression problem. Uh, absolute error is rarely used. Um, the only time you would use it is if you're very certain that you have a lot of

[5:39:23] a lot of outliers and you don't want those outliers to be amplified as much. And the mean squared error, you could use mean absolute error. But for the most part, I think the mean squared error is a very common loss function for

[5:39:37] regression. So obviously what we do is we subtract, we find the distances. we've talked about this over and over again is like we take the distances here and we square them uh and then we just average those. So for that same data,

[5:39:54] our mean squared error is actually a little bit higher. Uh and that's be and that makes sense. That's because the uh errors are kind of amplified when we square them rather than just taking the single absolute value. So here like if

[5:40:09] single absolute value. So here like if we were to do this manually in code just in numpy it would kind of look like this where we take the uh actual minus predicted we square that we just uh in python remember that's star star 2 is

[5:40:22] squaring it um and then we take the average of those squared errors so that makes sense again like when we define our loss function in in our tensorflow than this this is just to kind of explain the calculation

[5:40:38] explain the calculation Okay, so you can see uh there the errors are a little bit more amplified. Now from a training perspective of the neural net, that's not actually a bad thing. Like that would actually be

[5:40:50] preferred because that's going to encourage us to make better adjustments. Um you know, potentially better adjustments on our weights. Uh we have a little bit more error to go off of in the squared case. So So that's actually

[5:41:03] fine in that situation. But you can see like if you have an outlier um the square is really going to amplify it. So this is an extreme case where we change the predicted value and so we change the actual value to something much bigger.

[5:41:18] actual value to something much bigger. So this 31.5 is likely an outlier compared to the rest of the data. So when you have an outlier label like that um your prediction can be significantly amplified in terms of its

[5:41:33] error. So this this produces just a humongous MSE whereas it doesn't affect humongous MSE whereas it doesn't affect the MAE quite as much. So again like if you know if you if you suspect there's a lot of outliers in in the data that's

[5:41:47] where the MAE may be you know better as a loss function wouldn't amplify those as much. Okay. So just kind of showing like for data without if we have data that doesn't have outliers like on the right MSE is going to be our preferred

[5:42:02] loss function for any kind of regression problem we're going to lean towards MSE. It's a very one of the reasons by the way one of the reasons mean squared error is preferred if we go back to the formula for it um this square is very

[5:42:17] desirable from a gradient perspective if you think about like because a square is like a parabola right so having this square is actually really powerful from a loss perspective so our loss function kind of looks like this it's much easier

[5:42:33] to take steps towards the minimum and and basically guarantee we only have one minimum there. So it's desirable to have that square. Okay. So again, this is why mean squared error if you're doing if you're trying to train a neural net for

[5:42:47] regression, mean squared error is going to be your preferred loss function. One of the reasons is it's so effective in back propagation for that reason that the square is really nice. It the square makes the gradients really easy. the

[5:43:02] absolute value actually is a difficult gradient to do um because it it technically doesn't have a gradient in certain places um mathematically. So um

[5:43:14] this is a much easier function to work with under the hood for computing like those gradient updates. So the squared is actually preferred in in the loss is actually preferred in in the loss function. Okay. So that leads us into uh

[5:43:30] classification. So we talked about regression. If we're doing regression to use mean squared error as our loss function. That'll make uh training it really easy uh or not I shouldn't say really easy but much better than if we

[5:43:44] did mean absolute error. Let's talk about classification and kind of um talk about what the difference is with the loss function there. So let's start with the most basic case of binary and then we can work our way up to multiclass. So

[5:43:58] with a binary classification um we are checking uh we're we're trying um we are checking uh we're we're trying to compute our error where the the classification is going to be into only one of two options right it's going to

[5:44:12] be a cancer not cancer fraud not fraud it's going to be a one or a zero and it's only one of those two options right so in a binary classification those are the only two options we have but in multiclass we know it's more complex

[5:44:28] than that. So in a multiclass there could be more than two. Um we could be example I've always used is with images like cats, dogs, giraffes. Um but it could be many many categories that we have. And what we do in multiclass

[5:44:45] classification is usually predict the probability of belonging to each class. Right? So that's where that softmax comes in in the end of the network to kind of see see what is the probability of each class. So the reason to mention

[5:45:00] this is that's going to dictate what kind of loss function we use for classification. If we're doing binary or if we're doing multiclass, um there's going to be a slightly different loss function uh for each of

[5:45:14] these. Honestly it the multiclass is just really a multi-dimensional just really a multi-dimensional extension of the binary case. So binary case is really easy and the multiclass is just an extension of that in terms of

[5:45:26] the loss function. Okay. So for classification we have two kind of main categories of loss functions and these are these are going to be really important because these are where we're when we do training of neural networks

[5:45:40] especially for classification um we are going to need to define this um in order to train it like what loss function we want to use to compute our error and

[5:45:53] then that dictates what kind of adjustments we should be making to our weight Right. So, the two options that we have are going to be cross entropy and hinge loss. We're going to actually focus our attention on cross entropy

[5:46:07] because this is the one that's most widely used in classification. Um, hinge loss is rarely used. I I haven't really seen it used too much, but we'll talk

[5:46:19] bit different, but uh we'll focus our attention mainly on cross entropy widely used for classification. All right. So, um, won't be able to finish this today because we only have about a minute left, but where we're headed with

[5:46:34] minute left, but where we're headed with this is we need some way of keeping this is we need some way of keeping track of our error when we're producing probabilities. This is this is where um we need a special function to kind of

[5:46:47] we need a special function to kind of penalize us putting probability on the wrong result. So, we know like just to wrap this up, we know that for a wrap this up, we know that for a classification, especially multiclass,

[5:47:00] it's only going to be one of these three options as the actual label. So, um it's either going to be Samsung, Apple, or LG as the category of phone. Let's say we were analyzing a picture of a phone and putting into one of these three uh uh

[5:47:16] products, Samsung, Apple, or LG. It's only going to be one of those three. But our model is going to produce a probability for each one of those. And so what we need to do is have some type of function that can compute the error

[5:47:31] if we know the actual category and we know the probability. So and what we want to do is penalize situations where we're putting our probability like a high probability on the wrong class. So let's say we put a really high

[5:47:45] probability on LG like a 0.9 and it actually was an Apple. So it was a 1 Z as the class right for Apple and we had a 0.9 we had a really low probability

[5:47:58] a 0.9 we had a really low probability for Apple like maybe we had uh 07 and then we had 0.03 and so we would want to penalize a situation like this and we need some type of function to do that.

[5:48:10] Okay. So to get an intuition for cross entropy loss now this is again going to um be a loss function to measure our error and the way we're going to capture this error on classification is to basically look at the difference in

[5:48:24] basically look at the difference in probability. our model is going to be producing a probability um more often than not especially in the very typical multiclass classification where we're predicting amongst

[5:48:37] where we're predicting amongst potentially um many categories right so potentially um many categories right so in that situation our model produces on the right it produces something like this which is uh basically a

[5:48:53] distribution right to say okay I've produced do some probability of this class, let's say it's Samsung, some probability of this class, let's say it's Apple, and then some probability of this class, which is LG. And those need

[5:49:06] to add up to one. And of course, these are all fractions between zero and one. And we generate this from our network, right? And and usually this gets generated from a softmax activation that's in the kind of final

[5:49:20] layer, right? And so this is a pretty typical situation. Maybe it's three categories. we produce three probabilities. That's pretty typical. So in order to measure our error, we want to compare this to the true

[5:49:33] distribution. The true distribution is going to be where there's a 100% probability on the actual class. So if it's like a Samsung phone then I what I should get for the true so for the for the label let's say or the ground truth

[5:49:50] whatever you want to call that the label is going to be a distribution as well but it's it's going to have ones and zeros right so like if it's actually a Samsung there should be 100% probability there and then zero for the other guys

[5:50:04] that would be a pretty typical like label distribution right would be something like that And what the cross entropy is going to do is measure kind

[5:50:16] of the difference between these two distributions almost like a distance. difference between these two and generate a penalty whenever we have our probabilities that are different than this true label. Because what would be a

[5:50:32] perfect prediction is if our our output was 100% Samsung, 0% Apple, 0% LG. That something that's very close to that might be something like where where we might be something like where where we have 90% of our probability on Samsung

[5:50:47] have 90% of our probability on Samsung and then like 5% on Apple and another 5% on LG. That might be another that might be a prediction which is pretty good. I mean that's saying we're pretty confident this is a Samsung and in in

[5:51:01] the real world whenever we want to generate a prediction we take the maximum probability. So 0.9 is by far the maximum, right? So we would we would the maximum, right? So we would we would predict Samsung for this. But this gives

[5:51:14] us a way to kind of measure our error, the cross entropy. It's going to measure mathematically the difference between this prediction in terms of probabilities and this true label distribution. That's the goal of the

[5:51:29] cross entropy. So it's it's kind of like mean squared error um in terms of measuring a distance. It's just measuring like probability distances. So, think of it that way. Okay. So, we're going to I'm going to show you the

[5:51:43] formula coming up, but that's the intuition is to is the fact that we need a way to measure how far off our probabilities are in our prediction, right? That's that's kind of the goal of the cross entropy. Okay. So, with that

[5:51:56] the cross entropy. Okay. So, with that in mind, let's go forward. So again the actual um the actual labels would look something like these guys right on the right. So these would be pretty typical for our data where we have our labels um

[5:52:11] something like this depending on which class that that uh example is. So if it's like an image and it's a Samsung phone in the image um it should be a 100% probability like we just said. If it's an apple, it should be 100% on the

[5:52:28] second uh entry. If it's an LG, it' be 100% on the third. And we're going to produce these probabilities in this uh output and kind of measure that distance output and kind of measure that distance from this to these kind of labels.

[5:52:43] That's going to be the goal is kind of measure that distance there. Okay. So, measure that distance there. Okay. So, here's an example again where we uh now in this example, we're not quite as confident. So let's say it's actually an

[5:52:56] LG. So therefore there's a 100% probability in this third slot. Zero and zero here, which makes sense. This is this is the label, right? This is the label here. Let's say in this example that was the label. But our model

[5:53:11] that was the label. But our model predicted 60% LG, 30% Apple, 10% probabilities that came out of our our prediction. Now from a from a prediction perspective, we would get it right because technically if we

[5:53:26] take the maximum of this, we would still predict LG. But what we're really after is how good is our model because that's going to dictate adjusting our weights

[5:53:38] because what we would love is to get this very confidently correct. We barely get it correct because this 6 is just enough to be more than the others. So yes, we would predict LG in a and realistically, but it's we want a true

[5:53:54] kind of mathematical distance there between these two uh like our prediction and our label just like what we do with regression, but we're doing it with our probabilities this time, right? Um hopefully that makes sense when you see

[5:54:08] hopefully that makes sense when you see it this way. And so the intuition should be that there should be a decent penalty for this because yes, we're getting it for this because yes, we're getting it right, but not confidently correct. So

[5:54:21] there should be some level of penalty for this versus if we had way more confidence in LG. Like let's say we had even more confidence like 0.99. even more confidence like 0.99. So we had that and then maybe we had 1%

[5:54:36] on Apple and then 0% on Samsung. Let's say we had that. The intuition should be say we had that. The intuition should be that this setup should be penalized much less because we are very very confident it's LG. The model's doing a pretty good

[5:54:49] job. It's 99% confident that it it's that the prediction should be LG. So it's it's put all of its probability nearly all of the probability on LG. That's a pretty good prediction. this would be desirable and this should be

[5:55:03] penalized less than a situation like this with the 6.3.1 because it's not as probability that's assigned to the other labels and it's only 60% on LG. This

[5:55:16] labels and it's only 60% on LG. This should be a higher penalty than than 0.99.01 and zero which is a pretty good prediction. It's nearly perfect right

[5:55:28] nearly perfect. So to mathematically capture this we compute the cross entropy. So in terms of a formula um it looks like this. It's actually a very simple formula which is to multiply the uh to multiply the labels times the

[5:55:47] logarithm of the probabilities uh uh for those labels. So if we have uh let's say we [clears throat] had um by the way this formula is here

[5:56:01] um and you might be wondering why is there a negative in front of all this? The the reason there's a negative there is because these uh values here are all of our probabilities, right? And so um they're

[5:56:16] one. Anytime you take a logarithm of fraction, it's actually going to be a negative number. So these are all going to be negatives. These are all going to of fractions. And so you're going to you're going to basically take the

[5:56:30] negative of a bunch of negatives, which are which is going to make a positive loss, which is which makes sense. There should be a positive error. Um kind of like a distance, right? There should be a positive distance uh there. And what

[5:56:46] you can also see from this calculation is this calculation actually simplifies is this calculation actually simplifies quite a bit. If you think about it, um what's going to happen in the uh in the label is only one of these guys is going

[5:57:02] to be one. We saw that with our LG, Apple, Samsung example, right? Only one of these guys is going to be a one. The others are going to be zero. So what you're going to see in this formula is it actually reduces down quite a bit

[5:57:16] because the the things that are zero are essentially going to be canceled out. Right? So like if if the true label was a one for the first class and then zero and then zero and then zero for all the other classes. Um that's essentially

[5:57:31] going to wipe out all of these terms, right? And so you're really only going to get a contribution um from the you're only going to get a contribution from this term from this probability which is why you would

[5:57:46] really like that probability to be high. The closer that probability is to one the logarithm of one is the logarithm of exactly one is zero. So if if if this were one as well and we had a 100% probability on this then the error would

[5:58:02] actually be zero because it would just be one time zero. But something that's really close to one like a.99 if you take the logarithm of 0.99 um that's going to be pretty close to zero a very small number. So it's going

[5:58:16] to be a small penalty if you get it way off. So let's say um let's say it's the class is one but you predicted a very low probability for pretty bad mistake, right? So let's say you only predicted a 0.1 probability for

[5:58:33] that and you had all your other probabilities on these other classes, right? And the true label was a one and then 0 0 0 etc. The issue with that is yes all of these guys are zero but if you look at this

[5:58:47] guys are zero but if you look at this term this is a one but the logarithm of this really small fraction like point if you have a low probability there is going to be a big error the logarithm of that's going to be a pretty big negative

[5:59:00] number the close the smaller it is it's a larger negative number and so um and so then you get a much bigger

[5:59:12] contribution to your error to your loss. Okay, so mathematically that's why you Okay, so mathematically that's why you have logarithm there is because you want whenever you have a small probability you want that contribution to be more um

[5:59:28] for the target, right? You you would like this to line up so that most of target and that's a very small contribution to the cross entropy. Okay, Now will we need to calculate this by ourselves? Of course not. This will be

[5:59:44] handled by TensorFlow or PyTorch. But this is just the intuition for you know why we use this formula under the hood to do our training. Um remember this is our loss function. So this is the function that's going to be used during

[5:59:58] the gradient descent training of our neural network. Um and this is particularly useful for classification. Right? we had mean squared error which is very natural for regression. Um but in this setting of classification we

[6:00:13] need to use this uh loss function cross entropy. Okay. So here's an example entropy. Okay. So here's an example calculation. If we go back to um that setup we had before where the actual label was LG and we had these

[6:00:28] label was LG and we had these probabilities where we had um 6.3 probabilities where we had um 6.3 and 0.1. Um if we were to calculate this cross entropy, it looks a lot like this. So we have um again like I said really

[6:00:41] only the contribution from this term that where the label is on the LG and so these other terms don't contribute anything because they have a zero there. But what we can see is the logarithm of 6 is going to be a negative number. And

[6:00:58] therefore when we take the negative of this negative number we get a positive error contribution which makes sense there there is some level of error there and that error is going to be more than if we had.99

[6:01:13] if we had.99 sitting here because the logarithm of.99 closer um to zero. Okay, it in case you're in case you're wondering, the logarithm um the logarithm graph kind of looks

[6:01:30] um the logarithm graph kind of looks like this. Um and it crosses this axis like this. Um and it crosses this axis right here at one. Um so.99

[6:01:45] uh.99 is going to be a very tiny value that's very close to zero. um whereas 6 is going to be somewhere down here um that's going to contribute more error. that's going to contribute more error. Okay. So um given that we've talked

[6:01:58] about cross entropy the the cross entropy actually can be thought of in two different ways. The one that we just studied is the categorical cross entropy studied is the categorical cross entropy loss um which is used for multiclass

[6:02:11] classification. There's a simpler version of it whenever you have a binary meaning you only have one probability which is for between two classes like a spam not spam or cancer not cancer. It actually reduces even further into a

[6:02:25] simpler formula. But the one we just studied is is very much related to categorical cross entropy which is uh mainly for the multiclass. So the reason it's called categorical is because it's for multiple

[6:02:40] categories. Uh so so this is the one that would be used most of the time for multiclass classification and this the binary cross entropy is going to be a reduced simpler version that's going to

[6:02:54] be uh only in the case of binary uh classification. So we only have two classes and we produce a single probability uh and the label is only going to be a zero or a one, right? There's only two options.

[6:03:08] there's, you know, we have a prob we have a a multi-class classification. So there's multiple probabilities at play. So that's going to be this one.

[6:03:23] Okay. So let's study those. So if we were to look at some various cases, getting at what we were just talking about um, earlier. So if we're looking

[6:03:35] at all of our data, so let's say we went through every single example, every picture that we had in order to compute the loss for that iteration, we're going to we're going to basically average those errors together. So that's what's

[6:03:49] categorical cross entropy is we're basically going to average all those cross entropies together. So if you look at this table, you can see some scenarios where there's much higher error in some cases. I want I

[6:04:05] want to pinpoint a couple of cases that kind of jump out to us. Um based on what we were just talking about, look at this case here where we produce a 2.3 error. That's a significant error. Why did we produce that? If you look at this uh

[6:04:21] example, the true label was apple. So there's a one here in the second slot. That's an Apple prediction, right? An Apple label. There's a one there in that second slot. But what did we predict? We would have gotten this wrong and we

[6:04:36] would have predicted Samsung, right? Because the most of the probability here for this that our model might have produced would would have been here on produced would would have been here on the 6. Um we only had a probability of

[6:04:49] 0.1 for Apple, which is a which is a mistake, right? That's a very low probability um for the actual result and so we get penalized for that pretty heavily right we get a 2.3 penalty in the cross

[6:05:03] entropy which is which is contributed from this term right in the cross entropy calculation. So that should make some sense. um we So that should make some sense. um we get a very low error right here because

[6:05:17] in this like in this situation which aligns to what we just talked about we have most of our probability is on Samsung and that is the true label. There's a one in that first slot. So this is a much smaller contribution to

[6:05:31] our error. There's still some error because it's only 90% probability. Ideally, we would have 100% probability on that and then our error would be zero, right? Because a logarithm of one is going to be zero. But we so we still

[6:05:45] have some error, but it's much smaller, right? Much smaller than the others. But right? Much smaller than the others. But all we have to do is basically uh take all we have to do is basically uh take all of these guys and average them

[6:05:59] together uh in order to produce the actual um cross entropy loss uh quantity. It's just an average. Okay, an average amongst these guys. Um, I think

[6:06:11] there's a typo here, by the way. I don't think it's 4.76. That seems kind of high. I think that's I think that's this total. And we have to divide that. We need to divide this by seven to get the actual average. Okay. Okay. So whenever

[6:06:26] actual average. Okay. Okay. So whenever we um whenever we do the binary case the we um whenever we do the binary case the formula actually reduces uh quite a bit formula actually reduces uh quite a bit um into a simple uh a simple formula

[6:06:40] here that depends on the label. So when our label is one, we get this, which is just our because remember in a binary cross entropy, you're only going to cross entropy, you're only going to produce a single probability. And what

[6:06:54] the intuition here is is the same as usual. Whenever this um whenever this probability is uh high, whenever it's close to one, that's a good result in to be pretty small because whenever that's close to one, this is a small

[6:07:10] value, right? including when it's all the way at one. If we had 100% probability, this logarithm would be zero. So we'd have no error. Um, so hopefully that makes sense. Whenever the label is zero, we take one minus our

[6:07:27] probability. And so our contribution comes from one minus our probability. Uh comes from one minus our probability. Uh so if we're predicting uh like a 60% probability and the label is actually zero, that's a big mistake. Or like if

[6:07:42] if we're predicting a 90% probability and our label was actually a zero, that's a pretty big that's a huge mistake, right? Because in the binary case, remember the probability is just a single value that represents like the

[6:07:57] single value that represents like the the class. If it's if it's above, usually our threshold is 0.5, right? If it's just above 0.5, we predict the one. If it's below 0.5, we predict a zero. So that's why we do one minus the

[6:08:11] probability when the when the class is zero. And so that helps us penalize the right thing. So if it's again like a 90% probability, but our label was actually zero, that's a significant penalty um

[6:08:24] because we would get logarithm of 0.1, which is going to be a big negative number. So that's why we have a one minus there. Now 1 minus 0 is just one. So this this just is always going to be one in either case. But uh when the

[6:08:39] label is zero, we do one minus the probability inside this logarithm. But in in the same way we did for categorical cross entropy, we're still going to average that across all of our examples. So we still to compute our

[6:08:55] loss to compute our overall loss against all of our examples, we're still just going to average that for all of our data, right? To to get the average cross entropy loss. Okay, so that's really

[6:09:08] remains the same. Just a simplified formula a bit whenever we have a binary case because we're only producing one probability. Okay, so in code, this is kind of what it would look like. uh in in order to compute the uh binary

[6:09:25] uh in in order to compute the uh binary cross entropy. Now you may be wondering cross entropy. Now you may be wondering in this code, you know, why do we have in this code, you know, why do we have uh why do we have this number here? And

[6:09:37] so just to point this out, this is actually pretty typical in most implementations of cross entropy is they have a small number here usually to avoid taking the logarithm of zero because the logarithm of exactly zero

[6:09:54] is negative infinity. So we don't want that. Um we never want to take the logarithm of exactly zero. So most under the hood implementations of cross entropy will safeguard against ever plugging in

[6:10:09] exactly zero. And uh because we we don't want this to be exactly zero. So we'll kind of have this tiny tiny tiny amount to keep us away from zero. Uh if if just

[6:10:21] in case, right? Just in case to avoid this. Now, it's still going to be a huge negative number. It's just not going to be infinity. Um, it would still be a huge negative number if if we uh like let's say we predicted um a one

[6:10:38] probability, but it was actually zero class. That's a completely wrong answer. penalty for that. we still want a significant penalty which we would have

[6:10:50] um but we don't want an infinite um because that would just ruin the rest of the calculations right so that's why that's why that's that's why this is here okay but of course like we we aren't going to use this uh because

[6:11:03] pietorch and tensorflow have their own way of calculating this they have their own loss function that we'll be able to use this is just like if we were to codify the math that we just saw for especially

[6:11:17] binary cross country it would look like this. Okay. So the uh now inside of uh TensorFlow and PyTorch they have a reduced version of that formula that we

[6:11:31] saw earlier with the cross entropy and it looks like this. So the the one that we actually will use will be this version. Um, the reason that's the case is because remember the only contribution we get from every

[6:11:47] single term is there's a one sitting right in front of this, right? That's the only contribution we ever get is one times the logarithm of the probability. Um so instead of instead of writing out that formula like this where we do um y1

[6:12:04] that formula like this where we do um y1 times logarithm of um probability of uh times logarithm of um probability of uh probability of class one probability of class one um you know uh and then we had plus

[6:12:18] um y 2 logarithm of probability of class 2 plus blah blah blah. We remember this formula is for the cross entropy the the

[6:12:35] in all of those cases the only time we get a contribution is is the term where get a contribution is is the term where the for the actual class right so uh if if the label had the second class then no other terms contribute this is the

[6:12:49] only guy that's going to contribute so for that reason we can actually simplify this formula to just this it just reduces to this formula here where we're taking the logarithm of that class um probability and we we technically have a

[6:13:05] it, but that doesn't really do anything, right? It just multiplies by one. And then we have our negative out front, which is this guy. And we have our average here, which is uh dividing by n and and totaling all these up. So, uh

[6:13:21] the formula makes a lot of sense. It's just um it just is uh reduced significantly from what we it's it's much simpler uh notation for writing that out than we had earlier. Okay, so this by the way this version of it is um

[6:13:38] often referred to as the sparse version or the sparse categorical cross entropy. Um you can use either one there. They give you the same result as categorical cross entropy. It's just a little bit simplified of a notation. That's really

[6:13:53] honestly the only difference. Uh that's that's the only difference is um you know we we just simplify the notation a bit from what we had earlier. This is

[6:14:05] sometimes called sparse. So we can use either um whenever we use pietorrch or entropy or sparse categorical cross entropy. Either one gives you the same result. They're the same formula really.

[6:14:18] result. They're the same formula really. Okay. So the reason we use that is to um you know obviously call out mistakes in the probability. So if you're wondering like why don't we just use mean squared error or mean absolute error. Um it's

[6:14:32] because we don't want to calculate like a numerical distance that's not going to amplify those mistakes. What really is amplifying them is the logarithm that's heavily penalizing uh the mistakes. So, so that's why, you

[6:14:47] why wouldn't we just use the distance from the mean squared error because we kind of have almost like two vectors that we could calculate the distance between. Um, but that that's not going to emphasize the right penalty which is

[6:15:03] really a probabilistic penalty that we want to apply. So, so in that situation mean absolute error. we're going to resort to the cross entropy because it

[6:15:16] does what we want it to do as a loss function. It really penalizes those wrong probabilities. It amplifies them correctly. So you do not want to use mean squared error. You can get uh bad training in that way. So so we want to

[6:15:32] avoid that and make sure we're always using cross entropy whenever we're doing classification. We should always be using that. um we should never be using mean squared error, mean absolute error in any

[6:15:44] classification example, right? It's just not going to penalize the right not going to penalize the right mistakes. Okay, so for just to recap for squared error. That's fine. That's actually what we would want to use for

[6:15:58] regression problems where we're predicting uh some type of continuous value that's coming out of our neural network. That's fine. For classification, we should be using cross entropy loss. Um, and whether it's the

[6:16:11] sparse version or the regular categorical cross entropy or in the case of a binary problem, the binary cross entropy, one of those formulas we should be using for the loss, right? In a classification, we should

[6:16:24] absolute error. It's not going to do the right job and penalizing those wrong right job and penalizing those wrong probabilities. Okay. So, I also wanted to call out that there is another loss function out there that's uh sometimes

[6:16:39] function out there that's uh sometimes used. I would say rarely used um versus cross entropy for classification and and it's known as the hinge loss. I'm not going to spend too much time on it cuz it's I've rarely ever seen it used. The

[6:16:53] the idea behind the hinge loss is to basically take the um take the cross basically take the um take the cross entropy loss and and have some type of uh regularization to it almost like when we did lasso and ridge regression where

[6:17:09] we had this kind of additional penalty you know added on top of the original loss function of the mean squed error. If you guys remember all the way back in regression when we studied lasso and ridge, they added a penalty to just

[6:17:22] typical mean squared error, right? So hinge loss is going to be kind of that same idea. It's going to be a variation on cross entropy that aims to kind of

[6:17:34] on cross entropy that aims to kind of penalize uh penalize certain terms uh in I'll show you I'll show you what it looks like. Um, but hinge loss also by the way is inspired by the SVM. It's

[6:17:49] actually the loss function that's used in training an SVM uh in order to find in training an SVM uh in order to find that um hyper plane that separates the data um and increase the margin between the hyper plane and the points you know

[6:18:05] the hyper plane and the points you know away from that hyper plane. So it it uh away from that hyper plane. So it it uh penalizes mistakes still in the same way that cross entropy does, but it also penalizes ones that are close to what we

[6:18:19] call kind of the the decision boundary which is uh the sort of like an SVM like that um hyper plane that we're building to divide the classes. Um, so it just

[6:18:31] to divide the classes. Um, so it just has this kind of extra penalty uh there that I I don't think is very natural for neural networks. I think it makes sense for SVMs, but it's not very natural for neural

[6:18:46] networks, which is why I think it's not used as much as as cross entropy. Um, so what is the hinge loss? It is the hinge loss is given by the maximum between 0

[6:18:59] loss is given by the maximum between 0 and then 1 - y * f ofx um which is the f ofx is just a probability. So it's kind of a it's kind of an all or nothing loss in some ways too. Um it's it's also kind

[6:19:14] of a weird one in that it's very it's much harder to train using this because it doesn't have a very nice gradient. That's another issue with the hinge loss is this function is very ugly. It doesn't have a nice it doesn't have a

[6:19:29] nice gradient at all. Um and the the reason it has this max is kind of an almost like a relu where you don't want to have anything negative. So your loss to have anything negative. So your loss is um guaranteed to be positive uh in

[6:19:44] this situation and you don't want to have anything negative. But again, I because we're not ever really going to use it. We're gonna focus on cross entropy because that's that's what's used um I'd say 99 out of a hundred

[6:19:58] times I've seen I've rarely ever seen the hinge loss actually used on a especially on a neural network. It now that's not to say it's never used. It is actually used on a support vector machine under the hood to train it. Um

[6:20:11] it is the loss function that's used but uh it's not primarily useful. It's not that useful with neural networks. It hasn't shown to be that useful, but it could be used in theory. Okay, I'm just going to go through that. So, I'm just

[6:20:26] worried about hinge loss. So, I'm just going to go past it. So, one of the going to go past it. So, one of the issues uh so one of the issues that I had mentioned earlier is it's it doesn't have a gradient. It's has a very um ugly

[6:20:40] kind of gradient. So it's for this reason it's it doesn't really perform well with training of neural networks because we really rely on that gradient uh in the gradient descent algorithm. So using this loss loss function can

[6:20:55] neural net more than it can help it often. Oh that's interesting. Loss functions and metrics and deep learning. Thanks for sharing that. So you can there's now there's other variations on the hinge hinge loss. squared, which can

[6:21:08] the hinge hinge loss. squared, which can square this uh which attempts to make it better um and attempts to have slightly different penalties on mistakes and make it a little bit better in terms of the gradient, but still not nearly as good

[6:21:22] as what we see with cross entropy. So, it does like using the squared version helps a little bit for gradient descent, but it's still not nearly as good as the um as as the cross entropy. So we'll So bottom line is we will stick with cross

[6:21:37] entropy but just showing you that there are other loss functions out there that are other loss functions out there that are sometimes used albeit rare rarely there's there's rarely ever a reason not to use cross entropy. It's it's so

[6:21:49] to use cross entropy. It's it's so effective at at um pointing out our probabilities. There's really no reason to use anything else unless you have a really good reason to. Okay. So to wrap up the lesson, we're going to just

[6:22:02] review forward and backward propagation and then talk a little bit about some regularization ideas which are a couple of ideas to help prevent overfitting just to remind us that you know overfitting is a huge issue especially

[6:22:16] with neural networks. Um so we'll just briefly talk about that towards the end when we start building our own neural networks especially. Um but let's talk about forward propagation just to remind us of what that really means. So forward

[6:22:32] propagation really uh revolves around sending data forward through the network uh in order to make a prediction. So whenever you hear that word forward prop or forward propagation, uh that's just referring to the idea that data kind of

[6:22:48] referring to the idea that data kind of propagates forward from layer to layer actually happening there on every layer weighted sum, passing it through an activation. That output goes on to the

[6:23:02] to become the input to the next layer and on and on and on until we finally reach the output. And so this is just the forward propagation. Now when we have more advanced neural networks, there may be times where there's

[6:23:16] connections that actually go backwards. And so not everything is forward And the prime example of that's going to be an RNN when we get to that later. But right now, um, data kind of propagates forward for us always in the, uh, kinds

[6:23:32] of layers that we've seen so far. mainly just awaited sum through an activation and that just goes forward through the network to ultimately to to produce some type of output and remember the output is going to be different depending on

[6:23:47] the network. It could be a single value if it's like a binary um classification it could produce a single probability. It could even be uh a regression. Right? So if it's a single output like this, it could be a regression might be a binary

[6:24:03] could be a regression might be a binary classification. Um or we might have more than one output node or I should say neuron which would be like in a um in a multiclass classification. We might have three output neurons or 10 or a thousand

[6:24:17] depends on how many classes we have we're predicting. And remember those middle layers there are typically called the hidden layers of the network. We usually have our input layer up front which matches the size of the input. So

[6:24:30] if we have kind of n features have n input nodes and that just orients our data to kind of start flowing forward and our hidden layers are going to be configurable. So of course we're going

[6:24:43] to practice this today like building out these hidden layers especially in TensorFlow. Um, but that's something we're going to be interested in doing is kind of setting up and architecting these hidden layers and architecting the

[6:24:55] output. Um, of course, as I said, the output depends on the problem if it's regression or classification. Okay, so forward propagation is a good analogy to it if you're if you

[6:25:10] uh if you think about it is kind of like our linear regression, right? in linear regression it's and this is a very very basic example of it with one feature and and kind of one uh weight and one bias. So in this situation like this forward

[6:25:27] propagation is just putting in our data and computing the output from the line which is computing this y that's what we're doing in forward propagation of the network. So what we imagine for the neural network is this is much more

[6:25:43] complicated. it's awaited sum through inactivation over and over and over inactivation over and over and over again. So it's repeated um it's repeated quantities of this over and over and over again in our layers. Um so it's

[6:25:57] much more complicated, but ultimately it's the same kind of idea that we had in linear regression. We're just sending a data point in and getting an output back. And it's this is kind of like in linear regression that's kind of like

[6:26:10] forward propagating. we just put in an input and it it generates an output. propagation. Um we're generating an output. It's just obviously the model is much more complex. There's more layers. There's more computations going on than

[6:26:25] just MX plusb, right? Just a basic linear regression. Okay. So I wanted to linear regression. Okay. So I wanted to show this demo um that kind of goes through forward propagation and and just reminds us of sending data through these

[6:26:40] reminds us of sending data through these layers. Um so if you go into lesson 4, it should be the only notebook there. So so you want to pull that up. I will pull that up as well. All right. So it should be this notebook. I just wanted to go

[6:26:53] through this to show us um just remind us of forward propagation and kind of us of forward propagation and kind of how data flows uh throughout um the layers. So hopefully you have this demo. This is again in lesson four. It should

[6:27:05] be the only demo there. Okay. So we're going to use uh numpy and uh maybe some plotting. So we have piplot uh plt from mapplot lib.

[6:27:19] Pretty standard. Um now we're going to generate some random weights. So generate some random weights. So remember this is what happens typically remember this is what happens typically in a um typically in a neural network

[6:27:33] going to have random weights to begin with. So this is not out of the ordinary random weights. That's what's going to happen. Um, now this is mimicking this

[6:27:46] 11 here is just it's nothing special, but it's just saying we have 11 elements. So this this would be like we have 11 features and we have 11 weights and one bias. So this would kind of be like a linear regression that has that

[6:28:01] like a linear regression that has that has these 11 uh 11 features um 11 weights and one bias. So just kind of like that to initialize those random values. So if you look at this weights uh array, it's going to have 11

[6:28:16] elements. The bias has one element and the inputs have 11 elements as well. Now we could take a look at the relu activation. So remember the relu

[6:28:28] activation uh takes whatever input comes into it and it computes the maximum between zero and that input. This is the rel very very basic nonlinear function

[6:28:40] that most of the times just going to return whatever it was as long as that's Very simple function. So we're just defining that. Now hopefully this looks familiar here. So forward propagation for a single

[6:28:55] So forward propagation for a single layer. So for a this is really um mimicking a single layer in a neural net because

[6:29:07] if you think about it what's happening is we are taking the the sum of the weights times the inputs which is what the dotproduct does right it's the sum of the weights times the inputs plus the bias. So we take that weighted sum and

[6:29:23] that gives us this and we put that through the activation. So that value is then going into our relu and producing an activated

[6:29:35] an activated value which is the output of this layer. value which is the output of this layer. So this is our forward propagation. We get this value. So we are forward propagating this data through this

[6:29:48] single very very simple layer which is just um has some weights and a bias and then we pass that through a relu activation. Um now when we do tensorflow it's going to look much different than this when we build out our network going

[6:30:03] to be different but this is the main idea under the hood of what we're doing. what forward propagation is doing in the code. Weighted sum through an activation. We see that again.

[6:30:22] here. This would be like computing our loss. So if we come down here um if the target was one we could compute the uh the the mean absolute error which is

[6:30:35] kind of what this is doing is taking the absolute value between the prediction absolute value between the prediction and the y. So this is um mimicking the mean absolute error loss function.

[6:30:48] it's taking the absolute value and we would total that up. But there's there's really only one entry in this. So the sum is just the uh it's just the single value. Um it's not really there's only one error.

[6:31:04] absolute error, we would total this up and then divide by the number that there even take the sum. We would likely just call the mean. Just do the mean to get the average of the errors. Um but in this case, there's really only there's

[6:31:19] only one entry in the error. So this is our mean absolute error for the loss and we we'd be off and running with that. Like we could use that to go back and update our weights with a gradient descent update um all of that. So so

[6:31:35] just reminding us of kind of a typical forward propagation compute loss back propagate right that's what we would actually be doing in in reality.

[6:31:49] error which would be doing this. So instead of absolute value we could compute the square that that'd be another option for our loss that we could use for this kind of regression example. Um and so the uh sum of squared

[6:32:03] example. Um and so the uh sum of squared error is uh 2.47. but just wanted to remind us of kind of what forward propagation is doing, which

[6:32:15] is mainly this idea weighted sum through an activation. And when we have a larger network, um you know, we're going to that that would be occurring in many many layers potentially, right? This this weighted sum through an activation

[6:32:29] would be kind of managed doing that process. Okay, so let's remind ourselves sending data forward weighted sum through activations mostly. Let's remind

[6:32:41] ourselves of back propagation which is going to be uh remember it is our way of going back and updating weights throughout the network based on the loss that we calculate and particularly the gradient of that loss because we there's

[6:32:57] a gradient descent formula that dictates how every single weight should be adjusted. And remember on Monday we did that very very detailed calculation which I encourage you guys to go back and kind of look through uh because I

[6:33:13] think it's a really good example that goes step by step. Um goes step by step. Um and I was saying the math isn't that important in terms of those gradients. I don't think it is but I think the

[6:33:25] process is really important to understand that we are you know going understand that we are you know going backwards in the uh layers to uh we're going backwards in order to adjust all these weights right we start from the

[6:33:39] towards the front updating all those weights so back propagation is really important for training it's not important for predicting so once it's

[6:33:51] trained we will never do back propagation unless we are going back and retraining. So back propagation only occurs during the training process. Once a model has been trained, we only are doing forward propagation, right?

[6:34:05] Because we're producing predictions. So really only doing back propagation hopefully that makes sense because the training process is is what is adjusting all those weights. So of course in order

[6:34:19] to do back propagation we need to compute the error. So once we've done forward propagation we can compute the error based on our loss function that we error based on our loss function that we have. And of course the the the the

[6:34:33] function is going to be whatever our model produced from a forward propagation perspective. Whatever our neural network produced and the actual label. So this is where I said last time we really need the actual label. This is

[6:34:47] still a supervised learning problem. Really need those labels in order to compute our loss. Really need that loss in order to update all of our weights and adjust our network to actually learn anything meaningful.

[6:35:00] Right? We need this loss. Um so what we just learned is um we should be using cross entropy in a classification problem. we should be using something like mean squared error in a regression

[6:35:15] problem for our loss. And of course, we'll be able to set all that up inside of TensorFlow or PyTorch. They they manage that for us. So um remember the manage that for us. So um remember the back propagation is utilizing um or is

[6:35:29] being utilized by gradient descent because the gradient descent gives us because the gradient descent gives us the formula for what the weight update should be during the back propagation process. And if you guys remember um I

[6:35:42] drew this picture which I think is a really powerful picture is is this kind really powerful picture is is this kind of idea that if we have this is a graph of our loss um loss function. Um what the gradient

[6:35:56] descent is really doing is finding out where we are on this which is our loss value. So we know because we can compute the loss, we know where we are on this and we're taking steps towards the minimum, right? We should be taking

[6:36:12] these kind of small steps down this way towards the minimum of the loss down here, which once we get to that point, we have a very good model at that at that point, right? basically have no error um at or minimal error you know at

[6:36:27] the at the loss uh minimum back propagation though is going to take every one of these updates. So when we take a step and we know like when we take a step that the step is going to tell us the formula for how we should

[6:36:43] adjust the weights um and and really we need to utilize those calculations um those weights and that's what back propagation is going to do is propagate

[6:36:55] those weight updates from back to front according to this uh gradient descent uh formula here. So gradient descent plays a really So gradient descent plays a really important role in computing what our

[6:37:09] weight update should be and then back propagation propagates those weight updates from the back to the front. Okay. And then we had a picture like this. Remember, I drew a picture kind of like this earlier on Monday where we

[6:37:23] looked at um the back propagation really relying on forward propagation in order relying on forward propagation in order to produce uh the quantity of what do we have. Uh and then we we compare what we have

[6:37:37] to the label and then we use that that is what helps us compute the loss. Ultimately the loss is what influences the gradient descent update, right? And that update gets back

[6:37:52] adjust the weight. So you can think about like training a neural net. I think a really good analogy of it is kind of like we have a million little kind of like we have a million little knobs and we're coming up with a formula

[6:38:05] that's telling us how much should I turn every little knob. And back propagation is saying what I should do is start with the knobs that are closest to the end. Start with those knobs. Turn those however much I need to. Those knobs

[6:38:19] helps us compute how much I should be turning earlier knobs, right, that are in the front of the network. And so then by the time I do one training iteration, I've made all the little tweaks to my knobs that I need. And now I'm set up

[6:38:33] the next time around to hopefully do better. That's that's the idea for the training. So, you know, ideally when we this slide is saying. When we make an adjustment in back propagation,

[6:38:48] um, we should be doing better the next time. And what we should see visually, time. And what we should see visually, like if we were to plot the loss over the course of the training, um, we should see that decreasing.

[6:39:01] things should be getting better over the course of our iterations, right? So maybe the loss starts somewhere up here. It should be getting better and ideally um flattening out to kind of a minimum uh as we go through the course of the

[6:39:16] iterations. What we do not want to see is increases, right? We don't want it we would be bad. We don't want that to not really training. It's not really learning anything at that point. Um, we

[6:39:30] something that we're going to talk about coming up shortly is we really want to be cautious about this is like yes we want it to be decreasing on the training data but ideally if we like once we make

[6:39:45] all these adjustments something we can do we haven't really talked about this make these adjustments we can apply that network to test data that's been held out and we can generate an equivalent

[6:39:59] graph to say okay what is our loss function on test data and ideally that's also decreasing. That would be nice. Like so this blue line is is the loss

[6:40:11] that's generated on a hold out set of test data. And that's that's actually going to be one of our strategies is to do a train test split on our data just like we did in machine learning and then evaluate

[6:40:26] our performance on the test set kind of simultaneously. So we so we train on the training set make all of our back propagation changes on every iteration. What we should be able to do is apply that new network the adjusted values of

[6:40:42] the weights the network on both sets. We should be able to apply it on the training set. We should also be able to apply that on the test set and see what apply that on the test set and see what our performance is. Ideally we see a the

[6:40:55] same kind of trend. we see the loss decreasing on that test set um over the getting better and better and better as well. What would be bad is a a

[6:41:07] well. What would be bad is a a divergence of these two. So what I mean by that is um if we drew a graph like this, let me go back and redraw this. So this, let me go back and redraw this. So if we had our loss

[6:41:26] what we do not want to see is a decrease. So this would be on the training data and then what we don't want to see is a

[6:41:39] divergence on the test data. So in other words, if we apply this to the test data, it kind of tracks initially and then kind of diverges away. This this is kind of a a difference that's significant difference there, right? We

[6:41:53] don't want to see that because that would signal an overfitting

[6:42:08] why is it overfitting? It's because on the training data, our loss is decreasing very steadily and it actually gets pretty low um over the course of iterations. That's great, but there's a there's a big difference on our test

[6:42:21] data that signals that we're overfitting to our training data and not able to generalize our network to test data because our loss is much higher. So that divergence is a very clear picture of overfitting. So that's something we're

[6:42:36] really cool is like TensorFlow in particular actually gives you the ability to generate this plot during the course of the training. So, so every iteration it can update this plot and we can see it in real time how the training

[6:42:51] is progressing on these on these two sets of data and that way we can kind of see that divergence occurring and then kind of stop it. Say I don't want to keep training because I I've already seen enough on the overfitting this

[6:43:04] divergence. So we'll we'll see that when we get into TensorFlow is um there is we get into TensorFlow is um there is that ability to kind of uh visualize our training process which is pretty pretty helpful. Uh if it's underfitting what do

[6:43:20] you think what would be a good what would be a sign of underfitting? High. Yeah, it should be high because what does what does the loss function track? It it tracks our mistakes. it it gives us a distance which is a distance

[6:43:34] between the actual label and our predictions. So the higher this distance predictions. So the higher this distance the more error we have. So the picture the more error we have. So the picture for un for um underfitting would look

[6:43:47] something like this. Yeah it the basically for underfitting it would let me try to draw it alongside this graph. It would be let me do it in purple. So It would be let me do it in purple. So um for underfitting it might um be it

[6:44:03] might be much higher kind of like this. It basically never decreases. It basically never decreases and stays up way too high. The loss is just too high in general. Um this would be this would be kind of uh underfitting

[6:44:23] kind of started increasing that would that would be underfitting as well. It's just not performing very well. Our loss is either increasing or staying kind of stagnant and it's too high. Uh if we add a hidden layer is

[6:44:36] there any impact on this graph? There might be. Yeah. So, um, if we add a hidden layer, uh, there's there's no direct way to know if it's going to help direct way to know if it's going to help us on this graph, but, um, that if we

[6:44:50] are underfitting, remember what underfitting kind of signals, it signals that our model is too basic. So, if we're underfitting, one of the things we can do is maybe add a hidden layer. That that might be something we want to do if

[6:45:04] we're underfitting because maybe we need a more complex model. Okay. Now, on the reverse side, if we're overfitting, maybe our model is too complex and we actually want to remove a hidden layer. We have too deep of a network. That

[6:45:18] sense? So, like underfitting and overfitting tell us maybe what kind of adjustment we should make to our network. Should we remove a layer? Should we make the layer smaller and less neurons?

[6:45:32] You know, those things we can kind of tell from a graph like this. All right, we're kind of talking about that now in terms of overfitting and underfitting.

[6:45:44] Um, and that's really what regularization is getting at. We talked bit in regression. Remember, regularization um was involved us doing something like lasso or ridge regression where we added

[6:45:59] lasso or ridge regression where we added penalties um to combat overfitting. penalties um to combat overfitting. So with neural networks, you know, we can also talk about regularization because neural networks actually have a

[6:46:12] tendency to overfit. They are very very prone to overfitting because they are remember things that are very complex tend to overfit and neural nets are

[6:46:24] neural networks are very complex. They can be very complex with many many layers and many neurons. And so they can be easily susceptible to overfitting. And so we have a picture kind of like this um where we uh I just drew

[6:46:41] something exactly like this. So, you know, we can definitely overfit um in in a situation like this where we uh uh maybe on our training data we have a picture like this, but then on our test data it we're kind of tracking with it

[6:46:57] data it we're kind of tracking with it and then we kind of diverge would be um certainly an overfitting situation. um because there's a there's a huge gap here that's just growing and growing and growing between our training performance

[6:47:12] and our test data performance in terms of the loss. So that's overfitting of the loss. So that's overfitting and um this can easily happen if you think about also the training process of a neural network. There's usually many

[6:47:25] iterations and every iteration we're sending all of the data forward in order to generate the the average loss which updates all the weights. And and if you think about it, we're making all these adjustments to all these weights.

[6:47:40] Just the fact that we're doing that and we're doing it so many times in these means that the neural net just the training process makes it potentially susceptible to overfitting just just relatively easil easily to overfit. It's

[6:47:56] iterations. Every iteration seeing all the data, it it means it can overfit really easily to training data and and struggle to generalize to new data such

[6:48:08] as a test set or in worst case scenario like actual predictions that we need to make on real life data, right? So it can overfit pretty easily. Um we're going to discuss a couple of ways to combat it. One way to combat it that is shown here

[6:48:26] One way to combat it that is shown here on this slide is what is known as early stopping. So early stopping is a is a very interesting uh idea that um PyTorch and TensorFlow both have an early stopping algorithm. And essentially what

[6:48:42] early stopping will do is keep track of the loss on the test and the training and if it sees a large enough divergence meaning there's a divergence here that's

[6:48:54] greater than some threshold. So if this distance between the uh training loss if the distance there becomes greater than some distance there becomes greater than some threshold let's call it alpha

[6:49:09] if this distance becomes big enough then we kind of trigger a stop basically to say I don't want to continue because my my uh test um evaluation of loss is starting to diverge away from

[6:49:25] my training. So I should just stop it here and stop training here and and maybe that's good enough. Um or I should like I stop there so I can make some adjustments. I don't want to keep going. So um there is this is actually built

[6:49:40] into TensorFlow and PyTorch. Uh and it's called early stopping and and you're you're able to set up with early stopping it is basically like a grace

[6:49:52] period to say, okay, if I if I see this distance greater than some threshold, I'm going to trigger an early stop. If I don't see an improvement in the next uh don't see an improvement in the next uh x number of steps, so as we go through

[6:50:07] the next 10 iterations, if I don't see an improvement and I don't see this distance drop down back below alpha, I'm going to stop. So you there too in the early stopping to give it a chance to recover. Maybe it's just

[6:50:23] bad. It gets unlucky and it's just bad on that iteration and I make some adjustments or the training adjusts the weights and then it starts to do better on the test set and it goes back below that alpha in in the next couple steps

[6:50:37] and therefore we've recovered and everything's okay. So so you can give yourself some grace there. uh and and that's all configurable in the early going to see an example of that when we get in lesson five when we do

[6:50:51] get in lesson five when we do TensorFlow. Um so so that is one thing we can do to combat overfitting is to just stop our training early. We don't want to let it keep going if it's performing poorly on test data. Don't

[6:51:05] let it keep going. That's basically the the simple answer there. Just stop it. um and and either stop it and use that model from the point you've stopped or adjustments to your to your architecture. Uh maybe remove a layer,

[6:51:22] lower the number of neurons, something like that and try again. So early stopping, we're going to see that uh later. Um so that's one way to combat overfitting is to just stop the training early. A couple of others that are

[6:51:37] widely used are what are called um L2 regularization and dropout regularization. Um and and both of these uh are basically ideas that are help that are used to to mitigate against overfitting.

[6:51:53] used to to mitigate against overfitting. Now this one we've actually seen before Now this one we've actually seen before with ridge regression. So this is very analogous to uh ridge regression and remember when we did lasso and ridge

[6:52:07] regression this is actually the same idea is we're going to introduce a penalty in the loss function um in order to penalize large weights um

[6:52:19] which is supposed to help prevent overfitting. So it's kind of the same exact idea there in like when we did ridge and and regular linear regression is we wanted to discourage any one weight from dominating.

[6:52:33] weight from dominating. So we penalize very large weights. Um we can actually do the same thing in neural net training and penalize large weights um or any one weight from being dominant with a similar kind of penalty term in

[6:52:47] the loss function. That's one way to do it. And we're going to talk about dropout regularization which is a whole different concept. This this is actually a very interesting concept that is widely used in neural net training

[6:53:01] way more than L2 regularization. So so dropout is widely used and we'll actually use it quite a bit uh in our examples uh especially when we get into dropout regularization coming up and tell you more about what that is but it

[6:53:17] is definitely widely used. uh L2 regularization not quite used as much but just something to be aware of that we could do is just add basically adjust our loss function to add to to prevent overfitting. So what does that mean? We

[6:53:33] overfitting. So what does that mean? We basically have um some type of penalty term that we add to our existing loss function. So in mathematical terms this function. So in mathematical terms this uh C 0 is just our initial cost or loss

[6:53:47] function. whatever we had, we still have, but we add in this penalty that discourages um large weights. So, it penalizes weights with a squared penalty there um to to kind of discourage those. So, when you're

[6:54:03] training, you're going to get an extra penalty if you have a really large weight. And so it so this helps make the weight adjustments and control those so they're not overly you know overfitit to the training data that it that it's

[6:54:18] being trained on. So it's the same idea we had this formula might look a little familiar. It's because it is very familiar from the uh ridge regression. It's exactly the same idea that we had there. Um and

[6:54:33] the loss function there if you remember was just mean squared error. But here you know in classification it could be it could be cross entropy in regression it could be mean squared error again. So same kind of penalization that we're

[6:54:46] doing that we we had back then. Okay. So this is an option and it's actually not too bad to set this up in PyTorch TensorFlow to take an existing loss function and just make an adjustment to it. Um most of them have a regular

[6:55:03] regularized version of the existing loss functions like cross entropy and mean squared error um that you can just grab out of the box and use. One thing to be out of the box and use. One thing to be aware of is if you do use this basically

[6:55:17] uh it can take longer to train because it's it's has because it has this extra penalty the gradient is a little bit more complex and so the back propagation is going to take a little bit longer. So the training can take longer with this

[6:55:32] regularization in there. That's the only kind of caution. Okay. So that's one approach is to add a penalty to our loss function which might make the training take longer but it's designed to discourage large weights and prevent

[6:55:47] overfitting just like we saw with ridge regression. So that's one option. Now another option that is widely used is a very very interesting idea

[6:55:59] is randomly getting rid of neurons during the training process um on every iteration. So what you can imagine is that each iteration of the training I basically select neurons and

[6:56:16] with a certain probability I'm going to turn them off. So it's as if they're not turn them off. So it's as if they're not there. Now, doing that has a significant there. Now, doing that has a significant effect on the training because on that

[6:56:28] iteration, I'm going to basically have a much different architecture if I delete a bunch of neurons. Um, and then maybe on the next iteration I delete different ones and then on the next iteration I delete different ones. So, it's not like

[6:56:42] they're permanently deleted. It's just during that iteration they're going to be turned off and so they're not going to be involved and their connections are not going to be involved during every training iteration.

[6:56:56] And this is a very clever approach because essentially what happens during because essentially what happens during the training is your model learns how to how to adjust the weights across all of the iterations um with some of the

[6:57:11] network turned off on every iteration. And the idea behind that is to prevent overfitting because you don't have a full network with all connections on every single iteration and that basically prevents having over

[6:57:25] complexity in your model. It's a very interesting approach. So again just to just to reiterate what dropout is doing is it is drop the reason it's called dropout is because the the deletion of the neuron is actually called dropping

[6:57:41] the neuron is actually called dropping it out. So if I had a network like this and it let's say it produces one output and so there's um obviously a bunch of connections in this network. Um so it'd be you know

[6:57:56] in this network. Um so it'd be you know kind of like this. Um and then connections like this. Um,

[6:58:08] essentially what we're going to do is on every iteration we choose with a certain every iteration we choose with a certain probability to drop out a a neuron or probability to drop out a a neuron or multiple neurons from this uh from this

[6:58:20] uh from each layer from each hidden layer. So let's say we on this hidden drop out this neuron with a certain probability right it has a chance to be probability right it has a chance to be kept and that's the probability of

[6:58:35] keeping the neuron is is actually a hyperparameter that you have to choose whenever you use drop dropout and it it has an effect like if you were to lower low um then you're basically um you're

[6:58:53] risking not having anything and you can underfit underfit um which which could be bad. If you keep high probability then basically you're going to ensure all your neurons are

[6:59:06] going to ensure all your neurons are kept which uh basically will potentially lead to overfitting anyways. So you want that probability to be somewhere in the middle. You want a decent chance of being dropped out, but

[6:59:19] not um overly low or overly high. And remember, if it's overly low, then and have an underfit model, a very simple model, because you're going to just delete so many neurons. Um you're not going to be able to train

[6:59:32] worst case is you like drop out everything and all of a sudden you're not learning anything at all. You don't have any any neurons to use. So, you have a very underfit model in that case. So you don't want that to happen. So you

[6:59:47] So you don't want that to happen. So you want this probability to be ideally like something like 0.4 to 0.6 somewhere in that like middle range maybe 0.5. You want it to be somewhere in there. But it is something you can change during the

[7:00:01] training. You could play around with it and hyperparameter tune. Try out different probabilities for the dropout. Now one thing I should also clarify is dropout is not permanent. So it when this is only during the training that

[7:00:16] the neurons get dropped out and it it changes iteration to iteration. So they're not permanently going to be deleted because on the next iteration we may keep this guy and then turn off this guy and keep this guy. So it's every

[7:00:32] iteration that probability resets and you have a a new chance to drop things out on each iteration. And then when the model is done, you're not dropping out in prediction. So in prediction, you're using the full

[7:00:46] propagation and you're sending data through to make a prediction, you're not going to turn off things randomly during prediction. This is only used during training. Only used during training. And it's the purpose of it is to prevent

[7:01:02] overfitting. It's a very clever approach to preventing overfitting is to basically artificially kind of change our complexity of the model by turning things off and on. It's a very interesting approach. But one of the

[7:01:15] things we're going to see is setting up dropout in our network is a really easy process. It's it's not like in terms of training. It's not trial and error. Essentially, we'll just set up the dropout layer um and have the ability to

[7:01:30] drop out any layer that we want. So, we could we could set up and say, I want to drop out this layer, but you know, of course, I don't want to drop out the output. We never want to drop out the output, and I don't want to drop out the

[7:01:42] input. I don't want those to be cut off at all. I only want to drop out the hidden layers. So, we should only be doing this to hidden layers, right? I and certainly not the output and I shouldn't be dropping out the input.

[7:02:01] doing this to? Okay. So here's a good picture better than what I drew is like picture better than what I drew is like if you're um dropping out things and again the whole goal is to do this so your model generalizes better by uh

[7:02:16] during the training process by kind of strengthening its learning ability. It's not it's not getting overfit by having everything activated all the time. It's everything activated all the time. It's kind of learning um on the fly by, you

[7:02:30] know, getting stronger in different areas by turning off certain neurons areas by turning off certain neurons every single iteration. Um you get a picture that's kind of like this. When you turn off a neuron, you are severing

[7:02:42] their connections, right? So you can kind of see in this picture, you go from something that's really complex, which is the whole point of of preventing overfitting, right? You go from something really complex

[7:02:55] something really complex to something simpler. That's the idea. But we're we're not going to keep it simple permanently. So on the next iteration, it might look slightly different because we would drop

[7:03:10] out different neurons, right? And so we're basically going to do this over and over and over again. And it's a good method. It turns out to be a really effective method to prevent overfitting by going and the idea is just to kind of

[7:03:24] keep alternating from complex to simple complex to simple and that way you kind of distribute your learning very effectively which is supposed to make it generalize. Okay, does that make sense on the

[7:03:38] overfitting or the prevention of overfitting like how it's preventing overfitting? It's it's going from complex to simple and really alternating that over the course of the training by it's it's all random. So, we randomly

[7:03:53] choose which ones to to turn off and on. I really want to emphasize that it's not permanent cuz like it could be like this on one iteration and then the next iteration um completely different ones are turned off. And once the model has

[7:04:08] been trained, nothing will get dropped out. will use the full model to make a prediction. It's just the what's the idea is the training is supposed to prevent the overfitting so that by the time we get the full model um it it has

[7:04:24] a better chance of generalizing because it it's basically been taking on these different forms and and learning in different ways the whole time during the during the training process. So yeah, drop this is called dropout. Dropout is

[7:04:38] very popular. It it um came around in the late 2000s like early 2010s I think the late 2000s like early 2010s I think it was invented and um it's been widely used ever since in most uh in most training of any neural net uh will use

[7:04:54] training of any neural net uh will use dropout to some extent. It's a very popular approach to prevent overfitting. And again overfitting is a very prominent issue in neural nets. they tend to overfit very easily because

[7:05:06] they're so complex of a model. This helps eliminate complexity, right? It makes the model simpler every iteration is a very very neat approach.

[7:05:18] So again, one of the uh things to be aware of is that probability of retaining a neuron. So that is known as the parameter P. And so when you have a

[7:05:32] very low P, meaning you have a very low probability of keeping a neuron, so most of it gets dropped out, you have underfitting. That makes sense. So if you go back to this picture, if we had a low P,

[7:05:48] basically everything's being turned off. So you have a ve you have an overly simple model. Overly simple. So you're underfitting, right? So you don't want to do that. And so this this um graph on

[7:06:04] the next slide kind of shows that happening. Whenever they had a very low probability of being kept, they underfit, which is what this is showing. There's a high rate of error. The the the

[7:06:17] loss function is pretty high on the training and the test set. They're underfitting. Now, as they increased the P, which is going this way, and they got more into uh this realm, they got more into this

[7:06:33] realm. Um, they saw better results. So, they had low training error, low test error. It's still like not perfect. There's still some level of overfitting here because there's a gap between the two, but it was better. It wasn't as

[7:06:48] two, but it was better. It wasn't as underfit as in this this area here. So that middle ground is generally a good a good spot to be with a you know kind of a 50-50 chance of turning off a neuron. Um seems to be a good a good

[7:07:02] Um seems to be a good a good probability. Now as they kept going the overfitting got drastic more drastic right because what this means when the P right because what this means when the P is really high in this range is you have

[7:07:16] a really high chance of keeping the every neuron. So you're basically learning on every iteration with a full model. You're not turning anything off, which is keeping it overly complex, right? So when you have a very high P,

[7:07:32] turned on. It has a very high chance of being turned on, which means you have a very comp you basically have this, which is a complex more complex model. So you're more at risk of overfitting in

[7:07:45] those situations. So that's kind of what they discovered. Um, which makes a lot of sense intuitively. Like this this just kind of verifies what we would think would happen. Um, where you get you get more overfitting the the further

[7:08:00] to the right you go on those probabilities. More overfitting. So very interesting. And by the way, this number so if you're reading this slide, this so if you're reading this slide, this number refers to the size of the uh

[7:08:13] number refers to the size of the uh layers. So it's 784 because the images on the MNEST data set. So if you guys remember MNEST is the handwritten digits, right? Those are the images of things like a two or a one or a zero or

[7:08:27] a nine or whatever, right? Those are those are those images. So there's the look at what the output layer is. It's at the very end here is 10. That makes at the very end here is 10. That makes sense because it's the digits 0 to 9. So

[7:08:41] there's 10 different classes that we could predict. So, the output layer needs to be a 10. That hopefully resonates and makes sense. Um, and the input layer is 784 because it's a 28x 28 image resolution. So, 28x 28 is 784.

[7:09:01] image resolution. So, 28x 28 is 784. So, um, the input has 784 pixels in So, um, the input has 784 pixels in total that are in this 28x 28 image. Um, and then they have 24,048 neurons in this layer, 2408 neurons in this layer,

[7:09:15] 24,48 neurons in this layer, and then 10 output neurons for the classes 0 to9. output neurons for the classes 0 to9. Okay, so on that architecture is what they applied the dropout to these layers and came up with these numbers. Okay, so

[7:09:31] by the way, we're going to build a model that looks like this. We will build a model that looks almost exactly like this architecture. We'll we'll practice that on the MNEST data set with those images. Okay, so uh a lot of ground that

[7:09:45] we covered there in lesson four, but that pretty much wraps up lesson four. that pretty much wraps up lesson four. Um we talked about deep neural nets and how to do forward propagation uh back propagation to update all the weights.

[7:09:57] We talked about regularization which is really um helpful especially dropout um to basically turn off neurons to prevent overfitting. We also talked about loss functions which are really critical for uh doing uh the training process right

[7:10:14] we need to know what the loss function is um so that we can evaluate the error is um so that we can evaluate the error uh that is um very critical in making that gradient descent update for the network. Okay. So, if you want to pull

[7:10:29] up those notes, we're going to be moving on to lesson five, which is going to be the first of two lessons that really get into our two frameworks for building neural networks, which are going to be TensorFlow and PyTorch. So, this lesson

[7:10:43] will cover TensorFlow and lesson six will cover PyTorch. Um, and of course along the way we're going to do examples with both so we can start to see how we with both so we can start to see how we actually build um, neural networks and

[7:10:56] see what goes into the training process and building in the code right inside of our Python. So that'll be pretty exciting I think. Um, so by the end of this lesson um, we want to see the main thing we want to do is get some hands-on

[7:11:09] experience with TensorFlow. So this lesson actually has a lot of slides that are with some different aspects of TensorFlow that we won't actually be touching. Um like TensorFlow playground and TF learn won't actually be touching

[7:11:25] those too much. So in the slides you're going to see me skip around a little bit because I want to get to the parts that we are definitely going to use which are mainly going to be TensorFlow and the underlying uh or the uh library

[7:11:41] that kind of goes on top of it which is caris um as kind of the interface to tensorflow to help us build neural networks. Um so that's where we're going to focus our attention. I know there are other slides in here about different um

[7:11:57] actually be using those really for anything and so we'll be using just caris mostly to practice building our nets and training them. So that's where we're going to focus our attention in the lesson. Okay. So let's just get a

[7:12:12] basic introduction to uh TensorFlow itself. So if you guys remember um we talked about TensorFlow very at the very very beginning of this journey where we talked about um you know these

[7:12:24] talked about um you know these frameworks being developed to help um build and train neural nets and TensorFlow is one of them and so uh it is an open-source Python framework for building uh neural nets um and it was

[7:12:40] building uh neural nets um and it was developed by Google uh back in the day to help them um to help them build and train neural nets. And it was open- train neural nets. And it was open- sourced in 2015, so not that long ago.

[7:12:53] Um but you know, by now over 10 years ago, um it has been kind of open sourced ago, um it has been kind of open sourced and available for free use. Um now, and available for free use. Um now, TensorFlow itself is an remarkable

[7:13:08] library. It basically handles all of that computation of forward and backward propagation. Especially that backward propagation that we saw that was so tedious, right? Computing all those gradients and then multiplying all those

[7:13:23] pieces together to generate a weight update is very tedious and hard to keep track of even manually. But what TensorFlow does is actually uh keep track of that very effectively in kind of a graph structure. Um so so

[7:13:39] TensorFlow is really a library that manages the is really a library that manages the data graph and data flow of neural data graph and data flow of neural network data which is um primarily going

[7:13:53] to be tensors for us which are basically like matrices. Um so it's it's this really remarkable library to manipulate that data in this kind of graph network that data in this kind of graph network format um which mimics you know a neural

[7:14:07] format um which mimics you know a neural net. Um so TensorFlow has a bunch of stuff that comes with it. Obviously the core library is something we'll use mainly actually mainly the library that's built on top of it which is caris

[7:14:21] um that that kind of interacts with tensorflow but it also has surrounding tools and other community resources such as documentation that's really amazing um that we will look at from time to time as we need to. Um, one of the tools

[7:14:35] that TensorFlow has that we'll look at later is called Tensor Board, which is a really amazing tool to kind of visualize that training process. So the the kind of graphs that I mentioned where you can kind of see your your loss um over the

[7:14:51] course of the training iterations uh is something that tensorboard can do which something that tensorboard can do which is a tool that uh was created by um you know the folks who made TensorFlow and they have many other tools like that.

[7:15:03] they have many other tools like that. they have tools for like serving models uh um like in terms of deploying them and making them usable through like uh APIs and things like that. So have a lot of other tools out there but you know

[7:15:17] we'll focus our attention on primarily the libraries that enable us to build the libraries that enable us to build and train neural networks. Um so the re the reason TensorFlow is called TensorFlow is primarily because

[7:15:30] uh the fact that it manages this underlying graph of tensor. So all the data that flows through this graph um is actually in uh this tensor object which

[7:15:43] actually in uh this tensor object which is very similar to kind of a numpy array or kind of like a pandas dataf frame. very simple uh very similar kind of very simple uh very similar kind of idea. So, but in reality we can think of

[7:15:56] a tensor as kind of a kind of a a a matrix um like kind of data object that uh contains kind of multi-dimensional arrays inside of the tensor. So um you

[7:16:10] know we've been dealing with data that's in a matrix format in terms of a data in a matrix format in terms of a data frame. So this was our data frame. um the vector was kind of our numpy array

[7:16:22] um kind of structure. And now in TensorFlow we're going to be working with tensors which are going to be these kind of like matrices of matrices kind of like matrices of matrices essentially and uh so TensorFlow

[7:16:35] essentially and uh so TensorFlow manipulates data uh that is uh converted into tensor uh into tensors under the hood of TensorFlow whenever we pass data into our into the network. Um, so it's just useful to know that is that the

[7:16:51] data that TensorFlow is manipulating is actually tensors and not really data frames, not really numpy arrays. But that's not to say we can't start with TensorFlow. Of course, we can. And TensorFlow will manage converting data

[7:17:06] frames or converting numpy arrays into tensors under the hood. That's something tensors under the hood. That's something that they do. But it manages kind of the computations in this tensor kind of data structure. In the same way that you know

[7:17:20] the the models that we've built to this point mainly manipulate things like data frames, right? They they manage and manipulate data frames. TensorFlow manipulates tensors. Um but luckily, you know, it will convert things like data

[7:17:33] frames into a tensor as needed. So we don't really need to worry too much about that. Just pointing it out. That's where it kind of gets its name is that structure, that tensor structure. Okay. So, mainly what we're going to be using

[7:17:47] So, mainly what we're going to be using TensorFlow for is of course for uh building and training neural nets, but TensorFlow as a library has a lot of TensorFlow as a library has a lot of other features to it. Um, it has these

[7:18:01] uh it has things on top of it that make life so much easier when it comes to what we're talking about here with these highle um APIs which are which would be highle um APIs which are which would be something like uh caris.

[7:18:22] to manipulate like build and train neural networks. It'll be really easy to do with a library like caris uh built on top of tensorflow. TensorFlow also has a that are already trained. That'll be really useful for things like computer

[7:18:37] vision um where we want to pull things off the shelf and maybe fine-tune it um or do what's called transfer learning to it to to make it better on our own data. So that's really cool. It has a bunch of pre-trained models already built in. Um

[7:18:51] it can take advantage of GPUs really effectively. So, TensorFlow manages GPU uh compute really, really nicely. It interfaces with GPUs really easily, which is a great feature for it because uh we're going to need that

[7:19:07] especially to train um especially to train computer vision uh models later on down the line, like when we get into object detection, things like that. GPU is going to be really important. Um I already mentioned TensorBoard. That's a

[7:19:20] really important visualization uh kind of library that TensorFlow provides to be able to to plot a graph like this in real time where we can kind of see the loss over the course of the iterations and see that kind of updating in real

[7:19:35] time as iterations are finishing kind of see how that loss is progressing and and that can inform us if we need to stop if we should um let it keep going all those kind of things. um we can kind of visualize with the help of tensorboard.

[7:19:50] So we'll see that later on. As I said earlier um TensorFlow also has features that help for model deployment. Now we won't really use that too much um but they they do have that capability where you can build a neural network and kind

[7:20:05] of easily um deploy it behind an endpoint. We won't really need to do endpoint. We won't really need to do that. Um but uh it does facilitate that in in the library. So if we needed to, we could easily do that. Okay. So I hope

[7:20:20] it makes sense why TensorFlow was such a big deal, especially after going through our examples of like forward and backward propagation. When you're backward propagation. When you're building a neural network, there's so

[7:20:33] much computation to keep track of. And we we saw that, right? We saw that kind of firsthand is you know a forward propagation you're multiplying and doing this uh weighted sum through an activation function and you're doing

[7:20:47] that and passing that data forward throughout the network and that's a lot of computation especially if it's a deep network but man that back propagation network but man that back propagation was so so so tedious right computing the

[7:21:01] gradients that is challenging um and then keeping track of that and multip multiplying those together with the right terms and propagating that backward throughout the whole network is a very tedious and intensive process but

[7:21:18] libraries like TensorFlow and PyTorch manage that for us which is a huge deal and that's why they were created to really manage that computation and help

[7:21:30] us train and build neural networks. So that's why we need a library like TensorFlow is to ultimately manage the computation that is inherent in a neural network. There's a lot it's much more complex than any kind of

[7:21:45] machine learning model, right? There's a lot more going on with it. And so it necessitates these kind of libraries that are specialized in in neural nets beyond scikitlearn. Scikitlearn was great. We've used scikitlearn for you

[7:22:00] know so many machine learning models but we really need to go beyond it to the specialized libraries for neural nets because there's so much going on with those. Okay. So, as I mentioned before, um, under the hood of TensorFlow, and

[7:22:15] know this, but just showing you kind of how it works, is TensorFlow will manage the network and break it down into really manageable chunks of computation

[7:22:28] by um kind of organizing, organizing how that data flows and organizing that computation into really really manageable pieces. So for example, doing a weighted sum and passing that through an activation kind

[7:22:43] of gets broken down into a data flow graph that looks like this. So you can even see this here like we do a a a matrix multiplication between the weights and the inputs and that forms our weighted sum. That's just a

[7:22:57] our weighted sum. That's just a dotproduct, right? Weighted sum. Uh and then we add in a bias. So we add in a bias which is what this does. So it manages adding a bias term here and then it passes that into a relu function

[7:23:12] which is doing our activation on this weighted sum. Right? So it everything that we've been doing this this weighted sum through an doing this this weighted sum through an activation

[7:23:29] efficiently uh to produce an output. And of course the uh the the deeper the network the more sophisticated this graph looks. But this is just a simple

[7:23:41] this would be like the perceptron righted sum through an activation. It looks just like this a and on and on and on. So a more complex uh neural network is going to be represented by a much bigger graph here. But TensorFlow

[7:23:54] manages that under the hood and it does that on purpose so that it can manage all aspects of the neural net. It can manage training. It can manage prediction by doing forward propagation like this. It can manage back

[7:24:09] propagation by doing all these gradients really effectively really effectively um in of the graph. So by putting it in this graph structure, it's able to manage our computation so so

[7:24:21] effectively. Um that's the beauty of TensorFlow is this data flow graph of tensors. um and it manages it really really efficiently. Okay, so basically

[7:24:34] everything I said is uh TensorFlow manages the computation through this manages the computation through this graph. Um and it can also leverage graph. Um and it can also leverage parallelism by using GPUs. So it it can

[7:24:47] uh basically optimize this graph execution um as it needs to parallelize actually offload a lot of these calculations in the graph to a GPU and

[7:25:01] that's something we will uh work with later on. Um so it's in that way very effective at managing again managing the computation. So some of the benefits as computation. So some of the benefits as I said using GPUs it can uh parallelize

[7:25:15] things um it can be built in Python. This graph can actually be saved and reloaded really effectively. So the the idea of putting things in a graph like idea of putting things in a graph like this makes it really um persist

[7:25:29] persistent like we can persist this graph uh architecture very easily and then reload it as we need to. So in that sense we can always save our architectures and save any of our weights that are there like this weight

[7:25:43] matrix um we can save and so like loading and saving models actually becomes really really easy in TensorFlow and same thing for PyTorch. Everything that I'm saying here for TensorFlow is honestly applicable to PyTorch as well.

[7:25:56] I know we're going to cover that in lesson six, but um very similar where PyTorch manages everything as kind of a data graph and and remember PyTorch was made by Facebook. So they kind of made these libraries independent of each

[7:26:12] other to both try to tackle the neural network problem and then they open sourced them uh to make it available for the community later down the road. But both working on neural networks quite a bit. So they had their own libraries to

[7:26:28] manage those and they're both prominent today, right? TensorFlow and PyTorch. So when it comes to working with TensorFlow um there's kind of two approaches.

[7:26:40] um there's kind of two approaches. One is to work at that graph level um which is kind of the low level of TensorFlow. Um so working at the graph level is sometimes known as TensorFlow core. It's just the core of TensorFlow.

[7:26:54] We won't do that. It's very complicated to do that. Um it it gives you ultimate you're controlling that data tensor graph um directly. So you get a lot of control and fine grain control over

[7:27:09] building your models. We won't ever need to do that. um we'll mostly work at the caris level which is kind of an abstraction layer on top of the graph. So caris is going to be a library that sits on

[7:27:26] top of tensorflow almost in the way that like a seaborn sits on top of mapplot like a seaborn sits on top of mapplot lib or a panda sits on top of numpy very similar way caris kind of is an interface to the low-level graphs and it

[7:27:42] interface to the low-level graphs and it makes it much much easier to manage uh neural networks and training um because we don't need to go into of graph level details. Um we can just stick at kind of a high level of building the models. So

[7:27:57] most everyone actually building neural networks will utilize caris. If you're doing deep research and writing a paper and kind of really concerned with performance and coming up with new architectures, you may work at the graph

[7:28:13] level. Um we're not going to do that in this program because nobody really does that for the purpose of building a model. You you would generally work at the caris level uh which is what we're going to do. Okay. So makes it easier

[7:28:27] and that's honestly what is practical is to work at that level because um you a lot in terms of building really effective models. It's just we're not going to be coming up with brand. So if you were to come up with a brand new

[7:28:42] layer like an attention layer for transformers, you would be doing at that at the graph level at the very low low level. We're not assuming we're going to do that. We're just going to use what's already uh basically in in our toolbox

[7:28:56] inside of Caris and build models off of that which is plenty of complexity for that which is plenty of complexity for us. Okay. So just to uh oh sorry just to us. Okay. So just to uh oh sorry just to reiterate you know some of the the uh

[7:29:09] features of TensorFlow um is that we are going to be it's it's open source so it's easy to install even on your local machine um it'll be really install it just like you would any other Python library it is free um and it

[7:29:26] Python library it is free um and it supports uh building basically doing GPU training um which is uh really really nice. So, TensorFlow will interact with will interact with GPUs really really effectively which is nice. Okay, so

[7:29:41] working with TensorFlow. We've already talked about a lot of this open- source uh it does GPU computation really easily, very flexible library in terms easily, very flexible library in terms of the uh ability to build out different

[7:29:55] contains a lot of different layers that we may need. We're going to see that when we start building our own networks. The fact that it also has highle uh abstraction is really really nice too. The fact that it has a caris, we don't

[7:30:09] need to go into the graph level makes it super approachable for us uh to get started and kind of learning how we would build and train neural nets. It's so much more approachable than having to get into the graph level details which

[7:30:23] install locally or can we work in the cloud? No, you can work in the cloud. Uh so Collab has TensorFlow built into it because it you know um TensorFlow came from Google. So uh of course Google Collab is going to have TensorFlow built

[7:30:38] in. You don't need to install it. Um so every Collab notebook can work with TensorFlow. In fact, it's actually really nice to work in Collab for this stuff because I' I think I've said before Collab has GPUs for free. So you

[7:30:52] can connect to a GPU and kind of train your models on GPU really easily in the Collab environment. So I definitely encourage especially for deep learning to try to use Collab if you can. I think so TensorFlow is also

[7:31:06] installed in the SimplyLearn lab environment. So if you want to use that environment, it also has a GPU connected to it for free. Um so you can feel free to use that and and it has TensorFlow setup in it. you would just have to

[7:31:20] install. If you want to run locally, you probably would have to install because it's likely you don't have the TensorFlow package. Um, and I can share TensorFlow package. Um, and I can share some resources later on uh for utilizing

[7:31:32] the GPU uh locally, especially if you're like on a M1, M2, M3, M4 Mac. You can actually use the GPUs uh in the in the uh M1, M2, M3, M4 chips. um you can

[7:31:46] actually use GPUs locally on those which is really nice or or if you have like an Nvidia GPU, you can use that too either way, but I do recommend Collab strongly um you know because of it free GPU ability and it and it all just works

[7:32:01] ability and it and it all just works pretty seamlessly in Collab. Okay. So, what's nice about uh TensorFlow as well is obviously everything is going to be is obviously everything is going to be in Python for us to be able to do to

[7:32:13] handle all of our neural network building and training. um that'll all be in Python. But actually there's a lot under the hood, you know, not that we under the hood, you know, not that we need to know this, but under the hood of

[7:32:25] TensorFlow is a lot of like efficient C implementations that make work that make the graph data flow uh as efficient as possible. So it's Python, but it's powered by C honestly a lot under the hood. Um which is which is a lot of

[7:32:41] libraries are like that. Numpy is very like that. Um, it's basically an interface to low-level C code that's manipulating all the the numpy arrays. So, TensorFlow is kind of no different. Um, has very efficient uh C

[7:32:56] implementation. Um, it also has the parallel uh computation. It has a it has parallel computation capabilities. This is something we're going to see uh that we can easily offload our computations to a GPU to get to to have as much as it

[7:33:13] of it parallelized as we possibly can to speed up our computations. Um so what we're going to see is TensorFlow makes it and PyTorch PyTorch for that matter too they both make it really easy to do that. So we'll see that later on.

[7:33:29] TensorFlow can run on many different environments including mobile and it's really important mobile to call out mobile on this mainly to call out mobile on this mainly because a lot of sophisticated AI

[7:33:44] because a lot of sophisticated AI programs that run on phones like the Apple intelligence for example are often powered by neural networks that have to powered by neural networks that have to run on a mobile device. Um so that's so

[7:33:57] t TensorFlow would be responsible for running that model and so it has to be running that model and so it has to be compatible with uh kind of uh edge devices like like an iOS or Android or

[7:34:10] uh even like a Raspberry Pi which is kind of a small uh computation platform. different platforms which is great because people are interested in running neural networks on many different platforms including those kind of mobile

[7:34:25] edge devices. Now because it's open source it has a really vibrant community is one of the most popular open source projects on GitHub. Um it still has a very dedicated team at Google that that's dedicated to kind of maintaining

[7:34:41] it even though it's open source. there's developers there that are kind of, you know, focused on maintaining it and making sure it's in a good state. Um, community and one of the things that comes out of that community is really,

[7:34:53] really good examples and documentation. So, I'll show us that probably later on today. The the TensorFlow documentation is really, really good. So, we'll utilize that in certain areas when we need to. Okay. So what we're going to be

[7:35:07] we're going to actually get into this soon and then start building as well in our code. So that's coming up shortly uh is using highle abstractions like caris.

[7:35:19] So the the graphs are great but we don't want to manage things at a graph level that's very very uh complex as I've alluded to. So, we want to be able to um

[7:35:31] basically quickly prototype models, quickly develop them, and quickly train them. And we'll be able to do that using something like Caris because it'll make uh building a neural net and training it really only take a handful of lines of

[7:35:46] really only take a handful of lines of code, which is really amazing. Um it's it's actually remarkable how these libraries like TensorFlow and PyTorch how easy they make building and training a neural net. They make it really easy

[7:35:59] to do which is nice and that's we've seen that before with you know scikitlearn made doing machine learning really easy by basically making everything into a dofitit and predict right um so so caris will be kind of the

[7:36:14] same thing and pietorch kind of the same thing making our life really easy uh in terms of building and training neural nets so many different uh applications that use tensorflow models that are out there today So image recognition,

[7:36:29] videos, NLP, basically anything that uses neural nets, um transformers, of course, not listed on here, but anything that's doing it, you know, anything with neural nets. Um it often there is a TensorFlow model for it. Doesn't mean

[7:36:45] it's exclusively in TensorFlow. There may be a PyTorch version of it as well, but most the time there's there's both, you know, a TensorFlow version and a

[7:36:57] PyTorch version for many different models that are out there. Um, they're generally developed in one of those two frameworks, if not both. So many companies leverage TensorFlow because they are using neural nets to do

[7:37:09] they are using neural nets to do something. Um so like object detection uh Uber you know uh and a lot of self-driving uh is powered by neural nets that is that TensorFlow uh helps provide and many companies are using

[7:37:25] TensorFlow for different kinds of uh neural net applications of course so no surprise here to see a lot of big companies on this list Google not shocking they created it of course they're on the list so some application

[7:37:39] areas I I mean we've talked about these before like medical images uh diagnosing before like medical images uh diagnosing disease um they so Google for example created uh this application called germacist using tensorflow um basically

[7:37:54] to make diagnosis on skin conditions which is like pretty interesting using which is like pretty interesting using just pictures um so so many like neural tensorflow I mean so many are developed in pietorch as well those two libraries

[7:38:08] just dominate the market in terms of building and training neural nets. So, building and training neural nets. So, uh the the Twitter or now X their uh ranking of tweets uh to do like recommendation is uh all powered by

[7:38:24] recommendation is uh all powered by TensorFlow. Google using uh uses TensorFlow to do its search rankings. Photo sharing VS Code um uses TensorFlow to suggest filters. So so many TensorFlow applications that are out

[7:38:40] there uh that are in use today. I mean not not to say there's not so many PyTorch examples. There are definitely those as well, but a lot of them, you know, a lot of people chose TensorFlow, a lot of people chose PyTorch. They're

[7:38:55] both pretty represented in the in the community. Um so we can use it for text applications. Um and you know a lot of uh even up to

[7:39:07] LLMs like transformer-based models are built in TensorFlow um or PyTorch or both as I said both very very popular frameworks. So NLP applications tensorflow has been used for uh e-commerce use tensorflows for

[7:39:23] recommendations. So remember when I said recommendations, you know, we studied recommendations and saw a lot of techniques to do that like um factorization. Those are still used in some places, but

[7:39:37] a lot of places have graduated to doing neural networkbased techniques. And uh nets, chances are you're using TensorFlow or PyTorch um to do that. And

[7:39:49] so so many companies have uh adopted TensorFlow to do recommendation uh because they're using some type of neural net to do that. Pretty interesting. Okay. So what I wanted to do is um show you one short demo on

[7:40:07] kind of see what the tensor data structure looks like. Um so I'm going to do that. I'm going to do this demo to start with. So um we're going to start with the 5.0 02 demo from lesson 5, which is just going to be an

[7:40:21] introduction to tensors as a data type. And what you're going to see is tensors are very much like numpy arrays, very much like data frames. They're just a basic data structure that TensorFlow works with. Um, now these these two

[7:40:36] demos I'm going to hold off on because they get into building a neural net. What I want us to see first is to jump ahead to the caris notes here in lesson five. So we're going to cover caris first and then once we've covered

[7:40:52] caris in a few slides, work our way back to these two demos. So the plan for today is I'm going to cover this demo first. We're going to jump ahead to the first. We're going to jump ahead to the caris notes to get some information on

[7:41:06] nets and then get our hands on these two demos so we can see it in action. So these two demos are going to have kind of end toend examples on building and training neural nets. Obviously we haven't learned that yet. So I want to

[7:41:20] jump ahead to Cara so we can see some of that and then work our way back to these that and then work our way back to these two demos to to actually practice that. Um so so that'll be kind of the plan. Um, so with that being said, let me let

[7:41:32] Um, so with that being said, let me let me uh jump over to the 5.02 me uh jump over to the 5.02 um notebook um just so we can see some basic examples with just tensors as kind of a data structure. Okay, do you guys

[7:41:45] have this one? 5.02. This is the uh first demo in lesson five. first demo in lesson five. Okay, great.

[7:41:58] you shouldn't need to run this. In fact, you can see it already says everything is satisfied if you do run this. This is just like for your local if you're if machine, you probably need to install these because you probably don't have

[7:42:12] them. So, this would be uh for local machine. Um if you don't have these um you'd want to install them. They're already here in Collab. So, it just

[7:42:26] tells us, hey, we already have all these. Um, so nothing really happens. All right. So then we have some setup here. Now these things are optimizations

[7:42:43] that that are uh basically flags for TensorFlow to know if it should be using GPU or if it should be using CPU and if it should be making optimizations based it should be making optimizations based on the GPU. So, what we can do is um

[7:42:58] basically disable GPU for this demo because we're not going to need GPU to do anything with tensors yet. Um we're only going to need GPU for uh later on when we're actually training uh more sophisticated models that that are going

[7:43:13] to require it. So, this is basically setting um the we're going to turn off setting um the we're going to turn off GPU visibility here. Um so, we don't use a GPU in any in any way because we don't need to right now. It's kind of a waste.

[7:43:28] Uh it wouldn't be useful to us. And we're going to set the uh this is just some logging output from TensorFlow uh in case there's any warnings or in case there's any warnings or anything. Um so, we'll just run that.

[7:43:43] anything. Um so, we'll just run that. Okay. So what I wanted to show you is uh first of all TensorFlow as an import is kind of like many other packages in data science and machine learning. It has an alias that is kind of an industry

[7:43:57] alias that is kind of an industry standard alias which is TF. So um this standard alias which is TF. So um this is a industry standard alias is TF for um TensorFlow right? So we can see is we import

[7:44:12] right? So we can see is we import TensorFlow as TF um which is kind of the industry standard to alias it. So you know just like we do NumPy as MP, Pandis as PD um TensorFlow is typically imported as

[7:44:26] um TensorFlow is typically imported as TF. That's very industry standard. So if across and saw TF, they would know what that means. They know it's TensorFlow that means. They know it's TensorFlow that we're using. Okay.

[7:44:39] that we're using. Okay. So in TensorFlow um as I said the the main data structure that gets manipulated under the hood that kind of manipulated under the hood that kind of holds our data um for training and doing

[7:44:51] prediction with a neural net is known as a tensor. Um and you can create a tensor. Um and you can create different dimensional tensors uh inside of TensorFlows or inside of TensorFlow I should say. Um the very ba the most

[7:45:07] basic tensor you can possibly have basically is dimension zero also known basically is dimension zero also known as rank zero. So rank is equivalent to kind of the shape or the dimension that we had uh from like a numpy array or a

[7:45:23] we had uh from like a numpy array or a pandas dataf frame. So a rank zero is just a constant. It's it basically has um no dimensions to it. It's just a single value. So um in TensorFlow you create a single value with TF.stant.

[7:45:40] So this creates just a single scalar value. It basically has no array value. It basically has no array dimensions um no matrix dimensions just a sing it's like a zerodimensional numpy array um equivalent. So what you can see

[7:45:55] is we create the tensor here and then we print it out. What you can see when we print it out is look at the data structure that it prints. It prints a TF structure that it prints. It prints a TF tensor. That's mainly what um we're

[7:46:10] seeing here is is this is the kind of data that TensorFlow is going to work with is a TF.tensor structure. So the the TF I should call structure. So the the TF I should call that out here. The TF.tensor

[7:46:24] that out here. The TF.tensor structure is um equivalent to a mparray structure or a pd dataf frame

[7:46:37] uh structure. It's it's equivalent to those but it's just in TensorFlow this is the data the primary data structure that's used is a is a tensor TF tensor.

[7:46:49] This has no shape to it because it's zero dimensional. So it's just a single constant value of four. It's a scaler and you can see what its data type is is an is an int32 which is just the default integer uh type for for integer data.

[7:47:08] Okay. So we could make a onedimensional tensor which is basically just an array with basically a vector that has a few values in it. Um, so a onedimensional is values in it. Um, so a onedimensional is still a um a tf.stant,

[7:47:24] still a um a tf.stant, but you can see it now we have a a list or you could put a list. You could also put a numpy array there. Um, you can actually build a tensor out of either kind of data. Um, and so this is just a

[7:47:39] single dimensional vector. And what I want you to see is look at the tensor want you to see is look at the tensor now. So the tensor now has this array as now. So the tensor now has this array as its data and it now has a shape of three

[7:47:54] uh comma nothing which is the signal that it only has one dimension and that dimension has three elements in its shape right so it's a shape of three nothing cuz it's just a single dimension so that's a rank that's what we would

[7:48:07] call a rank one tensor it's just a single dimension but again the underlying data is a tftensor this is what is manipulated under the this is what is manipulated under the hood of any neural net is a tensor

[7:48:20] structure. So just showing you this so you can kind of see what actually is being manip manipulated. Um when we build a neural net it's really this build a neural net it's really this tf.tensors.

[7:48:38] tensor which is a matrix. So this is data that has rows and columns. It's data that has rows and columns. It's two-dimensional. So here we have a 3x two. We have three rows and two columns of data. And the other thing we can do

[7:48:52] when we build a a tensor. So we use tf.constant to build the tensor. Um and so we put in this two-dimensional array. We also specify that the data type

[7:49:05] should be float 16. So that's really interesting. The original data is just integers, but we actually have the flexibility to say, "Hey, I want you to flexibility to say, "Hey, I want you to put all of this in float 16 data type."

[7:49:18] So you can see when we create this tensor, it has this data and it's um tensor, it has this data and it's um kind of casted over to a float 16 data type. Okay. So that is a two-dimensional and we could create even a

[7:49:32] threedimensional just like we've done before with with uh you know threedimensional numpy arrays. So we have um you know this is now this is have um you know this is now this is basically uh every entry in this tensor

[7:49:47] is a matrix itself. So we have three matrices. Each matrix has two rows and five columns. That's what these dimensions really represent. dimensions really represent. So we have one matrix here, one matrix

[7:50:01] here and one matrix here. So we have three of them. Each of them are 2x5 three of them. Each of them are 2x5 and so we get a shape of 3x 2x 5. So um

[7:50:13] three dimensions are actually very popular in TensorFlow because um usually popular in TensorFlow because um usually the data will be in either three or actually a lot of times four dimensions because um this first dimension may

[7:50:28] represent like a batch like we may have um 100 images and each image is this resolution which is maybe like 28x 28 resolution um like as a matrix. So three dimensions

[7:50:43] is actually very very common for a tensor and and actually four dimensions tensor and and actually four dimensions is also very very common um because the first dimension is still the batch but every image may be a 28x 28 resolution

[7:50:59] and then have like three color channels like RGB, red, green, blue. Um so four dimensions is actually pretty common. 100 images each one is 28x 28 with three

[7:51:13] This is actually so four dimensions is actually very very uh common in TensorFlow when we're working with uh data and building like image based networks that have color images. Um

[7:51:28] that's actually uh pretty common to see four-dimensional data. you can actually go back and forth between tensors and numpy arrays. So,

[7:51:43] between tensors and numpy arrays. So, all you have to do is um just call numpy on it and this will convert any tensor over to a numpy array. So, this is a uh over to a numpy array. So, this is a uh numpy array and the the tensorflow float

[7:51:57] numpy array and the the tensorflow float 16 just converts over to a numpy float 16 type um very seamlessly. So it it just retains that kind of type just in numpy. Okay. So if you ever want to convert for whatever reason, maybe you

[7:52:11] want to manipulate that tensor in numpy directly, um you could just call numpy on it and it will turn that tensor into a numpy array with the same shape. So now we have that two-dimensional array there.

[7:52:31] course going to be operations between tensors which needs to happen in order tensors which needs to happen in order to do matrix uh manipulations and to do through activations and all the kind of

[7:52:44] computations that we need to do. There has to be uh mathematical operations between tensors. they need to support that in order to do any kind of meaningful computation with the neural network. Right? So what you can see in

[7:52:58] network. Right? So what you can see in this example is just some examples of uh adding multiplying and doing matrix multiplication between two tensors. So multiplication between two tensors. So we have two tensors which are 2x two uh

[7:53:12] we have two tensors which are 2x two uh rank two tensors and we can add them together which will do an elementwise addition. We could multiply them multiplication or we can do a traditional matrix multiplication

[7:53:27] multiplication is that row by column dotproduct constructing a new matrix out of that. Um and and we can do all those all of those operations. Um so this

[7:53:39] all of those operations. Um so this makes sense. This like TensorFlow needs to support these kind of operations to do any meaningful kind of manipulations do any meaningful kind of manipulations right of underlying data. Again will we

[7:53:51] need to do this like no we won't really need to do this because this will be handled under the hood by the highle library like caris. they will in TensorFlow will manage doing all these computations. But when

[7:54:07] they do those computations, this is what they're doing under the hood. Multiplication, matrix multiplication, addition, um passing it through an activation, all those kind of operations are are

[7:54:20] managed by TensorFlow, which is really nice. It has other uh operations as well. Things like finding the max which makes sense because sometimes we need to find the max in order to uh in order to

[7:54:37] calculate like the maximum probability or find out which class something should be which is the arg max. So this is finding the index where the maximum is. That's an important operation for figuring out like which which uh entry

[7:54:52] in this array has the highest probability. Um that's the class prediction is the argmax. And then what I wanted to show you too is the uh TensorFlow has builtin activation functions. So you can see one

[7:55:08] a bunch of them when we start building our neural nets is like the softmax activation. So you can see um we take this uh we take this um tensor C which

[7:55:22] is a two-dimensional tensor and we actually compute the soft max of it which will normalize it right into probabilities that uh total up to one uh

[7:55:34] across the rows. So because it's two dimensions um the softmax will make sure that like this adds up to one and then this adds up to one. So every row adds

[7:55:46] up to one. So this is just an example of applying an activation like a softmax. Um notice where this come from comes from is the NN module which is the neural net module. So uh that's from its built-in neural

[7:56:02] net module. It has a bunch of stuff like layers. It has a bunch of stuff like uh activations um those kind of things. We'll talk more um those kind of things. We'll talk more about that as we get into caris.

[7:56:18] to run the notebook? All these steps have ran for you guys. Hopefully, no issues. We're not doing anything fancy yet. Just manipulating tensors, which are hopefully you guys can see are mostly acting like a dataf frame or a

[7:56:32] numpy array. Mostly acting in the same way. Um and then finally just like numpy arrays um you know TensorFlow does

[7:56:45] support broadcasting. So we've talked about broadcasting before when we did about broadcasting before when we did numpy arrays. Essentially if you take um a vector or in this case I should say a tensor and multiply it by uh a constant

[7:57:00] um then that will just apply that constant to every entry in that vector. Um which makes sense. And then um otherwise like makes sense. And then um otherwise like if you multiply uh this times this um

[7:57:14] it'll just do an elementwise um if you do you know it if you try to multiply um different shapes together it will attempt to broadcast to make the

[7:57:27] shapes fit and it will give you an error if it cannot do that. Um, so most of the multiplication and then attempt to do some broadcasting if the shapes don't

[7:57:39] line up, which we saw exactly, you know, the same exact thing happen with numpy arrays. So that's that's not entirely shocking that that TensorFlow would support broadcasting in some way. So, and the so a couple more operations

[7:57:54] you can actually reshape tensors which sometimes is useful uh that for for um tensors to be reshaped and sometimes that'll happen underneath the hood of TensorFlow anyways like it needs to do a reshape to uh um manipulate properly. So

[7:58:12] we can take this tensor and kind of reshape it into a 1x3 um which if you print it out would now uh you know would just kind of transpose uh you know would just kind of transpose it um essentially from 3x one into 1x3.

[7:58:27] Um, so you can do so what I hope you're seeing with this and part of the reason to go through this is I hope what you're seeing is a lot of the same stuff that we had for dataf frames and numpy arrays

[7:58:41] is really carrying over to tensors. In fact, it's kind of inspired by that. When Google made TensorFlow um they were kind of inspired by what was already out there in terms of numpy and pandas. So it makes sense that

[7:58:56] they're going to carry over a lot of the same operations like reshape and and uh having things that look like numpy arrays and pandas data frames. That makes sense. Um

[7:59:11] finally there's uh technically in TensorFlow they have different types of tensors like you can have what's called a ragged tensor which is um basically having allowing for different size

[7:59:24] dimensions. Um so you can have instead of requiring everything to have the same number of uh columns essentially like this first and this first row has four columns but this next row only has two. this next row only has three and then

[7:59:39] this last row only has one. So we basically pack that in as kind of like a list of lists um in this what's called a ragged tensor. We won't really ever have a use for using ragged tensors, but just to say

[7:59:53] that TensorFlow supports having different size arrays and inside of a tensor object. Um you can even have strings um inside of a tensor. So you can see here um this string gets stored here um and you can see its data type is

[8:00:09] string. So it is possible to manipulate strings. That's not very common because most of the time when TensorFlow is manipulating strings uh they'll they'll mostly be like in an LLM they'll be converted over into vectors before

[8:00:24] they're ever manipulated. So very rare you would actually directly be working with strings inside of TensorFlow, but it does support you can actually put in string data into a tensor and it can hold that data. It just won't be able to

[8:00:37] do much with it until it's converted over to a vector. Uh which is what we see with kind of language models. They tend to convert strings into vectors before doing any manipulation with it. But um we'll learn about that later.

[8:00:54] But um we'll learn about that later. All right. So let's talk about uh Caris. Um so just a reminder of what Carias is. It's basically a an interface built on

[8:01:06] top of TensorFlow to help us build and train neural network models without train neural network models without having to go into the weeds of the graph interface of core TensorFlow. So it's very very nice in that way. It's really

[8:01:20] very very nice in that way. It's really easy to use. Um, really fast to build models and train them. Uh, in terms of getting up and running with it, really userfriendly. That's what it was built for is to be really user friendly. And

[8:01:33] so it's kind of everyone's preferred way of interacting with TensorFlow. Um, if you can because it's so nice to use as we're going to see. You know, as I said earlier, unless we're doing very very sophisticated research, there's no

[8:01:49] reason we shouldn't be using caris and uh people wouldn't people generally prefer using caris. It's so nice to use for building and and working with neural nets. Um, so of course it's in Python and it uh it now caris also does the

[8:02:06] same thing as TensorFlow in the sense that it allows us to use GPUs relatively easier e easier uh or easily I should easier e easier uh or easily I should say. Um and it's uh really really easy

[8:02:19] to basically prototype and build your your neural nets and get up and running really quickly that way. So I think one of the ways to think about kas is I I

[8:02:31] used the analogy earlier of kind of like how we have panda sitting on top of how we have panda sitting on top of numpy. The underlying data in a dataf frame is technically stored in a numpy array. Um but we don't work with it in a

[8:02:46] numpy array style. We work with it in a dataf frame because when it's in a dataf frame in pandas there's so much we can do with that data frame from a pandas perspective. Right? We can manipulate it. We can um aggregate it. We can join

[8:03:00] we can do with a data frame that we cannot do with just a basic numpy array. Same thing with caris. There's so much we can do with caris to help us build we can do with caris to help us build neural nets and train it um that it it's

[8:03:15] you know sits on top of tensorflow and the underlying tensor graphs and it just abstracts that all away for us and focuses on the core things we care about focuses on the core things we care about like layers loss functions

[8:03:29] um setting up the training and all the hyperparameters around training. So it working with just the underlying TensorFlow which can be challenging and and take more time to ramp up to for sure.

[8:03:43] Basically as I said t uh caris acts as an interface for it. Um and we only need running building any neural nets which is really awesome. So uh caris actually

[8:03:55] has extensions to other deep learning libraries. Um so not that we need to know that but it caris has been extended to be an interface to many other um

[8:04:07] backends that are neural network libraries. So many of them are are way less used in the community than TensorFlow. So of course we're going to focus our efforts on working with caris as an interface to tensorflow. But these

[8:04:22] guys here are other deep learning libraries that Caris has support for as well, but they're not nearly as popular as PyTorch or uh or or TensorFlow. Um,

[8:04:38] unfortunately there's no caris version for PyTorch, but that's okay because PyTorch is kind of its own has its own um abstractions on top of the underlying

[8:04:50] tensor graph structure of PyTorch. So we'll learn about that in lesson six. So just to say caris has grown really popular and has been extended to other to other deep learning libraries, but the other ones on this list are not

[8:05:04] nearly as popular as TensorFlow. TensorFlow kind of dominates the industry along with PyTorch. PyTorch and TensorFlow blow away all these others in terms of their usage. Um so just uh continuing the theme here of being

[8:05:17] really nice uh timesaving. We don't have to go into uh learning the underlying uh core TensorFlow which I think from like a learner perspective from your guys's perspective is so nice. you don't have to spend a lot of time getting ramped up

[8:05:32] to spend a lot of time getting ramped up on the the nuts and bolts of core TensorFlow which is really challenging. It's not an easy library to understand. Um but caris is caris is really easy to work with. So it makes it really user

[8:05:46] friendly to get up and running. We just talked about this. Many backends are talked about this. Many backends are supported from uh from TensorFlow and or from caris. And one of the reasons is like as I said before TensorFlow was

[8:06:00] open- sourced a lot of people were developing their own neural net libraries. So these have been around like the MXNet CNTK which is from like the MXNet CNTK which is from Microsoft. These are neural net

[8:06:13] building because they needed them to be able to manipulate um or to build their own neural nets. But TensorFlow emerged and PyTorch emerged as the dominant and PyTorch emerged as the dominant libraries. Um, Caris is the main way

[8:06:28] that we interact with TensorFlow, which is why we care about Caris. Um, but Caris has extensions to be able to work with other neural net libraries. It's just those others are a fraction of what we see um, PyTorch and TensorFlow

[8:06:46] uh, being used for these days. Okay, so no need to keep selling you on it. Cares is really important. uh and of course that's what we're going to uh be using here initially to build out some of our to build out our models and train them.

[8:07:01] we've talked about the other thing is it has really good documentation um and a good you know solid open- source community. Caris is open source just like TensorFlow is. So, uh has really good um support for uh people uh

[8:07:18] maintaining it and building out documentation and things um still to this day. Okay, so we've talked about all of this. We've talked about how userfriendly and how easy it is. Quality documentation we just talked about. In

[8:07:31] fact, I'm going to show you some of the documentation coming up before we uh finish today. So you can see kind of what that documentation looks like in terms of um a way to see all of the layers that are available, a way to see

[8:07:44] all the activations that are available, the way to see all the loss functions that are available from Caris. Um so I'll show us that multiple backends care about that because we're going to be using it with TensorFlow primarily.

[8:07:59] One thing that's cool is uh just like underlying tensorflow, caris has easy ways to access pre-trained models which will be important in like computer vision or even in NLP to grab models off the shelf so we can kind of use them for

[8:08:13] the shelf so we can kind of use them for our use cases. Um so that'll be a nice feature. Caris allows us to work with GPUs. It can even work with multiple GPUs if you have them. Um so that's also really nice and it makes sense like

[8:08:27] of course caris is going to make it easy to do that too. Okay so one of the let's start getting into some of the elements of caris that we are going to use to of caris that we are going to use to construct our neural networks. So the

[8:08:42] construct our neural networks. So the first place to start is going to be the first place to start is going to be the uh layer um class. Now there's many different versions of layers that are inside of caris and I'll show us a

[8:08:55] inside of caris and I'll show us a documentation of that later. But um a layer is a fundamental building block of any model. And so when we build a layer, one of the things we need to decide when we build a layer is how many neurons do

[8:09:11] we build a layer is how many neurons do we want in that layer. Um and so that's going to be an important input. The other input that we'll typically need when we construct a layer is the activation. So what activation do we

[8:09:25] to apply relu? Do you want to apply softmax? Do you want to apply sigmoid uh hyperbolic tangent? You know, what activation do we want to have inside of

[8:09:37] all these neurons across this layer? So that's going to be important. Um, and what's interesting is we will be able to connect layers together inside of a

[8:09:50] neural net, we're going to be building out basically declaring layers and then connecting them together inside of this network structure. So we're going to see that in caris. But the fundamental building block is going to come from

[8:10:05] this uh module right here, the TensorFlow caris layers. And then there's going to be many different kinds of layers that are possible. This is just a generic layer,

[8:10:18] but there's going to be many different kinds of layers that are available for us to use. For example, there's going to be recurrent layers, convolutional layers. There's going to be attention layers like in a transformer. The most

[8:10:34] basic kind of layer that is basically like a perceptron is known as a dense like a perceptron is known as a dense layer. So dense is the most basic kind

[8:10:46] of layer and we're going to be using dense almost exclusively for now. So what a dense layer is is basically like a perceptron. it. A dense layer just

[8:10:58] a perceptron. it. A dense layer just takes a weighted sum and passes it through an activation. So it's just a feed forward kind of layer that's just doing the most basic thing any neuron could do, which is weighted sum through

[8:11:12] an activation. So a dense layer is the most basic, but that's not to say many different layers that we will see when we build our models. As I just when we build our models. As I just said, convolution, attention, recurrent,

[8:11:26] um, LSTM, all there's all kinds of layers. And I'll show us the documentation of, uh, what layers are available, but this is attention on is the dense layer. It models just a weighted sum going through

[8:11:41] an activation. It models a collection of neurons where every neuron is just a activation. So this is going to be our most basic kind of layer in the inputs for a dense layer needs to be number of neurons usually number of neurons

[8:12:00] neurons usually number of neurons and then the activation function. Um, so I'll again we're going to see this inside of the code, but these are the two things we typically will um declare when we build a dense layer to

[8:12:12] basically say, okay, this layer is going to have 20 neurons or it's going to have to have 20 neurons or it's going to have 10 neurons or it's going to have 256 neurons and all of those neurons are going to have relu activation or it's

[8:12:26] going to be softmax activation or it's going to be sigmoid activation. um whatever activation that we want to supply. Okay, so uh again that's going supply. Okay, so uh again that's going to be tf.caris.layers.dense

[8:12:44] vision. We'll then we'll use something like a convolutional layer or we get into RNN recurrent layer or in transformers an attention layer things like that. But right now we'll mostly use dense which is just the most basic

[8:12:59] kind of layer you can have that's just basically short for feed forward which means weighted sum through an activation. Okay. So we need layers and activation. Okay. So we need layers and we also need models. So models will

[8:13:14] we also need models. So models will allow us to connect layers together. So think of the model as the overall network. So, so what we're going to do typically the typical pattern is you declare a model and then add layers to

[8:13:29] declare a model and then add layers to it that will be connected together. Um, and so uh think of think of the the model as our basically our object that contains our neural net. Um, our neural net is the model and then

[8:13:46] within the neural net obviously are all of our layers with our neurons. So those are connected together and then they um generate all those connections by uh automatically under the hood whenever we add them to the model. And something

[8:14:01] that we'll see is that we can we can add them into the model uh sequentially which means that they will be um connected together in that order. Meaning like we want this layer first then we want this layer then this layer

[8:14:16] then this layer. Um we can also do it recursively meaning we basically say here's my starting point here's my ending point and I want you to connect everything in between um all the layers that I've declared in between. Um so we

[8:14:32] will we'll I'll show you both of those ways of building the model but um we need typically for any neural net we need an overall model object and then we need an overall model object and then we need layers to be uh to every model has

[8:14:47] layers inside of that model object. Okay. And then we then we'll just need a way to connect those layers together which I'll I'll show you how to do. But the model has some very important functionality to it. Mainly it has these

[8:15:03] two functions which are going to be uh really really critical for us such as the training. So every model has a fit function which executes this just function which executes this just executes the training

[8:15:21] function which now does this look familiar to us? Hopefully it does. Fit and predict. What do you think that's inspired by? That should hopefully look familiar, right? A fit and apredict. That should look really, really familiar

[8:15:35] to us. And it's actually no surprise that they chose those words specifically. Ffit and predict. Those are directly Ffit and predict. Those are directly inspired by scikitlearn, right? Sklearn.

[8:15:50] Every model there had a fit and predict. really no different here. When we build our neural net training, it's actually just going to be a matter of executing a fit. The difference is when we execute that ffit, we need to set a bunch of

[8:16:03] stuff up. We need to set up what the loss function is. We need to set up what our gradient descent algorithm is and and there's a lot of hyperparameters there that are involved in the fitting of a neural net. Um so this ffit is a

[8:16:17] of a neural net. Um so this ffit is a little more complex and the the predict is also kind of complex because under the hood what predict is doing is forward propagation right it will generate the output for uh whatever

[8:16:30] input data we pass in so it's just forward propagation is essentially this forward propagation is essentially this predict um it also has a convenient evaluate function every model has an evaluate function um which uh basically

[8:16:47] evaluate function um which uh basically gives us back the um uh the loss. So we evaluate on a on a collection of data and it it basically um gives us what the loss and any like accuracy metrics that we set up. So what's our precision

[8:17:02] we set up. So what's our precision recall accuracy um confusion matrix those kind of things whenever we apply our model to some data that has labels. So assuming we have labels we can do an evaluation which is like what is our

[8:17:16] evaluation which is like what is our loss um and what is whatever any metrics we declare like accuracy um we can produce those pretty easily. So again I'm going to show us all this in the code. I'm just trying to orient us right

[8:17:29] now into the different elements of caris that we really care about. So right now we have two which is the layers and the models. We care about those two primarily in order to build and train a neural net. There's going to be other

[8:17:42] things we care about like declaring a loss function, declaring activations, those kind of things. So, you know, there's there's other small bits that we're going to see along the way. But at the highest level, like the two things

[8:17:55] we care about are the layers in the model that holds the layers that holds our neural network layers. Okay. Okay. So there's, as I kind of alluded to, there's two primary ways of building models, which is going to be

[8:18:09] sequentially or recursively, also known as functionally. Um, so sequentially is as functionally. Um, so sequentially is uh the simpler way of the two. Um, which

[8:18:21] I think is a really good way to get started with model building. um because it allows you to create models basically layer after layer after layer in a linear sequence. So what that means is you basically um declare that I want you

[8:18:39] basically build a list and say in this list this layer goes first then this layer then this layer then this layer which for some models makes a lot of sense like if it's a very simple model that's really all you want is you want

[8:18:54] layer then this layer and it's going to go forward through that sequence right is very natural. that makes sense. So the sequential way

[8:19:06] of building a model is kind of the simplest way to do it. Um but the recursive way to do it or what I should say the functional way to do it is um a

[8:19:18] little bit more flexible in the sense that um it allows you to have layers that um it allows you to have layers that can be uh shared and nonsequential. So for like recurrent nets that's really important. So what we're going to see is

[8:19:33] We're going to build a model sequentially and we're going to build a model using the functional API um so that we can get exposed to both. with we can get away with just doing sequentially because they're actually

[8:19:49] going to be very simple. Um it's not until we start building some more complex models that the functional will be actually better for that case. But the the functional um allows us to have um kind of exotic

[8:20:05] structure uh because all we need to do is basically say how one layer acts on another layer and and we can connect all those kind of recursively. And so it nonsequential because you can have a a layer that's

[8:20:21] further down the line act on an earlier layer by um having like a recurrent connection back to the previous layers and you can do that in the functional API uh when you cannot do that in the sequential. So I'm going to I'm going to

[8:20:36] show us both ways. Both are widely used. I think both like when you're building something relatively simple, I think sequential is the easiest for a reader sequential is the easiest for a reader to understand and see. Um, but if it's

[8:20:49] anything that's complex, you're almost always going to be using the functional always going to be using the functional way to do it. Uh, because you have to sequential because you're not going to have a flow that's just directly in feed

[8:21:03] forward. Sequential is basically only for feed forward where things are guaranteed to go from one to one to one to one to one and just go forward which the case. That's all we really care about in the beginning. But as we get to

[8:21:16] that functional way of doing it. Let me give you an example. So let me give you an example what the sequential um uh way of building a model looks like. And this is actually a very prototypical case

[8:21:31] is actually a very prototypical case where you build your model as a sequential object. So you can see directly here we we initialize a sequential model using this sequential object here. And look at what we pass in

[8:21:47] object here. And look at what we pass in is a list. So inside of sequential is literally a list of layers. That's that's what the sequential model takes. And the it's it's purposely a list because what the list signifies is this

[8:22:03] layer should be first and then this layer should follow it, which is exactly what we've seen in our drawings, right? Like there's a layer that's first on the left and then that flows into a layer on the right. So it's it's it's sequential.

[8:22:21] It's this layer and then this layer and then one more layer and then one more then one more layer and then one more layer. Right? So, so sequential is just kind of for exactly those use cases we've been drawing. What I want you to

[8:22:33] see here is an interesting uh very simple model, but what it's saying is here's our dense. So, here's where we're using that layer. So, we have a list of layers. Here's that dense, which is the most basic kind of layer we

[8:22:48] can have. What that means is I have a bunch of neurons. How many neurons do I have? I have 64, which is a choice. So I can choose how many neurons I want in that layer. So we're declaring 64 neurons to be modeled

[8:23:03] in that layer. And what we're saying here is there's an input shape, meaning here is there's an input shape, meaning the input data to this initial layer is this size. So this is just orienting the network to say my input layer is going

[8:23:19] to be this shape. So I should expect data to flow into this network that's this exact shape. Um so that's however many dimensions it is which would be like however many features we have. For example like if we had 20 features that

[8:23:34] input dimension would be 20. Um but this is saying we have 64 neurons and the activation of every one of those neurons is a relu activation which is pretty standard right a relu activation is pretty common and then here now what

[8:23:51] pretty common and then here now what this is signifying is this layer this is signifying is this layer flows into this layer because it is next in the list. So the outputs of that layer flow into this layer which has 10

[8:24:05] neurons and a softmax activation. Now what does that mean? That means that we are likely predicting That means that we are likely predicting 10 categories. Um because I have 10

[8:24:19] outputs which are softmax activated. So that means they are all going to be they're all going to be probabilities that total up to one. So this is likely

[8:24:34] this is the and by the way, this is the output layer because it's last in the output layer because it's last in the sequence in the list. It is last. So it is the output. There's nothing else. It's not connected to anything else.

[8:24:47] It's not connected to anything else. Right? So this has to be. So the last Right? So this has to be. So the last layer in the sequence

[8:24:59] nothing else beyond it. Last layer equals the output. And so you can actually see this is a model. This is going to be a neural net that is designed to predict 10 different classes. It's providing us a probability

[8:25:14] for 10 different classes. So for example, this could be used for example, this could be used for something like the MNEST data, like that where there's the the digits 0 to 9 which are 10 different categories,

[8:25:30] right? That could be used for something like that. Um I'm just throwing that out as an example, but hopefully um hopefully this makes sense. What you're hopefully this makes sense. What you're seeing here is two layers. You have

[8:25:44] really this initial uh hidden layer with 64 neurons and then uh hidden layer with 64 neurons and then you have a output layer that is size 10 with a softmax activation meaning you're going to produce 10

[8:25:59] 10 probabilities. So the so it's a multi this is likely a network that's used for this is likely a network that's used for multiclass classification.

[8:26:13] network, right? This is our neural net. This here is our neural net object. Look how easy that is to create in caris. It's just a sequential with some

[8:26:25] layers. That's really all it is is a sequential and a bunch of layers. Okay, let me show you the difference with the functional API and and you you can see the difference, but hopefully it makes sense. Now, this is the exact same

[8:26:40] network. It's just built in the functional API. Exact same network, though. It's just going to look a little different. So, what we do in the functional API is we actually are going to declare an input layer as the very

[8:26:57] beginning. So, we explicitly define an input layer, which is this. Same thing though it has a particular shape but notice it's there's an input layer we're explicitly defining in the functional and the reason we're doing that is

[8:27:12] because when we build a model we need we need a layer to tell it where to start. We basically need a starting and stopping point. Um so so this first layer is our starting point here. So we start here with this input layer. Now

[8:27:28] start here with this input layer. Now what I want you to see is we now declare what I want you to see is we now declare um a dense 64 activation but look at the notation of the code. We are basically applying this layer almost like a

[8:27:42] function right almost exactly like a function. What are we applying it to? function. What are we applying it to? The input layer. That's really critical in the functional setup because what this is saying is I want to take this

[8:27:57] this is saying is I want to take this layer and apply it to this layer which is the input layer. Now if you look at the next layer, this is our dense uh 10

[8:28:09] with the softmax. So this is our output. What are we applying that to? We're applying this to our hidden layer which is what we declared here. So, do you is what we declared here. So, do you guys see how this is more functionally

[8:28:24] defining how layers act on each other? This is rather than building it in sequence, we're saying, okay, this layer acts on this and then this layer acts on

[8:28:36] this. All we need to do then to build the model is to instantiate a generic model and tell us where the starting and stopping point is. And what it will do is connect everything together because we've already declared how they all act

[8:28:52] on each other. So what we're doing here is saying start here should be here. And because we've specified how everything acts on each

[8:29:06] other, this model is going to recursively go through and build the graph because it knows how everything acts on each other. It knows that the acts on each other. It knows that the input layer is an input to this and it

[8:29:20] knows that the hidden layer is an input to this. So it's going to connect all those together in this model here. So it's a it's a different way of declaring it, but it's the same exact model, but this is just the functional way of doing

[8:29:36] this. Now you might you might ask why would we do it this way rather than the sequential? It's because you now have an opportunity to build more complex architectures in this in this way of

[8:29:50] build in a much more complex architecture because what you can do is you can have things be shared. You can basically say I want to take one layer and add it to another layer um like plus

[8:30:05] and add it to another layer um like plus another layer. um you can do things like that which in more complex architectures they actually do and what we're going to see that later on when we get into like convolutions and recurrent nets. Okay.

[8:30:18] Okay. So just to compare and contrast again, the sequential is mainly for a linear stack of layers which is literally in that list. Usually because of that it limits itself to simple models, limited customization, very

[8:30:32] simple models that you can get up and running again which is going to be like in the beginning of when we're learning this. That's mostly when it's going to this. That's mostly when it's going to be um applicable. Um but the the

[8:30:44] functional has very high degree of customization which lends itself well to complex architectures um which is going to be useful down the road when we get into those kind of networks. I think it's good to know both

[8:30:57] sure. >> And that brings us to the end of this this video, we started from the fundamentals of deep learning and move step up into neural networks, artificial neurons, perceptrons, activation

[8:31:11] function, loss function, gradient design, and back propagation. We will also explore how to build and train models using TensorFlow, Keras, and PyTorch, and how these frameworks help us solve real classification problems.

[8:31:24] The key takeaway is simple. Deep learning may look complex at first, but once you start understanding neurons, layers, laws, and optimization, you can start building powerful AI models with confidence. If you found this video

[8:31:37] subscribe to SimplyLearn for more such videos on artificial intelligence, latest tech skill. Thank you for watching, and see you in the next watching, and see you in the next session.